lance-format/lance

Overview

Lance is an open lakehouse format for multimodal AI workloads. It defines a file format, table format and catalog spec so a complete lakehouse can run directly on object storage. The repository provides the core Rust implementation plus Python (PyO3) and Java (JNI) bindings.

It is aimed at teams that need vector similarity search, full-text BM25 search and SQL analytics on the same dataset, fast random access for ML training and sampling, or unified storage of images, videos, audio, text and embeddings. Unlike Parquet, Iceberg or Delta Lake, which are optimized mainly for SQL analytics, Lance folds AI-first features into the format itself. It also supports adding columns with backfilled values without rewriting the whole table, plus versioning with ACID transactions, time travel, tags and branches.

Getting started

Install the Python package:

pip install pylance

Convert an existing Parquet dataset with lance.write_dataset, then reopen it through the normal Arrow dataset API:

import lance
import pyarrow.dataset

parquet = pyarrow.dataset.dataset("/tmp/test.parquet", format="parquet")
lance.write_dataset(parquet, "/tmp/test.lance")
dataset = lance.dataset("/tmp/test.lance")
df = dataset.to_table().to_pandas()

The dataset can also be queried from DuckDB. For vector search, write vectors into a Lance dataset, call create_index with index_type="IVF_PQ", then pass a nearest argument to to_table to retrieve the top-k neighbors. The README walks through a SIFT1M example end to end. See the format specification and the documentation site for deeper usage.

Compatibility and constraints

The format is identified by the data_storage_version stored in each dataset; stable versions are a long-term read-compatibility contract, while the next alias is experimental and should not be used for production data. SDK and API changes follow semantic versioning separately from file-format stability. Mixed Lance versions should pin data_storage_version for deterministic writes.

Storage backends are selected automatically by URI scheme: local filesystem, S3 and S3-compatible stores, Azure Blob and ADLS Gen2, GCS, Alibaba OSS, Tencent COS, Volcengine TOS, Hugging Face Hub, GooseFS and an in-memory backend for testing. Rust and Java are available alongside Python, and integrations exist for Arrow, Pandas, Polars, DuckDB, Spark, Ray, Trino, Flink and several open catalogs. Credentials and per-store options live in the object-store configuration guide. The project is under active development, and the license is Apache-2.0.

HotRepo's take

On the 14 November 2026 Rust daily list, the project recorded 6 new stars and ranked 8th in that language list. The sample is a single snapshot, so it supports only that day's placement, not a trend. The more substantive signal is the scope of the repository: a published format spec, storage-versioning contract and bindings across Rust, Python and Java. Teams evaluating it should read the versioning rules before writing production data, since the chosen storage version is what future releases commit to supporting.