🤗 Lance x Hugging Face: A New Era of Sharing Multimodal Data on the Hub
Lance now has native support on the Hugging Face Hub, allowing large multimodal datasets — including blobs, embeddings, and indexes — to be published as a single searchable artifact.
Developers can scan Lance tables directly from the Hub using either the 🤗 datasets API or Lance’s dataset API and immediately run SQL and vector search in just a few lines of code, reducing friction between sharing datasets and running production-grade retrieval workflows.
Branching and Shallow Cloning in Lance: Towards a “Git for AI Data”
Lance introduces a branching model designed for large-scale AI experimentation, combining independent manifest trees, physical isolation per branch, immutable tags, and zero-copy shallow clones via multi-base snapshots.
Unlike traditional table formats that share metadata across branches, Lance avoids shared bottlenecks and cache invalidation, enabling full time travel and high-concurrency experimentation without impacting production workloads.
🏔️ How We Added Geospatial Support To Lance With No New Code
Geospatial support landed in Lance with no storage format changes. As an Arrow-native system, Lance supports GeoArrow types end-to-end, integrates geospatial functions directly into DataFusion, and uses a production R-Tree index to accelerate spatial predicates—enabling geospatial filters, vector search, and SQL over metadata to run in a single multimodal query plan.
This work was driven by Xin Sun (ByteDance), with contributions from Jay Narale (Uber), Kyle Barron (GeoArrow / GeoDataFusion), and Tim Saucer (Rerun & Apache DataFusion).
📅 Upcoming Events
We’re bringing together executive leaders building AI-native developer tools for a candid discussion on what it actually takes to move agentic systems from demo to dependable production infrastructure.
Thursday, March 26 | San Francisco | Register →
Join us with Anyscale and Exa for a technical walkthrough of the infrastructure behind Exa’s AI search engine, covering how Lance and Ray support distributed embedding pipelines and semantic retrieval at web scale across billions of documents.
Tuesday, March 31 | San Francisco | Register →
📺 Talks & Recordings
Almost Every AI Problem Is a Search Problem
Chang She (CEO @ LanceDB) and Bryan Bischof (Head of AI @ Theory Ventures) discuss multimodal data, hybrid search, and why agent workloads turn retrieval into a parallel, latency-sensitive system problem.
🏗️ LanceDB Enterprise Updates
| Feature | Description |
|---|---|
| Automatic backfill | Automatic backfill is available for preview in Feature Engineering |
| Major improvements to indexer and service stability | Fixes to indexer parallelism, debuggability, and failover behavior allow jobs and queries to execute faster during partial failures and under resource constraints |
| Better cache statistics in query plan analyses | Query plan analyses now include counts of cache hits and misses, allowing better understanding of query performance |
| New APIs for versioning direct writes to storage | New APIs for TableVersions allow systems to write directly to object storage and update the metadata stored by Query Nodes |
🌟 Open Source Releases
| Project | Description |
|---|---|
| Lance v2.0.0 - v2.0.1 Release notes |
• Format v2 is now default with new manifest design and storage options accessor (breaking changes in index validation and metadata APIs) (
#5656
,
#5728
,
#5609
). • Expanded indexing capabilities: GEO R-Tree support ( #5034 ), distributed vector index builds ( #5117 ), distributed range-based BTree index ( #5202 ), and FTS-as-filter in vector search ( #4928 ). • Blob v2 architecture adds external, dedicated, and packed blob support with GC for large multimodal objects ( #5385 , #5473 ). • Performance improvements: ~20% faster FTS indexing ( #5650 ), up to 15x faster WAND block scoring ( #5668 ), ~13x BTree benchmark improvement (19.5ms → 1.5ms) ( #5446 ), and ~30% faster HNSW indexing via IVF CPU offload ( #5551 ). |
| LanceDB v0.29 Release notes |
• Parallel inserts + new writer path improve local table ingestion throughput (
#3062
,
#3029
). • Arrow-native RecordBatch writes (breaking change) in create_table() and Table.add() for cleaner ingestion APIs (
#2948
). • Background dataset update checks for remote tables ( #3021 ). • Hybrid search explain plan now surfaces reranker details for better debugging and observability ( #3006 ). |
| lance-graph v0.5.1 - v0.5.3 Release notes |
• CypherEngine exposed in Python for multi-query execution and programmatic graph workflows (
#124
). • Vector rerank method added to CypherEngine API, enabling tighter integration between graph traversal and vector search ( #136 ). • Internal crate restructuring and catalog split to modularize graph + catalog components for cleaner extensibility. |
| lance-context v0.2.2 - v0.2.4 Release notes |
• Context.add() now supports embeddings, bot_id, and session_id, enabling richer conversational and agent memory workflows directly in the API ( #31 , #33 ). |
| lance-duckdb v0.5.1 Release notes |
• Updated to lance-format v2.0.0 (stable) for compatibility with the new default manifest and index features (
#142
). • Fixed table discovery issues after SHOW TABLES when using ATTACH TYPE LANCE, improving SQL workflow reliability in DuckDB (
#133
). |
| lance-namespace v0.5.0 - v0.5.2 Release notes |
• Introduced table version operations and managed versioning support in the REST API, including partition specs and table properties (
#301
,
#305
). • Improved versioning correctness and API responses for table version creation and manifest naming schemes ( #306 , #307 ). |
🫶 Community Contributions
Thank you to contributors from Uber, ByteDance, Netflix, Twitter, and Huawei for improvements across indexing, storage format evolution, distributed indexing, geospatial support, and API ergonomics in LanceDB, Lance, lance-graph, and lance-context.
Notable contributions this month:
- @ddupg – Advanced GEO R-Tree index support and Blob v2 APIs, expanding native geospatial indexing and large-object handling in Lance.
- @wojiaodoubao – Enabled FTS-as-filter in vector search and introduced partition specs in lance-namespace, improving hybrid retrieval and table management flexibility.
-
@jtuglu1
– Added
when_matched_deletesupport tomerge_insert, enabling more expressive upsert and deduplication workflows. - @steFaiz – Implemented distributed range-based BTree indexing, improving scalability for large ordered datasets.
- @niyue – Extended format support for larger minichunk sizes (u32), improving scalability for large tables.
-
@xloya
– Improved vector index behavior and exposed
distance_rangecontrols in Python, enabling finer-grained nearest-neighbor search tuning. - @Mesut-Doner – Introduced a type-safe Rust expression builder API, making programmatic query construction safer and more ergonomic.
- @ChunxuTang – Exposed CypherEngine and vector rerank APIs in lance-graph, enabling tighter integration between graph traversal and vector search.
-
@dcfocus
– Extended
Context.add()with embeddings, bot_id, and session_id support, improving agent and conversational memory workflows.
A heartfelt thank you to our community contributors of lance and lancedb this past month:
@omair445 • @veeceey • @ChinmayGowda71 • @dask-58 • @Abhisheklearn12 • @fzowl • @Ra5hidIslam • @ddupg • @Mesut-Doner • @amanharshx • @VedantMadane • @kysshsy • @wojiaodoubao • @jtuglu1 • @niyue • @majin1102 • @chenghao-guo • @steFaiz • @zhangyue19921010 • @xloya • @yanghua • @valkum • @jonded94 • @lichuang • @hh23485 • @timsaucer • @fenfeng9 • @touch-of-grey • @YinZheng-Sun • @camilesing • @fangbo • @hushengquan • @XuQianJin-Stars • @AndreaBozzo • @rongou • @beinan • @ChunxuTang • @dcfocus • @mikewhb
🤝 Lance Community Sync Recap
Announcing 3 new Lance Maintainers !
- Beinan Wang @beinan (Uber)
- Jinglun @wojiaodoubao (Bytedance Volcano Engine)
- Wyatt Alt @wkalt (LanceDB)
February’s Lance Community Syncs focused on scaling the Lance file format and stabilizing the 2.x release line, including a hint file to eliminate manifest scans, a multi-tenant cache redesign, and experimental column statistics for predicate pushdown.
The next Lance Community Sync will take place on Thursday, March 12, 2026.
