LanceDB

🛠 Breaking Changes

• feat(python)!: align Permutation.with_format("torch") with HuggingFace set_format("torch") by @omkar-334 in https://github.com/lancedb/lancedb/pull/3369

🎉 New Features

• feat: add an elastic dataloader as an iterable dataset by @westonpace in https://github.com/lancedb/lancedb/pull/3509
• feat: support date, datetime, bytes, and Decimal literals in expr builder by @lakshjain7 in https://github.com/lancedb/lancedb/pull/3235
• feat(python)!: align Permutation.with_format("torch") with HuggingFace set_format("torch") by @omkar-334 in https://github.com/lancedb/lancedb/pull/3369
• feat: add get_lsm_write_spec to read the installed LSM write spec by @hamersaw in https://github.com/lancedb/lancedb/pull/3631
• feat: add Tencent COS and GooseFS object store support via new feature flags by @XuQianJin-Stars in https://github.com/lancedb/lancedb/pull/3526
• feat: expose Lance metrics via OpenTelemetry in Python and Node by @wjones127 in https://github.com/lancedb/lancedb/pull/3609
• feat(python): blob v2 fetch API by @geruh in https://github.com/lancedb/lancedb/pull/3578
• feat(python): added support for WatsonxReranker component by @MateuszOssGit in https://github.com/lancedb/lancedb/pull/3642
• feat: add table FTS query tokenization by @jackye1995 in https://github.com/lancedb/lancedb/pull/3659
• feat: support distributed analyze plan metrics in clients by @jackye1995 in https://github.com/lancedb/lancedb/pull/3675
• feat(wheels): publish lancedb-compat for pre-Haswell x86_64 hosts by @tobocop2 in https://github.com/lancedb/lancedb/pull/3327
• feat: skill references to work with jobs (incl server connection) by @dantasse in https://github.com/lancedb/lancedb/pull/3683
• feat: add remote branch diff and merge client APIs by @geruh in https://github.com/lancedb/lancedb/pull/3686
• feat: branch skill updates for merge by @dantasse in https://github.com/lancedb/lancedb/pull/3685

🐛 Bug Fixes

• fix: handle empty/wrong-length vectors returned by embedding functions by @Spectual in https://github.com/lancedb/lancedb/pull/3192
• fix(python): resolve Ollama embedding serialization error in create_table by @Missing-Identity in https://github.com/lancedb/lancedb/pull/3583
• fix(python): flatten_columns raises when flatten=False by @PranavAchar01 in https://github.com/lancedb/lancedb/pull/3629
• fix: avoid manifest writes for read-only directory namespace opens by @Xuanwo in https://github.com/lancedb/lancedb/pull/3635
• fix(python): preserve zero distance bounds in hybrid search by @u70b3 in https://github.com/lancedb/lancedb/pull/3652
• fix(python): gemini batching, user agent and variable dims by @markmcd in https://github.com/lancedb/lancedb/pull/3618
• fix(python): preserve phrase semantics in sync queries by @u70b3 in https://github.com/lancedb/lancedb/pull/3654
• fix(python): fill bad vector values element-wise by @prrao87 in https://github.com/lancedb/lancedb/pull/3613
• fix(rust): skip embedding functions for empty batches by @u70b3 in https://github.com/lancedb/lancedb/pull/3646
• fix(python): route local sync namespace operations through rust by @jackye1995 in https://github.com/lancedb/lancedb/pull/3606
• fix(python): bound scanner memory for wide-row bulk ingestion by @wjones127 in https://github.com/lancedb/lancedb/pull/3625
• fix(python): raise clear ValueError when vector column cannot be infe… by @Vitaliy-Pikalo in https://github.com/lancedb/lancedb/pull/3567
• fix(rust): bound remote insert request size to avoid ingestion timeouts by @wjones127 in https://github.com/lancedb/lancedb/pull/3630
• fix(node): sanitize Map fields across Arrow versions by @u70b3 in https://github.com/lancedb/lancedb/pull/3650
• fix(python): allow selection of _rowid in Permutation by @erandagan in https://github.com/lancedb/lancedb/pull/3133

📚 Documentation

• docs: add LanceDB agent skill for portable pipelines by @prrao87 in https://github.com/lancedb/lancedb/pull/3662
• docs(java): fill Java API refe

<a href="https://cloud.lancedb.com" target="_blank">

<img src="https://github.com/user-attachments/assets/92dad0a2-2a37-4ce1-b783-0d1b4f30a00c" alt="LanceDB Cloud Public Beta" width="100%" style="max-width: 100%;">

</a>

<div align="center">

LanceDB

Website

Blog

Discord

Twitter

LinkedIn

<img src="docs/src/assets/lancedb.png" alt="LanceDB" width="50%">

The Multimodal AI Lakehouse

How to Install ✦ Detailed Documentation ✦ Tutorials and Recipes ✦ Contributors

The ultimate multimodal data platform for AI/ML applications.

LanceDB is designed for fast, scalable, and production-ready vector search. It is built on top of the Lance columnar format. You can store, index, and search over petabytes of multimodal data and vectors with ease.

LanceDB is a central location where developers can build, train and analyze their AI workloads.

</div>

<br>

Demo: Multimodal Search by Keyword, Vector or with SQL

<img max-width="750px" alt="LanceDB Multimodal Search" src="https://github.com/lancedb/lancedb/assets/917119/09c5afc5-7816-4687-bae4-f2ca194426ec">

Star LanceDB to get updates!

<details>

<summary>⭐ Click here ⭐ to see how fast we're growing!</summary>

<picture>

<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=lancedb/lancedb&theme=dark&type=Date">

<img width="100%" src="https://api.star-history.com/svg?repos=lancedb/lancedb&theme=dark&type=Date">

</picture>

</details>

Key Features:

• Fast Vector Search: Search billions of vectors in milliseconds with state-of-the-art indexing.
• Comprehensive Search: Support for vector similarity search, full-text search and SQL.
• Multimodal Support: Store, query and filter vectors, metadata and multimodal data (text, images, videos, point clouds, and more).
• Advanced Features: Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure. GPU support in building vector index.

Products:

• Open Source & Local: 100% open source, runs locally or in your cloud. No vendor lock-in.
• Cloud and Enterprise: Production-scale vector search with no servers to manage. Complete data sovereignty and security.

Ecosystem:

• Columnar Storage: Built on the Lance columnar format for efficient storage and analytics.
• Seamless Integration: Python, Node.js, Rust, and REST APIs for easy integration. Native Python and Javascript/Typescript support.
• Rich Ecosystem: Integrations with LangChain 🦜️🔗, LlamaIndex 🦙, Apache-Arrow, Pandas, Polars, DuckDB and more on the way.

How to Install:

Follow the Quickstart doc to set up LanceDB locally.

API & SDK: We also support Python, Typescript and Rust SDKs

| Interface | Documentation |

|-----------|---------------|

| Python SDK | https://lancedb.github.io/lancedb/python/python/ |

| Typescript SDK | https://lancedb.github.io/lancedb/js/globals/ |

| Rust SDK | https://docs.rs/lancedb/latest/lancedb/index.html |

| REST API | https://docs.lancedb.com/api-reference/rest |

Join Us and Contribute

We welcome contributions from everyone! Whether you're a developer, researcher, or just someone who wants to help out.

If you have any suggestions or feature requests, please feel free to open an issue on GitHub or discuss it on our Discord server.

Check out the GitHub Issues if you would like to work on the features that are planned for the future. If you have any suggestions or feature requests, please feel free to open an issue on GitHub.

Contributors

<a href="https://github.com/lancedb/lancedb/graphs/contributors">

<img src="https://contrib.rocks/image?repo=lancedb/lancedb" />

</a>

Stay in Touch With Us

<div align="center">

</br>

Website

Blog

Discord

Twitter

LinkedIn

</div>