Neural information retrieval (IR) has greatly advanced search and other knowledge-intensive language tasks. While many neural IR methods encode queries and documents into single-vector representations, late interaction models produce multi-vector representations at the granularity of each token and decompose relevance modeling into scalable token-level computations. This decomposition has been shown to make late interaction more effective, but it inflates the space footprint of these models by an order of magnitude. In this work, we introduce ColBERTv2, a retriever that couples an aggressive residual compression mechanism with a denoised supervision strategy to simultaneously improve the quality and space footprint of late interaction. We evaluate ColBERTv2 across a wide range of benchmarks, establishing state-of-the-art quality within and outside the training domain while reducing the space footprint of late interaction models by 6--10$\times$.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
Chunking strategy is really difficult and, like you say, so important to RAG. I'm currently battling with it in a "Podcast archive -> active social trend" clip-finder app I'm working on. You have to really understand your source material and how it's formatted, consider preprocessing, consider when and where semantic breaks happen and how you can deterministically handle that in the specific domain. Adjacency similarity is a must, otherwise you leave perfectly cromulent results on the table because they didn't have the right cosine score in a vacuum.
There is some early stuff from Apple's research labs and the ColBERT team in late attention embedding (https://arxiv.org/abs/2112.01488) which looks to ease that burden, and generate compressed token-level embeddings across a document.
We're building LintDB to productionize late interaction retrieval.
If you're not familiar with late interaction, here's the colBERTv2 paper: https://arxiv.org/pdf/2112.01488.pdf
We built LintDB because we had a hard time getting the retrieval performance we wanted in our RAG application. colBERT calculates a document's score based on the maximum similarity across multiple vectors per document, and this was more successful for our task. The only problem was that traditional vector databases didn't support it.
Vespa supports this type of indexing, actually. But the learning curve was high and I was concerned about how we'd operationalize it.
Some of the features of LintDB:…
Read the full comment
- Embeddable. You can import LintDB as a Python package and get started immediately.
- Bit level compression. We fully support PLAID and compressing token embeddings into bits. 128 dimension vectors can compress down to 16 bytes.
- Built on top of rocksDB for fast on-disk access. This also lets us take advantage of merging rocksdb indices. Index your data across multiple machines and merge them together for serving.
- Multi-tenant support.
We're seeing new research on bitvector searching and late interaction, and we want to be the first ones to support it. Our roadmap also includes adding explainability to the results, which is possible thanks to late interaction.
repo: https://github.com/DeployQL/LintDB
documentation: https://deployql.github.io/LintDB/
This is an area that's dear to me, I'm the cofounder of Vectara and have been working with embedding-based semantic search, aka neural search or neural IR, since 2017.
To whether Google uses semantic search, the answer is yes, very heavily [1][2]. Not only that, but they have led, and continue to lead, much of the pioneering research in NLP and neural IR for the past decade [3][4][5].
Technical challenges lie along a few primary dimensions. The first has been search quality, because, while early neural systems like Google Talk to Books [5][6] demonstrated the potential of these techniques, benchmarks like BEIR [7], released a few years later, in 2020, showed that the best…
Read the full comment
keyword retrieval algorithms still outperformed neural techniques in general settings.
The landscape since then has shifted very rapidly: In 2022, for the first time, neural search methods outperformed BM25 on BEIR. This includes late interaction [8], sparse encoding [9], and, most challengingly, dense encoding [10] systems.
The second technical challenge is scalability. After decades of infrastructure optimization, keyword systems scale well to very large corpora, while semantic systems struggle to achieve the same scale. The k-d tree approach presented in the article, for example, while good for experimentation, would be difficult to productionize, as-is, in a large-scale system.
However, research into scaling dense vector retrieval has received a lot of focus recently [11], so I'm confident this will change.
I'll close by saying your observation about being stuck with keyword search in a lot of apps is accurate, but I expect that to change soon. It's becoming easier to embed neural models everywhere, and I think that distilled models in the 5-50mb size range can feasibly power semantic search everywhere you press Ctrl-F today.
[1] https://blog.google/products/search/search-language-understa...
[2] https://blog.google/products/search/introducing-mum/
[2] https://arxiv.org/abs/1706.03762
[3] https://arxiv.org/abs/1810.04805
[4] https://arxiv.org/abs/1907.04307
[5] https://books.google.com/talktobooks/
[6] https://ai.googleblog.com/2018/04/introducing-semantic-exper...
[7] https://arxiv.org/abs/2104.08663
[8] https://arxiv.org/abs/2112.01488
[9] https://arxiv.org/abs/2109.10086
[10] https://arxiv.org/pdf/2112.09118.pdf
[11] https://www.microsoft.com/en-us/research/uploads/prod/2021/1...
Abstract If future net-zero emissions energy systems rely heavily on solar and wind resources, spatial and temporal mismatches between resource availability and electricity demand may challenge system reliability.