Precise Zero-Shot Dense Retrieval without Relevance Labels

HyDE apparently means “Hypothetical Document Embeddings”, which seems to be a kind of generative query expansion/pre-processing https://arxiv.org/abs/2212.10496 https://github.com/texttron/hyde From the abstract: Given a query, HyDE first zero-shot instructs an instruction-following language model (e.g. InstructGPT) to generate a hypothetical document. The document captures relevance patterns but is unreal and may contain false details. Then, an unsupervised contrastively learned encoder~(e.g. Contriever) encodes the document into an embedding vector. This vector identifies a neighborhood in the corpus embedding space, where similar real documents are retrieved based on vector similarity. This second step ground the generated document to the actual corpus, with the encoder's dense bottleneck filtering out the incorrect details.

I could not help but notice the Contriever curve is so much higher on y-axis Recall than the other methods (figure 11 in https://arxiv.org/pdf/2307.03172.pdf). Has anyone come across more recent experiments, results, or papers related to this? I'm acquainted with the: - Contriever 2021 paper https://aclanthology.org/2021.eacl-main.74.pdf - Hyde 2022 https://arxiv.org/pdf/2212.10496.pdf My suspicion is some pre-logic such as is the user's question dense enough then use Hyde with chat history. If anyone has more recent experience with Contrievers, would love to learn more about it! Feel free to contact me directly on LinkedIn. https://www.linkedin.com/in/christybergman/

If you know that your search queries will be actual questions (like in the example you listed), you can possibly use the HyDE[0] to create a hypothetical answer which will usually have an embedding that's closer to the RAG chunks you are looking for. It has the downside that an LLM (rather than just a embedding model) is used in the query path, but it has helped me multiple times in the past to strongly reduce problems with RAG like the ones you outlined, where it likes to latch onto individual words. [0]: https://arxiv.org/abs/2212.10496

The approach is based on hypothetical document embeddings (HyDE). Here is a good description of it in the context of langchain: https://python.langchain.com/docs/use_cases/question_answeri... The original paper proposing this technique can be found here: https://arxiv.org/pdf/2212.10496.pdf

Some people packaged this rather intuitive idea, named it Hyde (Hypothetical Document Embeddings) and wrote a paper about it — https://arxiv.org/abs/2212.10496 Summary — HyDE is a new method for creating effective zero-shot dense retrieval systems that generates hypothetical documents based on queries and encodes them using an unsupervised contrastively learned encoder to identify relevant documents. It outperforms state-of-the-art unsupervised dense retrievers and performs strongly compared to fine-tuned retrievers across various tasks and languages.

Query (→ GPT3 completion) → vector db lookups → GPT3 synthesis The optional step two is used when the lookups are more closely related to an answer's latent space than the original query text. This approach is called HyDE (first published here: https://arxiv.org/abs/2212.10496). The synthesis is also optional. You can essentially summarize your lookups or refine them or do whatever you want at this stage. If you skipped steps 2 and 4, it's just a semantic search engine. If you skip step 2, you're either doing it for latency/performance reasons, or because the user query's embeddings are more similar to the docs in the vector db.

it is a result of trying to align the AI model after it has been trained using self-supervised learning with an addition human feedback step. OpenAI has a model / process called InstructLM or InstructGPT see e.g. https://arxiv.org/pdf/2212.10496.pdf https://arxiv.org/pdf/2203.02155.pdf also an approach to solve the alignment problem. https://en.wikipedia.org/wiki/AI_alignment

Yeah I'm happy with the results I got from embeddings so far. The areas I want to explore there are: 1. What's the ideal size of text to embed? I'm doing whole blog entries right now but I'm confident I can get better results if I divide them up into smaller chunks first - I'm just not sure how best to do that. 2. There's a trick called Hypothetical Document Embeddings (HyDE) where you ask GPT-3 to invent an answer to the user's question, embed THAT fictional answer, then use that embedding to find relevant documents in your corpus. https://arxiv.org/abs/2212.10496

I’ve also used similar approach to build Q&A system for PDF files (my main use case - board game manuals). OpenAIs embeddings are nice to play with. Also there is an easy technique for getting better results with dense search - Hypothetical Document Embeddings (https://arxiv.org/abs/2212.10496).