Tip
TL;DR: HyDE (Hypothetical Document Embeddings, Gao et al. 2022) asks the LLM to draft a fake answer to the query, embeds that answer, and uses it for retrieval instead of the raw query. It wins when queries are short and documents are long-form prose — and loses when the hypothetical drifts. Treat it as a query-side transformation that composes with hybrid search and reranking, not as a replacement for them.
Key takeaways:
- Vector search struggles when the query and the document use different vocabulary and sentence shapes. HyDE is a direct attempt to fix that mismatch.
- The technique is prompt-time, not training-time. You can add it to an existing RAG system in an afternoon without retraining anything.
- The hypothetical answer does not need to be correct. It needs to look like a document in your corpus. That distinction is the whole trick.
- HyDE helps most with short, under-specified queries on prose-heavy corpora. It helps less — and sometimes hurts — when queries are already detailed or when the corpus is keyword-dominant.
- It is orthogonal to hybrid search and reranking. Real systems run all three in series.
- The dominant failure mode is hallucinated drift. A confident hypothetical with invented entities pulls retrieval in the wrong direction, and everything downstream inherits that drift.
The query-document vocabulary mismatch problem
Vector search works by embedding the query, embedding every document chunk, and returning the chunks whose embeddings are closest to the query's. It sounds neat. It often is not.
The problem is that a user's query and the document that answers it rarely look alike. The query is short — a noun phrase, a few keywords, maybe a fragment. The document is long, structured, written in a specific register. They sit in different neighborhoods of embedding space. An embedding model does its best to collapse that gap, but it is doing cross-register matching, which is harder than within-register matching.
Three concrete versions of the mismatch:
Length mismatch. A five-word query produces a dense vector with little surface to work from. A paragraph-long chunk has more signal. Similarity is averaging apples against an orchard.
Register mismatch. A user types "non-compete California valid?". A legal document says "A covenant not to compete with one's employer post-termination is generally void under California Business and Professions Code § 16600..." Same question. Wildly different surface form.
Phrasing mismatch. A user asks "ways to save on taxes as a freelancer". A blog post is titled "Self-Employment Tax Deductions: A Complete Guide". The embedding captures some overlap but not enough to rank it reliably above a post titled "How to Save Money on Everyday Purchases".
You can push against these problems with better embedding models, chunking tweaks, and index tuning. HyDE attacks from the other side — not by making the query space richer, but by making the query itself look more like a document.
How HyDE works, step by step
The full flow is four steps.
Step 1 — User query comes in. Say the query is "effects of vitamin D on muscle recovery?". Short, under-specified, typical of how real users type.
Step 2 — Prompt an LLM to draft a hypothetical answer. The prompt is small and opinionated: "Given this query, write a single paragraph that could plausibly appear in a research paper on this topic. Be specific; use the vocabulary a domain expert would use. Do not fact-check — a rough draft is fine." The model produces something like: "Recent studies suggest that adequate vitamin D levels may enhance muscle-protein synthesis and accelerate recovery after eccentric exercise. Supplementation protocols commonly use 2,000–4,000 IU per day over 8–12 weeks, with outcomes measured via creatine kinase clearance and subjective soreness scales..."
Step 3 — Embed the hypothetical answer, not the query. The paragraph-length hypothetical, now full of domain vocabulary and paper-shaped sentences, is passed to the embedding model. The resulting vector lives in the part of embedding space where real research-paper chunks live.
Step 4 — Retrieve from the vector database using the hypothetical's embedding. Top-k nearest neighbors come back. These are real documents, scored by similarity to a fake one. They flow into the rest of your RAG pipeline as normal — context assembly, generation, citation.
The hypothetical is discarded once retrieval is done. It never reaches the final generator and never appears in the user-facing answer. It is scaffolding for a better vector, nothing more. That framing matters for how you judge HyDE. The hypothetical's factual correctness is not the metric. The correctness of the retrieved documents is.
Worked example — a hypothetical medical-literature research assistant
Hypothetical scenario, not a shipped product. A research assistant runs over 400,000 biomedical paper abstracts. A clinician asks:
"effects of vitamin D on muscle recovery?"
Baseline, no HyDE. Embed the query as-is. Retrieve top-10. Six of the ten are loosely relevant — papers on deficiency and falls in the elderly, general supplementation, bone density, and one directly on recovery. Four are off-topic: vitamin D receptor polymorphisms, dairy intake, seasonal affective disorder, a mood meta-analysis. The top hit is a falls review, not a recovery paper. Top-10 recall against a hand-labeled relevant set: illustrative 0.41.
With HyDE. The same query is passed through Step 2. The small model drafts:
"Vitamin D status appears to influence skeletal-muscle repair after eccentric exercise through effects on calcium handling and myocyte proliferation. Randomized trials examining supplementation in athletes have measured recovery using creatine kinase, subjective soreness, and force production at 24–72 hours post-exercise. Observational data suggest serum 25-hydroxyvitamin D below 30 ng/mL is associated with slower recovery and greater soreness, though heterogeneity across study designs limits firm conclusions."
Embed that paragraph. Retrieve top-10. Eight of the ten are now on the specific recovery question. The falls and bone-density papers drop out. The creatine-kinase trials, which previously ranked around position 15, rise into the top-5.
Top-10 recall: illustrative 0.67.
The baseline was not broken. The HyDE run is not perfect. But the delta is real and directionally consistent with the kind of result reported in the Gao et al. paper. Short biomedical queries over prose-heavy corpora is squarely in HyDE's strike zone.
A useful follow-up exercise: vary the hypothetical. Ask for a longer draft, a shorter one, one in layperson register, one in expert register. Re-run retrieval each time. The variance across those runs is the ceiling on how much HyDE can help — and the floor on how much it can hurt when the hypothetical drifts.
When HyDE wins versus when it fails
Rough mental model.
HyDE wins when:
- Queries are short (under ~10 words) and under-specified.
- The corpus is prose-heavy — research papers, legal cases, internal memos, long-form documentation.
- Query vocabulary differs from document vocabulary (lay vs. expert, informal vs. formal, abbreviated vs. spelled out).
- The embedding model is a general-purpose model, not one fine-tuned on your exact query-document pairs.
HyDE hurts when:
- Queries are already long and specific. The query itself is near the right neighborhood; the hypothetical only adds noise.
- The corpus is keyword-dominant — product catalogs, SKU search, API reference, code, structured data. Vector search is already a weak tool there; HyDE makes it worse.
- Hypothetical answers drift into invented entities that do not exist in the corpus. This is the subtle failure: retrieval looks fine statistically, but it is pulling documents that match invented details, not real ones.
- The generator model and domain are badly matched. A general-purpose model drafting a hypothetical for a very specialized corpus (say, obscure Kubernetes internals) may produce a plausible-sounding but domain-wrong draft that pulls retrieval off-track.
The healthy instinct is to A/B HyDE on a golden set of real queries and measure whether top-k recall actually moves. Do not adopt it because the paper said so. Adopt it because it shows up in your metrics. The RAGAS evaluation walkthrough covers how to measure this for retrieval specifically — context precision and context recall are the numbers that will tell you.
Implementation notes
Prompt design for the hypothetical. Keep it small and opinionated. A prompt that produces consistent hypotheticals is worth more than a clever one that produces variable ones. Four things to specify:
- Target length. Ask for a single paragraph, not a full answer. Longer hypotheticals dilute the embedding signal; too-short ones fail to escape the query's neighborhood. A paragraph tends to land in the same length class as retrieved chunks.
- Register. Tell the model to match the register of documents in the corpus — "write as if this were a paragraph in a research paper" or "as if this were a section of a legal memo." Without this, the model drifts toward a generic chatbot register, which does not match any real corpus.
- Vocabulary permission. Invite domain jargon. "Use the vocabulary a domain expert would use." This is what unlocks the register match.
- No fact-check needed. Tell the model explicitly that the draft need not be correct. That is counterintuitive to most model training, and the explicit permission reduces a bias toward hedging and caveats, which hurt the embedding.
Which model to use. A small, fast model is usually enough. The hypothetical is throwaway scaffolding — paying premium tokens for it is waste. Haiku-class frontier models or small open-source 7B-13B instruction-tuned models are typically fine. The quality of the downstream retrieval is not very sensitive to whether the hypothetical was written by a small model or a large one, provided the prompt is good.
Caching. HyDE adds a generation step per query. Cache the hypothetical (or its embedding) keyed on the query. Most real systems have long-tail query distribution; a small cache covers a lot of traffic.
Latency budget. The added step is one LLM call plus one embedding call per query. On a small model, that is often under 500ms. If your system has a sub-second retrieval budget, prototype carefully.
Observability. Log the hypothetical alongside the retrieved documents. A retrieval miss is often diagnosable in the hypothetical: it invented a concept, adopted the wrong register, or was too short. Without logging, you are debugging blind.
Combining HyDE with other retrieval upgrades
HyDE is one layer of a stack, not a standalone fix. The layers compose.
HyDE plus hybrid search. HyDE improves the query vector; hybrid search adds a keyword signal. The keyword signal is HyDE's failure-mode backstop — when the hypothetical drifts, BM25 anchors retrieval to actual query terms. A fused score (reciprocal rank fusion or similar) from HyDE-vector + BM25 is more robust than either alone.
HyDE plus reranking. HyDE widens the net; a reranker tightens it. HyDE might pull 20 candidates that are all plausibly relevant; a cross-encoder reranker scores each (query, candidate) pair with higher fidelity than a bi-encoder similarity and reorders. The reranker sees the original query, not the hypothetical, so it corrects for HyDE-introduced drift.
HyDE plus query rewriting. Rewrite the query for clarity first — expand abbreviations, split compound questions — then run HyDE on the rewritten version. Rewriting cleans the input; HyDE changes its shape.
HyDE plus chunking strategy. Chunk granularity interacts with hypothetical length. If you chunk at 500 tokens, a paragraph-length hypothetical matches well. If you chunk at 50 tokens, it is length-mismatched. Adjust the hypothetical length to roughly match the median chunk length.
HyDE plus semantic search broadly. HyDE is a refinement within the semantic-search paradigm, not an escape from it. If your semantic search struggles because the embedding model is poor for your domain, HyDE will not rescue it. Fix the foundation first.
The right adoption order, from most-to-least recall gain per unit effort, is roughly: good chunking → hybrid search → reranker → HyDE. HyDE is a refinement, not a headline feature. Teams that try it before the stack is solid often conclude HyDE "does not work" when the real issue is that the rest of the pipeline was the bottleneck. This aligns with the stance in the agentic prompt stack post — retrieval upgrades compose, but only if applied in the right order with each layer's contribution measured independently.
Failure modes
Four anti-patterns worth naming.
Hallucinated drift in the hypothetical. A model confidently invents an entity or framing that does not exist in the corpus. Retrieval pulls documents about the invented concept — or nothing at all — and recall craters. Mitigation: log hypotheticals, spot-check them weekly, and watch for a regression pattern (retrieval getting worse when the query hits a topic the model has weak priors on).
Over-long hypotheticals. A prompt that does not cap length produces full-page hypotheticals. The embedding is an average over more content, which dilutes the signal and makes retrieval less discriminating. Cap at roughly one paragraph or the median chunk length, whichever is smaller.
Register mismatch. The hypothetical is written in a chatbot register while the corpus is formal. The embedding lands in an "AI answers" neighborhood that is sparse or empty in your corpus. Mitigation: explicitly instruct the model to match the register of documents in the corpus, ideally with a one-shot example from a real chunk.
Treating HyDE as a silver bullet. Adopting it without measuring retrieval on a real golden set. The technique helps in specific conditions and hurts in others. The RAGAS walkthrough covers the measurement: context precision and recall before and after HyDE tell you whether it earns its keep.
Our position
Five opinionated stances.
- HyDE is a refinement, not a foundation. Get chunking, embedding choice, and hybrid search right first. HyDE is cheap to add once the rest is solid and often disappointing when bolted onto a weak pipeline.
- Use a small model for the hypothetical. Paying frontier-tier token cost for throwaway scaffolding is waste. A Haiku-class model or a small open-source model does the job at a fraction of the latency and cost.
- Measure, do not assume. HyDE helps in well-defined conditions — short queries, prose corpus, vocabulary mismatch. Outside those, it can hurt. Every team should A/B it on a real golden set before making it default.
- Log the hypothetical. The drafted answer is invisible scaffolding to users but critical evidence for debugging. Every retrieval miss is faster to diagnose when you can read what the model drafted. Treat the hypothetical as a first-class log artifact.
- HyDE composes; it does not replace. Run it alongside hybrid search, reranking, and smart chunking. Each of those addresses a different failure mode. Anyone pitching HyDE as a replacement for the stack is selling a simplification that does not match how strong RAG systems are actually built. Score retrieval with RAGAS and prompts with the SurePrompts Quality Rubric to keep the whole system honest.