Most RAG write-ups focus on retrieval — embedding models, chunking, re-rankers, vector stores. Those matter, but they're the plumbing. What decides whether retrieved snippets become a useful answer or a confident lie is the prompt wrapped around them. Under the context engineering pillar, this post covers four prompt patterns that separate RAG that ships from RAG that demos: explicit citation, groundedness framing, chunk formatting, and negative handling. Skip them and you ship hallucinated citations. Apply them and the same retrieval stack gets noticeably more reliable.
RAG Prompts vs RAG Architecture
RAG has two layers that people collapse into one. Architecture is the pipeline — embed, store, retrieve top-k, re-rank, pass to a model. Prompting is what the model actually sees: the instructions that frame retrieved chunks, the format they arrive in, the rules for using them.
Architecture decides what gets retrieved. Prompting decides whether what's retrieved gets used correctly. State-of-the-art retrieval feeding a sloppy prompt still hallucinates; decent retrieval feeding a careful prompt is honest about its gaps. The prompt layer is where most failure modes live and the cheapest thing to fix — no reindexing, just text changes. The four patterns below operate entirely at that layer.
Pattern 1: Explicit Citation
The biggest lever: require the model to name the source chunk for every factual claim. Without this, the model paraphrases retrieved content and mixes in pretraining; you can't tell what came from the corpus. With it, each claim is tied to a chunk the model says supports it — which you can verify.
What the pattern looks like. Give each chunk a stable identifier — an ID, a title, or a label like [Source 1]. Instruct the model to include that identifier inline whenever it uses the chunk, tied to specific claims, not a bibliography at the end.
What this buys you. Auditable answers. A reviewer can check each citation against the chunk it names. The mere presence of the requirement tends to reduce hallucination because the model is now accountable for claims.
What this doesn't fix alone. The model can cite [Source 2] while pulling from pretraining. Pattern 1 composes with Pattern 2 to close that gap.
Pattern 2: Groundedness Framing
Tell the model explicitly that it can only use retrieved context, and if the answer isn't there, it must say so.
Why this is a separate pattern. Default model behavior is to answer — from retrieval if it helps, from pretraining if retrieval falls short. Groundedness framing overrides that default: retrieval is the only allowed source; outside knowledge is out of scope; "I don't know based on the provided sources" is preferred over guessing.
What the pattern looks like. Language like: "Answer using only the information in the provided sources. If the sources don't contain the answer, say 'The provided sources don't cover this.' Do not use outside knowledge." The phrasing matters less than the trio — scope restriction, explicit admission path, and prohibition.
What this buys you. Refusals instead of fabrications when retrieval misses. Refusals feel like demo failures but are correct production answers — they route to expansion or escalation while a wrong confident answer silently burns trust.
The trade-off. Strict groundedness can cause refusals when the model could have synthesized correctly across chunks. Tune to allow inference within sources while forbidding inference outside them. For orchestration around what to retrieve, see dynamic context assembly patterns.
Pattern 3: Chunk Formatting
Wrap each retrieved chunk in consistent, machine-navigable delimiters. Models handle structured input better than prose blobs — three paragraphs concatenated with newlines have blurry boundaries the model struggles to cite. The same three paragraphs wrapped in tags with IDs have clear starts, ends, and identifiers for the citation pattern to latch onto.
What the pattern looks like. Pick a delimiter convention and stick to it. XML-style tags work well (<source id="3" title="Billing FAQ">...</source>), as do consistent markers (--- Source 3: Billing FAQ ---). Every chunk the same shape, a stable ID, unambiguous boundaries.
Include useful metadata. Title, source, date, section heading. A chunk labeled [Source 1: Refund Policy, updated 2026-02-15] beats one labeled [Source 1]. The metadata costs a few tokens and buys the model something to reason about.
Put chunks in a stable position. Chunks belong in the variable region of the prompt, in a predictable place — typically after the question and before the final instruction. Stable structure compounds with hierarchical context loading when the prompt has multiple context tiers.
Pattern 4: Negative Handling
Plan explicitly for the case where retrieval returns nothing relevant. Every RAG system hits this: the user asks about something the corpus doesn't cover; retrieval returns its top-k anyway; the model receives irrelevant chunks and an instruction to answer. Without negative handling, the model either falls back on pretraining or stretches the chunks into an answer they don't support.
The three negative-handling moves.
- Admit the gap. Groundedness framing handles this — give the model explicit permission to say none of the provided sources answer this.
- Escalate. Route no-match cases to a different retrieval strategy, a different model, or a human. Surface the signal, don't swallow it.
- Fall back to a bounded source. When "I don't know" isn't acceptable, define an explicit fallback — "If the provided sources don't cover the question, answer from general knowledge but prefix with
[No source match — general knowledge]." Preserves the audit trail while allowing graceful degradation.
Detection matters as much as handling. Low retrieval scores, chunks that don't mention the query topic, or the model's own "the sources don't cover this" are catchable signals. Silent failures — answers that look fine but came from pretraining — are the dangerous case, and citations plus groundedness reveal them.
The Four Patterns Side by Side
| Pattern | Job | Failure mode without it |
|---|---|---|
| Explicit citation | Tie each claim to a chunk | Source vs pretraining indistinguishable |
| Groundedness framing | Restrict scope to retrieved context | Confident answers from pretraining |
| Chunk formatting | Make retrieved content navigable | Blurred boundaries, citation errors |
| Negative handling | Behave sensibly on misses | Stretched answers, silent fallback |
The patterns compose. Citations without groundedness still let the model add unsupported material. Groundedness without citations forces in-scope answers but leaves them unauditable. Chunk formatting without negative handling still hallucinates on misses. You want all four.
Example: A RAG Prompt Using All Four Patterns
Illustrative and hypothetical — a starting template to adapt, not a prescription.
You are a support assistant. Answer the user's question using ONLY the
information in the <sources> block below.
Rules:
1. For every factual claim, cite the source inline as [Source N] where
N is the source's id attribute.
2. Do not use information outside the provided sources. If the sources
do not answer the question, respond exactly:
"The provided sources don't cover this question. Please rephrase or
contact support."
3. You may combine information across sources if consistent. If they
conflict, note the conflict explicitly and cite both.
4. Keep answers under 150 words.
<sources>
<source id="1" title="Refund Policy" updated="2026-02-15">
Refunds are available within 30 days of purchase for any reason.
After 30 days, refunds are handled case-by-case by support.
</source>
<source id="2" title="Billing FAQ" updated="2026-03-01">
Subscriptions can be cancelled at any time from the billing page.
Cancellation takes effect at the end of the current billing period.
</source>
<source id="3" title="Trial Terms" updated="2026-01-10">
Free trials last 14 days and do not require a payment method.
</source>
</sources>
User question: Can I get a refund 45 days after purchase?All four patterns are visible: citation format (rule 1), scope restriction and refusal string (rule 2), delimited chunks with stable IDs and metadata, and a prescribed no-match response. Change the sources, question, and refusal text; the skeleton ports across domains. When the prompt also needs few-shot demos, see few-shot example selection guide.
Failure Modes to Watch
- Hallucinated citations. Model cites
[Source 2]for a claim not in Source 2. Catch with automated audit. - Ignoring retrieval. Generic answers that barely use chunk-specific phrasing. Grounded answers should echo the chunks.
- Citing the wrong chunk. Claim is correct, cited chunk is relevant, but a different chunk is the real source. Matters for audit.
- Over-refusing. Strict groundedness plus borderline-relevant chunks can refuse when synthesis was possible. Tune to allow in-scope inference.
- Swallowed no-match signals. Model recognizes no-match internally then answers anyway. Instruct for early exit.
Testing RAG Prompts
Eyeballing doesn't scale — you need a test set and a check for each pattern.
- Citation accuracy. Parse citations and verify the cited chunk supports the claim. Substring or fuzzy-match catches extractive cases; paraphrased claims need a model-as-judge.
- Groundedness. Feed questions you know aren't in the corpus and measure refusal rate. Low rate means the framing is too weak.
- Chunk coverage. For multi-chunk synthesis questions, check whether all relevant chunks get cited. Low coverage means cherry-picking.
- No-match behavior. Inject deliberate no-match cases. Verify the model follows your negative-handling move instead of hallucinating.
Run these as regression tests on every prompt, retrieval, or model change. Small tweaks can silently shift groundedness.
Common Anti-Patterns
- No chunk identifiers. Asking for citations without giving chunks stable IDs. Positional citations break when retrieval reorders.
- Groundedness-by-hope. Assuming retrieved chunks are "enough of a hint" without instructing scope. The default is to be useful — override it explicitly.
- Bibliography-only citations. Sources listed at the end instead of inline against specific claims. Harder to audit.
- Forgetting the negative case. A prompt that assumes retrieval always succeeds. It won't.
- Mixing chunks into prose. Concatenating retrieved text without delimiters. Citation becomes unreliable, audit impossible.
- One prompt for every corpus. Copy-pasting the same RAG prompt across different domains. Refusal text, metadata fields, and fallback policy should reflect the domain.
FAQ
Can I skip citations if I have a re-ranker?
No — they solve different problems. A re-ranker puts relevant chunks higher; citations let you verify the model actually used them. Even with perfect retrieval, the model can ignore chunks or mix in pretraining. Citations catch that independently.
What's the right chunk delimiter?
Whatever you apply consistently. XML-style tags (<source id="1">) are easy to parse for audit. Markdown headers or bracket markers work too. The pattern matters more than the syntax.
Won't groundedness framing hurt helpfulness?
It shifts the mix — more refusals, fewer hallucinations. For most production use cases, a refusal beats a confident wrong answer. If your use case genuinely tolerates best-effort answers, use the labeled-fallback negative-handling move instead.
How do I detect hallucinated citations at scale?
Substring or fuzzy-match the cited chunk against the claim for extractive cases. For paraphrased claims, use a small model as a judge. Run on a sample of production answers continuously; any drop in pass rate is a signal.
Does this replace the retrieval pipeline?
No — it sits on top. The patterns assume retrieval returns roughly relevant chunks most of the time. If retrieval is broken, prompting can't fix that. The patterns make the most of whatever retrieval produces and surface retrieval failures instead of hiding them.
Wrap-Up
RAG architecture gets chunks to the model; RAG prompting decides what happens next. Four patterns turn that handoff from fragile to reliable. Citation makes claims auditable. Groundedness keeps the model inside retrieved context. Chunk formatting gives it a clean structure to cite against. Negative handling defines what happens when retrieval whiffs, instead of quietly falling back on pretraining. None require better vectors or bigger models — they're prompt-layer changes you can ship today, usually the highest-leverage moves in a RAG stack.
For the broader frame, the context engineering pillar and glossary entry. For the layer above, hierarchical context loading and dynamic context assembly patterns.