Section 0
The Full Pipeline
RAG splits cleanly into two phases that run at very different times: ingestion (once, or whenever your source documents change) and query time (every single user question). Keeping this split in your head prevents a common mistake — re-embedding your whole document set on every request.
| Phase | Steps | Runs |
|---|---|---|
| Ingestion | Chunk documents → embed each chunk → store vectors + text | Once per document, or on update |
| Query time | Embed the query → retrieve top-k chunks → build a grounded prompt → generate | Every user question |
Section 1
Chunking Strategy
Topic 6 used a naive character-count chunker to build intuition. Production chunking respects the structure of the content, because retrieval quality is bounded by chunk quality — you can't retrieve a good answer out of a badly-cut chunk.
| Strategy | When it fits |
|---|---|
| Fixed-size with overlap | Unstructured text (transcripts, freeform notes) — Topic 6's approach, good enough as a baseline |
| Paragraph / section-based | Structured docs (markdown, HTML, PDFs with headers) — split on natural boundaries, not arbitrary character counts |
| One chunk per logical unit | FAQs, support tickets, product entries — each item is already the right retrieval granularity, don't split further |
| Semantic chunking | Long-form prose where topic shifts mid-document — split where embedding similarity between adjacent sentences drops, not on a fixed count |
Size tradeoff: chunks too small lose context (a sentence fragment retrieved with no surrounding meaning); chunks too large dilute relevance (a 2,000-word chunk where only one sentence answers the question, burning context budget on the rest). 200-500 tokens per chunk is a common starting range — tune it against your own content and queries.
Section 2
Retrieval: Top-K Search
Exactly Topic 6's search() function, with one new decision: how many chunks to retrieve. Too few and the answer might be missing the right fact; too many and you're back to Topic 2's context-budget and "lost in the middle" problems from Topic 3.
def retrieve(index: TinySearchIndex, query: str, top_k: int = 4,
min_score: float = 0.3) -> list[str]:
results = index.search(query, top_k=top_k)
# drop anything below a relevance floor — a "best available" match
# that's still unrelated is worse than no context at all
return [doc for doc, score in results if score >= min_score]
The empty-retrieval case matters: what happens when nothing clears
min_score? This isn't an edge case to handle later — it's the case Section 3's prompt has to explicitly instruct the model to say "I don't know" for, tying directly back to Topic 3 Section 1's unhappy-path rule.Section 3
Augmenting the Prompt
This is Topic 3's delimiter pattern, applied to retrieved chunks instead of a single document — numbered so the model (and your citation logic in Section 4) can refer back to a specific source.
def build_rag_prompt(query: str, chunks: list[str]) -> list[dict]:
if not chunks:
context = "No relevant documents were found."
else:
context = "\n\n".join(f"[{i+1}] {chunk}" for i, chunk in enumerate(chunks))
system = """Answer the user's question using ONLY the numbered sources below.
Cite sources inline using [1], [2] etc. matching the source numbers.
If the sources don't contain the answer, say "I don't have that information"
instead of guessing. Never use knowledge outside the provided sources.
Sources:
""" + context
return [
{"role": "system", "content": system},
{"role": "user", "content": query},
]
Section 4
Grounding & Citations
"Grounded" means every claim in the answer traces back to a retrieved source. Citations aren't just a UI nicety — they're a verification mechanism: a user (or an eval harness, Topic 10) can check whether [2] actually says what the model claims it says.
response = client.chat.completions.create(
model="gpt-4.1-mini",
messages=build_rag_prompt(query, retrieved_chunks),
temperature=0, # factual retrieval task — determinism matters, per Topic 2 Section 6
)
answer = response.choices[0].message.content
# answer now looks like: "Refunds are allowed within 30 days [1]."
# — [1] is traceable back to retrieved_chunks[0] for display or verification
A citation the model produces is still just generated text — it can cite the wrong number, or cite correctly while still slightly misrepresenting the source. Citations make grounding checkable, not guaranteed. Topic 10's faithfulness evaluation is how you actually measure whether citations are trustworthy at scale.
Section 5
Failure Modes
Retrieval missThe right document exists but didn't score high enough to make top-k — often a chunking or phrasing mismatch between how the doc is written and how users ask. Fix: better chunking, or Section 6's hybrid search.
Irrelevant-but-retrievedA chunk scores above your threshold but isn't actually useful for this specific question — the model may still try to use it, producing a technically-grounded but unhelpful answer.
Hallucination despite contextThe model ignores the provided sources and answers from its own training knowledge anyway — usually fixed by strengthening the system prompt's "ONLY use the sources" instruction and testing with
temperature=0.Stale indexSource documents changed but the vector store wasn't re-ingested — the system confidently cites information that's no longer true. This is an operational problem, not a prompting one.
Section 6
Hybrid Search & Re-ranking
Two upgrades worth knowing about before you need them — don't add either until plain vector search demonstrably isn't good enough for your data.
| Technique | Solves |
|---|---|
| Hybrid search | Combines vector similarity with traditional keyword search (BM25) — catches cases like exact product codes or acronyms that embeddings alone can blur together. |
| Re-ranking | Retrieve a larger candidate set cheaply (e.g. top-20 by vector similarity), then run a more expensive, more accurate re-ranking model over just those 20 to pick the final top-4 — better precision at a fraction of the cost of scoring everything that way. |
Section 7 · Checkpoint
Capstone: A Grounded Q&A System
Combine Topic 6's TinySearchIndex with this topic's retrieval threshold, grounded prompt, and citations into one callable function.
from openai import OpenAI
client = OpenAI()
def answer_question(index: TinySearchIndex, query: str,
top_k: int = 4, min_score: float = 0.3) -> str:
results = index.search(query, top_k=top_k)
chunks = [doc for doc, score in results if score >= min_score]
if not chunks:
context = "No relevant documents were found."
else:
context = "\n\n".join(f"[{i+1}] {chunk}" for i, chunk in enumerate(chunks))
system = """Answer using ONLY the numbered sources below, citing them
inline as [1], [2] etc. If the sources don't contain the answer, say
"I don't have that information." Never use outside knowledge.
Sources:
""" + context
response = client.chat.completions.create(
model="gpt-4.1-mini",
messages=[{"role": "system", "content": system},
{"role": "user", "content": query}],
temperature=0,
)
return response.choices[0].message.content
if __name__ == "__main__":
index = TinySearchIndex([
"Reset your password from the account settings page.",
"Our refund policy allows returns within 30 days of purchase.",
"The mobile app supports offline mode since version 4.2.",
"You can export your data as a CSV from the settings menu.",
])
print(answer_question(index, "can I get my money back"))
print(answer_question(index, "does the app support dark mode"))
Expected output: the first call answers correctly with a
[1]-style citation pointing at the refund chunk. The second call — asking about something not in the index — should return "I don't have that information," not a plausible-sounding guess.min_score to 0 and observed the second query's behavior change — connects retrieval threshold directly to hallucination risk