05. Production RAG Architecture | AI Systems Architecture

Android Architecture Mental Model: RAG is identical to the Offline-First Repository Pattern in Android Clean Architecture. When a ViewModel requests domain data, the repository doesn't rely solely on static in-memory parameters; it fetches fresh raw data from a local Room database or remote Retrofit API, filters and transforms the DTOs into a clean UI state, and binds it to the screen. In RAG, the LLM is merely the presentation engine; external vector storage and rerankers act as the local cache and query planner providing ground truth.

1. The End-to-End RAG Topology

A production RAG pipeline consists of two decoupled operational phases: the asynchronous Ingestion Pipeline (data preparation) and the synchronous Query Pipeline (runtime retrieval and generation).

flowchart TD subgraph Ingestion["Phase 1: Ingestion Pipeline (Offline/Async)"] DOC["Raw Knowledge Base (Markdown, PDFs, Code)"] --> CHUNK["Chunker Engine (512 tokens + 64 overlap)"] CHUNK --> EMB1["Embedding Model (text-embedding-3-small)"] EMB1 --> VDB[("Vector DB (pgvector / HNSW)")] end subgraph Runtime["Phase 2: Query & Inference Pipeline (Real-Time)"] USER["Client Query (Android App)"] --> EMB2["Query Embedding Generator"] EMB2 --> VSEARCH["Dense Retrieval (Top-K ~ 20)"] VDB -.->|Index Scan| VSEARCH VSEARCH --> RERANK["Cross-Encoder Reranker (Top-N ~ 4)"] RERANK --> PROMPT["Context Assembly (System + Context + Query)"] PROMPT --> LLM["LLM Inference Engine (Gemini / Claude / GPT)"] LLM --> CLIENT["Streaming Grounded Response"] end

2. Dense Retrieval vs. Cross-Encoder Reranking

Naive RAG retrieves the top $K$ chunks using cosine similarity and immediately dumps them into the LLM prompt. This frequently injects low-relevance noise. Production systems use a two-stage retrieval pipeline:

Architecture	Bi-Encoder (First Stage)	Cross-Encoder Reranker (Second Stage)
Mechanism	Encodes Query and Document separately into dense vectors ($\mathbf{u}$ and $\mathbf{v}$); computes dot product.	Feeds Query and Document simultaneously into the model layers; cross-attention scores relevance directly.
Speed / Throughput	Extremely fast ($O(1)$ lookup with vector index). Searches millions of rows in milliseconds.	Computationally expensive; scales linearly with candidates ($O(N)$ forward passes).
Role in Pipeline	Candidate generation: narrows 1,000,000 documents down to Top 20–50 candidates.	Precision filtering: scores and resorts the Top 20 down to the 3–5 most relevant chunks.

3. The Context Assembly Contract

Retrieval context must be formatted deterministically with strict boundary tags. This isolates user inputs, reinforces source attribution, and resists prompt injection.

flowchart LR SYS["System Invariants:
Answer ONLY using provided sources."] --> PROMPT["Final Prompt Envelope"] CTX["Retrieved & Reranked Context:
<context>...</context>"] --> PROMPT Q["User Query:
<question>...</question>"] --> PROMPT PROMPT --> LLM["LLM Engine"]

4. Dual-Stack Implementations: Minimal RAG Pipeline

Executing candidate retrieval, prompt formatting, and grounded generation:

import numpy as np
from openai import OpenAI

client = OpenAI()

# 1. Mock Vector Store
corpus = [
    {"id": "doc_1", "text": "WorkManager handles deferrable, guaranteed background work in Android."},
    {"id": "doc_2", "text": "Coroutines with Dispatchers.Main should only be used for UI interactions."},
    {"id": "doc_3", "text": "pgvector adds vector similarity search capabilities directly to PostgreSQL."}
]

def get_embedding(text: str) -> list[float]:
    res = client.embeddings.create(input=[text], model="text-embedding-3-small")
    return res.data[0].embedding

# Pre-compute corpus vectors
for doc in corpus:
    doc["vector"] = get_embedding(doc["text"])

def retrieve(query: str, top_k: int = 2) -> list[dict]:
    q_vec = np.array(get_embedding(query))
    scored = []
    for doc in corpus:
        score = np.dot(q_vec, np.array(doc["vector"]))
        scored.append((score, doc))
    scored.sort(key=lambda x: x[0], reverse=True)
    return [doc for _, doc in scored[:top_k]]

def generate_grounded_answer(query: str) -> str:
    retrieved_docs = retrieve(query, top_k=1)
    context_str = "\n".join([f"- {d['text']}" for d in retrieved_docs])

    prompt = f"""Use ONLY the following context to answer the user request.
If the answer cannot be found in the context, say 'I cannot find that in the documents.'

<context>
{context_str}
</context>

User Question: {query}"""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0
    )
    return response.choices[0].message.content

print(generate_grounded_answer("How should I execute guaranteed background tasks in Android?"))

import dev.langchain4j.data.segment.TextSegment
import dev.langchain4j.model.openai.OpenAiChatModel
import dev.langchain4j.model.openai.OpenAiEmbeddingModel
import dev.langchain4j.store.embedding.inmemory.InMemoryEmbeddingStore

fun main() {
    val apiKey = System.getenv("OPENAI_API_KEY")
    val embeddingModel = OpenAiEmbeddingModel.builder().apiKey(apiKey).modelName("text-embedding-3-small").build()
    val chatModel = OpenAiChatModel.builder().apiKey(apiKey).modelName("gpt-4o-mini").temperature(0.0).build()
    
    val store = InMemoryEmbeddingStore<TextSegment>()

    // 1. Ingestion Phase
    val docs = listOf(
        "WorkManager handles deferrable, guaranteed background work in Android.",
        "Coroutines with Dispatchers.Main should only be used for UI interactions.",
        "pgvector adds vector similarity search capabilities directly to PostgreSQL."
    )
    docs.forEach { text ->
        val segment = TextSegment.from(text)
        val embedding = embeddingModel.embed(segment).content()
        store.add(embedding, segment)
    }

    // 2. Query & Retrieval Phase
    val query = "How should I execute guaranteed background tasks in Android?"
    val queryEmbedding = embeddingModel.embed(query).content()
    val matches = store.findRelevant(queryEmbedding, 1)

    val context = matches.joinToString("\n") { "- ${it.embedded().text()}" }

    // 3. Grounded Synthesis
    val prompt = """
        Use ONLY the following context to answer the question.
        <context>
        $context
        </context>
        Question: $query
    """.trimIndent()

    val answer = chatModel.generate(prompt)
    println(answer)
}

5. Progressive Glossary

Term	Technical Definition	Mobile / Systems Analogy
Bi-Encoder	Model that embeds queries and documents into separate vectors for independent comparison.	Generating distinct SHA-256 hashes for two entities to quickly check inequality.
Cross-Encoder	Model that passes query and candidate pairs together through full multi-head attention.	Deep property-by-property equality comparison via custom `equals()` logic.
Hallucination	An output where the model asserts false facts with high linguistic confidence.	A dangling pointer or stale UI state rendering data from an invalid cursor snapshot.
Grounding	Constraining the LLM's response generation exclusively to explicit context provided in the prompt.	Validating user view states strictly against local Room DB entities without inventing state.

Sources & Reference Standards

Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS)
Cohere Architecture Whitepaper: Rerankers and Cross-Encoder Scoring
LangChain4j / LangChain Architecture Documentation: Retrieval Systems

← Module 04: Embeddings Proceed to Module 06: Postgres & pgvector →