12. AI System Design, Semantic Caching & Cost Architecture

Semantic caching architectures, prompt prefix KV-cache reuse, token bucket rate limiting, and RAG vs. Fine-Tuning trade-off matrices.

Android Architecture Mental Model: In high-scale consumer apps, you would never make duplicate network requests to a billing or search API if an identical request was fulfilled 2 seconds ago—you use OkHttp HTTP cache interceptors or in-memory LRU caches. In generative AI, traditional string-hash caching fails because queries like "How to cancel my membership?" and "Where do I terminate my subscription?" share 0% string overlap. Semantic Caching checks vector similarity against previous query embeddings: if cosine similarity exceeds a threshold (e.g. $> 0.96$), the gateway returns the cached response in < 20ms at zero API cost.

1. The Semantic Caching Lifecycle

A semantic cache sits in front of the LLM inference engine, saving latency and token expenses on common queries:

flowchart TD REQ["Inbound User Prompt"] --> EMB["Embed Query (Fast Embedding Model)"] EMB --> LOOKUP[("Semantic Cache: Redis / pgvector")] LOOKUP --> CHECK{"Nearest Vector Distance <= Threshold? (e.g. <= 0.05)"} CHECK -- Yes: Cache Hit --> CACHED["Return Cached Response (< 25ms, $0.00 cost)"] CHECK -- No: Cache Miss --> LLM["Execute Foundation Model Inference"] LLM --> STORE[("Store New Query Vector + Response")] STORE --> RESP["Return Generated Stream"]

2. Prompt Prefix Caching (Hardware KV-Cache Optimization)

Modern foundation models (Anthropic Claude, Google Gemini, OpenAI) support Prompt Caching. When large system prompts, extensive documentation, or few-shot examples remain identical across calls, the server hardware preserves the pre-computed Key-Value (KV) attention states:

Operation Standard Non-Cached Call Prompt Caching Enabled
Token Cost 100% of input token price billed on every call. Up to 90% discount on cached prefix tokens.
Inference TTFT Full quadratic attention re-computed for prefix tokens. Up to 80% reduction in Time-to-First-Token.
Requirement None. Prefix tokens must match exactly (deterministic ordering required).

3. The Architectural Decision Matrix: Prompting vs. RAG vs. Fine-Tuning

Engineering leaders often face the question: "Should we fine-tune our own model?" The answer is usually no. This matrix guides architectural choices:

Dimension In-Context Prompting Retrieval-Augmented Gen (RAG) Fine-Tuning (PEFT / LoRA)
Primary Objective Guiding style, formatting, and single-turn task instructions. Injecting dynamic, real-time, or private external knowledge. Specialized domain jargon, nuanced tone, or latency reduction.
Knowledge Recency Static (bounded by model training date). Real-time (updates instantly as documents are indexed). Static (requires periodic retraining runs).
Hallucination Risk Moderate to High. Lowest (answers are strictly grounded in citations). Moderate (model internalizes facts into weights).
Engineering Overhead Lowest (immediate prompt changes). Moderate (vector store and ingestion pipeline management). High (training datasets, GPU infrastructure, evals).

4. Dual-Stack Implementations: Semantic Caching Engine

import psycopg
from openai import OpenAI

client = OpenAI()
DB_URL = "postgresql://postgres:postgres@localhost:5432/ai_db"

def get_embedding(text: str) -> list[float]:
    res = client.embeddings.create(input=[text], model="text-embedding-3-small")
    return res.data[0].embedding

def get_semantic_cached_answer(query: str, threshold: float = 0.05) -> str | None:
    query_vector = get_embedding(query)
    
    with psycopg.connect(DB_URL) as conn:
        with conn.cursor() as cur:
            # Query the nearest cosine distance (<=>)
            cur.execute("""
                SELECT response, embedding <=> %s::vector AS distance
                FROM prompt_semantic_cache
                ORDER BY distance ASC
                LIMIT 1;
            """, (query_vector,))
            row = cur.fetchone()
            
            if row and row[1] <= threshold:
                print(f"[CACHE HIT] Cosine distance: {row[1]:.4f}")
                return row[0]
                
    return None # Cache Miss
import kotlin.math.sqrt

data class CachedEntry(val query: String, val vector: FloatArray, val response: String)

class SemanticCache(private val similarityThreshold: Float = 0.95f) {
    private val cache = mutableListOf<CachedEntry>()

    fun get(queryVector: FloatArray): String? {
        for (entry in cache) {
            val similarity = cosineSimilarity(queryVector, entry.vector)
            if (similarity >= similarityThreshold) {
                println("[CACHE HIT] Similarity: $similarity")
                return entry.response
            }
        }
        return null // Cache Miss
    }

    fun put(query: String, queryVector: FloatArray, response: String) {
        cache.add(CachedEntry(query, queryVector, response))
    }

    private fun cosineSimilarity(v1: FloatArray, v2: FloatArray): Float {
        var dot = 0f; var nA = 0f; var nB = 0f
        for (i in v1.indices) {
            dot += v1[i] * v2[i]
            nA += v1[i] * v1[i]
            nB += v2[i] * v2[i]
        }
        return dot / (sqrt(nA) * sqrt(nB))
    }
}

5. Progressive Glossary

Term Technical Definition Android / Systems Analogy
Semantic Cache A cache keyed on vector similarity distance rather than identical raw text strings. Fuzzy search caching with threshold-based hit validation.
Prompt Caching Server infrastructure reusing the attention Key-Value state of static prompt prefixes. Reusing compiled DEX bytecode or static resources without re-parsing.
PEFT / LoRA Parameter-Efficient Fine-Tuning: freezing base model weights and training small low-rank adapter matrices. Android dynamic feature modules or runtime plugin patching.
Token Bucket Limiter A rate-limiting algorithm enforcing maximum Requests-Per-Minute (RPM) and Tokens-Per-Minute (TPM). Network interceptors throttling background WorkManager request bursts.

Sources & Reference Standards

← Module 11: Edge AI & SLMs Curriculum Complete • Return to Index →