10. AI Evaluation & System Observability

Automated benchmark testing, the RAG Triad, LLM-as-a-Judge guardrails, latency tracing, and telemetry pipelines.

Android Architecture Mental Model: In mobile platforms, code quality is guarded by deterministic test runners (JUnit, Espresso) and CI/CD quality gates (SonarQube, Crashlytics, Firebase Performance). If an assertion fails, the build breaks immediately. In generative AI systems, testing is probabilistic: identical prompts can yield varying syntactic phrasing while maintaining semantic correctness. AI Evaluation (Evals) replaces binary assertions with statistical confidence scoring, semantic similarity assertions, and LLM-as-a-Judge automated test harnesses to prevent silent performance regressions.

1. The RAG Evaluation Triad

Evaluating an AI application by reading final responses leads to misleading conclusions. The industry standard framework evaluates three decoupled vectors of the pipeline:

flowchart TD subgraph Triad["The RAG Triad Architecture"] Q["User Query"] C["Retrieved Context Chunks"] A["Model Answer Output"] end Q -->|Context Relevance: Is context relevant to the query?| C C -->|Groundedness / Faithfulness: Is answer derived ONLY from context?| A Q -->|Answer Relevance: Does answer address the query?| A
Evaluation Metric Evaluation Formula / Objective Root Cause When Score Drops
Context Relevance Fraction of retrieved chunks that contain direct evidence for the query. Poor embedding model, improper chunk sizing, or poor vector indexing.
Groundedness (Faithfulness) Percentage of claims in the generated answer that can be traced to retrieved context. Model hallucination, weak system prompt rules, or high temperature ($T > 0.3$).
Answer Relevance Degree to which the final response matches the user's explicit question. Model drifted, produced verbose fluff, or misidentified intent.

2. Latency Anatomy & Tracing

Evaluating performance in an AI streaming gateway requires tracking two distinct latency components:

gantt title AI Request Lifecycle Telemetry dateFormat X axisFormat %s ms section Pre-Processing Auth & Rate Limit :0, 150 Vector Embedding & Retrieval :150, 450 section Upstream Inference TTFT (Time-to-First-Token) :450, 1200 Inter-Token Latency (Streaming Delivery) :1200, 3200 section Post-Processing Telemetry Export & Metric Logging :3200, 3250

3. Dual-Stack Implementations: LLM-as-a-Judge Evaluation Test

An automated CI/CD evaluation harness checking whether an AI response introduces ungrounded hallucinations:

import pytest
from pydantic import BaseModel, Field
from openai import OpenAI

client = OpenAI()

class FaithfulnessJudge(BaseModel):
    is_grounded: bool = Field(description="True if all claims exist in context; False if hallucinated.")
    reasoning: str = Field(description="Step-by-step evidence justification.")

def evaluate_faithfulness(context: str, answer: str) -> FaithfulnessJudge:
    prompt = f"""You are a strict QA auditor evaluating an AI response.
Determine whether EVERY claim in the candidate answer is directly supported by the context.

<context>
{context}
</context>

<answer>
{answer}
</answer>
"""
    completion = client.beta.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format=FaithfulnessJudge,
        temperature=0.0
    )
    return completion.choices[0].message.parsed

def test_rag_faithfulness_ci_gate():
    context = "Android WorkManager is intended for deferrable, guaranteed background work."
    # Candidate answer contains an ungrounded hallucination
    untrusted_answer = "WorkManager is guaranteed, but drops tasks when battery saver is on."

    eval_result = evaluate_faithfulness(context, untrusted_answer)
    
    print(f"Audit Result: {eval_result.reasoning}")
    assert eval_result.is_grounded, f"CI Gate Failed: Hallucination detected! {eval_result.reasoning}"
import dev.langchain4j.model.openai.OpenAiChatModel
import dev.langchain4j.service.AiServices
import dev.langchain4j.service.SystemMessage
import dev.langchain4j.service.UserMessage
import org.junit.jupiter.api.Assertions.assertTrue
import org.junit.jupiter.api.Test

data class EvalVerdict(val isGrounded: Boolean, val score: Double, val reason: String)

interface FaithfulnessAuditor {
    @SystemMessage("You are a strict CI test judge. Verify if the answer is completely supported by the context.")
    @UserMessage("Context:\n{{context}}\n\nAnswer:\n{{answer}}\n\nRespond ONLY with: PASS or FAIL")
    fun audit(context: String, answer: String): String
}

class AiSystemRegressionTest {

    @Test
    fun `verify RAG answer does not hallucinate beyond context`() {
        val chatModel = OpenAiChatModel.builder()
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .modelName("gpt-4o-mini")
            .temperature(0.0)
            .build()

        val auditor = AiServices.builder(FaithfulnessAuditor::class.java)
            .chatLanguageModel(chatModel)
            .build()

        val context = "Android WorkManager is intended for deferrable, guaranteed background work."
        val candidateAnswer = "WorkManager provides guaranteed background execution."

        val verdict = auditor.audit(context, candidateAnswer).trim()

        assertTrue(verdict.contains("PASS"), "Regression Failure: Answer contains ungrounded claims.")
    }
}

4. Production Tracing & Observability Standards

Just as mobile apps emit Crashlytics non-fatals and performance traces, AI backends instrument spans using OpenTelemetry (OTel) or specialized frameworks (Langfuse, Arize Phoenix).

Span / Metric Data Logged Alerting Threshold
gen_ai.prompt.tokens Exact token count consumed by input prompts and context chunks. Alert when single-request tokens exceed 80% of budget.
gen_ai.completion.tokens Tokens emitted by inference engine. Alert on runaway generations hitting max_tokens limit.
gen_ai.server.ttft Latency until first byte arrives from foundation provider. P95 > 1500ms requires model fallback or caching review.
gen_ai.evaluation.score Rolling average of user thumbs-up/down or async LLM audit score. Drop below 0.90 triggers canary rollout rollback.

5. Progressive Glossary

Term Technical Definition Mobile / Systems Analogy
LLM-as-a-Judge Using an advanced, deterministic LLM to grade and audit the responses of another pipeline or model. Automated static analysis linters (SonarQube / Android Lint) inspecting source code.
Golden Dataset A curated benchmark set of questions, reference contexts, and ground-truth answers for regression testing. Espresso UI mock test fixtures with validated assertions.
Semantic Drift Gradual divergence in model behavior caused by underlying foundation model updates or prompt modifications. API contract degradation across backward-incompatible library version bumps.
Trace Span A structured unit of contiguous work capturing input parameters, execution time, and output metadata in a request pipeline. Android Trace.beginSection() / Trace.endSection() performance profiling tags.

Sources & Reference Standards

← Module 09: FastAPI & Cloud Run Proceed to Module 11: Edge AI & SLMs →