01. LLM API Protocols & Token Mechanics

Android Architecture Mental Model: In mobile platforms, text is represented as UTF-8 or UTF-16 character sequences, and memory bottlenecks surface via Binder transaction buffer exhaustion (the 1MB IPC limit) or Garbage Collection pressure. In Large Language Models (LLMs), text does not exist at runtime—models operate entirely on discrete integer token IDs. Passing input to an LLM is analogous to dispatching a streaming network request through OkHttp with strict body byte limits, where every single sub-word chunk incurs discrete compute latency and monetary cost.

1. Tokenization Fundamentals

Neural networks cannot process raw strings. Tokenization is the preprocessing step that parses continuous text into discrete integer representations using algorithms like Byte-Pair Encoding (BPE).

flowchart LR A["Input String:
'Structured concurrency in Kotlin'"] --> B["BPE Tokenizer Engine"] B --> C["Token Array:
['Struct', 'ured', ' concurrency', ' in', ' Kot', 'lin']"] C --> D["Token IDs:
[14205, 1142, 45129, 294, 38102, 1109]"] D --> E["Embedding Layer Matrix Lookup"]

Key properties of tokens:

Sub-word decomposition: Common words map to a single token (e.g., "the" → ID 1820), while uncommon or specialized code identifiers are split into multiple fragments (e.g., CoroutineScope → ["Cor", "outine", "Scope"]).
Token Rule of Thumb: In English text, 1 token ≈ 0.75 words, or approximately 4 characters. 1,000 tokens represent ~750 words.
Language bias: Non-English languages and code blocks often produce significantly more tokens per character due to lower occurrence in base BPE vocabulary dictionaries.

2. Model Sampling Hyperparameters

At each generation step, the model computes raw numerical logits across its entire vocabulary (e.g., 32k to 128k distinct tokens). A Softmax function converts logits to probabilities. Hyperparameters determine which token is chosen next.

Parameter	System Mechanism	Low Value Impact	High Value Impact
Temperature ($T$)	Divides logits before Softmax: $P(w_i) = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$.	$T \to 0$: Deterministic argmax choice. Ideal for structured JSON and code generation.	$T \ge 1.0$: Flattens probability curve. Increases diversity, creativity, and hallucination risk.
Top-P (Nucleus)	Selects dynamically from the smallest candidate pool whose cumulative probability exceeds $P$.	Tight candidate pool; discards low-probability tail tokens. Highly stable.	Considers longer-tail tokens; prevents repetitive loops in open-ended generation.
Top-K	Hard-truncates candidate set to the top $K$ absolute highest-probability tokens.	$K=1$: Exact greedy decoding. Extremely predictable.	Broader token options, balancing variety with sensible alternatives.

3. Asynchronous Streaming & SSE Protocols

Generating 500 tokens takes several seconds because LLMs generate text auto-regressively (one token per forward pass). Standard synchronous blocking HTTP creates high Time-to-First-Token (TTFT) latency, degrading UX. Production systems use Server-Sent Events (SSE) over HTTP/2.

sequenceDiagram autonumber participant App as Android Client (ViewModel) participant Engine as AI Backend Gateway (FastAPI) participant LLM as Inference Engine (Vertex AI / Claude) App->>Engine: POST /v1/chat/completions (stream=true) Engine->>LLM: Stream Inference Request loop Token Generation LLM-->>Engine: Raw Token Logit -> Token String Engine-->>App: SSE data: {"delta": {"content": "token"}} Note over App: Emit to Flow<String>, Render in UI end LLM-->>Engine: [DONE] Engine-->>App: SSE data: [DONE]

Dual-Stack Implementations: Consuming SSE Streams

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def stream_tokens(prompt: str):
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a concise technical architect."},
            {"role": "user", "content": prompt}
        ],
        stream=True,
        temperature=0.2
    )

    # Asynchronously iterate over arriving SSE deltas
    async for chunk in response:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)

if __name__ == "__main__":
    asyncio.run(stream_tokens("Explain structured concurrency in 2 sentences."))

import kotlinx.coroutines.channels.awaitClose
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.callbackFlow
import kotlinx.coroutines.flow.flowOn
import kotlinx.coroutines.Dispatchers
import okhttp3.*
import okhttp3.sse.EventSource
import okhttp3.sse.EventSourceListener
import okhttp3.sse.EventSources
import java.io.IOException

fun streamModelResponse(okHttpClient: OkHttpClient, request: Request): Flow<String> = callbackFlow {
    val eventSourceListener = object : EventSourceListener() {
        override fun onEvent(eventSource: EventSource, id: String?, type: String?, data: String) {
            if (data == "[DONE]") {
                channel.close()
                return
            }
            // Parse partial delta token from raw JSON
            val token = parseDeltaFromJson(data)
            trySend(token)
        }

        override fun onFailure(eventSource: EventSource, t: Throwable?, response: Response?) {
            channel.close(t ?: IOException("SSE connection dropped"))
        }
    }

    val eventSource = EventSources.createFactory(okHttpClient)
        .newEventSource(request, eventSourceListener)

    awaitClose { eventSource.cancel() }
}.flowOn(Dispatchers.IO)

4. Progressive Glossary

Term	Technical Definition	Android / Systems Analogy
Autoregressive	Inference mode where each generated token is appended to the context window to predict the next token sequentially.	Recursive list accumulator emitting partial items down a reactive stream.
Context Window	The absolute token limit (input prompt + output generation) processed in a single inference cycle.	Heap memory allocation limit or SQLite maximum SQLite query parameter count.
TTFT	Time To First Token: Duration between user dispatch and the arrival of the first generated token.	Time to Initial Display (TTID) in Android app performance benchmarking.
Logits	Raw, unnormalized score vectors output by the final transformer layer before probability mapping.	Unmapped domain entities before domain-to-UI DTO transformation.

5. Architecture Trade-offs & Production Considerations

Client Connection Lifecycle: When an Android device enters the background or loses network connection, the OkHttp client terminates the socket. Backend gateways must capture this ClientDisconnect to terminate the upstream LLM inference call; otherwise, upstream token generation continues running and incurring cost.
Context Window Degradation: Larger context windows (e.g., 1M+ tokens in Gemini 1.5) enable wide context ingestion, but retrieval accuracy ("needle-in-a-haystack") degrades if critical prompts are buried in the middle of long documents.

Sources & Reference Standards

Anthropic Documentation: Claude API Architecture & Token Mechanics
Google Vertex AI Architecture Guide: Gemini Foundational Capabilities
OpenAI Developer Platform: Understanding Tokens & Latency Optimization

← Back to Curriculum Proceed to Module 02: Prompt Engineering →