11. Edge AI & On-Device Small Language Models

Quantized weights, Android LiteRT / MediaPipe runtime execution, NPU hardware acceleration, and hybrid edge-cloud orchestration.

Android Architecture Mental Model: Cloud-only AI architectures break down under mobile realities: cellular dead zones, airplane mode, battery constraints, and strict data privacy regulations. Edge AI brings inference directly to Android runtime silicon (Qualcomm Hexagon NPU, ARM Mali GPU). Just as an offline-first Room repository serves local cached data instantly and syncs changes in the background via WorkManager, an edge SLM (Small Language Model) runs sub-100ms intent classification and document parsing on-device, calling out to cloud models only when high-compute reasoning is required.

1. The Hybrid Edge-Cloud Routing Topology

Production architectures do not force a binary choice between edge and cloud. A local edge router inspects task complexity, network availability, and privacy tags before deciding execution targets:

flowchart TD INPUT["User Request (Android Client)"] --> ROUTER{"Edge Routing Engine
(Task Complexity & Network State)"} ROUTER -->|Offline OR Sensitive PII OR Low Complexity| EDGE["On-Device SLM (Gemma 2B / LiteRT)"] ROUTER -->|Online AND Complex Multi-Step Task| CLOUD["Cloud Run Gateway (Frontier Model)"] EDGE -->|Immediate Local Response| UI["Jetpack Compose UI"] CLOUD -->|Grounded Structured SSE Stream| UI

2. Weight Quantization: Compressing Models for Mobile RAM

A 2-billion parameter model stored in standard 32-bit floating point (FP32) requires:

2,000,000,000 × 4 bytes = 8.0 GB RAM

This exceeds the total available memory ceiling on most mobile devices, triggering Android Low Memory Killer (LMK). Quantization maps continuous floating-point weights into low-bit discrete integers:

Precision RAM Footprint (2B Model) Android Target Hardware Quality & Perplexity Impact
FP16 (16-bit Float) ~4.0 GB High-end flagship GPUs only. Baseline ground-truth reference quality.
INT8 (8-bit Integer) ~2.0 GB Standard Android GPUs & Mid-tier SOCs. < 1% degradation in benchmark accuracy.
INT4 (4-bit Integer) ~1.1 GB to 1.3 GB Qualcomm / MediaTek NPUs & Mobile CPUs. Optimal production balance: 3× faster token generation.

3. Android Edge Acceleration: CPU vs. GPU vs. NPU

Google LiteRT (formerly TensorFlow Lite) and MediaPipe GenAI runtimes map model operators to hardware acceleration delegates via the Android Neural Networks API (NNAPI) or native hardware drivers:

Compute Unit Throughput Profile Thermal & Battery Impact Deployment Role
CPU (Neon SIMD) ~3 - 6 tokens/second High CPU utilization; quickly causes thermal throttling. Universal fallback when hardware acceleration fails.
GPU (OpenCL / Vulkan) ~12 - 25 tokens/second Moderate power draw; good memory bandwidth. Standard delegate for mid-range and premium devices.
NPU (Hexagon / DaVinci) ~35 - 60+ tokens/second Dedicated silicon; lowest battery draw per generated token. Optimal runtime delegate for continuous on-device inference.

4. Dual-Stack Implementations: On-Device Inference

import android.content.Context
import com.google.mediapipe.tasks.genai.llminference.LlmInference
import kotlinx.coroutines.channels.awaitClose
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.callbackFlow
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext

class OnDeviceAiEngine(private val context: Context) {

    private var llmInference: LlmInference? = null

    suspend fun initialize(modelPath: String) = withContext(Dispatchers.IO) {
        val options = LlmInference.LlmInferenceOptions.builder()
            .setModelPath(modelPath) // Path to quantized .bin file (e.g. gemma-2b-it-cpu-int4.bin)
            .setMaxTokens(512)
            .setTemperature(0.2f)
            .build()
        llmInference = LlmInference.createFromOptions(context, options)
    }

    // Non-blocking token streaming via Coroutines Flow directly on-device
    fun generateStreaming(prompt: String): Flow<String> = callbackFlow {
        val inferenceEngine = llmInference 
            ?: throw IllegalStateException("Model runtime not initialized")

        inferenceEngine.generateResponseAsync(prompt) { partialResult, done ->
            trySend(partialResult)
            if (done) {
                channel.close()
            }
        }
        awaitClose { /* Free native delegate allocations if cancelled */ }
    }
}
# Preparing and quantizing an SLM for edge deployment using BitsAndBytes & ONNX
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "google/gemma-2b-it"

# 1. Load weights in 4-bit NormalFloat (NF4)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    load_in_4bit=True,
    torch_dtype=torch.float16
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

print(f"Model successfully loaded and quantized. Footprint: {model.get_memory_footprint() / 1e6:.2f} MB")

# Ready for conversion to LiteRT / ExecuTorch runtime format

5. Progressive Glossary

Term Technical Definition Android / Systems Analogy
SLM Small Language Model: architectures under 4B parameters designed for low-memory environments. A specialized microservice vs. a massive monolith.
Quantization Converting neural network weights from continuous 32-bit floats to discrete 4-bit or 8-bit integers. Downsampling 32-bit ARGB_8888 Bitmaps to RGB_565 to avoid OOM errors.
LiteRT Google's high-performance runtime for on-device ML deployment (formerly TensorFlow Lite). The Android ART runtime or Android NDK executing native compiled code.
NPU Neural Processing Unit: specialized silicon tailored for matrix dot-product operations. The GPU handling hardware-accelerated Compose UI render nodes.

Sources & Reference Standards

← Module 10: Evals & Observability Proceed to Module 12: AI System Design & Caching →