Section 0
Why Containerize
| Problem | What Docker fixes |
|---|---|
| "Works on my machine" | The container bundles the exact Python version and dependencies — no environment drift between dev and production |
| Manual server management | Cloud Run runs the container and scales instances automatically based on traffic, including down to zero when idle |
| Onboarding a new engineer | docker build reproduces the environment exactly, instead of a README with seventeen setup steps |
Section 1
Writing the Dockerfile
FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . # Cloud Run sets $PORT; default to 8080 for local runs ENV PORT=8080 CMD ["sh", "-c", "uvicorn main:app --host 0.0.0.0 --port $PORT"]
# requirements.txt fastapi uvicorn[standard] openai pydantic
Binding to
0.0.0.0 instead of 127.0.0.1 matters here — the default localhost binding only accepts connections from inside the container, which means Cloud Run's traffic can never reach it. This one detail is the most common "deployed fine, returns nothing" bug.Section 2
Environment Variables & Secrets
Topic 2 Section 0's .env file was fine on your laptop. It should never end up inside a container image — anyone who can pull the image can read every layer, including anything baked in at build time.
Baking secrets into the image
COPY .env . in a Dockerfile puts the API key in the image itself — extractable by anyone with pull access, and permanently in the image's layer history even if a later layer deletes it.The fixInject secrets at runtime — Cloud Run's
--set-secrets flag (backed by Google Secret Manager) or plain environment variables set on the running service, never copied into the image.echo -n "sk-..." | gcloud secrets create openai-api-key --data-file=- gcloud run deploy ai-service \ --set-secrets=OPENAI_API_KEY=openai-api-key:latest \ ...
Section 3
Building & Running Locally
docker build -t ai-service . docker run -p 8080:8080 \ -e OPENAI_API_KEY="$OPENAI_API_KEY" \ ai-service curl http://localhost:8080/health
Checkpoint: confirm
/health and /extract-task from Topic 11's service both work through the container before moving to a cloud deployment — debugging a broken container locally is dramatically faster than debugging one already deployed.Section 4
Deploying to Cloud Run
gcloud run deploy ai-service \ --source . \ --region us-central1 \ --allow-unauthenticated \ --set-secrets=OPENAI_API_KEY=openai-api-key:latest \ --memory 512Mi \ --max-instances 10
| Flag | Why it matters for an LLM-backed service |
|---|---|
--allow-unauthenticated | Only for a public endpoint with its own auth layer (Topic 11's job) — omit it and use IAM if the caller is another internal service |
--max-instances | A hard ceiling on concurrent scale-out — protects against a traffic spike (or an attack) turning into an unbounded LLM API bill |
--memory | LLM calls are I/O-bound, not memory-hungry — start small and raise it only if profiling says otherwise |
Section 5
Vertex AI: When It Earns Its Place
Vertex AI is Google Cloud's managed platform for accessing models (including Gemini, and hosted open-weight models) with enterprise features layered on — it's an alternative front door to some of the same models from Topic 2, not a different product category.
| Situation | Direct provider API (Topic 2) | Vertex AI |
|---|---|---|
| Small team, prototyping fast | Simpler, fewer moving parts | More setup for little immediate benefit |
| Already deep in Google Cloud (IAM, VPC, billing) | A separate API key and billing relationship to manage | Unified IAM, billing, and networking with the rest of your GCP infrastructure |
| Need enterprise data residency / compliance controls | Depends on the provider's own offering | Often stronger guarantees, built for enterprise procurement |
| Want to swap between Gemini and other hosted models easily | Separate SDKs per provider (Topic 2's comparison) | More unified access, still provider-specific model behavior underneath |
Don't add Vertex AI on day one just because it's available — it's an infrastructure and procurement decision, not a capability the direct SDKs from Topic 2 lack. Reach for it when the organizational reasons (IAM, billing, compliance) actually apply to your situation.
Section 6
Basic Observability
Cloud Run captures stdout/stderr as logs automatically — the minimum useful step is logging enough about each LLM call to debug and cost-track it in production, extending Topic 2 Section 5's "log usage on every call" habit from a script to a service.
import logging
logger = logging.getLogger("ai-service")
@app.post("/extract-task", response_model=TaskResponse)
async def extract(request: ExtractRequest, client: AsyncOpenAI = Depends(get_client)):
completion = await client.beta.chat.completions.parse(...)
logger.info("extract_task", extra={
"prompt_tokens": completion.usage.prompt_tokens,
"completion_tokens": completion.usage.completion_tokens,
"model": completion.model,
})
...
This is the seed of what Topic 10's evaluation harness needs in production: without logged token usage and outcomes, you can measure quality in a test set but not track whether it's holding up against real traffic.
Section 7 · Checkpoint
Capstone: Deploy Topic 11's Service
Take the FastAPI service from Topic 11's checkpoint, containerize it, and deploy it to Cloud Run with a secret-backed API key.
# Dockerfile FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . ENV PORT=8080 CMD ["sh", "-c", "uvicorn main:app --host 0.0.0.0 --port $PORT"]
# deploy.sh echo -n "$OPENAI_API_KEY" | gcloud secrets create openai-api-key --data-file=- \ || echo "secret already exists, skipping create" gcloud run deploy ai-service \ --source . \ --region us-central1 \ --allow-unauthenticated \ --set-secrets=OPENAI_API_KEY=openai-api-key:latest \ --memory 512Mi \ --max-instances 10 SERVICE_URL=$(gcloud run services describe ai-service --region us-central1 --format 'value(status.url)') curl "$SERVICE_URL/health"
Expected behavior: the deploy command prints a live HTTPS URL, and the final
curl to /health returns {"status": "ok"} from the deployed instance, not localhost.docker history ai-service — only injected at runtime via the secret/extract-task against the live URL, not localhost--max-instances deliberately low (e.g. 1) and confirmed the reasoning for why that cap exists rather than leaving it unbounded