Production GenAI: design for the day a dependency fails

Draw the deployable components

Use a frontend for interaction, an authenticated application API for orchestration, an ingestion worker for document updates, storage for source records and indexes, and model/tool clients behind clear interfaces. Keep long ingestion jobs out of ordinary request handling. A queue can help isolate background work, but also needs retry and dead-letter handling.

Version documents, embeddings, prompts, and model configuration together in release records. Build a new index separately, validate it, then switch traffic through an atomic version pointer or equivalent mechanism. Keep the previous version available for rollback. Updating half the records in place can expose inconsistent evidence to users.

Bound failures and avoid retry storms

Set request deadlines and shorter dependency timeouts inside them. Retry transient errors with bounded backoff and jitter where appropriate. A circuit breaker can stop repeatedly calling a failing dependency. Distinguish a safe read retry from a write whose outcome is uncertain.

Fallback behavior must preserve the task's boundaries. If generation is unavailable, showing permitted source excerpts may be more honest than producing an ungrounded answer through an untested fallback model. If retrieval is unavailable, do not silently answer a private-policy question from general model knowledge. Explain the limitation and offer a useful next step.

Cache with the right identity and version

Embedding caches can key on content hash and embedding version. Answer caches need additional care: permissions, tenant, policy version, prompt version, and relevant request state can affect correctness. A shared cache keyed only by question text can leak private answers or return stale policy.

Start with measured bottlenecks. Shorter context, avoiding duplicate tool calls, batching ingestion, and limiting unnecessary generation can reduce cost. Horizontal scaling helps stateless application workers, but downstream quotas and database capacity may remain the bottleneck. Streaming improves perceived responsiveness; it does not reduce the total work automatically.

Instrument the path users depend on

Use request traces to connect retrieval, model calls, tool calls, and validation. Metrics summarize rates and distributions; logs capture selected events. OpenTelemetry provides concepts and tooling for these signals. Collect identifiers, durations, versions, status, and usage where useful; avoid making raw private prompts the default diagnostic record.

Define release gates from your evaluation set and operational requirements. Test cancellation, dependency outages, stale documents, unauthorized access, and rollback. Watch both service availability and answer quality after release. A green HTTP dashboard can coexist with systematically wrong answers if nobody measures the content path.

References and further reading

Use these primary references for deeper study and current API details. Examples in this lesson use fictional Northstar data.

• OpenTelemetry concepts —