← All posts

A Reference Architecture for Agents in Production

The pieces around the model loop — state, sandboxing, observability, evals, and cost controls — that turn a demo into a dependable system.

The agent loop itself is a few dozen lines: call the model, run any tools it asks for, append the results, repeat. Everything that makes it dependable lives around that loop.

1. State and memory

Decide what survives between turns and between sessions. Conversation history is short-term memory. Durable facts (user preferences, project notes) belong in a store the agent reads and writes through tools. For long sessions, use compaction or context editing so the history doesn't outgrow the context window.

2. Sandboxing and permissions

Anything that executes code or touches external systems runs in an isolated environment with least-privilege credentials. Gate destructive actions (sending email, merging, deleting) behind explicit approval, and keep secrets out of the model's context entirely.

3. Observability

Log every model request, tool call, tool result, token count, and latency, tied together by a run ID. When a run goes wrong you need to replay exactly what the model saw.

4. Evals

A fixed set of representative tasks with automatic grading is the only way to know whether a prompt or model change helped. Run it on every change, the same way you run unit tests.

5. Cost and latency controls

  • **Prompt caching** for the stable prefix (system prompt, tool definitions).
  • **Effort or reasoning settings** tuned per route: low for chat, higher for hard agentic work.
  • **Budgets** on tokens, tool calls, and wall-clock time, with graceful wrap-up when they run out.

6. Failure handling

Retry transient API errors with backoff, handle refusals and truncation explicitly, and design tools to be idempotent so a retried step can't double-charge a customer.

Putting it together

```text

request → router → agent loop ⇄ tools (sandboxed)

│

state store · traces · evals · budgets

```

None of these pieces are exotic. They're the same disciplines as any distributed system, applied to a component that happens to be non-deterministic.