Building a Production Agent Runtime

A useful first implementation question for an agent is not “which model should I use?” It is “what is the runtime that owns a run?” The model is only one dependency. The runtime is the application layer that accepts a user request, assembles context, calls the model, executes tools, persists run state, enforces limits, and emits traces.

Start with the execution contract

1Client2↓3API edge / load balancer4↓5Agent API6↓7Agent Runtime Worker8├─ Model provider call9├─ Retrieval10├─ Tool adapters11└─ Durable state / traces

The runtime should be mostly stateless. A worker should be able to disappear between turns without losing the run. Persist the durable pieces outside the worker: run ID, conversation/session references, pending action IDs, workflow state, and audit metadata.

Pick the compute shape by workload

Split synchronous and asynchronous paths

A two-second constrained CRM lookup should not be treated like a 60-second research workflow.

1FAST PATH2User → Agent API → Model → Tool → Model → Stream response

1LONG-RUNNING PATH2User → Agent API → Create job → durable queue3↓4Worker pool5↓6Durable workflow engine7↓8Persist result / notify

The moment a request becomes durable, the UX contract changes. Return a job identifier or a clear pending state, persist progress, and let the user continue interacting. Do not keep scarce interactive capacity hostage to a long-running workflow.

Scale components independently

• Agent API workers scale on request rate, active streams, CPU/memory, and latency.
• Background workers scale on queue depth and queue age.
• Retrieval scales on query throughput and index latency.
• Tool workers scale on downstream capacity, not just local CPU.
• Model calls are constrained by provider rate limits and token throughput.

This is why “10,000 employees” is not a capacity number. Peak concurrency, requests per second, tool fan-out, model turns per task, and downstream quotas are.

Protect the runtime with an autonomy budget

1run_budget = {2"max_model_turns": 8,3"max_tool_calls": 12,4"max_wall_clock_seconds": 45,5"max_side_effecting_calls": 2,6"max_estimated_cost_usd": 0.257}

The exact values vary by product. The important point is that agent execution is bounded. A malformed task should not be able to loop through search, reasoning, retries, and writes indefinitely.

Measure the bottleneck you actually have

• active runs and request rate
• queue depth and oldest-message age
• model latency, errors, rate-limit responses, and token use
• tool latency, error rate, and calls per task
• database connection pressure
• task success and cost per successful task

A concrete burst walkthrough

Assume 500 employees submit requests at once. The API tier admits the traffic up to its configured limits. Short interactive runs are distributed across the agent worker pool. Long-running work is converted into jobs and buffered. Tool fan-out is bounded per downstream dependency. If one SaaS system becomes the limiting resource, only that tool path queues rather than the entire agent platform.

1500 user requests2↓3Agent API4┌───┴──────────────┐5↓ ↓6interactive durable jobs7workers ↓8↓ durable queue9model/tools ↓10background workers

Implementation checklist

• Externalize run state.
• Separate interactive and durable execution.
• Use queues for buffering, not as a substitute for concurrency control.
• Bound model turns, tool calls, cost, and wall-clock time.
• Scale tool paths according to downstream capacity.
• Instrument task success, not just infrastructure health.