Agents amplify ordinary model risks
A chat request usually creates one generation. An agent can generate, call tools, inspect results, retry, branch, and repeat dozens of times. Small error rates and token inefficiencies compound across the trajectory.
Measure the completed task, not the individual call. An inexpensive model may be ideal for tool selection while a stronger model is needed for planning or final review.
The seven layers
Treat each layer as independently observable and replaceable.
- Orchestration: state machine, plan, stop conditions, and concurrency.
- Tools: narrow schemas, permissions, timeouts, and idempotency.
- State: durable checkpoints and context compaction.
- Inference: model access, routing, fallback, and protocol normalization.
- Policy: allowed models, tools, data, and actions.
- Economics: budgets per run, tenant, and environment.
- Evaluation: task success, safety, and regression testing.
Bound every loop
Set maximum steps, tokens, elapsed time, tool calls, and spend. Require explicit approval for irreversible actions. Persist checkpoints so a recoverable provider error does not force the whole task to restart.
A stop condition should be testable. 'Continue until done' is not a production policy unless 'done' has a machine-verifiable definition.
Route by agent step
Planning, retrieval synthesis, tool selection, execution review, and user-facing prose have different model requirements. Evaluate them separately and route accordingly. This often cuts cost without asking one smaller model to imitate a flagship across the entire trajectory.
Keep fallbacks step-aware too. A tool-selection substitute must match schema reliability; a final-writing substitute must preserve tone and safety policy.