Bounding AI agent loops: step, time, token and spend limits

A chat request is one model call. An agent run is a loop: the model asks for a tool, your code runs it, the result goes back, and the model decides what to do next. Nothing in that loop stops by itself. A model that misreads a tool error, a tool that returns the same page forever, or a planner that keeps "double-checking" will run until something outside it says no.

OWASP files this risk as unbounded consumption, and one of its named examples is usage that runs up a pay-per-use bill until it becomes unsustainable¹. Its mitigations are the ones this guide implements — rate limits and quotas, timeouts and throttling for expensive operations, and monitoring for unusual consumption¹.

Why agent cost grows faster than step count

Each model call in an agent run re-sends everything the model needs: the system prompt, the tool definitions, and the whole conversation so far, including every tool result. Tool definitions alone are billed as input tokens on every call; OpenAI says function definitions are injected into the system message and "billed as input tokens"², and Anthropic adds a tool-use system prompt on top of your tool schemas³.

So step 20 does not cost what step 1 cost. It costs step 1 plus everything the run has accumulated. Total input over a run grows roughly with the square of the step count.

That shape is why a step limit alone is a weak control. A run with a modest step count and one tool that returns 40,000 tokens of HTML can cost more than a long run of small steps. Bound the resources, not only the iterations.

The five limits a run needs

Agent frameworks already expose the first one. Anthropic's guidance on agents recommends stopping conditions "such as a maximum number of iterations" and warns that autonomy brings "higher costs, and the potential for compounding errors"⁴. The OpenAI Agents SDK raises MaxTurnsExceeded when a run passes the max_turns given to the runner⁵. Those are step limits. A production run needs four more.

A run guard in Python

The guard below is framework-neutral: call before_model_call() before each request, after_model_call() with the provider's reported usage, and before_tool_call() before executing anything the model asked for. The spend check uses a worst case — input tokens you are about to send plus the max_tokens you are about to allow — so a single call cannot jump past the limit.

import hashlib
import json
import time
from dataclasses import dataclass, field


class RunLimitReached(Exception):
    def __init__(self, reason: str, detail: str):
        super().__init__(f"{reason}: {detail}")
        self.reason = reason


@dataclass
class RunLimits:
    max_steps: int = 25
    deadline_seconds: float = 120.0
    max_total_tokens: int = 400_000
    max_spend_usd: float = 1.00
    max_identical_tool_calls: int = 2
    min_output_tokens: int = 256  # stop rather than ask for a useless reply


@dataclass
class RunGuard:
    limits: RunLimits
    input_usd_per_token: float
    output_usd_per_token: float
    started: float = field(default_factory=time.monotonic)
    steps: int = 0
    tokens: int = 0
    spend_usd: float = 0.0
    seen_calls: dict[str, int] = field(default_factory=dict)

    def remaining_seconds(self) -> float:
        return self.limits.deadline_seconds - (time.monotonic() - self.started)

    def before_model_call(self, input_tokens_estimate: int, wanted_output: int) -> int:
        """Return the max_tokens this call may use, or raise."""
        if self.steps >= self.limits.max_steps:
            raise RunLimitReached("steps", f"{self.steps} steps used")
        if self.remaining_seconds() <= 0:
            raise RunLimitReached("deadline", "wall-clock deadline passed")
        token_room = self.limits.max_total_tokens - self.tokens - input_tokens_estimate
        spend_room = self.limits.max_spend_usd - self.spend_usd
        spend_room -= input_tokens_estimate * self.input_usd_per_token
        output_by_spend = int(spend_room / self.output_usd_per_token)
        max_tokens = min(wanted_output, token_room, output_by_spend)
        if max_tokens < self.limits.min_output_tokens:
            raise RunLimitReached("budget", f"only {max(max_tokens, 0)} output tokens left")
        self.steps += 1
        return max_tokens

    def after_model_call(self, prompt_tokens: int, completion_tokens: int, cost_usd: float | None):
        self.tokens += prompt_tokens + completion_tokens
        if cost_usd is None:  # unpriced: fall back to your own rates, never to zero
            cost_usd = (prompt_tokens * self.input_usd_per_token
                        + completion_tokens * self.output_usd_per_token)
        self.spend_usd += cost_usd

    def before_tool_call(self, name: str, arguments: str) -> float:
        """Return the timeout for this tool call, or raise on a repeat."""
        try:
            canonical = json.dumps(json.loads(arguments), sort_keys=True)
        except json.JSONDecodeError:
            canonical = arguments
        key = hashlib.sha256(f"{name}:{canonical}".encode()).hexdigest()
        self.seen_calls[key] = self.seen_calls.get(key, 0) + 1
        if self.seen_calls[key] > self.limits.max_identical_tool_calls:
            raise RunLimitReached("repeat", f"{name} called again with identical arguments")
        remaining = self.remaining_seconds()
        if remaining <= 1:
            raise RunLimitReached("deadline", "no time left for a tool call")
        return min(30.0, remaining)

Three details matter more than they look:

• max_tokens comes from the guard. Asking for the model's maximum output on every step reserves, and can spend, far more than the run has left. The guard hands each call only what fits.
• Unpriced is not free. When a response carries no cost estimate, the guard charges your own rate rather than zero, so an unpriced model cannot run unbounded.
• Repeat detection canonicalises arguments. {"id": 7, "page": 1} and {"page": 1, "id": 7} are the same call. Two identical calls can be legitimate (a retry after a transient error); a third is a loop.

The loop that uses it

The loop is ordinary tool calling; the only additions are the guard calls, a timeout around each tool, and treating limit errors as a normal way for a run to end. It uses the OpenAI Python SDK against any OpenAI-compatible endpoint.

import concurrent.futures
import json

from openai import OpenAI, APIStatusError

from run_guard import RunGuard, RunLimitReached, RunLimits

client = OpenAI()  # base_url and api_key from the environment
pool = concurrent.futures.ThreadPoolExecutor(max_workers=4)
TERMINAL_STATUSES = {400, 401, 402, 403, 404}


def run_agent(task: str, tools: list[dict], handlers: dict, model: str, run_id: str) -> dict:
    guard = RunGuard(RunLimits(), input_usd_per_token=2e-6, output_usd_per_token=10e-6)
    messages = [{"role": "system", "content": "You are a support agent."},
                {"role": "user", "content": task}]
    last_prompt_tokens = 2_000  # first-call estimate; later calls use reported usage
    try:
        while True:
            estimate = last_prompt_tokens + 1_500  # plus what the last tool result added
            max_tokens = guard.before_model_call(estimate, wanted_output=2_048)
            try:
                response = client.chat.completions.create(
                    model=model, messages=messages, tools=tools, max_tokens=max_tokens,
                    timeout=max(5.0, guard.remaining_seconds()),
                    extra_body={"metadata": {"trace_id": run_id, "workflow": "support-agent",
                                             "step": f"turn_{guard.steps}"}},
                )
            except APIStatusError as err:
                if err.status_code in TERMINAL_STATUSES:
                    return {"status": "refused", "http_status": err.status_code, "run_id": run_id}
                raise
            usage = response.usage
            cost = getattr(usage, "cost", None)  # present when the endpoint reports it
            guard.after_model_call(usage.prompt_tokens, usage.completion_tokens, cost)
            last_prompt_tokens = usage.prompt_tokens + usage.completion_tokens

            message = response.choices[0].message
            messages.append(message.model_dump(exclude_none=True))
            if not message.tool_calls:
                return {"status": "done", "answer": message.content, "run_id": run_id}

            for call in message.tool_calls:
                timeout = guard.before_tool_call(call.function.name, call.function.arguments)
                handler = handlers.get(call.function.name)
                if handler is None:
                    result = {"error": f"unknown tool {call.function.name}"}
                else:
                    future = pool.submit(handler, json.loads(call.function.arguments))
                    try:
                        result = future.result(timeout=timeout)
                    except concurrent.futures.TimeoutError:
                        result = {"error": f"{call.function.name} timed out after {timeout:.0f}s"}
                messages.append({"role": "tool", "tool_call_id": call.id,
                                 "content": json.dumps(result)[:20_000]})
    except RunLimitReached as stop:
        return {"status": "stopped", "reason": stop.reason, "steps": guard.steps,
                "spend_usd": round(guard.spend_usd, 4), "run_id": run_id}

A timed-out tool thread keeps running in the pool; for tools with side effects, pass a cancellation token or make the tool itself honour a deadline. The [:20_000] cap on tool results is crude, and deliberately so: a result that large should be summarised or paginated by the tool, and the cap stops one bad page from dominating every later step.

In-process limits are not enough

A run guard lives in the same process as the bug it protects against. A code path that forgets to call it, a retry wrapper that restarts the loop with fresh counters, a sub-agent spawned with its own guard, or a deploy that sets the limits from a missing environment variable — each silently removes the bound. The guard is the precise control; it needs a blunt one behind it that sits outside your code.

The outer layers answer different questions. A per-key ceiling bounds what one deployment of one agent can spend in total. A per-end-customer cap bounds what any single user of your product can cause. Provider-side quotas are the last backstop, and you set those in each provider account.

Setting hard ceilings for agents in INFRO

INFRO sits between your agents and the provider accounts you connect, so its limits apply to every model call regardless of which code path made it. Your providers still bill you directly; these limits act on INFRO's estimate of that provider cost.

A key per agent, with a spend limit. A gateway key can carry a spend limit that resets daily, weekly or monthly, or a lifetime limit that never resets; key limits always block, and every applicable blocking limit must allow the request (spend controls). A lifetime limit on a key used only by one agent worker bounds what that worker can ever spend, whatever its loop does. GET /v1/key reports limit, usage, limit_remaining and limit_resets_at for the calling key (limits), so a guard can read the outer ceiling before it starts a run.

Reservation, not after-the-fact accounting. A blocking budget admits a request only if its worst-case estimated cost fits the room left, counting the worst case of every request still in flight. When a chat request sets no max_tokens, the worst case assumes the model's maximum output and INFRO sends that ceiling to the provider. Setting max_tokens from your guard, as above, reserves less per call, so more concurrent agent steps fit under the same budget (spend controls).

Per end customer. Send your user's opaque id as metadata.user_id and an end-customer cap limits what each of your customers can spend per day, week or month; a request over the cap returns 402 with code end_user_budget, plus end_user_id, period and resets_at so the application can tell that user when they can continue (end-customer caps).

Refusals are terminal. A budget refusal is 402 budget_exceeded with a code naming the ceiling, such as key_budget, key_lifetime_limit, project_budget or end_user_budget. It does not clear on retry, which is why the loop above ends the run on a 402 instead of backing off.

{
  "error": {
    "type": "budget_exceeded",
    "code": "key_lifetime_limit",
    "message": "Budget exceeded: … (names the key and what to raise)"
  }
}

See what a run cost. The loop tags each call with metadata.trace_id, workflow and step. GET /v1/traces/{trace_id} then returns every request in the run with estimated cost by step, and flags the total as a lower bound with cost_is_partial when any call was unpriced (request traces). Tracing a run end to end is covered in tracing an AI agent run.

These estimates are not the provider's invoice. Provider-side charges INFRO cannot observe — a built-in tool surcharge, or an attempt the provider billed after a timeout — sit outside them, so keep provider quotas and billing alerts on as well.

Checklist

• Every agent loop checks steps, a wall-clock deadline, tokens, estimated spend and repeated tool calls before each model call.
• max_tokens on each call is computed from the run's remaining budget.
• Every tool call has a timeout, and tool results are capped or summarised before they enter the history.
• Limit hits end the run with a labelled partial result and a run id, and they are counted and alerted on per workflow.
• 400, 401, 402, 403 and 404 responses end the run; they are never retried in a loop.
• Each agent deployment uses its own gateway key with a spend limit; user-facing agents send an end-customer id and have a cap.
• Provider account quotas and billing alerts are set as the last backstop.
• A test forces each limit — steps, deadline, tokens, spend, repeats, a 402 — and asserts the run stops cleanly.