Tool calling

Every tool-capable model on INFRO uses the same OpenAI tools and tool_choice schema. Define tools once. Most provider kinds speak the OpenAI protocol, so INFRO forwards the schema to them as is; for Anthropic connections INFRO translates tools, tool_choice, tool calls and tool results to and from Anthropic's tool_use blocks. The shapes on this page are the same whichever of your connections serves the request.

Tool calling is a loop: send tool definitions with a chat completion, the model responds with one or more tool_calls instead of text, you execute them and return the results, and the model produces its final answer. This page walks the full loop, then covers tool_choice, parallel calls, and streaming.

Check model support

INFRO does not record tool support per model yet — the capabilities array in GET /v1/models is empty — so check the model vendor's documentation. Most current-generation text models support tools, but capability sets differ between a vendor's flagship and its small models, so test at integration time rather than assuming. If a provider rejects the tools parameter, INFRO returns a 400 carrying the provider's own message.

Define a tool

Tools go in the top-level tools array. Each entry describes one function the model may call; arguments are specified as JSON Schema.

typestringrequired

Always "function".

function.namestringrequired

The name the model uses to call the tool. Letters, numbers, underscores, and dashes; 64 characters max.

function.descriptionstring

What the tool does and when to use it. The model leans on this heavily when deciding whether to call — write it for the model, not for humans.

function.parametersobject

JSON Schema for the arguments. Keep it flat where you can, and use enum and per-property description fields — they measurably improve argument quality.

{
  "type": "function",
  "function": {
    "name": "get_weather",
    "description": "Get the current weather for a city. Use when the user asks about weather conditions.",
    "parameters": {
      "type": "object",
      "properties": {
        "city": {
          "type": "string",
          "description": "City name, e.g. \"Lisbon\""
        },
        "unit": {
          "type": "string",
          "enum": ["celsius", "fahrenheit"],
          "description": "Temperature unit. Defaults to celsius."
        }
      },
      "required": ["city"]
    }
  }
}

Tool definitions are part of the prompt prefix, so keep the tools array byte-stable across requests — system prompt first, then tools, then history — and provider-side prompt caching can keep hitting.

The tool-calling loop

When the model decides to call a tool, the response has finish_reason "tool_calls" and the assistant message carries a tool_calls array instead of (or alongside) text content. Each entry:

idstring

Unique id for this call. Echo it back as tool_call_id on the result message.

typestring

Always "function".

function.namestring

Which tool to run.

function.argumentsstring

JSON-encoded arguments — a string, not an object. It is model-generated: parse defensively and validate before executing anything with side effects.

Sending results back has three rules: append the assistant message exactly as received (it carries the tool_calls your results answer), append one "tool" role message per call with tool_call_id set to the call's id and content set to the result as a string, then request the next completion with the same tools array. The complete loop:

import os
import json
from openai import OpenAI

client = OpenAI(
    base_url="https://api.infro.io/v1",
    api_key=os.environ["INFRO_API_KEY"],
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "City name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["city"],
            },
        },
    }
]

def get_weather(city: str, unit: str = "celsius") -> str:
    # Replace with a real lookup
    return json.dumps({"city": city, "temp": 21, "unit": unit, "conditions": "clear"})

messages = [{"role": "user", "content": "What's the weather in Lisbon right now?"}]

response = client.chat.completions.create(
    model="anthropic/claude-sonnet-5",
    messages=messages,
    tools=tools,
)
message = response.choices[0].message

while message.tool_calls:
    # 1. Append the assistant turn that requested the calls
    messages.append(message)

    # 2. Execute every call, one tool message per tool_call_id
    for tool_call in message.tool_calls:
        if tool_call.function.name == "get_weather":
            args = json.loads(tool_call.function.arguments)
            result = get_weather(**args)
        messages.append({
            "role": "tool",
            "tool_call_id": tool_call.id,
            "content": result,
        })

    # 3. Ask for the next turn with the same tools
    response = client.chat.completions.create(
        model="anthropic/claude-sonnet-5",
        messages=messages,
        tools=tools,
    )
    message = response.choices[0].message

print(message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.infro.io/v1",
  apiKey: process.env.INFRO_API_KEY,
});

const tools: OpenAI.ChatCompletionTool[] = [
  {
    type: "function",
    function: {
      name: "get_weather",
      description: "Get the current weather for a city.",
      parameters: {
        type: "object",
        properties: {
          city: { type: "string", description: "City name" },
          unit: { type: "string", enum: ["celsius", "fahrenheit"] },
        },
        required: ["city"],
      },
    },
  },
];

function getWeather(city: string, unit = "celsius"): string {
  // Replace with a real lookup
  return JSON.stringify({ city, temp: 21, unit, conditions: "clear" });
}

const messages: OpenAI.ChatCompletionMessageParam[] = [
  { role: "user", content: "What's the weather in Lisbon right now?" },
];

let response = await client.chat.completions.create({
  model: "anthropic/claude-sonnet-5",
  messages,
  tools,
});
let message = response.choices[0].message;

while (message.tool_calls?.length) {
  // 1. Append the assistant turn that requested the calls
  messages.push(message);

  // 2. Execute every call, one tool message per tool_call_id
  for (const toolCall of message.tool_calls) {
    if (toolCall.type !== "function") continue;
    const args = JSON.parse(toolCall.function.arguments);
    messages.push({
      role: "tool",
      tool_call_id: toolCall.id,
      content: getWeather(args.city, args.unit),
    });
  }

  // 3. Ask for the next turn with the same tools
  response = await client.chat.completions.create({
    model: "anthropic/claude-sonnet-5",
    messages,
    tools,
  });
  message = response.choices[0].message;
}

console.log(message.content);

The loop runs until the model answers in plain text (finish_reason "stop"); multi-step tasks commonly take two or three iterations — the model calls a tool, reads the result, and calls another. If you use fallbacks, pick fallback models that also support tools — one that can't call tools answers in plain text instead — and read the response model field to see which model actually served each turn.

tool_choice

tool_choice controls whether the model may, must, or must not call tools. OpenAI-protocol providers receive it unchanged; on Anthropic connections INFRO maps each value to Anthropic's equivalent (auto, none, any, a named tool). A tool_choice sent without tools is dropped on Anthropic connections.

Forcing a tool guarantees a call, not good arguments — a model pushed into a tool it wouldn't have chosen will invent parameters it doesn't have. Validate arguments before executing, especially with "required" or a forced function.

Parallel tool calls

Where the underlying model supports it, one assistant turn can carry several entries in tool_calls — independent lookups batched into a single round trip. Execute them all (concurrently if you like) and append one "tool" message per call. Order doesn't matter; results are matched by tool_call_id, not position. The loop above already handles this — its for loop over message.tool_calls returns every result before the next request.

Streaming tool calls

With stream: true, tool calls arrive incrementally in delta.tool_calls. Fragments are keyed by index within the turn — the first fragment of each call carries the id, type, and function.name; subsequent fragments append pieces of the function.arguments string. Concatenate arguments per index until the chunk with finish_reason "tool_calls", then parse.

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_9f2b","type":"function","function":{"name":"get_weather","arguments":""}}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"city\":\"Lis"}}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"bon\"}"}}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

data: [DONE]

Parallel calls interleave in the same stream, distinguished by index. The SSE format, usage accounting via stream_options, and mid-stream failure behavior are covered in streaming.

Which models to use

Tool-use quality varies more across models than plain-text quality does, and it shifts between versions — benchmark on your own tools before committing. Frontier models such as anthropic/claude-sonnet-5, anthropic/claude-opus-5 and openai/gpt-5.6-sol are the usual starting point for long multi-step chains; open-weight models such as moonshot/kimi-k2.6 are worth including in the comparison.

For high-volume, single-call workloads — one lookup, one answer — a smaller model such as openai/gpt-5.6-luna or google/gemini-3.7-flash is often enough; compare the vendors' published rates and the estimated cost INFRO records per request. And if what you need is validated JSON output rather than executed actions, use structured outputs instead: tool schemas describe actions, response_format describes shapes.