# Integrate INFRO into this codebase

You are working in this repository as a senior engineer. Route this application's AI model calls through INFRO with the smallest correct change, verify it, and report what you did. Follow every rule below. Do not guess beyond the facts given here; when something is unclear, read the documentation linked at the end or stop and report.

Prompt version 2026-10-06 · https://infro.io/integrate.md

## What INFRO is

INFRO is an AI gateway and control plane. The team connects AI provider accounts they already hold (OpenAI, Anthropic, Google, Azure OpenAI, AWS Bedrock and others) in the INFRO console. The application then calls one OpenAI-compatible API with an INFRO gateway key. INFRO routes each request to those connections, fails over between them, records every request with its estimated provider cost, and enforces budgets and model policy. Providers keep billing the team directly for model usage.

## Facts to rely on

- Base URL: `https://api.infro.io/v1`
- Authentication: `Authorization: Bearer $INFRO_API_KEY`. Gateway keys start with `sk_infro_`.
- Model ids are `vendor/model` (for example `openai/gpt-5.6-terra`, `anthropic/claude-sonnet-5`, `google/gemini-3.7-flash`), or the slug of a route the team created in the console. `GET https://api.infro.io/v1/models` lists exactly the catalog ids this key can call. For a provider whose ids INFRO cannot derive (LaoZhang, Together, Fireworks, Bedrock, an OpenAI-compatible endpoint and others), `<provider>/<that provider's own model id>` is also valid and is sent exactly as written; those ids are not in `/models`, so take them from the codebase or the provider's own model list.
- Text generation is OpenAI-compatible: `POST /chat/completions` with the OpenAI request and response shape, including streaming, tool calls, structured outputs and image inputs.
- Other endpoints: `POST /images/generations`, `POST /images/edits`, `POST /audio/speech`, `POST /audio/transcriptions`, and `POST /videos`, which returns an async job read with `GET /jobs/{id}` or delivered to a signed webhook.
- TypeSafe's Jev: `POST /systemone` with TypeSafe's own request body and model names (`jev-latest`, `jev-1.13.0`), on the team's TypeSafe connection. Code that uses TypeSafe's SDK needs no code change: set `TYPESAFE_BASE_URL=https://api.infro.io` and `TYPESAFE_API_KEY` to the INFRO gateway key.
- Not available through INFRO: the OpenAI Responses API (`/responses`), embeddings, moderations, Assistants and threads, Realtime, Batch, Files, fine-tuning and vector stores, and provider-native APIs such as Anthropic Messages or Gemini `generateContent` called with their own SDKs.
- Every response carries the header `X-INFRO-Request-Id`. Response bodies add `provider`, `connection`, `route`, `route_name` and `usage.cost`, an estimate in USD of what the provider will bill. `usage.cost` is null when a model is unpriced; that does not mean free.
- Optional top-level body fields: `fallbacks` (up to 5 alternate model ids, tried after every connection for the requested model fails; ignored when `model` is a route slug, whose failover is configured on the route), `metadata` (an object recorded on the request and never sent to the provider; `user_id`, `session_id` and `tags.<key>` can be filtered in the request log; the reserved keys `workflow`, `trace_id`, `step` and `feature` group cost by run and by product feature, and requests without a JSON body can send them as `X-INFRO-Workflow`, `X-INFRO-Trace-Id`, `X-INFRO-Step`, `X-INFRO-Feature` and `X-INFRO-User-Id` headers; `user_id` is also the end-customer id that per-end-customer spend caps count against, so send the application's own opaque user id, never an email address or a name), and `logging: false` (opt this request out of payload capture). To keep one request on particular providers the customer has connected, send `routing: { providers: { allow: ["bedrock"] } }` (or `deny`) with provider ids; leave it out to let the gateway choose.
- Streaming: send `stream_options: { include_usage: true }` to get the cost estimate in the final chunk. Failover happens only before the first byte. A stream that fails midway ends without `data: [DONE]`; treat that as an error, not a complete answer.
- Errors use `{ "error": { "message", "type", "code" } }`. Branch on the HTTP status and `code`, never on the message.
  - 400 `invalid_request_error` — fix the request.
  - 401 `authentication_error` — missing or revoked key.
  - 402 `budget_exceeded` (a budget the team set, an unpriced model under a blocking budget, or codes `subscription_required`, `trial_payment_failed`) — do not retry. Code `end_user_budget` means the end customer in `metadata.user_id` reached its own spend cap: the error also carries `end_user_id`, `period` and `resets_at`, so surface a message the application can show that user ("daily limit reached, resets at …").
  - 400 `end_user_id_required` — the team requires an end-customer id on every request; send `metadata.user_id` (or `X-INFRO-User-Id`). Do not retry without it.
  - 403 `permission_denied`, code `model_not_allowed`, `route_not_allowed`, `project_archived`, `key_disabled` or `ip_not_allowed` — model, key, project or network policy (`key_disabled`: someone turned the key off; it can be enabled again on the Keys screen) (`ip_not_allowed`: the key accepts requests only from its allowed networks); code `guardrail_blocked` or `guardrail_unavailable` — the organization's guardrails refused the request or the response; code `organization_suspended` — INFRO suspended the organization's access, tell the user to contact support@infro.io. Do not retry any of them.
  - 413 `request_too_large` — the body is over the endpoint's limit; do not retry.
  - 404 `model_not_found` — wrong model id.
  - 429 `rate_limit_exceeded` — INFRO's per-key limit, or code `provider_rate_limited` when the team's own provider account is throttling. Retry after `Retry-After`.
  - 408, 502, 503 — retry with exponential backoff and jitter, EXCEPT 503 with code `no_provider_connected`, `connection_out_of_scope` or `model_not_served`, which no retry clears: a provider connection is missing, the existing one is out of reach (another environment, disabled, missing a model map, or temporarily ejected — the message says which), or its key has no access to the model on the provider's side.
- `Idempotency-Key` is honoured on `POST /videos`. Send one per logical job.

## Rules

1. Never print, log, echo, commit or paste the value of `INFRO_API_KEY` or any provider key. Do not read `.env` values into your output. Use placeholders such as `sk_infro_...` in examples.
2. The INFRO key is a server-side secret. Never expose it to browser or mobile code (for example `NEXT_PUBLIC_`, `VITE_`, `EXPO_PUBLIC_` variables, client components, or bundled front-end code). If a model call runs in the browser today, move it behind a server route or function. If the project has no server side, stop and report instead of shipping the key to clients.
3. Make the smallest correct change. Keep prompts, message content, parameters, tools, output parsing and business logic exactly as they are. Do not upgrade SDK major versions unless a base URL override is impossible otherwise, and say so if you do.
4. Follow the repository's existing conventions: language, formatting, config and secrets handling, error and logging style, test framework.
5. Do not add provider API keys for INFRO. Providers are connected in the INFRO console. Keep an existing provider key only where a call stays direct.
6. Do not invent model ids. If you cannot confirm an INFRO id for a model, keep the call direct or leave a clearly marked TODO in one mapping, and list it in your report.
7. If the change needs a large refactor (for example converting many Anthropic-native or Gemini-native calls), do the straightforward call sites, leave the rest direct, and report them with an estimate instead of forcing it.

## Steps

### 1. Inventory

Search the whole repository for model calls, including:
- SDK imports: `openai`, `@anthropic-ai/sdk`, `anthropic`, `@google/genai`, `@google/generative-ai`, `google-genai`, `ai` and `@ai-sdk/*`, `langchain` and `@langchain/*`, `llama_index`, `litellm`, `groq`, `mistralai`, `cohere`, `together`, and OpenAI-compatible clients in Go, Java, .NET, Ruby, PHP or Rust.
- Raw HTTP calls to `api.openai.com`, `api.anthropic.com`, `generativelanguage.googleapis.com`, `openai.azure.com`, `bedrock-runtime`, `openrouter.ai`, and other model hosts.
- Environment variables such as `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, and hard-coded model names.

For each call site record: file, SDK or client, API used (chat completions, responses, embeddings, images, audio, native messages), model, streaming, tools or structured output, and where it runs (server, edge, browser, mobile, background job).

### 2. Classify

- Route through INFRO: OpenAI-compatible chat completions from any language, OpenAI SDK image, speech and transcription calls, and framework adapters that use chat completions.
- Convert, only when straightforward: provider-native text calls (Anthropic Messages, Gemini) that use features the chat completions shape covers (messages, system prompt, streaming, tools, JSON output, images). Use the OpenAI client pointed at INFRO with the matching `vendor/model` id.
- Leave direct and report: every surface listed under "Not available through INFRO", and native calls that depend on provider-only features.

### 3. Implement

- Add one small INFRO client module or factory in the project's style that reads `INFRO_API_KEY` and `INFRO_BASE_URL` (default `https://api.infro.io/v1`). Reuse it at every routed call site instead of constructing clients inline.
- If the project already has a config layer, add a single switch (for example `AI_GATEWAY=infro|direct`) so the change can be reverted without a code change. Skip this if there is no config layer to put it in.
- Replace model ids with INFRO ids in one mapping or in configuration, not scattered through the code.
- Where the call site already knows them, attach `metadata` such as `user_id`, `session_id`, `workflow`, `feature` and `tags` for cost attribution. `feature` is a reserved key of its own (up to 64 characters of letters, digits and `_ . : -`); do not put it under `tags`. Never put secrets or raw personal data in metadata, and do not add new data flows to get it.
- Client specifics:
  - OpenAI Python: `OpenAI(api_key=os.environ["INFRO_API_KEY"], base_url=INFRO_BASE_URL)`. Pass INFRO fields with `extra_body={"fallbacks": [...], "metadata": {...}}`.
  - OpenAI Node or TypeScript: `new OpenAI({ apiKey: process.env.INFRO_API_KEY, baseURL: INFRO_BASE_URL })`. Extra body fields are sent as given; type them with an intersection instead of `any`.
  - Vercel AI SDK: use `createOpenAICompatible` from `@ai-sdk/openai-compatible` with `baseURL` and `apiKey`, or `createOpenAI({ baseURL, apiKey }).chat(modelId)`. Do not use the default `openai(modelId)` call, which targets the Responses API.
  - LangChain: `ChatOpenAI` with the INFRO key and base URL (`base_url` in Python, `configuration.baseURL` in JavaScript). Do not enable the Responses API.
  - LiteLLM: the `openai/` provider prefix with `api_base` set to INFRO, for example `model="openai/anthropic/claude-sonnet-5"`.
  - Any other language: an OpenAI-compatible client with a base URL override, or plain HTTP to the endpoints above.
- Errors and retries: keep the client's existing retry behaviour, and make sure no custom retry loop retries 400, 401, 402, 403, 404, 413, `no_provider_connected`, `connection_out_of_scope` or `model_not_served`. Surface budget and model-policy refusals as distinct, readable errors. Log `X-INFRO-Request-Id` on failures where the client exposes response headers (for example `.withResponse()` in openai-node, `with_raw_response` in openai-python).
- Streaming: request usage in the final chunk where cost is recorded, and handle a stream that ends without `[DONE]` as a failure.
- Configuration and documentation: add `INFRO_API_KEY=` and `INFRO_BASE_URL=https://api.infro.io/v1` to `.env.example` or the project's equivalent, the config schema or validation, deployment manifests and CI secrets lists where the project defines them, and the README's setup section. Remove a provider key from those files only if no call still uses it.

### 4. Test

- Update mocks and fixtures that assert on the old base URL, key variable or model ids.
- Add focused tests for the client factory (key and base URL wiring) and for error classification (which statuses and codes are retried). Do not make live network calls in the test suite.
- Run the project's typecheck, lint and tests, and fix what this change broke.

### 5. Verify with one real request

If `INFRO_API_KEY` is present in your environment, send one minimal request with a model from the inventory and `max_tokens` of 16, without printing the key. Report the HTTP status, the returned `model`, `provider`, `connection` and `usage.cost`, and the `X-INFRO-Request-Id` header. On 503 `no_provider_connected`, tell the user which provider to connect in the console. On 401, tell them to check the key. If the key is not available, give the user this command to run instead:

```bash
curl -sS -D - https://api.infro.io/v1/chat/completions \
  -H "Authorization: Bearer $INFRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "openai/gpt-5.6-terra", "max_tokens": 16, "messages": [{"role": "user", "content": "Reply with OK"}]}'
```

### 6. Report

Finish with:
1. Summary: how many call sites now go through INFRO, and how many stay direct.
2. Inventory table: file, client, API, model before and after, status (routed, converted, direct), reason.
3. Files changed, grouped by purpose.
4. Verification: commands run and their results, including the real request if you made one.
5. Steps for the user in the INFRO console (https://dash.infro.io): connect every provider the routed models need, create a gateway key for each environment, set `INFRO_API_KEY` in each deployment, and optionally add a route, a budget and a fallback.
6. How to roll back: the switch, or the files to revert.
7. Open questions and TODOs.

## Documentation

- Quickstart: https://infro.io/docs/quickstart
- Connect a provider: https://infro.io/docs/providers
- Text generation: https://infro.io/docs/api/chat-completions
- Streaming: https://infro.io/docs/api/streaming
- Errors: https://infro.io/docs/api/errors
- Routes and failover: https://infro.io/docs/routes and https://infro.io/docs/features/fallbacks
- SDKs and frameworks: https://infro.io/docs/sdks
- Full documentation as plain text: https://infro.io/llms-full.txt