# Multi-region AI traffic: latency, quotas and data location

URL: https://infro.io/blog/multi-region-llm-traffic
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Architecture and migration
Format: Guide

> Decide first why you want more than one region. Availability and quota call for a second connection in another Region or provider behind one route with failover; latency calls for measuring every hop before moving anything; data location is set at the provider, by deployment type, inference profile or residency project, and every fallback target has to satisfy it too. Gateway-level regional processing is a separate guarantee, and in INFRO it is on the roadmap, not shipped.

Key takeaways:
- Availability, quota, latency and data location are four goals; name yours before choosing a design.
- A request has up to five locations: client, gateway, provider endpoint, processing region, and stored records.
- Provider controls decide where inference runs: Azure deployment types, Bedrock inference profile prefixes, OpenAI residency projects.
- Global options usually offer the most capacity and the least location control.
- Every failover target must meet the same location policy as the primary, or failover quietly moves the data.

"We need to be multi-region" usually arrives as one requirement and turns out to be four. A reliability review wants the feature to survive a regional outage. A capacity plan wants more tokens per minute than one Region grants. A product manager wants faster answers for users on another continent. A legal review wants personal data processed inside a jurisdiction.

Each of those is a legitimate goal, and each needs a different control. Worse, they conflict: the option with the most capacity is often the one with the least control over location, and a failover target added for availability can move data somewhere the legal review never approved. Separate the goals before you design anything.

[Diagram: Start from the reason. Each branch leads to a different control, and only the last one is about where data is processed.]

## Where is a request, exactly?

For a request that goes through a gateway to a hosted model, there are up to five locations worth naming. Most multi-region confusion comes from treating them as one.

[Diagram: Five places a request can be said to be. Provider controls govern layer 4; your gateway's own guarantees govern layers 2 and 5.]

Layer 4 is the one most regulations care about, and it is set almost entirely at the provider. Layer 3, the endpoint you call, is not the same thing: an Azure resource in an EU region with a Global deployment may still process prompts in any Azure region[1].

## Goal 1: availability across regions

A regional outage at a provider affects everything routed to that Region. Microsoft notes that with Global Standard and Data Zone Standard deployments, "if the primary region experiences an interruption in service, all traffic initially routed to this region is affected"[1]. The remedy is a second, independent place to send the request: a deployment in another Region, an account at another provider, or both.

What makes the second target useful is independence. Two deployments behind the same quota, the same account policy and the same model version share more failure modes than two different providers do. And the second target has to be proven, not assumed: disable the primary in a drill and confirm the standby answers.

## Goal 2: more quota than one Region grants

Provider quotas are regional. Azure assigns quota per region, per model and per deployment type, in tokens per minute, and its guidance for workloads that need more than one deployment supports is to "spread requests across multiple deployments or regions"[2]. Bedrock's `bedrock-runtime` quotas are per model, per Region, with on-demand and cross-Region inference counted separately[3].

The provider-managed alternative is a global option. Azure's Global Standard "provides the highest default quota and eliminates the need to load balance across multiple resources"[1]; Bedrock's global inference profiles route to any supported commercial Region[4]. Both buy capacity by giving up control of layer 4. If location does not matter for a workload, that is often the simplest answer. If it does, spread load across regional deployments yourself.

> **Example: Splitting load across two regional deployments**
> A workload needs about 1.5 million tokens per minute of one model. A single regional deployment can be granted 1 million. Two deployments in two regions inside the same data zone, weighted 60/40, keep each below its limit with headroom, and each acts as the other's failover target. Illustrative figures; your granted quota is in the provider console.

## Goal 3: latency for distant users

Latency is where intuition fails most often, because a model call's time is dominated by the model: queueing, prompt processing and token generation. Network distance matters, but it is usually a smaller share than it is for an ordinary API call, and it can be spent in places that are not obvious. A gateway is one more hop, and where it runs matters as much as where the provider runs.

Measure before moving anything. From the client, record time to first token and total time per provider endpoint. Then compare with the provider-side timing where one is available. Moving a workload to a nearer Region only helps if distance, not generation, is where the time goes.

## Goal 4: keeping data in a jurisdiction

Each provider offers its own control over where inference runs.

[Diagram: Where each provider lets you pin processing, as documented on 2026-09-29. None of these is set by the gateway; all of them are set on your provider account.]

- **Azure OpenAI** sets it by deployment type. Data Zone deployments process prompts and responses only within the US, EU or APAC data zone; Standard keeps processing within the chosen Azure geography; Global may process in any Azure region. Data stored at rest remains in the designated geography for all types[1].
- **Amazon Bedrock** sets it by the model id you send. A geographic inference profile keeps routing within a geography such as the US or EU; a global profile routes to any supported commercial Region[4]. CloudTrail records the Region that processed each cross-Region request[4].
- **OpenAI** sets it per project. Data residency is a project configuration with regional domains such as `eu.api.openai.com`; regional processing is offered for the US, Europe and the UAE, while several other regions offer storage only, and non-US regions require approval for abuse-monitoring controls and a retention amendment[5].

Availability of a location option differs by model, too. On Bedrock's `bedrock-runtime` endpoint, GPT-5.6 Sol is offered through a US geographic profile or a global profile; in the EU Regions the model card lists only global inference for it[6]. A location policy therefore has to be checked per model, not per provider.

The rule that ties this to failover: **every target in a chain must satisfy the same location policy as the primary.** A route that falls back from an EU Data Zone deployment to a global profile keeps the feature up by moving the data. If that is acceptable during an outage, write it down as a decision. If it is not, the fallback has to be another in-zone target, or an error.

## Multi-region routing with INFRO today

INFRO calls the provider accounts and deployments you connect, so layer 4 is governed by the provider settings above, on your accounts, under your agreements ([EU compliance](https://infro.io/docs/compliance/eu)). What INFRO adds is the chain: several connections, one name, and failover before the first token.

- **One connection per regional deployment or account.** An Azure connection names one resource; a Bedrock connection names one Region; the model id you map decides geographic or global processing ([connect a provider](https://infro.io/docs/providers)).
- **A route per workload.** `priority` for a primary with in-policy standbys, `weighted` to split load across regional deployments; a target at weight 0 is a standby that never takes a first attempt ([routes](https://infro.io/docs/routes)).
- **Scoped by environment.** A connection or route scoped to `production` is invisible to a key that does not name that environment, which keeps a staging key from reaching production deployments ([connect a provider](https://infro.io/docs/providers)).
- **Evidence per request.** Each record names the connection that served and every failed attempt, so an audit can show which Region-bound deployment answered ([request traces](https://infro.io/docs/observability/traces)).

[Diagram: Adding a second region as a standby first, then as a share of traffic. Illustrative timing.]

What INFRO does not do today matters as much:

- **No regional processing guarantee at the gateway.** INFRO runs on Cloudflare; requests enter at the nearest edge location and the gateway Worker is placed near its primary database in Frankfurt, which is a performance setting, not a residency guarantee ([EU compliance](https://infro.io/docs/compliance/eu)). For users and providers far from Frankfurt, that placement is part of the latency you should measure. Pinning an organization or project to a region is on the roadmap and is not claimed before it is enforced.
- **`routing.regions` is not a residency control.** The legacy request field is recorded and checked against a stored region policy, but it does not constrain or rank your connections ([smart routing](https://infro.io/docs/features/routing)). Location comes from which connections a route contains.
- **The OpenAI connection has no region setting.** It calls `api.openai.com`, so an OpenAI data residency project's regional domain is not selectable through it today.

On layer 5, prompt and response capture is off by default, can be disabled per request with `logging: false`, and INFRO's settings do not change what your provider retains ([privacy](https://infro.io/docs/privacy)).

## Checklist

- [ ] The goal is named: availability, quota, latency, data location, or a stated combination.
- [ ] All five locations written down for each workload: client, gateway, endpoint, processing, records.
- [ ] Provider location control chosen per model: deployment type, profile prefix or residency project.
- [ ] Every fallback target checked against the same location policy as its primary.
- [ ] Quota headroom per Region recorded, with a weighted split where one Region is not enough.
- [ ] Client-side time to first token measured per endpoint before any move for latency.
- [ ] A failover drill that disables the primary Region and finds the standby on the request record.
- [ ] Gateway-level residency treated as absent until your gateway enforces it.

## Sources

1. Understanding deployment types in Microsoft Foundry Models — Microsoft Learn. https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/deployment-types (accessed 2026-09-29)
2. Manage Azure OpenAI in Microsoft Foundry Models quota — Microsoft Learn. https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/quota (accessed 2026-09-29)
3. Quotas for the bedrock-runtime endpoint — AWS. https://docs.aws.amazon.com/bedrock/latest/userguide/quotas-runtime.html (accessed 2026-09-29)
4. Route model inference requests across AWS Regions with cross-Region inference — AWS. https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html (accessed 2026-09-29)
5. Data controls in the OpenAI platform — OpenAI. https://developers.openai.com/api/docs/guides/your-data (accessed 2026-09-29)
6. GPT-5.6 Sol model card — Amazon Bedrock — AWS. https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-56-sol.html (accessed 2026-09-29)