Skip to content

Evaluation and modelsAnalysis

Cohere Model Vault open models: when regional hosting beats an API

Cohere added open-weight models to Model Vault on 3 November 2026, with regional serving in the US, EU and Canada. This piece sets out when single-tenant hosting is worth leaving a per-token API for, three tests to run first, and which parts stay on your own contracts.

By INFRO Editorial

Published

9 min read

A single amber path on a dark ground forking into two glass branches of slightly different brightness

In short

Regional single-tenant hosting beats a per-token API when a residency or isolation requirement rules the API out, or when steady volume keeps dedicated capacity busy. Otherwise run a cost, latency and consistency test on your own prompts first. Pricing for Model Vault was not public on the pages read, so get a quote before modelling anything.

Key takeaways

  • Model Vault is single-tenant, region-chosen inference that Cohere operates, and since 3 November 2026 it also serves open-weight models from DeepSeek, Z.ai, Google, OpenAI and others.
  • The deciding case is a residency or isolation requirement the shared API cannot meet; cost only decides it when utilisation of reserved capacity is high and steady.
  • Run three tests on your own traffic before moving: break-even utilisation, time-to-first-token percentiles at your concurrency, and output equivalence against the API you would leave.
  • Model Vault Encrypted covers only a subset of the open models and, at launch, only the US region, so check the model-by-region cell you actually need.
  • Keep residency-class traffic from failing over to an out-of-region API: configure the route to stop, not to move on.

The short answer

Regional single-tenant hosting beats a per-token API in two situations. The first is when a residency or isolation requirement rules the shared API out, so the comparison is no longer about price. The second is when your volume is steady enough to keep reserved capacity busy, and the per-token bill for that volume is larger than the capacity fee. Everywhere else, a per-token API is the better default, because you pay for tokens used and carry no idle capacity.

Cohere made this comparison live on 3 November 2026. Model Vault, its dedicated inference platform, now serves open-weight models alongside Cohere's own, and regional serving is live in the United States, the European Union and Canada.1 This article is about how to decide, not a recommendation to move. The pages read for it did not publish a Model Vault rate or any latency figure. The pricing page I read routes enterprise buyers to a sales contact,2 so every cost number below is a formula or an illustration until you hold a quote.

What Model Vault offers, as published

Cohere describes Model Vault as a dedicated, fully managed SaaS inference platform: "the convenience of an API with the security of private hosting" without the operational overhead.3 Three published properties matter for an architecture review:

  • Single tenant, region of your choice. Workloads run on single-tenant infrastructure; you choose the region and prompts and outputs stay there, while Cohere operates the infrastructure.1
  • Zero data retention by default. Cohere says neither prompts nor outputs are retained in Model Vault.1
  • A wider model catalogue than Cohere's own. DeepSeek, GLM, MiMo, Gemma, Nemotron, GPT-OSS, Phi and Tencent's Hy3 are listed beside Cohere's Command, Embed, Rerank, Parse, Transcribe and Translate.1

There is also a second tier, Model Vault Encrypted, which uses confidential computing so that, per Cohere, prompts and outputs stay encrypted while the model processes them.1 The confidential computing feature was announced in public beta on 22 October 2026, running on NVIDIA H100 GPUs on Microsoft Azure, with remote attestation that the client checks before sending data.4

The tiers do not cover the same ground. Check the exact cell you need:

Model familyModel VaultModel Vault Encrypted
Cohere CommandYesNo
Cohere Embed, Rerank, Parse, Transcribe, TranslateYesYes
DeepSeek, GLM, MiMo, Nemotron, Hy3YesNo
Gemma, GPT-OSS, PhiYesYes
Model availability by tier, from Cohere's 3 November 2026 post.1

Regions differ too. Model Vault is available in the US, EU and Canada. Model Vault Encrypted was live only in the US at launch, with EU and Canada listed as coming soon, and larger models in Encrypted are a stated future item.1 If your requirement is "EU residency plus encryption in use for a DeepSeek model", the published table says that combination does not exist yet.

When residency or isolation decides it

Start with the requirement, not the bill. A per-token API from a model's original vendor, or a multi-provider router, processes your request on shared infrastructure that you do not choose. Some routers now offer in-region paths. OpenRouter, for example, documents an EU base URL for Business and Enterprise plans with EU-eligible models,5 and its Private Endpoints add-on routes a catalog model to your own dedicated deployment.6 So "residency" alone no longer implies "dedicated hosting". Ask four questions:

  1. Is the requirement about location, isolation, or both? Location can be met by an in-region API. Isolation (no other tenant on the hardware) needs dedicated capacity.
  2. Who must be unable to see the data? Contract-level promises differ from technical ones. Encrypted tiers with attestation address the second kind; plain Model Vault relies on single tenancy and Cohere's operational controls.
  3. Which model must run? If the model you need is only in the non-encrypted column, that settles the tier.
  4. Who signs off? Single-tenant hosting is evidence for a reviewer, not a compliance outcome. Cohere says the regional options help organisations "meet" residency and regulatory requirements,1 so your own legal review still decides.

If the answer to the first question is location only, price an in-region API first. It keeps pay-per-token economics and no idle cost.

The cost test: break-even utilisation

Dedicated capacity converts a variable cost into a mostly fixed one. The question is how many tokens per month make the fixed cost cheaper than per-token billing, and whether you will really reach that volume. The arithmetic is short:

breakeven.py
def breakeven_tokens_per_month(fixed_fee_per_month, blended_usd_per_million):
    """Tokens/month at which dedicated capacity costs the same as the API."""
    return fixed_fee_per_month / blended_usd_per_million * 1_000_000

def monthly_cost_per_million(fixed_fee_per_month, tokens_per_month):
    """Effective rate you actually pay on dedicated capacity."""
    return fixed_fee_per_month / (tokens_per_month / 1_000_000)

The blended rate should weight input, output and cached tokens by your real mix, as a bill would, not the headline input price. See tokens are not the billing unit and provisioned throughput vs pay-as-you-go for the same reasoning on other providers.

Four things move the answer that the formula hides:

  • Peak versus average. Capacity is sized for peaks. If your traffic is bursty and the deployment cannot scale on demand, you pay for the peak all month. Cohere says Model Vault takes care of model scaling and serving,3 but whether that scaling is included in the fee or billed separately is a contract question to ask.
  • Several models, one fee each. If you use five models, you may need five deployments, each with its own utilisation.
  • Engineering and review cost. A second code path, a second set of evals and a second vendor review are real costs even when the managed service removes cluster operations.
  • Price changes on the API side. A per-token competitor can re-price or add a cheaper tier, which moves your break-even. Re-run the numbers on each renewal.

The latency test

A dedicated deployment removes noisy neighbours but adds a network hop if your clients sit far from the chosen region, and it does not by itself guarantee faster first tokens. Measure with streaming, at your real concurrency, over several days, since shared APIs vary by hour. Capture time to first token and total time, and record percentiles, not means. Measuring time to first token covers timeout settings.

ttft.py
import time, statistics
from openai import OpenAI

def run(base_url, api_key, model, prompts):
    client = OpenAI(base_url=base_url, api_key=api_key)
    ttft, total = [], []
    for p in prompts:
        t0 = time.perf_counter()
        first = None
        stream = client.chat.completions.create(
            model=model, stream=True, max_tokens=300,
            messages=[{"role": "user", "content": p}],
        )
        for chunk in stream:
            if first is None and chunk.choices and chunk.choices[0].delta.content:
                first = time.perf_counter() - t0
        total.append(time.perf_counter() - t0)
        ttft.append(first if first is not None else float("nan"))
    q = lambda xs, n: statistics.quantiles(xs, n=100)[n - 1]
    return {"ttft_p50": q(ttft, 50), "ttft_p95": q(ttft, 95), "total_p95": q(total, 95)}

The script assumes an endpoint that accepts the OpenAI chat shape. Whether Model Vault's does is not stated on the pages read, so confirm the contract before reusing it; otherwise adapt the call. Run it from the network location your production traffic uses, once against the API you would leave and once against the candidate region, and test at concurrency equal to your peak, since single-tenant capacity can queue where a shared pool absorbs bursts. Also compare 429 and 5xx rates, not only speed.

The consistency test

The same open-weight name can behave differently on different hosts: quantisation, context limits, chat templates, tool-call parsing and default sampling can all differ. Moving from a shared API to Model Vault changes the host even though the weights are nominally the same. Replay a few hundred real prompts at both endpoints, then compare:

  • structured-output validity rate against your JSON schema
  • tool-call rate and argument validity
  • share of answers ending with a length cutoff
  • output length distribution, which drives your cost
  • your own task score, graded by a method you have calibrated

The method is set out in same open-weight model, different hosts and can you replay production traffic to test a new model safely. Pin the exact model version and ask which version Model Vault serves, because a catalogue entry such as "DeepSeek V4.1 Flash" names a family member, not a snapshot you can audit. Treat a miss on any of these as a migration cost to price in, not a footnote.

What stays on your own contract

Moving to Model Vault adds a contract; it does not replace the ones you hold. Sort the spend by where the commitment lives:

ItemWhere it lives
Model Vault capacity fee, region, support termsNew contract with Cohere
Per-token API traffic that has no residency needYour existing provider contracts, at your negotiated rates
Open-weight model licencesThe model owner's licence, whichever host runs it
Fallback capacity for non-residency trafficYour existing providers

A gateway helps here only with the parts it can see. INFRO connects provider accounts you already hold, and it accepts any OpenAI-compatible endpoint, so a Model Vault deployment can be a connection if its endpoint takes that shape. Your provider bills you directly and INFRO takes 0% of model spend.7 The estimated cost per request uses the rates you enter per model and provider, ahead of list rates, so a fixed capacity fee has to be entered as an effective per-token rate you refresh as utilisation changes. Daily reconciliation against provider-reported cost covers OpenAI and Anthropic, so Model Vault spend would be checked against Cohere's invoice, not against an API. Commitment tracking compares a committed amount with reported cost, which may not exist for this provider; check before relying on it.

Failover without breaking residency

The failure mode that matters most is quiet. A route built for availability will, on a timeout, retry the next target, and the next target may be an API in another jurisdiction. That fixes the outage and breaks the residency promise in one step.

For residency-class traffic, build the route so that the in-region deployment is the only target, or so that the only fallbacks are other in-region deployments. In INFRO, routing rules can send matching requests, selected by metadata, key or project, to their own targets, and failure rules decide per failure kind whether to retry, move on or stop.7 Pick "stop" for kinds where moving on would leave the region, and handle the error in the application. A model allowlist per project then stops anyone calling an out-of-policy model by name. What does failover do to your bill describes the cost side of the same choice.

Two limits apply. Request tracing records metadata and, only if you enable it, prompts and responses for a retention window you choose;8 keep that capture off for regulated payloads unless your policy allows it. And the gateway is itself a hop in the path, so its own location belongs in the same review.

A decision checklist

  • The requirement is written down: location, isolation, or in-use encryption, and who must be unable to see the data.
  • The model you need exists in the tier and region you need, in the published table.
  • You hold a written quote and know whether scaling, peak capacity and support are included.
  • Break-even tokens per month are computed from your real input, output and cache mix, and your forecast clears it with headroom.
  • Time to first token p50 and p95, 429 rate and error rate are measured at peak concurrency from your real network location.
  • Output equivalence is replayed on real prompts: schema validity, tool calls, truncation and task score.
  • The exact model version served is recorded, with a notice period for changes.
  • Residency-class routes stop rather than fall back out of region, and this has been tested by forcing a failure.
  • The new contract, the unchanged contracts and the cost allocation method are documented for finance.

If all of it holds, regional hosting earns its place. If only the first line holds, the cheapest move may be an in-region API or Private Endpoints-style dedicated deployment from your current router; see OpenRouter Private Endpoints for that comparison.

Questions readers ask

Is Model Vault OpenAI-compatible?
The pages read do not say. Cohere states that customers keep using the Model Vault API and familiar request formats 4, which is not the same as the OpenAI chat shape. Ask for the exact request contract, including streaming and tool-call fields, before you plan a drop-in switch.
Does zero data retention in Model Vault mean nothing is logged anywhere in my stack?
No. Cohere describes zero retention of prompts and outputs inside Model Vault 1. Your own gateway, tracing and application logs are separate. INFRO's optional prompt and response capture is off until an owner or admin turns it on, and disabling it does not change storage at your provider 8.
Can a gateway keep requests in-region?
A gateway can only send a request to the endpoints you give it, so pointing a route at an in-region target and refusing fallbacks keeps the model call in-region. The gateway is still a hop in the path. INFRO's regional processing is planned, not available, so treat the gateway's own location as a separate review item.

From the team that builds it

Where this shows up in INFRO

INFRO is a control plane in front of the provider accounts you already hold. The capabilities below are the parts of it this article touches, each with the documentation that defines its behaviour.

INFRO accounts are self-serve. Provider usage is billed to you by your providers; INFRO charges a software subscription and takes 0% of model spend. See plans.

Sources

Numbered in order of first citation. Each was opened and checked against the claim on the date shown.

  1. 1.The world's leading open models, fully under your control (opens in a new tab) — Cohere. Accessed Nov 8, 2026.
  2. 2.Pricing (opens in a new tab) — Cohere. Accessed Nov 8, 2026.
  3. 3.Model Vault | Dedicated Model Inference Platform (opens in a new tab) — Cohere. Accessed Nov 8, 2026.
  4. 4.Introducing Confidential Computing on Cohere Model Vault (opens in a new tab) — Cohere. Accessed Nov 8, 2026.
  5. 5.OpenRouter Quickstart Guide (opens in a new tab) — OpenRouter. Accessed Nov 8, 2026.
  6. 6.Private Endpoints - Bring Your Own Model Endpoints to OpenRouter (opens in a new tab) — OpenRouter. Accessed Nov 8, 2026.
  7. 7.Connect a provider (opens in a new tab) — INFRO. Accessed Nov 8, 2026.
  8. 8.Privacy & data (opens in a new tab) — INFRO. Accessed Nov 8, 2026.

About this article

Read on 2026-11-08: Cohere's blog index, pricing page, Model Vault page, the 3 November open-models post and the 22 October confidential computing post; OpenRouter's quickstart and Private Endpoints docs; INFRO's docs overview, provider and privacy pages. Model Vault rates, latency figures, SLAs and whether the endpoint is OpenAI-compatible were not on the pages read, so they are left out.

Last checked against its sources on . The next review is due by . No separate named reviewer is recorded for this article.

How articles are written, sourced and corrected is set out in the editorial standards.