An on prem LLM is a language model you run on hardware you control, in your own data centre or private cloud, instead of calling a provider's API. It makes sense when data must stay on your infrastructure, when a regulator or contract requires it, or when steady volume keeps the GPUs busy; otherwise an API is usually simpler to operate and costs less at low or uneven volume.
This guide covers when on-premises is worth it, the serving stack, how to size GPU memory for the weights and the KV cache, the hybrid pattern most teams end up with, and the costs that do not appear on a hardware quote.
When does an on-prem LLM make sense?
Name the reason before buying hardware. Each one leads to a different design.
| Reason | What it requires | What it does not require |
|---|---|---|
| Data must stay on your infrastructure | Every request carrying that data served in-house, with no fallback outside | Every request in the company served in-house |
| A regulator, customer contract or sovereignty rule | Documented control of where processing happens, and evidence of it | The largest model available |
| Latency inside your network | The model near the callers, with capacity headroom | Multi-region failover to the internet |
| Steady, high volume | High utilisation, sustained for months | Burst capacity, which an API absorbs better |
| A fixed, validated model version | Pinned weights you upgrade on your schedule | Running the weights yourself; some hosts pin versions too |
Two observations follow. First, most of these reasons apply to some traffic, not all of it, which is why a hybrid design is common. Second, "air-gapped" is a stronger requirement than "on-prem": no outbound network at all, so weights, container images and updates arrive through a controlled transfer process, and nothing in the request path may depend on an outside service.
What you give up is most closed models. Their weights are not published, so on-prem in practice means open weight models, unless a vendor offers a private deployment under its own agreement. Open weight models in 2026 compares the current families and, just as important, their licenses.
The on-prem LLM stack
From the bottom up:
- Accelerators. GPUs with enough memory for the weights and the KV cache, and fast interconnect between them if a model spans several.
- A serving engine. The software that loads the weights, batches requests and streams tokens.
- An API edge. TLS, authentication and the paths you choose to publish.
- Routing and policy. Which application may call which model, failover, budgets and request records.
- Observability. Engine metrics such as queue depth and KV cache use, plus per-request records.
Serving engines
| Engine | Use it for | What its documentation states |
|---|---|---|
| vLLM | Production serving | PagedAttention for KV memory, continuous batching, chunked prefill and prefix caching; NVIDIA and AMD GPUs, CPUs and plugins for TPU, Gaudi and Ascend; FP8, INT8, INT4, GPTQ and AWQ; an OpenAI-compatible server1 |
| SGLang | Production serving | RadixAttention and prefix caching, multi-GPU parallelism; NVIDIA, AMD, Intel Xeon, TPU and Ascend hardware; OpenAI-compatible APIs2 |
| Text Generation Inference | Existing deployments only | In maintenance mode; Hugging Face recommends vLLM and SGLang going forward3 |
| Ollama | Development and single machines | A subset of the OpenAI API: chat completions, completions, responses, models and embeddings4 |
Model cards point the same way: DeepSeek V4.1 Flash, for example, gives serve commands for vLLM and SGLang and points to quantized builds for llama.cpp and Ollama for local use5. Pick vLLM or SGLang for shared production capacity, test the specific model on the engine you choose, and keep Ollama for laptops and prototypes. The production details around vllm serve (a stable model name, an API key, which paths the key covers, tool-call parsers, metrics) are in self-hosted models behind an OpenAI-compatible endpoint.
How much GPU memory does an on-prem LLM need?
Memory is the first constraint, and it has two parts: the weights, which are fixed, and the KV cache, which grows with every token in flight.
Weights
Weights take parameters times bytes per parameter. NVIDIA's worked example: a 7-billion-parameter model at 16-bit precision needs about 14 GB6. At 8-bit it is half that, at 4-bit a quarter, before overhead. Vendors increasingly publish the fit directly: gpt-oss-20b runs within 16 GB7, gpt-oss-120b fits on one 80 GB GPU because its expert weights ship in MXFP48, and Mistral Large 3 at FP8 runs on a single node of B200 or H200 GPUs, with its vLLM example using eight H200s9.
For mixture-of-experts models, size memory for the total parameters, not the active ones. Active parameters set compute per token; every expert still has to be loaded.
KV cache
During generation the engine keeps keys and values for every token of every running request. NVIDIA gives the size as batch size × sequence length × 2 × layers × hidden size × bytes per element, the 2 covering keys and values6. Models with grouped-query attention store fewer KV heads than attention heads, so replace hidden size with KV heads × head dimension:
KV bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_element
KV cache total = KV bytes per token × tokens in flight
GPU memory needed ≈ weights + KV cache total + engine overheadThe example shows why concurrency and context length are capacity questions, not just product settings. Before buying hardware, decide the longest context you will accept, the number of requests you need running at once, and the precision you have evaluated for quality. Then cap the engine's maximum model length to match, so one long prompt cannot crowd out everything else.
The hybrid pattern: on-prem for sensitive traffic, APIs for the rest
Few organisations need every request served in-house. The common design puts sensitive workloads on owned capacity and everything else on provider APIs, with one endpoint in front:
- One OpenAI-compatible endpoint. Applications use one client and one base URL. Because vLLM, SGLang and Ollama all speak the OpenAI API124, the on-prem server is one more target behind the gateway.
- Rules decide the destination. Route by project, application, metadata such as a data-classification tag, or key, so the decision is made once and enforced for every caller rather than remembered in each codebase.
- Failover is a policy, not a reflex. For general traffic, a failed on-prem attempt can move to a hosted model. For traffic that must stay in-house, a failure should stop and return an error. Failing over to a public API would quietly break the requirement that justified the hardware. Routing rules and failure rules covers how to express both on one route.
- Overflow is explicit. When the on-prem queue is full, non-sensitive work can spill to an API, which lets you size owned capacity for the steady load rather than the peak. Automatic failover explains which failures are safe to retry elsewhere.
There is one constraint to settle early: the gateway sits in the path of every request it routes. If your sensitive traffic may not leave your network, the gateway that routes it has to run inside your network too.
The honest costs of an on-prem LLM
The hardware quote is the smallest surprise. The rest:
- Utilisation. Cost per token is total monthly cost divided by tokens served. A cluster at half load costs twice as much per token as a full one, and GPUs cost the same idle. Provider APIs bill per use; owned capacity bills for time.
- People. Someone owns driver and engine upgrades, model upgrades and re-evaluation, capacity planning, security patching and the pager. Budget engineering time as part of the cost, not as an afterthought.
- Model churn. Open weight families keep shipping new versions. Each upgrade means downloading, quantizing, evaluating and rolling out, and an air-gapped site adds a transfer process to each step.
- Lead times and lock-in to a size. Hardware bought for today's model fixes what you can run for years. A model that outgrows a node needs a new node, not a configuration change.
- License review. On-prem, compliance with each model's license is entirely yours, including attribution and revenue-threshold clauses that some 2026 licenses carry.
On-prem and hybrid with INFRO
INFRO is a control plane over provider accounts you already hold, and an on-prem server is one of them: it connects as an OpenAI-compatible endpoint, with a base URL and an optional key, next to the hosted providers you use (connect a provider). Routes put it beside hosted targets with failover, if-rules send matching requests (by metadata, project, key and more) to their own targets, and failure rules decide per failure kind whether to retry, move on or stop (routes).
On the hosted service, INFRO's gateway runs on Cloudflare and reaches your server over the public internet: the base URL must be HTTPS, and private, loopback and link-local addresses are refused. So the hosted gateway fits the hybrid pattern when the on-prem server may be published through an authenticated HTTPS edge. It does not fit traffic that must never leave your network, or an air-gapped site.
For those, self-hosted and hybrid deployment of INFRO itself (the gateway in your own infrastructure, the console hosted by INFRO) is available under an enterprise agreement. It is scoped and delivered as part of that agreement, not a package you download, and self-serve plans run on the hosted service. The enterprise page describes how that conversation starts.