# Self-hosted models behind an OpenAI-compatible endpoint

URL: https://infro.io/blog/self-hosted-vllm-openai-compatible
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Architecture and migration
Format: Guide

> Run the model with `vllm serve`, give it a stable `--served-model-name` and an `--api-key`, publish only its `/v1` paths through an HTTPS reverse proxy, and connect that URL to your gateway as an OpenAI-compatible endpoint. Then send it a small weighted share of traffic with a hosted model as the failover target, and grow the share as its error rate and latency hold.

Key takeaways:
- Name the model with --served-model-name so the id your gateway maps to does not change when the weights path does.
- vLLM's --api-key protects /v1, /v2 and /inference only. Publish those paths and nothing else.
- Tool calling with tool_choice auto needs --enable-auto-tool-choice and the right --tool-call-parser for the model.
- A self-hosted target should start as a small weighted share with a hosted model behind it, not as the only target.
- Cost of a self-hosted model is your hardware, so measure tokens served per model and divide your own bill.

Self-hosting an open-weight model used to mean writing an inference server and a client for it. vLLM removed most of that: its server speaks the OpenAI API, so the OpenAI SDK talks to it by changing a base URL and a key[1]. That makes the self-hosted model look like one more provider to your application.

Looking like a provider and behaving like one in production are different things. A hosted API comes with authentication, TLS, a stable model name, an error model and somebody else's capacity planning. This guide covers what you add around `vllm serve` so it can take real traffic, and how to put it beside hosted providers so a busy GPU does not become an outage.

[Diagram: Five layers between your application and the weights. Each one has a job the others cannot do.]

## Serve the model with a stable name and a key

The vLLM documentation's own starting point is short[1]:

```bash
vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
  --dtype auto \
  --api-key token-abc123
```

For anything that other systems will call, add the flags that make the server's contract explicit:

```bash
export VLLM_API_KEY="$(cat /run/secrets/vllm_api_key)"

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --served-model-name llama-3.1-8b \
  --max-model-len 32768 \
  --enable-auto-tool-choice \
  --tool-call-parser llama3_json \
  --chat-template examples/tool_chat_template_llama3.1_json.jinja \
  --host 127.0.0.1 \
  --port 8000
```

What each addition buys:

- **`--served-model-name`** sets "the model name(s) used in the API"; without it, the name is the model argument[2]. Clients and gateways should depend on `llama-3.1-8b`, not on a Hugging Face path you will change when you move to a new checkpoint.
- **`--max-model-len`** fixes the context length for prompt plus output; otherwise it is derived from the model config[2]. Whatever you choose, any fallback target needs at least the same window, or a request that fits today fails during an outage.
- **`--api-key`**, or the `VLLM_API_KEY` environment variable, requires a key on requests[1]. The environment variable keeps the key out of the process list.
- **`--host 127.0.0.1`** keeps the server itself off the network. The port defaults to 8000[2]; only the proxy in front of it should be reachable from outside.

The same server in a container uses vLLM's image. The documented command passes the Hugging Face cache, a token and `--ipc=host`[3]:

```bash
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env "VLLM_API_KEY=$VLLM_API_KEY" \
  -p 127.0.0.1:8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --served-model-name llama-3.1-8b
```

Test it locally with the client your application already uses:

```python
import os
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key=os.environ["VLLM_API_KEY"])

reply = client.chat.completions.create(
    model="llama-3.1-8b",
    messages=[{"role": "user", "content": "Reply with the single word: ready"}],
    max_tokens=5,
)
print(reply.choices[0].message.content, reply.usage)
```

Parameters vLLM supports beyond the OpenAI set, such as `top_k`, go in `extra_body`[1].

## Know exactly what the key protects

vLLM documents a limit that matters the moment the server leaves localhost: `--api-key` "only authenticates requests to endpoints under the `/v1`, `/v2`, and `/inference` path prefixes", and other endpoints on the same server are not authenticated[1]. The Prometheus endpoint at `/metrics` is one of them[4].

So the edge should forward the API paths you use and refuse everything else. A reverse proxy with automatic TLS does both:

```text
models.example.com {
	@api path /v1/chat/completions /v1/models
	handle @api {
		reverse_proxy 127.0.0.1:8000 {
			flush_interval -1
		}
	}
	handle {
		respond 404
	}
}
```

`flush_interval -1` passes streamed tokens through as they arrive instead of buffering them. Scrape `/metrics` from inside the host or private network, never through the public hostname.

## Tool calling needs server flags

Hosted APIs make tool calling look like a request field. On vLLM it is also a server setting. `--enable-auto-tool-choice` lets the model decide when to call a tool, and `--tool-call-parser` names the parser that extracts tool calls from the model's text; different model families need different parsers, such as `hermes`, `llama3_json`, `mistral` or `pythonic`[5]. Named function calling and `tool_choice="required"` work by default through vLLM's structured outputs backend[5].

[Diagram: The flags that decide how the server behaves behind a gateway. A wrong tool parser is the usual reason tool calls arrive as plain text.]

Test tools on the self-hosted target with the same fixtures you use for the hosted model before sending it any agent traffic. A fallback between a hosted model and a self-hosted one is only safe if both emit `tool_calls` for the same inputs.

## Watch the server, not just the requests

vLLM exposes Prometheus metrics with a `model_name` label, including `vllm:num_requests_running`, `vllm:num_requests_waiting`, `vllm:time_to_first_token_seconds`, `vllm:e2e_request_latency_seconds` and `vllm:kv_cache_usage_perc`[4]. Two of them explain most production trouble:

- **Waiting requests** rising while running requests stay flat means the server is at its batch capacity and new requests queue. Time to first token climbs before anything errors.
- **KV cache usage** near 1 means long contexts are competing for memory, which lowers how many requests run at once.

Alert on those, and give the gateway a per-attempt timeout that is shorter than your users' patience, so a queued request moves to the fallback target instead of waiting.

> **Example: A saturation afternoon**
> A support assistant sends 10% of first attempts to a self-hosted model. At 15:00 a batch job starts on the same GPUs; `vllm:num_requests_waiting` climbs and time to first token on the self-hosted target passes the route's 8-second attempt timeout. Those attempts fail over to the hosted target, users see slower first tokens but no errors, and the request records show the self-hosted connection's failures stacking up until it is ejected for a cool-off. Illustrative scenario, not a measurement.

## Connecting a vLLM server to INFRO

INFRO connects a self-hosted server as an **OpenAI-compatible endpoint**: a base URL, including `/v1`, and an optional key ([connect a provider](https://infro.io/docs/providers)). A few rules decide how you publish the server:

- **HTTPS on a public hostname.** The connection's base URL must be HTTPS, and the production gateway refuses loopback, private and link-local addresses and `localhost`. INFRO's gateway runs on Cloudflare and reaches your server over the public internet, so a server that only listens on a private network cannot be connected directly: publish it through a proxy like the one above, protected by the vLLM key.
- **The key is sent as a bearer token**, which is what vLLM's `--api-key` expects. It is sealed at the gateway and never returned ([connect a provider](https://infro.io/docs/providers)).
- **Nothing is guessed.** An OpenAI-compatible connection serves only the models you name. A catalog model reaches it through the connection's model map; a model the catalog does not list, such as your own fine-tune, is named directly as a route target and sent exactly as written.

[Diagram: The route slug is the only name the application knows. The gateway adds the vLLM key and moves a failed attempt to the next target before the first token.]

A connection and a route through the management API:

```bash
curl https://api.infro.io/manage/v1/provider_connections \
  -H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "openai-compatible",
    "name": "vLLM eu-west gpu-1",
    "environment": "production",
    "secret": "'"$VLLM_API_KEY"'",
    "config": {"base_url": "https://models.example.com/v1"},
    "timeout_ms": 30000,
    "validate": true
  }'
```

```json
{
  "slug": "internal-chat",
  "strategy": "weighted",
  "max_retries": 0,
  "timeout_ms": 8000,
  "targets": [
    {"connection_id": "conn_openai", "model": "openai/gpt-6-luna", "weight": 90},
    {"connection_id": "conn_vllm", "model": "llama-3.1-8b", "weight": 10}
  ]
}
```

Under the `weighted` strategy the weight decides which target is tried first, and the others remain the failover order ([routes](https://infro.io/docs/routes)). Both targets must serve the same modality, and each target's model is still checked against your model allowlists. The two targets here are different models, so the pairing is a product decision: use it only when the hosted model passed the same evaluation as the self-hosted one.

Two consequences for cost reporting. The self-hosted target is not a catalog model, so its requests are recorded with `usage.cost` of `null`, meaning unpriced, not free ([chat completions](https://infro.io/docs/api/chat-completions)). And under a blocking budget, a request to an unpriced model is refused by default with `402 unpriced_model_under_budget` ([errors](https://infro.io/docs/api/errors)). What a self-hosted model costs is your GPU bill; the request log gives the other half of that division, with `prompt_tokens` and `completion_tokens` per request filterable by connection ([request traces](https://infro.io/docs/observability/traces)).

[Diagram: The order that keeps each step testable on its own. Traffic arrives only in the last step, and starts small.]

## Checklist

- [ ] `--served-model-name` set, and every client and route uses that name.
- [ ] `--api-key` or `VLLM_API_KEY` set; the server bound to localhost or a private interface.
- [ ] Only `/v1` API paths published over HTTPS; `/metrics` scraped privately.
- [ ] Tool parser chosen for the model family and tested with your real tool fixtures.
- [ ] `--max-model-len` no larger than what every fallback target accepts.
- [ ] Alerts on `vllm:num_requests_waiting` and `vllm:kv_cache_usage_perc`.
- [ ] Route with a short attempt timeout, a small self-hosted weight and an evaluated fallback.
- [ ] Unpriced-traffic budget setting chosen deliberately.

## Sources

1. OpenAI-Compatible Server — vLLM — vLLM. https://docs.vllm.ai/en/latest/serving/online_serving/openai_compatible_server/ (accessed 2026-09-29)
2. vllm serve — vLLM CLI reference — vLLM. https://docs.vllm.ai/en/latest/cli/serve/ (accessed 2026-09-29)
3. Using Docker — vLLM — vLLM. https://docs.vllm.ai/en/latest/deployment/docker/ (accessed 2026-09-29)
4. Metrics — vLLM — vLLM. https://docs.vllm.ai/en/latest/design/metrics/ (accessed 2026-09-29)
5. Tool Calling — vLLM — vLLM. https://docs.vllm.ai/en/latest/features/tool_calling/ (accessed 2026-09-29)