Self-hosted models behind an OpenAI-compatible endpoint

Self-hosting an open-weight model used to mean writing an inference server and a client for it. vLLM removed most of that: its server speaks the OpenAI API, so the OpenAI SDK talks to it by changing a base URL and a key¹. That makes the self-hosted model look like one more provider to your application.

Looking like a provider and behaving like one in production are different things. A hosted API comes with authentication, TLS, a stable model name, an error model and somebody else's capacity planning. This guide covers what you add around vllm serve so it can take real traffic, and how to put it beside hosted providers so a busy GPU does not become an outage.

Serve the model with a stable name and a key

The vLLM documentation's own starting point is short¹:

vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
  --dtype auto \
  --api-key token-abc123

For anything that other systems will call, add the flags that make the server's contract explicit:

export VLLM_API_KEY="$(cat /run/secrets/vllm_api_key)"

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --served-model-name llama-3.1-8b \
  --max-model-len 32768 \
  --enable-auto-tool-choice \
  --tool-call-parser llama3_json \
  --chat-template examples/tool_chat_template_llama3.1_json.jinja \
  --host 127.0.0.1 \
  --port 8000

What each addition buys:

• --served-model-name sets "the model name(s) used in the API"; without it, the name is the model argument². Clients and gateways should depend on llama-3.1-8b, not on a Hugging Face path you will change when you move to a new checkpoint.
• --max-model-len fixes the context length for prompt plus output; otherwise it is derived from the model config². Whatever you choose, any fallback target needs at least the same window, or a request that fits today fails during an outage.
• --api-key, or the VLLM_API_KEY environment variable, requires a key on requests¹. The environment variable keeps the key out of the process list.
• --host 127.0.0.1 keeps the server itself off the network. The port defaults to 8000²; only the proxy in front of it should be reachable from outside.

The same server in a container uses vLLM's image. The documented command passes the Hugging Face cache, a token and --ipc=host³:

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env "VLLM_API_KEY=$VLLM_API_KEY" \
  -p 127.0.0.1:8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --served-model-name llama-3.1-8b

Test it locally with the client your application already uses:

import os
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key=os.environ["VLLM_API_KEY"])

reply = client.chat.completions.create(
    model="llama-3.1-8b",
    messages=[{"role": "user", "content": "Reply with the single word: ready"}],
    max_tokens=5,
)
print(reply.choices[0].message.content, reply.usage)

Parameters vLLM supports beyond the OpenAI set, such as top_k, go in extra_body¹.

Know exactly what the key protects

vLLM documents a limit that matters the moment the server leaves localhost: --api-key "only authenticates requests to endpoints under the /v1, /v2, and /inference path prefixes", and other endpoints on the same server are not authenticated¹. The Prometheus endpoint at /metrics is one of them⁴.

So the edge should forward the API paths you use and refuse everything else. A reverse proxy with automatic TLS does both:

models.example.com {
	@api path /v1/chat/completions /v1/models
	handle @api {
		reverse_proxy 127.0.0.1:8000 {
			flush_interval -1
		}
	}
	handle {
		respond 404
	}
}

flush_interval -1 passes streamed tokens through as they arrive instead of buffering them. Scrape /metrics from inside the host or private network, never through the public hostname.

Tool calling needs server flags

Hosted APIs make tool calling look like a request field. On vLLM it is also a server setting. --enable-auto-tool-choice lets the model decide when to call a tool, and --tool-call-parser names the parser that extracts tool calls from the model's text; different model families need different parsers, such as hermes, llama3_json, mistral or pythonic⁵. Named function calling and tool_choice="required" work by default through vLLM's structured outputs backend⁵.

Test tools on the self-hosted target with the same fixtures you use for the hosted model before sending it any agent traffic. A fallback between a hosted model and a self-hosted one is only safe if both emit tool_calls for the same inputs.

Watch the server, not just the requests

vLLM exposes Prometheus metrics with a model_name label, including vllm:num_requests_running, vllm:num_requests_waiting, vllm:time_to_first_token_seconds, vllm:e2e_request_latency_seconds and vllm:kv_cache_usage_perc⁴. Two of them explain most production trouble:

• Waiting requests rising while running requests stay flat means the server is at its batch capacity and new requests queue. Time to first token climbs before anything errors.
• KV cache usage near 1 means long contexts are competing for memory, which lowers how many requests run at once.

Alert on those, and give the gateway a per-attempt timeout that is shorter than your users' patience, so a queued request moves to the fallback target instead of waiting.

Connecting a vLLM server to INFRO

INFRO connects a self-hosted server as an OpenAI-compatible endpoint: a base URL, including /v1, and an optional key (connect a provider). A few rules decide how you publish the server:

• HTTPS on a public hostname. The connection's base URL must be HTTPS, and the production gateway refuses loopback, private and link-local addresses and localhost. INFRO's gateway runs on Cloudflare and reaches your server over the public internet, so a server that only listens on a private network cannot be connected directly: publish it through a proxy like the one above, protected by the vLLM key.
• The key is sent as a bearer token, which is what vLLM's --api-key expects. It is sealed at the gateway and never returned (connect a provider).
• Nothing is guessed. An OpenAI-compatible connection serves only the models you name. A catalog model reaches it through the connection's model map; a model the catalog does not list, such as your own fine-tune, is named directly as a route target and sent exactly as written.

A connection and a route through the management API:

curl https://api.infro.io/manage/v1/provider_connections \
  -H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "openai-compatible",
    "name": "vLLM eu-west gpu-1",
    "environment": "production",
    "secret": "'"$VLLM_API_KEY"'",
    "config": {"base_url": "https://models.example.com/v1"},
    "timeout_ms": 30000,
    "validate": true
  }'

{
  "slug": "internal-chat",
  "strategy": "weighted",
  "max_retries": 0,
  "timeout_ms": 8000,
  "targets": [
    {"connection_id": "conn_openai", "model": "openai/gpt-6-luna", "weight": 90},
    {"connection_id": "conn_vllm", "model": "llama-3.1-8b", "weight": 10}
  ]
}

Under the weighted strategy the weight decides which target is tried first, and the others remain the failover order (routes). Both targets must serve the same modality, and each target's model is still checked against your model allowlists. The two targets here are different models, so the pairing is a product decision: use it only when the hosted model passed the same evaluation as the self-hosted one.

Two consequences for cost reporting. The self-hosted target is not a catalog model, so its requests are recorded with usage.cost of null, meaning unpriced, not free (chat completions). And under a blocking budget, a request to an unpriced model is refused by default with 402 unpriced_model_under_budget (errors). What a self-hosted model costs is your GPU bill; the request log gives the other half of that division, with prompt_tokens and completion_tokens per request filterable by connection (request traces).

Checklist

• --served-model-name set, and every client and route uses that name.
• --api-key or VLLM_API_KEY set; the server bound to localhost or a private interface.
• Only /v1 API paths published over HTTPS; /metrics scraped privately.
• Tool parser chosen for the model family and tested with your real tool fixtures.
• --max-model-len no larger than what every fallback target accepts.
• Alerts on vllm:num_requests_waiting and vllm:kv_cache_usage_perc.
• Route with a short attempt timeout, a small self-hosted weight and an evaluated fallback.
• Unpriced-traffic budget setting chosen deliberately.