Why the prototype is misleading
Forwarding one JSON request to two providers is easy. Production depth includes streaming translation, tool calls, structured output, cancellations, media jobs, authentication, rate limits, retries, circuit breakers, price changes, usage reconciliation, roles, budgets, and incident response.
Estimate the complete operating surface, not the first proxy. Every new provider and capability increases the compatibility matrix.
When building is rational
Build if inference routing is part of your product's unique advantage, if workloads must remain fully inside a controlled network, or if you need a specialized scheduling or hardware integration no external gateway offers.
Assign a durable owner and budget. Internal infrastructure without an owner becomes an invisible dependency that fails during provider incidents and audit requests.
When buying is rational
Buy when model access is necessary but undifferentiated, the team is small, time to market matters, provider changes are frequent, or finance and security need a shared control boundary.
A standard client contract and exportable data reduce buyer risk. The gateway should remove vendor lock-in, not replace it with a proprietary application interface.
Run a total-cost comparison
Model engineering, on-call, provider certification, billing reconciliation, security review, feature lag, and opportunity cost. Compare those with platform pricing, data-path risk, control fit, latency overhead, and exit cost.
Use a proof of concept with real tool calls, streams, and failures. Happy-path benchmarks hide most of the ownership cost.