What should be unified
Authentication, projects, model identifiers, access policy, budgets, request IDs, error envelopes, usage records and cost estimates benefit from one vocabulary. This is the layer product and platform teams should not have to rebuild for every vendor.
A shared catalog also makes capabilities explicit: modality, inputs, limits, lifecycle, and current availability.
What should remain modality-specific
Text can stream tokens over seconds. Image jobs may return one or more assets. Video can run for minutes and needs durable status. Audio may be synchronous for speech or asynchronous for longer processing. Forcing all four into a chat-completions shape produces leaky abstractions.
Use synchronous responses where the work fits and asynchronous jobs with signed webhooks where it does not. The API can still share auth, status, errors, and usage accounting.
Design the job lifecycle
For asynchronous media, create a job, return a stable identifier, expose status, sign webhook deliveries, make handlers idempotent, and support safe polling. Record model, parameters, resolution or duration, cost, and asset expiry.
Plan for cancellation and duplicate events. A webhook can be delivered more than once, and a client timeout does not mean the media generation stopped.
Govern cost and safety across modalities
Token budgets are not enough. Image count, resolution, video seconds, frames, and audio duration need their own units and ceilings. Access policy should prevent development keys from accidentally invoking expensive media models.
Keep modality-aware safety checks and provenance requirements near the job layer so every application does not invent its own inconsistent controls.