Async media jobs: webhooks, idempotency and retries

A text completion either returns or it does not. A video render is different: it is accepted in a second, runs for minutes, is billed by your provider per second of output, and finishes long after the HTTP request that started it has closed. Every failure mode of distributed systems shows up in that gap — duplicate submits, lost notifications, events arriving twice or out of order, and output that expires before anyone downloads it.

OWASP's entry on unbounded consumption lists resource-intensive operations and runaway pay-per-use cost among the ways an AI application runs up cost it did not intend¹. With async media, the most common version is not an attacker. It is your own retry logic submitting the same render twice.

The shape of an async job

A job API turns one long request into three short interactions: a submit that returns immediately, a series of state changes you learn about later, and a result you fetch while it is still available.

Design around four facts that hold for most async media APIs, including INFRO's (async jobs):

• The submit is not idempotent by default. POST is not safe to repeat, and the render it starts costs money whether or not you saw the response.
• Completion is an event, not a response. You learn about it from a webhook or by asking.
• The output is temporary. A result link is a handoff with an expiry, not storage.
• State can always be re-read. The job record is the source of truth; events are notifications about it.

Submit once: idempotency keys

The dangerous moment is a submit that times out on your side. The provider may have accepted the render; you do not know. Retrying without protection starts a second render and pays for both.

An idempotency key fixes this: the client sends a unique key with the POST, and the server returns the stored first response for any repeat with the same key instead of doing the work again. The IETF has a draft standardising an Idempotency-Key header. It recommends a UUID or similar random value, and suggests 400 when a required key is missing, 422 when a key is reused with a different payload, and 409 when a retry arrives while the original is still being processed². It is a draft, not an RFC: the latest revision, 07, has expired³. Treat the status codes as a common convention and read each API's documentation for what it actually returns.

INFRO honours Idempotency-Key on POST /v1/videos: any unique string up to 255 characters. It stores the first 202 for a key and replays it for 24 hours. Its error codes differ from the draft's suggestion: a reused key with a different body returns 400 invalid_request_error with code idempotency_key_reuse, and a retry while the first request is still in progress returns 400 with code idempotency_key_in_progress (async jobs). Branch on the code, not only the status.

The mistake to avoid is generating a new UUID on each attempt, which defeats the purpose. Derive the key from the record that asked for the render, so every retry of that request carries the same key.

import os
import time
import uuid

import requests

API = "https://api.infro.io/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['INFRO_API_KEY']}"}
NAMESPACE = uuid.UUID("5d2a3c1e-7b0f-4c55-9f0e-2f4b8f6a9c11")  # fixed for your service


def submit_render(render_request_id: str, prompt: str, model: str) -> dict:
    # Same record -> same key, on every retry and every worker.
    key = str(uuid.uuid5(NAMESPACE, f"render:{render_request_id}"))
    body = {"model": model, "prompt": prompt, "duration_seconds": 5}

    for attempt in range(4):
        try:
            res = requests.post(f"{API}/videos", json=body, timeout=30,
                                headers={**HEADERS, "Idempotency-Key": key})
        except requests.RequestException:
            time.sleep(2 ** attempt)  # the render may exist; the key makes retrying safe
            continue
        if res.status_code == 202:
            return res.json()  # the job, as first created (status queued)
        code = (res.json().get("error") or {}).get("code")
        if code == "idempotency_key_in_progress":
            time.sleep(2 ** attempt)
            continue
        res.raise_for_status()  # idempotency_key_reuse or a real 4xx: a bug, do not loop
    raise RuntimeError(f"submit for {render_request_id} did not complete")

Store the returned job id on your record in the same transaction that marks the record as submitted. If the process dies between the POST and that write, the next attempt replays the same 202 and recovers the job id.

Learn about completion: webhooks

Polling works, but each poll is a request against your rate limits, and a render that takes eight minutes needs dozens of them. A webhook replaces the loop with one inbound request per terminal event.

INFRO sends job.succeeded, job.failed and job.canceled events, either to endpoints configured for the organization in the console or to a webhook.url attached to a single request (webhooks). Subscribe to failures too; an endpoint that hears only about successes never learns that a render died.

Verify the signature

Every widely used webhook scheme is some variant of the same idea: an HMAC-SHA256 over the raw body and a timestamp, keyed by a shared secret, with the timestamp bounding how long a captured delivery can be replayed. The Standard Webhooks specification signs {webhook-id}.{timestamp}.{body}, encodes the digest in base64 after a v1, prefix, allows several space-delimited signatures for key rotation, and recommends constant-time comparison and a timestamp tolerance⁴.

INFRO's scheme is documented as INFRO-Signature: t=<unix seconds>,v1=<hex>, where v1 is the lowercase hex HMAC-SHA256 of {t}.{raw body} keyed by the endpoint's full signing secret, whsec_ prefix included. During a secret rotation the header carries one v1 per secret. The documented tolerance is five minutes.

import hashlib
import hmac
import os
import time

import psycopg
from fastapi import FastAPI, Request, Response

app = FastAPI()
SECRET = os.environ["INFRO_WEBHOOK_SECRET"].encode()  # the full whsec_... value
TOLERANCE = 300
db = psycopg.connect(os.environ["DATABASE_URL"], autocommit=True)


def verify(raw: bytes, header: str) -> bool:
    pairs = [part.split("=", 1) for part in header.split(",") if "=" in part]
    ts = next((v for k, v in pairs if k == "t"), None)
    sigs = [v for k, v in pairs if k == "v1"]
    if not ts or not sigs or not ts.isdigit():
        return False
    if abs(time.time() - int(ts)) > TOLERANCE:
        return False  # stale or replayed
    expected = hmac.new(SECRET, ts.encode() + b"." + raw, hashlib.sha256).hexdigest()
    return any(hmac.compare_digest(expected, s) for s in sigs)


@app.post("/hooks/infro")
async def infro_hook(request: Request) -> Response:
    raw = await request.body()  # raw bytes: never re-serialise before verifying
    if not verify(raw, request.headers.get("INFRO-Signature", "")):
        return Response(status_code=400)

    event = await request.json()
    with db.transaction():
        inserted = db.execute(
            "INSERT INTO webhook_events (event_id, type, job_id, created_at, payload) "
            "VALUES (%s, %s, %s, to_timestamp(%s), %s) ON CONFLICT (event_id) DO NOTHING",
            (event["id"], event["type"], event["data"]["id"], event["created_at"], raw.decode()),
        ).rowcount
        if inserted:
            db.execute("INSERT INTO job_outbox (event_id) VALUES (%s)", (event["id"],))
    return Response(status_code=200)  # duplicates are acknowledged too

The handler does three things and nothing else: verify, record durably, acknowledge. A worker reads job_outbox and downloads the output. That split is not style. INFRO counts a response slower than 10 seconds as a failed delivery and does not follow redirects (webhooks); Standard Webhooks recommends a timeout between 15 and 30 seconds⁴. A handler that fetches a large video inline will miss the window, earn a retry, and then race its own retry mid-download.

Deduplicate, and do not trust order

INFRO documents delivery as at-least-once and unordered: a delivery your handler processed but failed to acknowledge is sent again, and two jobs finishing close together can arrive in either order. The event id stays the same across retries of one delivery, which is why the unique index on event_id is the whole deduplication strategy. Standard Webhooks makes the same recommendation with its webhook-id header⁴.

For order, compare the event's created_at and, better, the job's own status: a terminal state never changes, so an event for a job you already recorded as succeeded changes nothing.

Know the retry schedule

A deploy that breaks signature verification for an hour does not lose events — they retry — but it does delay them. The most common cause of a verifier that "suddenly" fails is body-parsing middleware that consumed or re-serialised the raw bytes before the handler saw them. Alert on your webhook route's 4xx and 5xx rate so a broken verifier is found in minutes, not when a customer asks where their video is.

Close the gap: a reconciler

Webhooks reduce the need to poll; they do not remove it. A delivery can exhaust its retries, an endpoint can be misconfigured, and a job submitted by a worker that crashed has no record of its id on your side at all. A reconciler handles all three.

import random
import time

import requests

from submit import API, HEADERS


def in_flight_jobs() -> list[dict]:
    jobs, after = [], None
    while True:
        params = [("status", "queued"), ("status", "running"), ("limit", "100")]
        if after:
            params.append(("after", after))
        page = requests.get(f"{API}/jobs", params=params, headers=HEADERS, timeout=30).json()
        jobs.extend(page["data"])
        if not page["has_more"]:
            return jobs
        after = page["next_cursor"]


def wait_for(job_id: str, deadline_s: float = 1800) -> dict:
    delay, started = 2.0, time.monotonic()
    while time.monotonic() - started < deadline_s:
        job = requests.get(f"{API}/jobs/{job_id}", headers=HEADERS, timeout=30).json()
        if job["status"] in ("succeeded", "failed", "canceled"):
            return job
        time.sleep(delay + random.uniform(0, delay / 4))  # jitter
        delay = min(delay * 1.5, 30.0)
    raise TimeoutError(job_id)

Run in_flight_jobs() at worker startup to re-attach work orphaned by a crash, and on a schedule to catch terminal jobs whose events never arrived. The backoff — start near two seconds, multiply by 1.5, cap at 30 seconds, add jitter — is the one the jobs documentation recommends. Set your own deadline; a job with no terminal state after it is something to surface, not to poll forever.

Output, cancellation and cost

Copy the output immediately. INFRO keeps output files for 24 hours after completion; after expires_at the link returns 404, while the job record stays readable with output set to null. Regenerating an expired render is billed again by your provider and will not be identical unless you pinned a seed.

Cancel does not stop the provider. Cancelling stops INFRO tracking the job and records no cost, but it does not currently stop the render at your provider, which may still bill for it. Branch on the status the cancel call returns, because a render that finished a moment earlier comes back succeeded.

Cost appears on success only. usage.cost on a succeeded job is an estimate of what your provider will bill, in seconds of output at your rate or the vendor's list rate, and null when the model is unpriced. Failed and canceled jobs record no cost in INFRO, which is not the same as your provider billing nothing for them.

Failover happens before running. A job's id is stable even when the submit fails over to another target on your route; once a provider has accepted the render it is never moved (async jobs).

Checklist

• Idempotency keys are derived from your own record id, identical on every retry.
• The job id is stored on your record in the same transaction that marks it submitted.
• The webhook endpoint subscribes to failed and canceled events, not only succeeded.
• Signatures are verified on raw bytes, in constant time, with a five-minute window, accepting any v1 during rotation.
• Events are deduplicated with a unique index on the event id; duplicates are acknowledged with 2xx.
• The handler acknowledges within the sender's timeout and does downloads on a queue.
• A reconciler lists queued and running jobs at startup and on a schedule, with backoff and a deadline.
• Output is copied to your own storage as soon as a job succeeds.
• A test flips one signature character and confirms a 400, and replays one real delivery to confirm deduplication.