# Async media jobs: webhooks, idempotency and retries

URL: https://infro.io/blog/async-media-jobs-webhooks
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Agents and multimodal systems
Format: Guide

> Submit each render once with an Idempotency-Key derived from your own record, receive completion by webhook, verify the HMAC over the raw body with a short timestamp window, deduplicate on the event id with a unique index, acknowledge fast and do the work on a queue — and keep a reconciler that lists in-flight jobs, because delivery is at-least-once and can still fail.

Key takeaways:
- A retried POST after a timeout is how duplicate renders happen. Derive the idempotency key from your own record, not from a fresh UUID per attempt.
- Verify the signature over the raw request bytes, in constant time, and reject timestamps outside a few minutes.
- Delivery is at-least-once and unordered: deduplicate on the event id with a unique index and never infer sequence from arrival.
- Acknowledge within the sender's timeout and move downloads to a queue; a slow handler earns a retry that races itself.
- Webhooks can still be lost. A reconciler that lists queued and running jobs at startup and on a schedule closes the gap.

A text completion either returns or it does not. A video render is different: it is accepted in a second, runs for minutes, is billed by your provider per second of output, and finishes long after the HTTP request that started it has closed. Every failure mode of distributed systems shows up in that gap — duplicate submits, lost notifications, events arriving twice or out of order, and output that expires before anyone downloads it.

OWASP's entry on unbounded consumption lists resource-intensive operations and runaway pay-per-use cost among the ways an AI application runs up cost it did not intend[1]. With async media, the most common version is not an attacker. It is your own retry logic submitting the same render twice.

## The shape of an async job

A job API turns one long request into three short interactions: a submit that returns immediately, a series of state changes you learn about later, and a result you fetch while it is still available.

[Diagram: The documented INFRO job lifecycle. queued and running are the only non-terminal states; a terminal state never changes.]

Design around four facts that hold for most async media APIs, including INFRO's ([async jobs](https://infro.io/docs/api/jobs)):

- **The submit is not idempotent by default.** POST is not safe to repeat, and the render it starts costs money whether or not you saw the response.
- **Completion is an event, not a response.** You learn about it from a webhook or by asking.
- **The output is temporary.** A result link is a handoff with an expiry, not storage.
- **State can always be re-read.** The job record is the source of truth; events are notifications about it.

## Submit once: idempotency keys

The dangerous moment is a submit that times out on your side. The provider may have accepted the render; you do not know. Retrying without protection starts a second render and pays for both.

An idempotency key fixes this: the client sends a unique key with the POST, and the server returns the stored first response for any repeat with the same key instead of doing the work again. The IETF has a draft standardising an `Idempotency-Key` header. It recommends a UUID or similar random value, and suggests `400` when a required key is missing, `422` when a key is reused with a different payload, and `409` when a retry arrives while the original is still being processed[2]. It is a draft, not an RFC: the latest revision, 07, has expired[3]. Treat the status codes as a common convention and read each API's documentation for what it actually returns.

INFRO honours `Idempotency-Key` on `POST /v1/videos`: any unique string up to 255 characters. It stores the first `202` for a key and replays it for 24 hours. Its error codes differ from the draft's suggestion: a reused key with a different body returns `400 invalid_request_error` with code `idempotency_key_reuse`, and a retry while the first request is still in progress returns `400` with code `idempotency_key_in_progress` ([async jobs](https://infro.io/docs/api/jobs)). Branch on the `code`, not only the status.

The mistake to avoid is generating a new UUID on each attempt, which defeats the purpose. Derive the key from the record that asked for the render, so every retry of that request carries the same key.

```python
import os
import time
import uuid

import requests

API = "https://api.infro.io/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['INFRO_API_KEY']}"}
NAMESPACE = uuid.UUID("5d2a3c1e-7b0f-4c55-9f0e-2f4b8f6a9c11")  # fixed for your service


def submit_render(render_request_id: str, prompt: str, model: str) -> dict:
    # Same record -> same key, on every retry and every worker.
    key = str(uuid.uuid5(NAMESPACE, f"render:{render_request_id}"))
    body = {"model": model, "prompt": prompt, "duration_seconds": 5}

    for attempt in range(4):
        try:
            res = requests.post(f"{API}/videos", json=body, timeout=30,
                                headers={**HEADERS, "Idempotency-Key": key})
        except requests.RequestException:
            time.sleep(2 ** attempt)  # the render may exist; the key makes retrying safe
            continue
        if res.status_code == 202:
            return res.json()  # the job, as first created (status queued)
        code = (res.json().get("error") or {}).get("code")
        if code == "idempotency_key_in_progress":
            time.sleep(2 ** attempt)
            continue
        res.raise_for_status()  # idempotency_key_reuse or a real 4xx: a bug, do not loop
    raise RuntimeError(f"submit for {render_request_id} did not complete")
```

Store the returned job id on your record in the same transaction that marks the record as submitted. If the process dies between the POST and that write, the next attempt replays the same `202` and recovers the job id.

## Learn about completion: webhooks

Polling works, but each poll is a request against your rate limits, and a render that takes eight minutes needs dozens of them. A webhook replaces the loop with one inbound request per terminal event.

[Diagram: Submit with a key, let the job run, receive one signed event, verify it and acknowledge quickly.]

INFRO sends `job.succeeded`, `job.failed` and `job.canceled` events, either to endpoints configured for the organization in the console or to a `webhook.url` attached to a single request ([webhooks](https://infro.io/docs/api/webhooks)). Subscribe to failures too; an endpoint that hears only about successes never learns that a render died.

### Verify the signature

Every widely used webhook scheme is some variant of the same idea: an HMAC-SHA256 over the raw body and a timestamp, keyed by a shared secret, with the timestamp bounding how long a captured delivery can be replayed. The Standard Webhooks specification signs `{webhook-id}.{timestamp}.{body}`, encodes the digest in base64 after a `v1,` prefix, allows several space-delimited signatures for key rotation, and recommends constant-time comparison and a timestamp tolerance[4].

[Diagram: The same idea, two encodings. Write the verifier for the scheme your sender documents; a Standard Webhooks library will not verify an INFRO delivery unchanged.]

INFRO's scheme is documented as `INFRO-Signature: t=<unix seconds>,v1=<hex>`, where `v1` is the lowercase hex HMAC-SHA256 of `{t}.{raw body}` keyed by the endpoint's full signing secret, `whsec_` prefix included. During a secret rotation the header carries one `v1` per secret. The documented tolerance is five minutes.

```python
import hashlib
import hmac
import os
import time

import psycopg
from fastapi import FastAPI, Request, Response

app = FastAPI()
SECRET = os.environ["INFRO_WEBHOOK_SECRET"].encode()  # the full whsec_... value
TOLERANCE = 300
db = psycopg.connect(os.environ["DATABASE_URL"], autocommit=True)


def verify(raw: bytes, header: str) -> bool:
    pairs = [part.split("=", 1) for part in header.split(",") if "=" in part]
    ts = next((v for k, v in pairs if k == "t"), None)
    sigs = [v for k, v in pairs if k == "v1"]
    if not ts or not sigs or not ts.isdigit():
        return False
    if abs(time.time() - int(ts)) > TOLERANCE:
        return False  # stale or replayed
    expected = hmac.new(SECRET, ts.encode() + b"." + raw, hashlib.sha256).hexdigest()
    return any(hmac.compare_digest(expected, s) for s in sigs)


@app.post("/hooks/infro")
async def infro_hook(request: Request) -> Response:
    raw = await request.body()  # raw bytes: never re-serialise before verifying
    if not verify(raw, request.headers.get("INFRO-Signature", "")):
        return Response(status_code=400)

    event = await request.json()
    with db.transaction():
        inserted = db.execute(
            "INSERT INTO webhook_events (event_id, type, job_id, created_at, payload) "
            "VALUES (%s, %s, %s, to_timestamp(%s), %s) ON CONFLICT (event_id) DO NOTHING",
            (event["id"], event["type"], event["data"]["id"], event["created_at"], raw.decode()),
        ).rowcount
        if inserted:
            db.execute("INSERT INTO job_outbox (event_id) VALUES (%s)", (event["id"],))
    return Response(status_code=200)  # duplicates are acknowledged too
```

The handler does three things and nothing else: verify, record durably, acknowledge. A worker reads `job_outbox` and downloads the output. That split is not style. INFRO counts a response slower than 10 seconds as a failed delivery and does not follow redirects ([webhooks](https://infro.io/docs/api/webhooks)); Standard Webhooks recommends a timeout between 15 and 30 seconds[4]. A handler that fetches a large video inline will miss the window, earn a retry, and then race its own retry mid-download.

### Deduplicate, and do not trust order

INFRO documents delivery as at-least-once and unordered: a delivery your handler processed but failed to acknowledge is sent again, and two jobs finishing close together can arrive in either order. The event `id` stays the same across retries of one delivery, which is why the unique index on `event_id` is the whole deduplication strategy. Standard Webhooks makes the same recommendation with its `webhook-id` header[4].

For order, compare the event's `created_at` and, better, the job's own `status`: a terminal state never changes, so an event for a job you already recorded as `succeeded` changes nothing.

### Know the retry schedule

[Diagram: INFRO's documented schedule: 12 attempts over roughly 16 hours, after which the delivery is marked failed. The job itself is unaffected and can always be read.]

A deploy that breaks signature verification for an hour does not lose events — they retry — but it does delay them. The most common cause of a verifier that "suddenly" fails is body-parsing middleware that consumed or re-serialised the raw bytes before the handler saw them. Alert on your webhook route's 4xx and 5xx rate so a broken verifier is found in minutes, not when a customer asks where their video is.

## Close the gap: a reconciler

Webhooks reduce the need to poll; they do not remove it. A delivery can exhaust its retries, an endpoint can be misconfigured, and a job submitted by a worker that crashed has no record of its id on your side at all. A reconciler handles all three.

```python
import random
import time

import requests

from submit import API, HEADERS


def in_flight_jobs() -> list[dict]:
    jobs, after = [], None
    while True:
        params = [("status", "queued"), ("status", "running"), ("limit", "100")]
        if after:
            params.append(("after", after))
        page = requests.get(f"{API}/jobs", params=params, headers=HEADERS, timeout=30).json()
        jobs.extend(page["data"])
        if not page["has_more"]:
            return jobs
        after = page["next_cursor"]


def wait_for(job_id: str, deadline_s: float = 1800) -> dict:
    delay, started = 2.0, time.monotonic()
    while time.monotonic() - started < deadline_s:
        job = requests.get(f"{API}/jobs/{job_id}", headers=HEADERS, timeout=30).json()
        if job["status"] in ("succeeded", "failed", "canceled"):
            return job
        time.sleep(delay + random.uniform(0, delay / 4))  # jitter
        delay = min(delay * 1.5, 30.0)
    raise TimeoutError(job_id)
```

Run `in_flight_jobs()` at worker startup to re-attach work orphaned by a crash, and on a schedule to catch terminal jobs whose events never arrived. The backoff — start near two seconds, multiply by 1.5, cap at 30 seconds, add jitter — is the one the jobs documentation recommends. Set your own deadline; a job with no terminal state after it is something to surface, not to poll forever.

## Output, cancellation and cost

**Copy the output immediately.** INFRO keeps output files for 24 hours after completion; after `expires_at` the link returns 404, while the job record stays readable with `output` set to null. Regenerating an expired render is billed again by your provider and will not be identical unless you pinned a seed.

**Cancel does not stop the provider.** Cancelling stops INFRO tracking the job and records no cost, but it does not currently stop the render at your provider, which may still bill for it. Branch on the `status` the cancel call returns, because a render that finished a moment earlier comes back `succeeded`.

**Cost appears on success only.** `usage.cost` on a succeeded job is an estimate of what your provider will bill, in seconds of output at your rate or the vendor's list rate, and `null` when the model is unpriced. Failed and canceled jobs record no cost in INFRO, which is not the same as your provider billing nothing for them.

**Failover happens before running.** A job's id is stable even when the submit fails over to another target on your route; once a provider has accepted the render it is never moved ([async jobs](https://infro.io/docs/api/jobs)).

## Checklist

- [ ] Idempotency keys are derived from your own record id, identical on every retry.
- [ ] The job id is stored on your record in the same transaction that marks it submitted.
- [ ] The webhook endpoint subscribes to failed and canceled events, not only succeeded.
- [ ] Signatures are verified on raw bytes, in constant time, with a five-minute window, accepting any `v1` during rotation.
- [ ] Events are deduplicated with a unique index on the event id; duplicates are acknowledged with 2xx.
- [ ] The handler acknowledges within the sender's timeout and does downloads on a queue.
- [ ] A reconciler lists queued and running jobs at startup and on a schedule, with backoff and a deadline.
- [ ] Output is copied to your own storage as soon as a job succeeds.
- [ ] A test flips one signature character and confirms a 400, and replays one real delivery to confirm deduplication.

## Questions

### Should I use webhooks or polling for AI video generation?

Use webhooks wherever your code can receive an inbound HTTPS request, and keep polling as the fallback and as the reconciler's tool. A worker with no public endpoint polls with backoff; everything else waits for the signed event.

## Sources

1. LLM10:2025 Unbounded Consumption — OWASP GenAI Security Project. https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/ (accessed 2026-09-29)
2. The Idempotency-Key HTTP Header Field (draft-ietf-httpapi-idempotency-key-header-07) — IETF HTTPAPI Working Group. https://www.ietf.org/archive/id/draft-ietf-httpapi-idempotency-key-header-07.html (accessed 2026-09-29)
3. draft-ietf-httpapi-idempotency-key-header — document status — IETF Datatracker. https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/ (accessed 2026-09-29)
4. Standard Webhooks specification — Standard Webhooks. https://github.com/standard-webhooks/standard-webhooks/blob/main/spec/standard-webhooks.md (accessed 2026-09-29)