Skip to content

Delivery and retries

Each matching subscription gets its own delivery of an event (whd_…). Deliveries are claimed under a lease, POSTed with no database transaction open, and then recorded, so a crash between the receiver’s answer and the record means the same body, event ID and delivery ID are sent again. That’s why receivers deduplicate on the event ID.

The receiver Result
Answers 2xx within 10 seconds Delivered
Times out, refuses the connection, or answers 5xx, 408 or 429 Retried with backoff
Answers anything else (3xx, other 4xx) Failed at once, not retried
Resolves to an address it may not reach (private, loopback, link-local) Failed at once

Redirects are never followed: a 3xx is a failure.

The wait before the next attempt starts at 30 seconds and doubles, up to two hours:

  1. 30 safter the 1st failure
  2. 1 minafter the 2nd
  3. 2 minafter the 3rd
  4. 4 min … 64 mindoubling
  5. 2 hfrom then on

Retries stop 24 hours after the delivery was created; a delivery still failing then ends as retries_exhausted. The exact schedule is in the event reference, generated from the server’s code.

runbook webhook redeliver <whd> --webhook <whk> --project <p> sends a delivery’s event again under a new delivery ID, with a fresh 24-hour retry window. The event ID stays the same, so a receiver that already handled it skips it.

A subscription whose deliveries fail continuously for three days is disabled. The subscription shows a notice for project admins, and deliveries stop until someone fixes the receiver and re-enables it:

Terminal window
runbook webhook update whk_… --project prod --enable

Enabling resets its delivery health. Quiet gaps don’t end a failing streak unless they’re longer than two retry windows (48 hours).

approval.requested, approval.escalated, approval.expired, step.blocked, step.reminder and step.escalated are notices for people. When one reaches no subscriber at all (none subscribed, or every delivery failed), the run page says so, so nobody waits on a message that was never sent. Email and the in-app inbox carry the same notices.