What should happen once?
Charge this invoice, create this follow-up, ingest this transcript, or apply this usage debit.
The Retry-Safe Automation Playbook
Let it fail. Never let it do the same costly thing twice.
A practical field guide for making payments, webhooks, jobs, and AI workflows safe when networks retry, workers restart, or events arrive again.
Get the playbookSee the contract
The ambiguity
A timeout does not mean the action failed.
The caller may lose the response after the server commits the work. Retrying is necessary, but a new attempt must remain the same business intent. Otherwise resilience becomes duplication.
How many deliveries occurred?
Networks, queues, webhooks, users, and schedulers can deliver the same intent more than once.
What does replay return?
The same key and parameters should return the stored result without repeating the side effect.
The retry-safe contract
Make one intent recognizable at every mutating boundary.
Assign a stable intent ID
Use the upstream event, usage record, call, order, or operation ID - not a new random value for every attempt.
Claim atomically
Create a unique record or lock before the side effect so two workers cannot both become the winner.
Fingerprint parameters
Reject reuse of the same key with different business inputs. A key identifies one specific intent.
Store the outcome
Persist status, result reference, error class, and timestamps so a replay can return meaning rather than execute again.
Bound retries
Retry only retryable failures with timeouts, exponential backoff, jitter, and a clear terminal state.
Test the ambiguity window
Force timeout-after-success, duplicate delivery, concurrent workers, restart mid-step, and late replay.
Built from first-party corrections
One pattern protected five different kinds of work.
Public PONO build history documents retry-safe boundaries for payment events, usage debits, call follow-up tasks, transcript ingestion, and synchronization. Each chose a stable upstream identity appropriate to the business intent. The lesson is shared without exposing customers, amounts, transcripts, tasks, endpoints, or credentials.
Supporting guidance: Stripe API - Idempotent requests and AWS Builders' Library - Making retries safe with idempotent APIs.