A webhook is an untrusted request claiming that something happened elsewhere. It can be forged, duplicated, delayed, reordered, or omitted. This debugging walkthrough starts at a signature failure, follows raw bytes into an inbox and queue, and ends with replay and reconciliation. The design goal is not exactly-once delivery; it is one explainable business outcome despite messy delivery.
The core idea
Webhook ingestion is a custody transfer. The receiver first proves who sent the exact bytes, then makes the event durable before acknowledging it. Everything after acceptance is asynchronous, duplicate-tolerant work with explicit retry, ordering, dead-letter, and reconciliation behavior.
Security boundary: verify the exact body
HMAC authenticates bytes, not a parsed-and-reserialized object. Assume input and redirects can be controlled by an attacker. capture the raw body, validate timestamp and header grammar, then compare MACs in constant time
const signed = timestamp + '.' + rawBody
const expected = hmacSha256(secret, signed)
if (!timingSafeEqual(expected, supplied)) reject(401)Parsing happens only after this check; timestamp tolerance limits replay.
Use maintained cryptographic libraries and keep secrets out of examples, traces, and error responses. Add negative cases for official fixtures plus altered bytes, stale timestamps, and malformed signatures. The control is bypassable if middleware normalizes whitespace before verification or logs the secret. Monitor safe signature-failure categories and key identifier by safe reason categories, rate-limit abusive paths, and retain enough context for investigation without retaining sensitive payloads.
Data flow: durable acceptance before 2xx
Follow one logical operation. A success response should mean the event survives process death. At each arrow, persist a provider/event envelope before acknowledging and process asynchronously
flowchart LR
P[Provider]-->V[Verify raw bytes]
V-->I[(Inbox unique ID)]
I-->Q[Queue]
Q-->H[Handlers]
H-->D[(Domain DB)]An inbox-backed queue or outbox closes the crash gap between storage and publication.
The diagram is a map of failure and ownership boundaries, not decoration. Annotate where data becomes durable, where it can be duplicated, and where ordering can change. Validate it with kill points before insert, after insert, and before response. The model is wrong if the endpoint returns 200 while the event exists only in memory. Correlate accepted/enqueued discrepancy, ingress latency, and inbox errors across the flow using a safe operation identifier.
Comparison: retry, dead-letter, or discard
Timeouts may recover; invalid schemas will not. A comparison is only useful when the criteria match the workload, so classify errors and bound attempts by count or age
| Failure | Action |
|---|---|
| 503/timeout | Backoff with jitter |
| Rate limit | Honor guidance |
| Invalid payload | Dead-letter |
| Stale version | Acknowledge and skip |
| Revoked account | Review policy |
Treat each row as a hypothesis, not a universal verdict. Weight correctness guarantees before convenience when a mistake is expensive. Prototype brownouts, permanent 4xx responses, and worker restart. Beware the conclusion when every exception retries tightly and starves healthy events; benchmark data without representative distribution and contention is especially misleading. Revisit the choice using attempt distribution, oldest retry, dependency status, and queue saturation.
Failure reconstruction: duplicate after lost acknowledgement
Imagine the alert has fired. The handler committed, but the provider did not receive 2xx and redelivered. The first response should be to look up provider plus event ID and handler identity before applying effects, preserving evidence before retries or manual repair change the state.
INSERT INTO webhook_receipts(provider,event_id,handler)
VALUES ($1,$2,$3) ON CONFLICT DO NOTHING;The unique claim elects one handler execution; domain writes and completion should share a transaction where possible.
Walk the timeline twice: once from the caller's view and once from the durable system of record. The two stories often diverge at a timeout, commit, queue acknowledgement, or cache boundary. Recreate concurrent duplicates and crash after domain commit. The likely trap is that one global processed flag hides partial success across several handlers. Confirm recovery by watching duplicates, handler claims, replays, and invariant violations, not merely by seeing one successful request.
Debugging checkpoint: version 12 arrived before 11
Network arrival is not business order. Resist changing configuration immediately. First compare monotonic object versions or fetch authoritative current state, then form a hypothesis that one additional observation can disprove.
Apply subscription version 12, then acknowledge version 11 as stale. If the provider exposes no version, schedule reconciliation rather than inventing total order from low-resolution timestamps.
Keep timestamps in one time base and preserve request, transaction, or query identifiers. Reproduce with all permutations with gaps and duplicates. A false lead appears when a delayed event rolls state backward or timestamps tie. The decisive signals are stale skips, detected gaps, and reconciliation drift; inspect their distributions and dimensions rather than a single average.
Runbook phase: drain the dead-letter queue
Objective. Restore valid work without replaying poison events as a flood.
Procedure. redact payload display, group by cause, fix the handler, dry-run, and rate-limit replay
inspect -> classify -> patch -> dry-run
-> replay 1 -> verify domain state
-> replay cohort -> watch dependenciesKeep the original event identity so normal deduplication remains active.
Name the person authorized to stop the phase and make every command repeatable. Before proceeding, complete backup restore, provider outage, and replay cancellation. Stop rather than improvise if operators delete records or bulk replay without dependency capacity. The dashboard for this phase should expose oldest dead letter, count by cause, replay result, and domain mismatch. Capture start and finish checkpoints so another operator can determine exactly how far the run advanced.
Game-day cards for the webhook team
1. Rehearse verify the exact body
Before sign-off, restate this article-specific claim in one sentence: HMAC authenticates bytes, not a parsed-and-reserialized object. Build a controlled exercise that includes official fixtures plus altered bytes, stale timestamps, and malformed signatures. Record the expected durable state, response, and operator action before running it. During the exercise, deliberately introduce the condition where middleware normalizes whitespace before verification or logs the secret. The team must explain whether to retry, reject, compensate, roll back, or wait. Preserve safe signature-failure categories and key identifier with enough dimensions to compare the expected and observed path. Finish by assigning ownership and documenting the threshold that would force the design to be reconsidered.
2. Rehearse durable acceptance before 2xx
Before sign-off, restate this article-specific claim in one sentence: A success response should mean the event survives process death. Build a controlled exercise that includes kill points before insert, after insert, and before response. Record the expected durable state, response, and operator action before running it. During the exercise, deliberately introduce the condition where the endpoint returns 200 while the event exists only in memory. The team must explain whether to retry, reject, compensate, roll back, or wait. Preserve accepted/enqueued discrepancy, ingress latency, and inbox errors with enough dimensions to compare the expected and observed path. Finish by assigning ownership and documenting the threshold that would force the design to be reconsidered.
3. Rehearse retry, dead-letter, or discard
Before sign-off, restate this article-specific claim in one sentence: Timeouts may recover; invalid schemas will not. Build a controlled exercise that includes brownouts, permanent 4xx responses, and worker restart. Record the expected durable state, response, and operator action before running it. During the exercise, deliberately introduce the condition where every exception retries tightly and starves healthy events. The team must explain whether to retry, reject, compensate, roll back, or wait. Preserve attempt distribution, oldest retry, dependency status, and queue saturation with enough dimensions to compare the expected and observed path. Finish by assigning ownership and documenting the threshold that would force the design to be reconsidered.
4. Rehearse duplicate after lost acknowledgement
Before sign-off, restate this article-specific claim in one sentence: The handler committed, but the provider did not receive 2xx and redelivered. Build a controlled exercise that includes concurrent duplicates and crash after domain commit. Record the expected durable state, response, and operator action before running it. During the exercise, deliberately introduce the condition where one global processed flag hides partial success across several handlers. The team must explain whether to retry, reject, compensate, roll back, or wait. Preserve duplicates, handler claims, replays, and invariant violations with enough dimensions to compare the expected and observed path. Finish by assigning ownership and documenting the threshold that would force the design to be reconsidered.
5. Rehearse version 12 arrived before 11
Before sign-off, restate this article-specific claim in one sentence: Network arrival is not business order. Build a controlled exercise that includes all permutations with gaps and duplicates. Record the expected durable state, response, and operator action before running it. During the exercise, deliberately introduce the condition where a delayed event rolls state backward or timestamps tie. The team must explain whether to retry, reject, compensate, roll back, or wait. Preserve stale skips, detected gaps, and reconciliation drift with enough dimensions to compare the expected and observed path. Finish by assigning ownership and documenting the threshold that would force the design to be reconsidered.
6. Rehearse drain the dead-letter queue
Before sign-off, restate this article-specific claim in one sentence: Restore valid work without replaying poison events as a flood. Build a controlled exercise that includes backup restore, provider outage, and replay cancellation. Record the expected durable state, response, and operator action before running it. During the exercise, deliberately introduce the condition where operators delete records or bulk replay without dependency capacity. The team must explain whether to retry, reject, compensate, roll back, or wait. Preserve oldest dead letter, count by cause, replay result, and domain mismatch with enough dimensions to compare the expected and observed path. Finish by assigning ownership and documenting the threshold that would force the design to be reconsidered.
Conclusion: accept quickly, process deliberately
Reliability comes from layered guarantees: raw-byte verification establishes authenticity, durable enqueue separates provider deadlines from business work, inbox uniqueness contains duplicates, bounded retries handle transient faults, and dead-letter ownership makes poison events visible. Ordering and completeness must come from explicit provider guarantees or reconciliation, never from a tidy development trace. The durable lesson is to state guarantees in language clients and operators can verify, then build the smallest mechanism that satisfies them under retries, concurrency, deployment, and partial failure. Revisit the decision when traffic, risk, or ownership changes.
Continue with Idempotency: Making Retries Safe in APIs and Workflows to connect this topic to another production boundary. The Prime Axiom resource library collects practical implementation material.
Authoritative references
Provider documentation is authoritative for signature formats and retry behavior; test against captured fixtures whenever configuration changes.
