A message queue moves work out of the request that discovered it. Its most important effect is temporal decoupling: the producer and worker no longer need to be healthy at the same instant. An order endpoint can validate and commit an order, enqueue fulfillment, and answer without waiting for a warehouse API. The queue absorbs a short outage and exposes a backlog. It does not, by itself, make the workflow correct.
Begin with the business promise. A password-reset email may tolerate a minute of delay but not disclosure to the wrong address. A card capture must not happen twice. A thumbnail may be dropped and regenerated. Write down maximum useful age, acceptable duplicate behavior, ordering scope, recovery owner, and what the user sees while work is pending.
For an order workflow, commit the order and an outbox row in one database transaction. A relay publishes the outbox event and marks it sent only after broker confirmation. A crash after publish can create a duplicate, so the consumer records a stable message identifier before applying effects. “Exactly once” inside one broker cannot guarantee exactly once across the database, payment provider, and email service.
BEGIN;
INSERT INTO orders(id, state) VALUES (:order_id, 'accepted');
INSERT INTO outbox(id, topic, payload) VALUES (:event_id, 'order.accepted', :payload);
COMMIT;The transaction makes order acceptance and the intent to publish indivisible; the relay can safely retry publication, while consumers still deduplicate because acknowledgement can be lost.
Asynchronous work also changes economics. Capacity must cover arrival bursts without paying for idle workers all day. Retention, cross-region transfer, broker operations, and poison-message investigation are real costs. Compare a managed service with a self-operated broker using staff time and recovery responsibility, not only per-request price. Read event-driven architecture before turning every interaction into a message.
Production review: from acceptance to recovery
1. Choose an acknowledgement boundary
A worker should acknowledge only after its durable business effect commits; acknowledging on receipt can silently lose work when the process exits. For Choose an acknowledgement boundary, establish oldest-message age and retry rate as the primary evidence and publish its expected range before rollout. During a controlled exercise, terminate a worker immediately after the business commit; compare the resulting customer-visible behavior with the written promise. Review retained payload bytes, worker idle time, and broker operations during capacity and budgeting work, because a technically successful control can still be economically wrong. The acceptance record should name an owner, preserve the observation, and state when the decision must be revisited.
2. Make effects idempotent
Store an operation key beside the result and return the prior result on redelivery instead of charging, emailing, or reserving twice. Make effects idempotent should be reviewed through duplicate-effect rejections and outbox lag, not through a screenshot of one successful run. Rehearse the boundary by choosing a safe environment and then delay broker acknowledgement while publishing duplicate deliveries. Document containment and recovery in the same language operators see during an alert. Include support time for ambiguous outcomes and manual replay in the trade-off, and retain enough evidence to show whether the observed result came from the design or an unrelated dependency.
3. Bound retries
Use exponential backoff with jitter, a maximum attempt count, and an expiry based on business usefulness rather than retrying forever. Turn Bound retries into a falsifiable operating claim. Instrument worker saturation and dependency latency, define failure before running the test, and throttle the downstream database below arrival rate. A reviewer should be able to connect the signal to one business consequence and one recovery action without private team knowledge. Account for peak compute required to drain backlog within the promise; otherwise the architecture may shift expense into support, response labor, or another service while appearing inexpensive on its own dashboard.
4. Separate transient and permanent errors
Retry timeouts and throttling, but route invalid addresses, failed schema validation, and forbidden operations to deliberate resolution. The verification plan for Separate transient and permanent errors begins with dead-letter growth and reconciliation gaps measured across ordinary and peak traffic. Next, inject one poison payload into an ordered partition and inspect both the immediate symptom and the state left after recovery. Keep the runbook specific about authority, evidence, and rollback. Re-estimate cross-region transfer and dead-letter investigation when volume, regions, or ownership change, because limits selected for the first deployment rarely remain neutral as the system grows.
5. Control concurrency
Set worker concurrency from dependency capacity and transaction duration; autoscaling against queue depth alone can overwhelm the database. Treat oldest-message age and retry rate as the release gate for Control concurrency, with thresholds derived from the user journey rather than a vendor default. Prove degraded behavior when you terminate a worker immediately after the business commit, including what an operator can see before customers report it. The design review must compare retained payload bytes, worker idle time, and broker operations with the consequence of a missed or delayed operation. Record exceptions explicitly so emergency action does not silently become permanent configuration.
6. Measure age, not only depth
Oldest-message age reveals customer delay when traffic volume and processing rates make a raw queue count difficult to interpret. A production acceptance test for Measure age, not only depth should capture duplicate-effect rejections and outbox lag before, during, and after recovery. In the fault phase, delay broker acknowledgement while publishing duplicate deliveries; in the review phase, reconcile durable state rather than stopping when the process turns green. Model support time for ambiguous outcomes and manual replay at current load and forecast peak. This makes the decision reversible: later teams can see why the limit existed and what evidence would support changing it.
7. Design dead-letter ownership
A dead-letter queue needs alerts, payload-safe inspection, replay tooling, retention, and an owner who can resolve the underlying defect. Use Design dead-letter ownership to connect implementation detail with operations. Dashboard worker saturation and dependency latency, but keep the underlying event or state available for investigation. Then throttle the downstream database below arrival rate in a bounded drill and have someone outside the implementing team follow the runbook. Compare the resulting risk reduction with peak compute required to drain backlog within the promise. Passing means the system behaves predictably and the responder can explain why, not simply that an automated check returned success.
8. Limit ordering promises
Preserve order only within a required entity key, because global ordering serializes throughput and still needs duplicate handling. Before approving Limit ordering promises, ask which assumption is most likely to expire. Baseline dead-letter growth and reconciliation gaps, then inject one poison payload into an ordered partition without changing several variables at once. Capture timelines and identifiers needed to reconstruct the event. Budget explicitly for cross-region transfer and dead-letter investigation, including human ownership that provider calculators omit. Schedule review after material traffic, dependency, or policy changes so the control remains aligned with its original business purpose.
9. Carry trace context safely
Propagate correlation and trace identifiers in message attributes while keeping credentials and sensitive customer data out of diagnostic fields. Operational evidence for Carry trace context safely comes from oldest-message age and retry rate correlated with a representative transaction or workflow. Validate it when you terminate a worker immediately after the business commit, checking partial success and delayed recovery rather than only total outage. Discuss retained payload bytes, worker idle time, and broker operations with product and finance owners where the consequence is material. The final runbook should state the safe manual action and the conditions under which automation must stop and request judgment.
10. Plan poison-message isolation
One malformed message must not block a partition indefinitely; quarantine it while preserving evidence and allowing later messages to progress. For Plan poison-message isolation, separate prevention from detection. Use duplicate-effect rejections and outbox lag to discover drift from the intended behavior, and delay broker acknowledgement while publishing duplicate deliveries to prove containment. Recovery should preserve auditability and avoid converting one failure into duplicates, data loss, or uncontrolled load. Compare that protection against support time for ambiguous outcomes and manual replay. A concise decision record should link configuration, test evidence, dashboard, and accountable owner so the control survives staff rotation.
11. Test broker and worker crashes
Kill a worker before and after its commit, interrupt publication, throttle a dependency, and prove that the resulting state remains reconcilable. Test broker and worker crashes is ready only when worker saturation and dependency latency can be interpreted by the on-call engineer without reverse-engineering implementation internals. Run a game day in which you throttle the downstream database below arrival rate, and require an explanation of impact, state, and next action. Track peak compute required to drain backlog within the promise separately from normal application spend so reliability overhead remains visible. If the drill needs privileged improvisation, improve the interface or runbook before increasing automation.
12. Provide reconciliation
Periodically compare accepted business records with completed effects so rare gaps become repairable cases rather than permanent invisible loss. Close the Provide reconciliation review with a recovery proof. Observe dead-letter growth and reconciliation gaps, inject one poison payload into an ordered partition, and verify the final business records against their authority. Report cross-region transfer and dead-letter investigation beside latency and correctness rather than after the architecture is committed. Keep an explicit exit condition: if evidence, platform capability, or economics change, the team should be able to simplify or replace this mechanism without losing the contract it protects.
Recovery chain
producer -> durable queue -> idempotent worker -> effect -> reconciliationThe diagram makes the recovery boundaries explicit: every arrow can delay, duplicate, or fail.
Queue vendor guarantees and limits
The Google Cloud Tasks documentation defines the platform or specification behavior used in this design. The AWS SQS developer guide provides a second primary reference for implementation limits and operating guidance. These sources support the guarantees described here; product policy and risk decisions remain specific to the organization.
A compact operating model
request -> transaction + outbox -> broker -> idempotent worker -> durable effectThis chain distinguishes acceptance from completion. The outbox closes the database-to-broker gap, while idempotency and reconciliation handle duplicates and the rare outcome whose acknowledgement is lost.
