Consistency decision matrix
| Decision | Minimum control |
|---|---|
| Write the invariant in business language | State what must never happen, such as issuing two active refunds for one capture, and identify the authority that enforces it. |
| Define staleness tolerance | Give each projection a maximum acceptable age and specify whether stale data is merely inconvenient, financially harmful, or legally misleading. |
| Protect read-your-writes journeys | Return a committed version to the client and ensure subsequent reads meet or exceed it for profile, settings, and confirmation screens. |
| Use conditional mutation | Reserve scarce stock with a version check or atomic predicate rather than reading availability and writing later in separate unprotected steps. |
“Strong” and “eventual” consistency are incomplete requirements. The useful question is which facts must agree, for whom, within what time, and what harm occurs while they disagree. A social feed can show a new post seconds late. A customer who just changed a shipping address expects read-your-writes behavior. Two warehouse workers must not both receive the final unit. An account ledger must preserve conservation of money even when a reporting projection lags.
Start with invariants, not datastore features. “Available quantity never goes below zero” may require conditional writes at one inventory owner. “Search reflects catalog changes within five minutes” can be an asynchronous service objective. “A user sees their own update immediately” might be handled by session affinity, a version token, or reading from the leader without making every global reader strongly consistent.
CAP is often used too casually. During a network partition, a distributed operation cannot simultaneously guarantee every response and linearizable behavior. It does not say that one database is always consistent and another always available, nor does it choose the behavior of each business operation. A product may reject checkout during ambiguity while continuing to serve cached product descriptions.
Model a transfer as a ledger transaction inside one authoritative boundary where possible. Publish committed entries to statements, analytics, and notifications afterward. If ownership spans services, use a saga with explicit states and compensations, but remember that compensation is a new business action, not time travel. Refunding a captured payment may incur fees and leave an audit trail.
Consistency has a price: coordination adds latency, cross-region writes add network distance, locks reduce concurrency, and globally replicated products charge for storage and operations. Weakening guarantees transfers cost into reconciliation, support, fraud exposure, or customer confusion. Compare total consequence rather than benchmark throughput alone.
Decision probes for each invariant
5. Choose a single writer deliberately
Assign one service ownership of each invariant and reject direct table writes from neighboring services that bypass its rules. For Choose a single writer deliberately, establish compressed image bytes and layer churn as the primary evidence and publish its expected range before rollout. During a controlled exercise, build with an empty cache and unavailable package mirror; compare the resulting customer-visible behavior with the written promise. Review registry storage, transfer, and build minutes during capacity and budgeting work, because a technically successful control can still be economically wrong. The acceptance record should name an owner, preserve the observation, and state when the decision must be revisited.
6. Make saga states inspectable
Persist pending, completed, compensating, and intervention states so operators can understand a workflow without reconstructing it from scattered logs. Make saga states inspectable should be reviewed through build duration and cache hit rate, not through a screenshot of one successful run. Rehearse the boundary by choosing a safe environment and then start under a read-only filesystem as non-root. Document containment and recovery in the same language operators see during an alert. Include scanner triage and base-image maintenance in the trade-off, and retain enough evidence to show whether the observed result came from the design or an unrelated dependency.
7. Design compensation with product owners
Specify refunds, releases, credits, and customer communication as real policies, including cases where automatic reversal is unsafe or impossible. Turn Design compensation with product owners into a falsifiable operating claim. Instrument runtime vulnerabilities and package count, define failure before running the test, and send termination while the process has active work. A reviewer should be able to connect the signal to one business consequence and one recovery action without private team knowledge. Account for developer delay from poor cache boundaries; otherwise the architecture may shift expense into support, response labor, or another service while appearing inexpensive on its own dashboard.
8. Version asynchronous projections
Include source sequence or entity version so a delayed event cannot overwrite a newer search, cache, or reporting representation. The verification plan for Version asynchronous projections begins with startup time, memory, and failed architecture tests measured across ordinary and peak traffic. Next, rebuild after a base-digest security update and inspect both the immediate symptom and the state left after recovery. Keep the runbook specific about authority, evidence, and rollback. Re-estimate runtime waste from oversized artifacts when volume, regions, or ownership change, because limits selected for the first deployment rarely remain neutral as the system grows.
9. Expose uncertainty honestly
Show pending or last-updated state when data is not authoritative instead of presenting a stale value with false precision. Treat compressed image bytes and layer churn as the release gate for Expose uncertainty honestly, with thresholds derived from the user journey rather than a vendor default. Prove degraded behavior when you build with an empty cache and unavailable package mirror, including what an operator can see before customers report it. The design review must compare registry storage, transfer, and build minutes with the consequence of a missed or delayed operation. Record exceptions explicitly so emergency action does not silently become permanent configuration.
10. Exercise partitions
Block replication or a service link and observe which operations reject, queue, degrade, or proceed; compare behavior with the written policy. A production acceptance test for Exercise partitions should capture build duration and cache hit rate before, during, and after recovery. In the fault phase, start under a read-only filesystem as non-root; in the review phase, reconcile durable state rather than stopping when the process turns green. Model scanner triage and base-image maintenance at current load and forecast peak. This makes the decision reversible: later teams can see why the limit existed and what evidence would support changing it.
11. Reconcile independent records
Compare payment, order, inventory, and ledger authorities using stable identifiers and route mismatches through an auditable repair process. Use Reconcile independent records to connect implementation detail with operations. Dashboard runtime vulnerabilities and package count, but keep the underlying event or state available for investigation. Then send termination while the process has active work in a bounded drill and have someone outside the implementing team follow the runbook. Compare the resulting risk reduction with developer delay from poor cache boundaries. Passing means the system behaves predictably and the responder can explain why, not simply that an automated check returned success.
12. Measure consistency cost
Track coordination latency, abort rate, replication lag, compensations, support cases, and write pricing before changing the guarantee. Before approving Measure consistency cost, ask which assumption is most likely to expire. Baseline startup time, memory, and failed architecture tests, then rebuild after a base-digest security update without changing several variables at once. Capture timelines and identifiers needed to reconstruct the event. Budget explicitly for runtime waste from oversized artifacts, including human ownership that provider calculators omit. Schedule review after material traffic, dependency, or policy changes so the control remains aligned with its original business purpose.
13. Trace an oversell incident
Reconstruct concurrent reads, reservation writes, event delivery, cache age, and customer remedies to identify the exact unenforced invariant. Operational evidence for Trace an oversell incident comes from compressed image bytes and layer churn correlated with a representative transaction or workflow. Validate it when you build with an empty cache and unavailable package mirror, checking partial success and delayed recovery rather than only total outage. Discuss registry storage, transfer, and build minutes with product and finance owners where the consequence is material. The final runbook should state the safe manual action and the conditions under which automation must stop and request judgment.
14. Document regional behavior
Explain leader loss, replica lag, failover authority, and degraded writes so responders know when availability intentionally yields to correctness. For Document regional behavior, separate prevention from detection. Use build duration and cache hit rate to discover drift from the intended behavior, and start under a read-only filesystem as non-root to prove containment. Recovery should preserve auditability and avoid converting one failure into duplicates, data loss, or uncontrolled load. Compare that protection against scanner triage and base-image maintenance. A concise decision record should link configuration, test evidence, dashboard, and accountable owner so the control survives staff rotation.
15. Test client retry ambiguity
Interrupt responses after commit and verify operation keys let clients discover the original result without creating a second mutation. Test client retry ambiguity is ready only when runtime vulnerabilities and package count can be interpreted by the on-call engineer without reverse-engineering implementation internals. Run a game day in which you send termination while the process has active work, and require an explanation of impact, state, and next action. Track developer delay from poor cache boundaries separately from normal application spend so reliability overhead remains visible. If the drill needs privileged improvisation, improve the interface or runbook before increasing automation.
Container build and supply-chain references
The Google Cloud Spanner transactions defines the platform or specification behavior used in this design. The AWS prescriptive guidance on saga patterns provides a second primary reference for implementation limits and operating guidance. These sources support the guarantees described here; product policy and risk decisions remain specific to the organization.
A compact operating model
locked sources -> builder stage -> tests -> minimal runtime stage -> signed digestTools and dependencies live in the builder, while only verified runtime output crosses the stage boundary. The signed digest is the artifact promoted between environments.
