A service-level indicator measures behavior users care about. A service-level objective sets a target for that indicator over a window. An error budget is the allowed gap between perfect performance and the objective. Together they translate “make it reliable” into a decision framework. They are not a promise that every internal component stays green.
For checkout, define a good event as an eligible request that returns a correct terminal response within two seconds. The SLI is good events divided by valid events. An objective of 99.9 percent over 28 days permits 0.1 percent bad events; it does not permit knowingly losing orders. Exclude traffic only for principled reasons, such as clearly malformed requests, and keep exclusions reviewable. If the denominator can be manipulated during incidents, the objective is not trustworthy.
Availability and latency often need separate indicators. A fast error is not a successful request. Batch systems may use records processed before a deadline. Data pipelines may measure freshness and correctness. A dependency metric is useful diagnostically but usually not the top-level SLI because customers experience the whole journey.
Error budgets create a conversation between feature velocity and risk. A team consuming budget quickly may pause risky launches, prioritize capacity, or reduce change size. A team preserving nearly all budget may be overinvesting in reliability or simply measuring the wrong thing. Budget policy needs product and engineering agreement before an incident, including who decides exceptions.
Objectives have costs. Another nine can demand redundancy, testing, staffing, and lower-risk change processes. Setting 100 percent hides trade-offs and makes every change theoretically unacceptable. Set the least stringent objective that protects the business and customer expectation, then use observed demand, incident impact, and contract obligations to revise it.
Objective worksheet
- Select a user-centered event: Measure completed searches, accepted writes, fresh reports, or processed records at the boundary where the customer receives value. For Select a user-centered event, establish conflicting-write aborts and invariant violations as the primary evidence and publish its expected range before rollout. During a controlled exercise, partition the authoritative writer from one application region; compare the resulting customer-visible behavior with the written promise. Review cross-region coordination latency and write charges during capacity and budgeting work, because a technically successful control can still be economically wrong. The acceptance record should name an owner, preserve the observation, and state when the decision must be revisited.
- Define good precisely: Specify status, correctness, latency threshold, and eligibility so two engineers calculate the same numerator and denominator. Define good precisely should be reviewed through replica lag and read-version age, not through a screenshot of one successful run. Rehearse the boundary by choosing a safe environment and then return a successful commit after the client connection closes. Document containment and recovery in the same language operators see during an alert. Include manual repair, refunds, and customer-support handling in the trade-off, and retain enough evidence to show whether the observed result came from the design or an unrelated dependency.
- Choose a meaningful window: Use a rolling window for continuous control or a calendar window for contractual reporting, and explain the behavior at boundaries. Turn Choose a meaningful window into a falsifiable operating claim. Instrument saga compensation rate and pending duration, define failure before running the test, and deliver an older projection event after a newer version. A reviewer should be able to connect the signal to one business consequence and one recovery action without private team knowledge. Account for reserved failover capacity and replication storage; otherwise the architecture may shift expense into support, response labor, or another service while appearing inexpensive on its own dashboard.
- Separate objectives from contracts: Keep internal SLOs capable of providing warning before a legal or commercial SLA is breached and remedies are triggered. The verification plan for Separate objectives from contracts begins with reconciliation mismatches by business operation measured across ordinary and peak traffic. Next, pause one saga participant after money has moved and inspect both the immediate symptom and the state left after recovery. Keep the runbook specific about authority, evidence, and rollback. Re-estimate lost availability when correctness requires rejection when volume, regions, or ownership change, because limits selected for the first deployment rarely remain neutral as the system grows.
What happens after budget burn
5. Validate data completeness
Detect missing telemetry, collector outages, and biased sampling because absent bad events can make reliability appear artificially excellent. Treat conflicting-write aborts and invariant violations as the release gate for Validate data completeness, with thresholds derived from the user journey rather than a vendor default. Prove degraded behavior when you partition the authoritative writer from one application region, including what an operator can see before customers report it. The design review must compare cross-region coordination latency and write charges with the consequence of a missed or delayed operation. Record exceptions explicitly so emergency action does not silently become permanent configuration.
6. Alert on budget burn
Combine fast and slow burn-rate windows so severe incidents page quickly while persistent smaller degradation still creates action. A production acceptance test for Alert on budget burn should capture replica lag and read-version age before, during, and after recovery. In the fault phase, return a successful commit after the client connection closes; in the review phase, reconcile durable state rather than stopping when the process turns green. Model manual repair, refunds, and customer-support handling at current load and forecast peak. This makes the decision reversible: later teams can see why the limit existed and what evidence would support changing it.
7. Avoid percentile averaging
Calculate distributions from events or mergeable histograms because averaging instance percentiles produces mathematically misleading service latency. Use Avoid percentile averaging to connect implementation detail with operations. Dashboard saga compensation rate and pending duration, but keep the underlying event or state available for investigation. Then deliver an older projection event after a newer version in a bounded drill and have someone outside the implementing team follow the runbook. Compare the resulting risk reduction with reserved failover capacity and replication storage. Passing means the system behaves predictably and the responder can explain why, not simply that an automated check returned success.
8. Segment without hiding harm
Inspect region, tenant tier, device, and operation slices so a healthy global aggregate cannot conceal a badly affected customer group. Before approving Segment without hiding harm, ask which assumption is most likely to expire. Baseline reconciliation mismatches by business operation, then pause one saga participant after money has moved without changing several variables at once. Capture timelines and identifiers needed to reconstruct the event. Budget explicitly for lost availability when correctness requires rejection, including human ownership that provider calculators omit. Schedule review after material traffic, dependency, or policy changes so the control remains aligned with its original business purpose.
9. Publish an error-budget policy
Define thresholds for launch review, reliability work, leadership exception, and restoration of normal delivery before pressure arrives. Operational evidence for Publish an error-budget policy comes from conflicting-write aborts and invariant violations correlated with a representative transaction or workflow. Validate it when you partition the authoritative writer from one application region, checking partial success and delayed recovery rather than only total outage. Discuss cross-region coordination latency and write charges with product and finance owners where the consequence is material. The final runbook should state the safe manual action and the conditions under which automation must stop and request judgment.
10. Connect incidents to consumption
Estimate which events and minutes consumed budget, then prioritize recurrence prevention by user impact rather than dramatic internal symptoms. For Connect incidents to consumption, separate prevention from detection. Use replica lag and read-version age to discover drift from the intended behavior, and return a successful commit after the client connection closes to prove containment. Recovery should preserve auditability and avoid converting one failure into duplicates, data loss, or uncontrolled load. Compare that protection against manual repair, refunds, and customer-support handling. A concise decision record should link configuration, test evidence, dashboard, and accountable owner so the control survives staff rotation.
11. Review objective fitness
Compare support cases, revenue journeys, and incident narratives with the SLI to discover important harm the current measurement misses. Review objective fitness is ready only when saga compensation rate and pending duration can be interpreted by the on-call engineer without reverse-engineering implementation internals. Run a game day in which you deliver an older projection event after a newer version, and require an explanation of impact, state, and next action. Track reserved failover capacity and replication storage separately from normal application spend so reliability overhead remains visible. If the drill needs privileged improvisation, improve the interface or runbook before increasing automation.
12. Rehearse a decision
Use historical or synthetic burn to walk product and engineering leaders through the exact release and escalation policy. Close the Rehearse a decision review with a recovery proof. Observe reconciliation mismatches by business operation, pause one saga participant after money has moved, and verify the final business records against their authority. Report lost availability when correctness requires rejection beside latency and correctness rather than after the architecture is committed. Keep an explicit exit condition: if evidence, platform capability, or economics change, the team should be able to simplify or replace this mechanism without losing the contract it protects.
Primary consistency and transaction guidance
The Google SRE service level objectives defines the platform or specification behavior used in this design. The Google SRE workbook on SLOs provides a second primary reference for implementation limits and operating guidance. These sources support the guarantees described here; product policy and risk decisions remain specific to the organization.
A compact operating model
command -> invariant owner -> committed record -> versioned projections
\-> reconciliation reportThe invariant owner decides correctness once. Projections may lag, but version checks and reconciliation prevent a delayed copy from silently redefining authoritative state.
