Structured Logging: Events Humans and Machines Can Investigate

Incident prompt

A responder has an alert but no request body and no database console. The event record must carry enough safe context to decide the next query.

1. Create stable event names

Use controlled past-tense or outcome-oriented identifiers and evolve fields compatibly so saved queries survive wording changes. For Create stable event names, establish good-event ratio and eligible request count as the primary evidence and publish its expected range before rollout. During a controlled exercise, make a dependency return errors for five minutes; compare the resulting customer-visible behavior with the written promise. Review redundancy, capacity headroom, and on-call staffing during capacity and budgeting work, because a technically successful control can still be economically wrong. The acceptance record should name an owner, preserve the observation, and state when the decision must be revisited.

2. Separate message from attributes

Keep a readable summary while placing provider, operation, outcome, and identifiers in typed fields that do not require regular expressions. Separate message from attributes should be reviewed through fast-window and slow-window burn rates, not through a screenshot of one successful run. Rehearse the boundary by choosing a safe environment and then add latency that crosses only the user threshold. Document containment and recovery in the same language operators see during an alert. Include feature delay caused by reliability intervention in the trade-off, and retain enough evidence to show whether the observed result came from the design or an unrelated dependency.

3. Standardize correlation

Include trace, request, workflow, and message identifiers where applicable, but avoid treating one identifier as a substitute for all scopes. Turn Standardize correlation into a falsifiable operating claim. Instrument latency distribution around the objective, define failure before running the test, and remove telemetry from one serving region. A reviewer should be able to connect the signal to one business consequence and one recovery action without private team knowledge. Account for commercial remedies and customer attrition; otherwise the architecture may shift expense into support, response labor, or another service while appearing inexpensive on its own dashboard.

4. Control exception records

Capture class, category, sanitized message, and stack once at the handling boundary instead of repeating the same exception through every layer. The verification plan for Control exception records begins with budget consumption by release and incident measured across ordinary and peak traffic. Next, ship a canary that affects one customer cohort and inspect both the immediate symptom and the state left after recovery. Keep the runbook specific about authority, evidence, and rollback. Re-estimate monitoring needed to calculate trustworthy indicators when volume, regions, or ownership change, because limits selected for the first deployment rarely remain neutral as the system grows.

A structured log is an event with named fields, not a sentence decorated with JSON braces. Machines can filter payment.capture_failed by provider and error category; humans can read the same record with order, attempt, trace, and deployment context. The goal is not to log everything. It is to preserve enough trustworthy evidence to explain behavior while controlling privacy, ingestion cost, and noise.

Define an event catalog around decisions and state transitions. Useful events include authentication rejected, order accepted, retry scheduled, circuit opened, migration completed, and configuration loaded. Give each a stable event name and schema. Keep the human message, but do not force investigations to parse it. Severity should represent operational urgency: a handled validation rejection is usually information, while inability to persist accepted work is an error.

{
  "event.name": "payment.capture_failed",
  "severity": "ERROR",
  "order.id": "ord_42",
  "payment.provider": "gateway_a",
  "error.category": "timeout",
  "attempt": 3,
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

The event separates a stable machine-queryable name and typed context from the provider’s variable error wording, while the trace identifier connects it to distributed evidence.

Never log secrets, session tokens, authorization headers, card data, or unrestricted request bodies. Hashing identifiers is not automatic anonymization; low-entropy values can be guessed, and stable hashes still support tracking. Prefer allowlisted fields, classify them, restrict access, and set retention from incident, legal, and operational needs.

Logging has two cost curves. Too little evidence increases investigation time and unresolved incidents. Too much increases ingestion, indexing, storage, and human search cost while expanding security exposure. Sample repetitive successful events, aggregate high-volume counters as metrics, and preserve rare errors and audit events according to policy. A concise event with stable context is more valuable than a multi-line dump emitted on every request.

Evidence controls

• Protect sensitive values: Use an allowlist, automated secret scanning, field-level redaction, access controls, and deletion procedures rather than relying on developer memory. Treat good-event ratio and eligible request count as the release gate for Protect sensitive values, with thresholds derived from the user journey rather than a vendor default. Prove degraded behavior when you make a dependency return errors for five minutes, including what an operator can see before customers report it. The design review must compare redundancy, capacity headroom, and on-call staffing with the consequence of a missed or delayed operation. Record exceptions explicitly so emergency action does not silently become permanent configuration.

• Make time unambiguous: Emit UTC timestamps with explicit precision and retain event occurrence time separately from collector ingestion time when delayed delivery matters. A production acceptance test for Make time unambiguous should capture fast-window and slow-window burn rates before, during, and after recovery. In the fault phase, add latency that crosses only the user threshold; in the review phase, reconcile durable state rather than stopping when the process turns green. Model feature delay caused by reliability intervention at current load and forecast peak. This makes the decision reversible: later teams can see why the limit existed and what evidence would support changing it.

• Define severity semantics: Document what debug, information, warning, and error mean operationally and prevent normal user mistakes from paging responders. Use Define severity semantics to connect implementation detail with operations. Dashboard latency distribution around the objective, but keep the underlying event or state available for investigation. Then remove telemetry from one serving region in a bounded drill and have someone outside the implementing team follow the runbook. Compare the resulting risk reduction with commercial remedies and customer attrition. Passing means the system behaves predictably and the responder can explain why, not simply that an automated check returned success.

• Handle logger backpressure: Use bounded buffers and a declared drop or block policy so a logging outage cannot exhaust application memory or freeze every request. Before approving Handle logger backpressure, ask which assumption is most likely to expire. Baseline budget consumption by release and incident, then ship a canary that affects one customer cohort without changing several variables at once. Capture timelines and identifiers needed to reconstruct the event. Budget explicitly for monitoring needed to calculate trustworthy indicators, including human ownership that provider calculators omit. Schedule review after material traffic, dependency, or policy changes so the control remains aligned with its original business purpose.

• Test schema compatibility: Validate required fields and types in automated tests, including older consumers and dashboards that depend on the event contract. Operational evidence for Test schema compatibility comes from good-event ratio and eligible request count correlated with a representative transaction or workflow. Validate it when you make a dependency return errors for five minutes, checking partial success and delayed recovery rather than only total outage. Discuss redundancy, capacity headroom, and on-call staffing with product and finance owners where the consequence is material. The final runbook should state the safe manual action and the conditions under which automation must stop and request judgment.

• Run an evidence drill: Ask an engineer unfamiliar with the service to explain a staged failure using only approved production views and record missing context. For Run an evidence drill, separate prevention from detection. Use fast-window and slow-window burn rates to discover drift from the intended behavior, and add latency that crosses only the user threshold to prove containment. Recovery should preserve auditability and avoid converting one failure into duplicates, data loss, or uncontrolled load. Compare that protection against feature delay caused by reliability intervention. A concise decision record should link configuration, test evidence, dashboard, and accountable owner so the control survives staff rotation.

• Tune indexes and retention: Index fields used for urgent filtering, archive lower-value evidence cheaply, and delete data when its justified retention period ends. Tune indexes and retention is ready only when latency distribution around the objective can be interpreted by the on-call engineer without reverse-engineering implementation internals. Run a game day in which you remove telemetry from one serving region, and require an explanation of impact, state, and next action. Track commercial remedies and customer attrition separately from normal application spend so reliability overhead remains visible. If the drill needs privileged improvisation, improve the interface or runbook before increasing automation.

• Audit configuration changes: Log who changed verbosity, routing, redaction, and retention while protecting the audit stream from ordinary application modification. Close the Audit configuration changes review with a recovery proof. Observe budget consumption by release and incident, ship a canary that affects one customer cohort, and verify the final business records against their authority. Report monitoring needed to calculate trustworthy indicators beside latency and correctness rather than after the architecture is committed. Keep an explicit exit condition: if evidence, platform capability, or economics change, the team should be able to simplify or replace this mechanism without losing the contract it protects.

Google SRE reliability source notes

The OpenTelemetry logs data model defines the platform or specification behavior used in this design. The Google Cloud structured logging provides a second primary reference for implementation limits and operating guidance. These sources support the guarantees described here; product policy and risk decisions remain specific to the organization.

A compact operating model

user events -> SLI calculation -> SLO window -> remaining error budget -> release decision

Raw events become a reliability ratio over a declared window. The remaining budget then informs a product decision; it is not merely another operations chart.