A cache stores a cheaper copy and therefore creates a consistency problem. Speed is the benefit; freshness, invalidation, capacity, and failure are the bill. This incident-driven tutorial follows a product-price cache through read-through loading, write invalidation, a stampede, Redis failure, and recovery. Every pattern begins with what stale data means to the user.
The core idea
A cache is a copy with a declared truth relationship. Before selecting Redis, HTTP caching, or an in-process map, define the authoritative source, acceptable age, invalidation owner, miss behavior, and outage policy. Performance follows only after stale and unavailable outcomes are safe.
Decision point: what may be stale
The question is not which option is fashionable. A product description may tolerate minutes; authorization or a final price may not. Choose by define truth source, maximum age, miss path, and consequence per field, and record the assumption that would cause the choice to change.
| Data | Freshness | Outage policy |
|---|---|---|
| Product copy | 10 min | Serve stale |
| Inventory hint | seconds | Label uncertainty |
| Checkout price | Revalidate | Fail closed |
| Permissions | Very short | Source/policy decision |
The comparison should include steady-state cost, failure behavior, migration effort, and the skills of the team operating it. Run stale values at every business transition before committing. If one TTL is applied to an object containing fields with different risk, the option's apparent simplicity has moved complexity into operations. Use served age and stale outcome by data class as the review signal after real traffic arrives.
Data flow: cache-aside read and invalidation
Follow one logical operation. The application owns miss loading and write invalidation. At each arrow, version keys, constrain tenant scope, and delete/update only after source commit
flowchart LR
R[Read]-->C{Cache}
C-->|hit|A[App]
C-->|miss|D[(Database)]
D-->C
W[Write]-->D
D-->I[Invalidate]Delete-after-commit avoids caching uncommitted truth, but races still require acceptable TTL or versioned keys.
The diagram is a map of failure and ownership boundaries, not decoration. Annotate where data becomes durable, where it can be duplicated, and where ordering can change. Validate it with barrier-controlled read/write interleavings. The model is wrong if a failed database write still invalidates or repopulation races with an older reader. Correlate hit ratio, age, invalidations, and source/cache mismatches across the flow using a safe operation identifier.
Step: HTTP validators and private responses
Start with a small, reversible implementation. Shared and browser caches obey HTTP directives and validators, not application folklore. The immediate action is to set Cache-Control deliberately, emit ETag, and vary on representation-changing headers
Cache-Control: private, max-age=60
ETag: "product-42-v9"
GET /products/42
If-None-Match: "product-42-v9"An unchanged representation returns 304 without a body. Personalized data must not leak through a shared cache.
Read the example from top to bottom and identify which line establishes the guarantee and which lines merely implement it. In a real system, repeat the exercise with users, locales, encodings, revalidation, and intermediary behavior. The step is incomplete if authorization or locale changes content but the cache key does not vary. During rollout, record Age, cache status, 304 rate, and cross-user safety tests; compare it with a baseline before claiming success.
Failure reconstruction: popular key expires and melts the origin
Imagine the alert has fired. Thousands of callers miss together, multiplying one refresh into a stampede. The first response should be to coalesce refreshes, jitter expirations, prewarm known hot keys, and serve bounded stale data when safe, preserving evidence before retries or manual repair change the state.
hit -> return
miss -> one loader acquires short lease
others -> wait briefly or receive acceptable stale value
loader -> fetch, publish, releaseThe lease has a timeout; it is not a permanent distributed lock.
Walk the timeline twice: once from the caller's view and once from the durable system of record. The two stories often diverge at a timeout, commit, queue acknowledgement, or cache boundary. Recreate hot-key expiry with slow origin and loader death. The likely trap is that all keys share an exact TTL or waiters retry in a tight loop. Confirm recovery by watching concurrent loaders, origin QPS, wait time, stale serves, and lease expiry, not merely by seeing one successful request.
Operating reality: cache outage and recovery
Treating the cache as optional can still overload the database when it disappears. The mechanism is not production-ready until the team can budget origin capacity, circuit-break misses, restore traffic gradually, and avoid empty-cache floods
cache errors rise
-> cap origin concurrency
-> serve safe stale/local fallback
-> shed low-priority work
-> recover cache
-> warm/restore traffic in cohortsFail-open versus fail-closed is chosen per data class, especially for security decisions.
Alerts should indicate customer impact and a plausible action; lower-severity diagnostics can remain on dashboards. Exercise total outage, latency spike, eviction storm, and cold restart during a game day. Operations will stall if every request falls through to the source or recovered nodes are instantly saturated. Measure cache errors, origin saturation, evictions, memory, latency, and rejected work with rates, percentiles, age, and saturation as appropriate, and link every alert to an owned recovery procedure.
Cache chaos drills
1. Rehearse what may be stale
Before sign-off, restate this article-specific claim in one sentence: A product description may tolerate minutes; authorization or a final price may not. Build a controlled exercise that includes stale values at every business transition. Record the expected durable state, response, and operator action before running it. During the exercise, deliberately introduce the condition where one TTL is applied to an object containing fields with different risk. The team must explain whether to retry, reject, compensate, roll back, or wait. Preserve served age and stale outcome by data class with enough dimensions to compare the expected and observed path. Finish by assigning ownership and documenting the threshold that would force the design to be reconsidered.
2. Rehearse cache-aside read and invalidation
Before sign-off, restate this article-specific claim in one sentence: The application owns miss loading and write invalidation. Build a controlled exercise that includes barrier-controlled read/write interleavings. Record the expected durable state, response, and operator action before running it. During the exercise, deliberately introduce the condition where a failed database write still invalidates or repopulation races with an older reader. The team must explain whether to retry, reject, compensate, roll back, or wait. Preserve hit ratio, age, invalidations, and source/cache mismatches with enough dimensions to compare the expected and observed path. Finish by assigning ownership and documenting the threshold that would force the design to be reconsidered.
3. Rehearse http validators and private responses
Before sign-off, restate this article-specific claim in one sentence: Shared and browser caches obey HTTP directives and validators, not application folklore. Build a controlled exercise that includes users, locales, encodings, revalidation, and intermediary behavior. Record the expected durable state, response, and operator action before running it. During the exercise, deliberately introduce the condition where authorization or locale changes content but the cache key does not vary. The team must explain whether to retry, reject, compensate, roll back, or wait. Preserve Age, cache status, 304 rate, and cross-user safety tests with enough dimensions to compare the expected and observed path. Finish by assigning ownership and documenting the threshold that would force the design to be reconsidered.
4. Rehearse popular key expires and melts the origin
Before sign-off, restate this article-specific claim in one sentence: Thousands of callers miss together, multiplying one refresh into a stampede. Build a controlled exercise that includes hot-key expiry with slow origin and loader death. Record the expected durable state, response, and operator action before running it. During the exercise, deliberately introduce the condition where all keys share an exact TTL or waiters retry in a tight loop. The team must explain whether to retry, reject, compensate, roll back, or wait. Preserve concurrent loaders, origin QPS, wait time, stale serves, and lease expiry with enough dimensions to compare the expected and observed path. Finish by assigning ownership and documenting the threshold that would force the design to be reconsidered.
5. Rehearse cache outage and recovery
Before sign-off, restate this article-specific claim in one sentence: Treating the cache as optional can still overload the database when it disappears. Build a controlled exercise that includes total outage, latency spike, eviction storm, and cold restart. Record the expected durable state, response, and operator action before running it. During the exercise, deliberately introduce the condition where every request falls through to the source or recovered nodes are instantly saturated. The team must explain whether to retry, reject, compensate, roll back, or wait. Preserve cache errors, origin saturation, evictions, memory, latency, and rejected work with enough dimensions to compare the expected and observed path. Finish by assigning ownership and documenting the threshold that would force the design to be reconsidered.
Conclusion: cache with an explicit truth model
Caching is safe when the source of truth, freshness budget, invalidation owner, miss behavior, and outage policy are explicit. Keys need schema versions and tenant boundaries. TTL limits age but does not guarantee freshness. Jitter, request coalescing, stale-while-revalidate, and bounded fallbacks protect dependencies. Measure user latency together with origin load and stale outcomes. The durable lesson is to state guarantees in language clients and operators can verify, then build the smallest mechanism that satisfies them under retries, concurrency, deployment, and partial failure. Revisit the decision when traffic, risk, or ownership changes.
Continue with SQL vs NoSQL: Choose From Access Patterns and Guarantees to connect this topic to another production boundary. The Prime Axiom resource library collects practical implementation material.
Authoritative references
HTTP and server-side caches have different controls; use the cited standards and product guidance only for the layer they actually govern.
