The distributed cache invalidation decision record: why the invalidation strategy you chose determines your stale data failure surface and your thundering herd exposure

A 40-person e-commerce SaaS company introduced Redis caching after a database performance review identified product listing pages as the primary source of slow queries — each page required six to nine database queries to assemble product data, category breadcrumbs, inventory counts, and pricing. The engineering team implemented a cache layer that stored the assembled page payload as a serialized JSON blob, keyed by product ID, with a 24-hour TTL. Cache hit rates reached 94% within a week of deployment. Database query volume dropped by 89%. Page load times improved from an average of 1.8 seconds to 140 milliseconds. The engineering team considered the implementation complete.

The caching decision session had focused on the performance problem. The team had discussed key design, serialization format, Redis cluster configuration, and TTL duration. The 24-hour TTL had been chosen as a reasonable default — product data changed infrequently, the engineering team had reasoned, so a 24-hour cache was aggressive enough to deliver database relief without being so long that stale data would be a practical problem. The session did not include anyone who tracked the frequency of price changes, promotional adjustments, or inventory updates. It did not include the marketing team that ran flash sales. It did not produce a document enumerating which fields in the cached payload had different acceptable staleness windows than others: product descriptions changed rarely, but prices changed multiple times per day during promotional periods, and available inventory changed with every purchase.

Three months after the caching layer was deployed, the company ran its largest promotion of the year — a 36-hour flash sale with price reductions of 15% to 40% across 800 products. The promotions team updated the prices in the admin panel at 2 PM on a Tuesday. The admin panel's price update endpoint called the database, updated the price records, and returned a success response. It did not invalidate the Redis cache keys for the updated products. The founding session that designed the cache had specified TTL-only invalidation; no write path had been wired to trigger cache invalidation. The cache continued serving the pre-promotion prices. Customers browsing the site saw the original prices. The price shown on product listing pages, product detail pages, and search result cards was the cached pre-promotion value. The checkout flow fetched prices from the database directly (it did not use the page cache), which returned the live promotional prices. Customers who reached checkout saw a different price than the one displayed on the product page.

By 4 PM, the support queue contained 340 tickets from customers asking why the price in their cart differed from the price they had seen on the product page. The engineering team identified the cache as the source of the discrepancy. Flushing the Redis cache for all 800 affected products would solve the immediate problem. The team flushed the cache at 4:17 PM. The cache flush resulted in a stampede — 94% of traffic hitting empty keys simultaneously, every request generating 6 to 9 database queries for the first time in three months — and the database CPU peaked at 98% for 4 minutes before query throughput returned to normal as cache keys were repopulated. By 11 PM — nine hours after the price update — the TTL-based cache would have expired naturally and the promotion prices would have appeared. The company identified 4,200 orders placed between 2 PM and 4:17 PM at prices that differed from the promotional price displayed on the checkout page versus the product listing page; the legal and customer success teams spent three weeks working through refund requests and customer complaints. The founding session that chose the 24-hour TTL had not specified a staleness window for price data. The same TTL that was appropriate for product descriptions was appropriate for prices — until it was not, and the consequence was visible to thousands of customers simultaneously.

A 30-person B2B SaaS platform added explicit cache invalidation on write after a support incident in which a customer service manager updated a customer account configuration in the admin panel and the changes were not visible in the customer-facing portal for up to 24 hours. The engineering team implemented write-through invalidation: whenever any record was saved through the admin panel's update endpoint, the endpoint called a cache invalidation function that deleted the relevant Redis keys for the affected entity. The fix worked. Admin panel updates were now immediately visible in the customer portal. The engineering team closed the incident ticket and marked the write-through invalidation as the standard for new cache-backed features.

The write-through invalidation was implemented in the admin panel's update endpoint. The endpoint was the canonical write path for human-initiated configuration changes. It was not the only write path that modified configuration records. The platform also had a bulk import feature — a CSV importer that allowed account managers to update hundreds of customer account configurations simultaneously by uploading a formatted spreadsheet. The bulk importer had been built six months before the caching layer was added. It operated by reading the uploaded CSV row by row and issuing SQL INSERT or UPDATE statements directly against the database, bypassing the admin panel's update endpoint entirely. It had been implemented this way because the admin panel's endpoint processed one record at a time and adding 500 records through it would require 500 HTTP requests; the bulk importer's direct database path was faster and simpler. When the write-through cache invalidation was added to the admin panel's update endpoint, no one updated the bulk importer. The bulk importer was not mentioned in the cache invalidation decision session. The session had discussed "writes through the admin panel" as the scope of the invalidation trigger. It had not produced a list of all write paths that modified cached data.

Six weeks after the write-through invalidation was deployed, an account manager uploaded a CSV containing configuration updates for 5,000 customer accounts. The bulk importer ran successfully in 12 minutes. The database contained the updated configurations. The customer-facing portal continued serving the cached pre-import configurations for all 5,000 accounts, because no invalidation trigger had fired for any of them. Over the following 24 hours — the cache TTL — customers attempting to use newly configured features found them inactive. Customers whose configurations had been changed to remove a feature they were currently using found the feature still appearing in their portal (cached pre-removal state). The support team received 300 tickets over the first 8 hours before the on-call engineer identified the pattern, checked the cache, and manually flushed the affected keys. The full scope of the discrepancy was not known until the cache TTL had expired and all 5,000 accounts were reading from the database. The incident post-mortem identified the bulk importer as a write path that was not in scope for the cache invalidation trigger. The founding session that designed write-through invalidation had documented "invalidate on admin write." It had not enumerated the complete set of write paths that modified cached data, so the bulk importer — which also wrote to the data that was cached — was not included in the scope of the invalidation contract.

The remediation required auditing every code path that issued write operations against tables whose data was cached, identifying all paths that bypassed the admin panel's update endpoint, and adding explicit cache invalidation calls to each. The bulk importer required the most significant change: because it operated by issuing SQL directly, the invalidation call had to be added inside the row-processing loop immediately after each successful database write. The audit identified three additional bypass paths: a data migration script that recomputed derived account fields monthly, a webhook handler that updated account status in response to payment events from the billing provider, and an internal tool used by the customer success team to apply bulk discounts. All four paths were modified to call the invalidation function. The post-mortem added a checklist requirement to the code review process: any pull request that adds a write operation against a cached table must include evidence that the relevant cache keys are invalidated on write in the new code path.

A 25-person consumer platform implemented Redis-backed caching for its most expensive page render — the user profile page, which aggregated a user's activity, posts, follower counts, and engagement metrics from seven database tables. Profile page renders had been the source of the platform's slowest queries; the cache brought P99 load time from 4.2 seconds to 180 milliseconds. Shortly after deployment, a traffic spike during a marketing campaign caused all cached profile keys for featured users to expire within a narrow window, generating a thundering herd of database queries that degraded the entire service for 8 minutes. The engineering team added stampede protection: a Redis-based distributed mutex that ensured only one request per cache key could trigger a rebuild at a time. Other requests that found a key missing and the lock held would wait for the lock to be released, then read the freshly populated key.

The stampede protection implementation used a 10-second lock timeout. The timeout had been chosen to "prevent deadlock" — the engineer's concern was that if a rebuild request crashed or hung, a held lock would block all subsequent requests for the affected profile forever. A 10-second timeout felt conservative but safe: if the rebuild took longer than 10 seconds, something was wrong and the lock should be released. The rebuild duration had not been measured in production at the time the lock timeout was set. The rebuild process made seven database queries (one per aggregated table), computed engagement metrics from the query results, serialized the result, and wrote it to Redis. In the development environment, the entire rebuild completed in under 2 seconds. In production, during normal load, the rebuild completed in 4 to 8 seconds. During the previous traffic spike incident, database query response times had degraded to 8 to 15 seconds per query under peak load. Seven queries at 15 seconds each totaled 105 seconds. The stampede protection had been tested and deployed after the development-environment rebuild time was measured at 2 seconds. The lock timeout had been set to 5x the development rebuild time, which seemed like a large safety margin. It was 10x smaller than the production rebuild time under the load conditions that caused the original stampede.

Three months after the stampede protection was deployed, the platform ran a partnership promotion that drove a 15x traffic increase to profiles of partner accounts. The cache keys for the six featured partner profiles, all with the same TTL, expired within a 90-second window. For the first expiring key, one request acquired the rebuild lock; the other 200 concurrent requests for that profile waited. At 10 seconds, the lock expired. The rebuild was still in progress — it had completed two of seven database queries. The 200 waiting requests were released. Each found the key missing and the lock unset. Each competed for a new rebuild lock. One acquired it; 199 waited. At 20 seconds, the second lock expired before the second rebuild completed. The pattern repeated. Within 60 seconds, there were 40 concurrent rebuild operations running for the same profile key, each issuing 7 database queries. The 280 concurrent database queries exceeded the connection pool limit of 120 connections; rebuild operations began failing with connection pool exhaustion errors. The failed rebuilds released their locks immediately. New requests acquiring locks found no available database connections and failed in turn. For 11 minutes — until the traffic spike subsided and the rebuild operations could complete without contention — every request for the featured partner profiles returned a 503 error. The stampede protection had made the stampede worse: without the mutex, the 200 concurrent rebuild requests would have competed simultaneously in the first 10 seconds; with the mutex, the lock release cycle spread the same number of rebuild operations over 11 minutes, keeping the connection pool saturated for the full duration rather than for a single 10-second burst.

The engineering team's post-mortem identified the core failure: the lock timeout had been set shorter than the rebuild duration under degraded conditions. The invariant that makes mutex-based stampede protection work is that the lock is held for the full duration of the rebuild — so waiting requests are blocked until the key is populated, at which point they read the cached value and issue zero database queries. A lock timeout shorter than the rebuild duration breaks this invariant. The founding session that added stampede protection had documented "mutex with 10s timeout to prevent deadlock." It had not documented the rebuild duration, the production database query latency under load, the relationship between the lock timeout and the rebuild duration that must hold for the protection to be effective, or the failure mode when the timeout is shorter than the rebuild — which is that the protection does not prevent stampede under the exact load conditions that make rebuilds slow.

Structural properties set by the cache invalidation decision

Three structural properties are determined when a team decides how to invalidate cached data. None appear explicitly in the sessions that set a TTL, add an invalidation trigger to one write path, or deploy a mutex for stampede protection — they are the operational consequences of choices made under the assumption that the mechanism present is sufficient for the data categories and traffic patterns that will actually be encountered.

Property 1: The invalidation surface and the write path completeness. A cache invalidation trigger installed on one write path does not cover write paths that bypass it. Application-layer invalidation — a function called by the admin UI's update endpoint, triggered by an ORM's after-save hook, or fired by an event handler attached to a specific service method — covers the write paths that go through that application layer and nothing else. The invalidation surface must be defined as the complete set of code paths that issue write operations against data that is also cached, not as the set of write paths that were in scope when the invalidation trigger was implemented. The gap between these two sets is the invalidation bypass surface: writes that update cached data without triggering cache invalidation, leaving the cache in a state that misrepresents the underlying data until the TTL expires.

Common bypass surfaces that are frequently missing from the invalidation scope include bulk import and data migration scripts that operate by writing SQL directly to the database, external integrations and webhook handlers that modify application data through a separate service or direct database connection, background computation jobs that recompute or denormalize fields by executing SQL without going through the application's service layer, and administrative tools that perform bulk updates using database queries for efficiency. The data pipeline decision record documents the interaction between ETL and application cache invalidation: when a pipeline job writes transformed data directly to an application database table, it must either include cache invalidation calls for the affected keys or be documented as an exception to the invalidation contract with an explicit staleness consequence and an acceptable staleness window for the affected data category. The bypass surface must be audited whenever a new write path is added to the codebase.

Property 2: The staleness contract and the data category classification. A uniform TTL policy imposes the same staleness budget on data with fundamentally different freshness requirements. The TTL that is appropriate for static content — blog post bodies, product category descriptions, help documentation — will be inappropriate for transactional content that changes in response to real-world events with immediate business consequences. The staleness contract must classify cached data by its acceptable staleness window: the maximum age of a cached value that can be served to a user without causing a business-relevant discrepancy between what the user sees and what the underlying system reflects.

The staleness classification has three practically relevant categories for most applications. High-staleness-tolerance data (24 hours or more) includes content that changes infrequently and where a stale read has low cost: marketing copy, static product descriptions, documentation. Medium-staleness-tolerance data (5 minutes to 1 hour) includes content that changes periodically but where a stale read causes at most a user-visible inconsistency rather than a business transaction error: social counts, non-transactional profile data, recommendation lists. Zero-staleness-tolerance data includes any content where a stale read causes a transaction at an incorrect value, a conflict between what a user was shown and what the system records, or a compliance violation: prices during active promotions, inventory availability for products with finite stock, user-specific data modified within the current session, regulatory and policy content that must reflect the current state of a legal requirement. For zero-staleness-tolerance data, TTL-only invalidation is insufficient regardless of TTL duration — the only correct invalidation strategy is write-through invalidation that fires on every write path that modifies the data, supplemented by a TTL as a backstop for undetected bypass writes. The caching strategy decision record documents the broader selection matrix for cache-aside versus write-through versus write-around strategies; the invalidation decision record must specify which strategy applies per data category and why.

Property 3: The stampede protection mechanism and the lock-versus-rebuild-duration invariant. Cache stampede — the thundering herd triggered when a popular key expires and all concurrent requests for that key simultaneously attempt to rebuild it — is not a failure of the TTL policy; it is a failure of the invalidation strategy to account for the load distribution at the moment of expiry. Stampede protection based on a distributed mutex (a lock that allows only one request to proceed with the rebuild while others wait) works correctly only if the lock is held for the full duration of the rebuild. The invariant is: lock_timeout > max_rebuild_duration, where max_rebuild_duration is the worst-case rebuild time under degraded conditions — not the typical rebuild time in development or under normal load.

The rebuild duration is bounded by the sum of I/O operation durations in the rebuild path — typically a series of database queries or external API calls whose latency is itself a function of the system load at the time of the rebuild. Under load spike conditions (the exact conditions that generate stampede), database query latency degrades. The rebuild duration under peak load may be 5x to 20x the rebuild duration under normal load, because the database is handling elevated concurrency from non-rebuild traffic simultaneously. A lock timeout set from development measurements or normal-load production measurements will almost always be shorter than the rebuild duration under the peak-load conditions that actually generate stampede, making the protection ineffective under exactly the conditions it was designed for. The correct approach is to bound the rebuild duration explicitly — add timeouts to each I/O operation in the rebuild path so that the rebuild either completes within a known maximum or fails fast and allows the waiting requests to fail fast rather than waiting indefinitely — and set the lock timeout to 2x the bounded maximum. An alternative that avoids the lock-timeout problem entirely is distributed lock lease renewal: the rebuild process renews the lock every N seconds throughout its execution, so the lock is held for the actual duration of the rebuild regardless of how long that takes, at the cost of requiring the rebuild process to be written as a cooperative lock holder that performs periodic renewals. The distributed locking decision record documents the full comparison of lock implementations, including fencing tokens that prevent lock-expiry races and the semantics of lock renewal under network partition.

What the founding session records and what it omits

The founding cache invalidation session typically records the cache technology selected (Redis, Memcached, in-process cache), the key design (how keys are constructed from entity identifiers), the serialization format (JSON, MessagePack, binary), the TTL duration chosen, and the performance rationale for the caching decision. It may record the cache hit rate target, the database load reduction expected, and the deployment plan for the cache cluster. What it does not record is the staleness classification — the data categories covered by the cache and the acceptable staleness window for each, with the invalidation strategy that enforces that window. It does not record the invalidation surface — the complete set of write paths that modify data covered by the cache and the invalidation trigger for each path. It does not record the stampede protection specification — whether any protection is in place, the mechanism used, and the invariants that must hold for it to be effective under peak load.

The staleness classification omission produces a failure that is latent until the business context changes. A 24-hour TTL on product data is appropriate for a business that never runs time-sensitive promotions. The same TTL becomes a liability the first time the business runs a flash sale, applies an emergency price correction, or updates content that has a legal or contractual freshness requirement. The failure is not in the caching decision itself but in the absence of a staleness contract that would have flagged price data as requiring a shorter TTL or write-through invalidation. The flash sale incident is not a surprising failure given the implementation; it is a predictable consequence of applying a static-content TTL to transactional content. The founding session that chose the TTL did not ask the question "what is the acceptable staleness window for price data?" because the team was thinking about performance, not about the business events that would require prices to change with immediate effect.

The invalidation surface omission produces a failure whenever a write path that bypasses the invalidation trigger is used in a context where users depend on the cache reflecting the latest data. The failure is typically triggered by a routine operational event — a bulk import that has run hundreds of times before without incident — that happens to interact with a cache layer that was added after the write path was established. The bulk importer predated the cache; the cache invalidation trigger postdated the importer; no session connected the two. The incident is discovered not through monitoring but through customer support tickets, because the cache serves correct-looking data (it was correct 24 hours ago) and no alarm fires on data that is stale but syntactically valid. The observability strategy decision record documents the synthetic stale-content probe that can detect this class of failure: a probe that reads a known value from the live database and from the cache at regular intervals and alerts when the values differ — the only monitoring control that can detect a cache that is serving data that is stale beyond the acceptable window without waiting for a customer to report the discrepancy.

The stampede protection invariant omission produces a failure that is worse than no protection. A stampede without protection causes a transient load spike whose duration is bounded by the rebuild time — all concurrent requests rebuild simultaneously, the database absorbs the burst, the cache repopulates, and the spike subsides. A stampede with mutex protection and a lock timeout shorter than the rebuild duration causes a sustained outage whose duration is bounded by the time it takes for the traffic spike to subside, because the lock expiry cycles continuously release new batches of rebuild requests into an already-saturated connection pool. The protection converts a sharp, brief spike into a sustained outage. The founding session that added stampede protection had tested it in development, where rebuilds completed in 2 seconds and the 10-second timeout was a 5x safety margin. It had not tested it under the production load conditions that would actually trigger stampede, where rebuilds take 45 seconds and the 10-second timeout is 22% of the rebuild duration.

The cache invalidation decision record does not need to enumerate every Redis command or configuration option. It needs to answer four questions: what is the acceptable staleness window for each category of data covered by the cache, what is the complete set of write paths that modify cached data and how does each path trigger invalidation, what stampede protection is in place and what invariant must hold between the lock timeout and the rebuild duration for it to be effective, and what synthetic probe or monitoring control will detect a cache that is serving data outside the accepted staleness window. Four answers written in the founding session prevent the flash-sale price discrepancy, the bulk-import staleness incident, and the stampede protection failure that extended a transient spike into an eleven-minute outage. The performance optimization decision record documents the broader context: caching is a performance optimization whose correctness contract — the guarantee that the cache reflects the underlying data within the accepted staleness window — must be as carefully specified as the correctness contract for any other data consistency mechanism.

The WhyChose decision extractor finds the founding cache sessions in your ChatGPT and Claude export — the "how should we add caching?" architecture conversation, the "we need to reduce database load" performance review thread, the "let's add stampede protection" incident follow-up session. It extracts the decision and the options considered, not the surrounding performance discussion that buries the staleness classification question in forty messages about Redis cluster topology and key expiry strategies.

The five ADR sections for a cache invalidation decision

Section 1: Cache invalidation strategy per data category. Specify the invalidation strategy for each category of data covered by the cache. Three strategies are commonly in use and their selection depends on the acceptable staleness window for the data category: TTL-only invalidation (appropriate for high-staleness-tolerance data where the TTL enforces the freshness window and no write path is expected to trigger an immediate invalidation requirement), write-through invalidation (appropriate for zero-staleness-tolerance data where the cache must reflect the current state of the underlying data; every write path that modifies data in this category must trigger cache key deletion or update), and event-driven invalidation (appropriate for medium-staleness-tolerance data where invalidation is triggered by domain events — a message on a message broker, a database change data capture event, a webhook from an external system — rather than by the write path itself; the invalidation signal is decoupled from the write path, which reduces coupling but introduces the latency of the event delivery pipeline into the effective staleness window).

For each data category, document: the category name and the data it includes, the acceptable staleness window in seconds or minutes, the selected strategy, and the rationale for the strategy selection. For write-through categories, list the write paths enumerated in Section 2. For event-driven categories, document the event topic or queue, the consumer that performs invalidation, and the maximum expected delivery latency that bounds the effective staleness window. The event-driven architecture decision record documents the delivery guarantee model for event-driven invalidation: at-least-once delivery with idempotent invalidation (delete the key regardless of whether it exists — deletion of a missing key is a no-op in Redis) is the standard pattern, because at-most-once delivery creates the risk of a missed invalidation event leaving the cache stale indefinitely.

Section 2: Invalidation surface specification. Enumerate every code path that issues write operations against data covered by the cache. For each path, specify the data category affected, the cache keys that must be invalidated, and the mechanism by which invalidation is triggered in that path. The path enumeration must cover: web request handlers and API endpoints in the application layer, bulk import and data migration scripts, background and scheduled jobs, webhook and event handlers that process external system notifications, administrative and developer tools that perform database operations directly, and any ETL or data pipeline job that writes to tables covered by the cache. For write paths that cannot practically be wired to the application's invalidation function — for example, a SQL migration run directly against the database — document the exception explicitly: specify why the path cannot be wired, the consequence for cache freshness (the cache may serve stale data for up to TTL seconds for records modified by this path), and any compensating control (for example, a post-migration script that explicitly flushes affected cache keys, or a reduced TTL for data modified by this path).

Maintain the invalidation surface specification as a living document that is updated when new write paths are added. The code review checklist for any pull request that adds a write operation against a cached table must include verification that the new write path is either wired to the invalidation function or documented as an exception with a rationale. The database migration strategy decision record documents the interaction between schema migrations and cache invalidation: migrations that modify columns included in cached values must include cache invalidation as a step in the migration runbook, because the migration's SQL execution bypasses all application-layer invalidation triggers.

Section 3: Stampede protection mechanism. Specify the stampede protection mechanism for cache keys that are both popular (many concurrent requests) and expensive to rebuild (the rebuild requires multiple database queries or external API calls). Two mechanisms are commonly applicable: distributed mutex with lease renewal (a lock is acquired by the first request to find a key missing; subsequent requests wait; the lock holder periodically renews the lock throughout the rebuild to prevent expiry; the key is populated and the lock is released; waiting requests read the populated key) and probabilistic early expiration (a probability function makes it increasingly likely that a request will proactively recompute a key as its TTL approaches zero, preventing the key from ever reaching simultaneous expiry under consistent traffic).

For the mutex approach, document: the lock implementation (Redis SET NX with expiry, Redlock, or a dedicated distributed lock service), the lock timeout for initial acquisition (which must equal the rebuild duration budget plus a safety margin — see below), the lease renewal interval (typically the lock timeout divided by 4, renewed by the rebuild process on each iteration), the rebuild duration budget (the maximum time the rebuild is permitted to run before being aborted and the lock released), and the behavior of waiting requests when the lock is released without a populated key (because the rebuild was aborted or failed): either retry with exponential backoff or return a degraded response from the underlying data source. Document the invariant explicitly: lock_timeout > rebuild_duration_budget, where rebuild_duration_budget is the P99 rebuild time measured in production under peak load conditions, multiplied by a safety factor of 2. For the PER approach, document the beta tuning parameter, the rebuild duration used in the probability calculation, and the traffic volume assumption under which early recomputation is reliably triggered before expiry. The circuit breaker resilience decision record documents the interaction between stampede protection and circuit breakers: when the database is degraded, the correct response to a cache miss may be to open a circuit and return a cached-or-stale value rather than attempting a rebuild that will fail and hold the stampede lock for the full duration of the rebuild budget.

Section 4: Cold-start and cache warming policy. Specify the behavior of the system when the cache is empty — after a full cache flush, after a deployment that changes key format or serialization schema and requires invalidating all existing keys, or after a cache cluster failure that empties the key space. Cold start under traffic generates the same database load profile as a stampede, but across all cache keys simultaneously rather than for a single expiring key. Document: whether cache warming runs proactively before traffic is routed to the freshly started or flushed cache (pre-warming), the order in which keys are warmed (most-frequently-requested first, to maximize early hit rates), the read-through fallback behavior for cache misses during warming (requests that miss during warming fall through to the database; the response is cached; subsequent requests hit the cache), and whether the rate of warming requests is throttled to prevent the warming process itself from saturating the database during the cold-start window. For critical data categories where a cold start would result in database overload, document the circuit breaker configuration that limits the rate of rebuild operations: a maximum concurrency limit on simultaneous cache-miss rebuilds, with a queue for excess requests rather than immediate fallthrough to the database.

Section 5: Cache observability. Specify the monitoring controls for the cache invalidation contract. Required metrics include: cache hit rate by key category (the primary health metric; a sustained drop in hit rate indicates either expiry-related or invalidation-related key churn), cache miss rate by key pattern with an alert threshold for sustained miss rates on a single key pattern (the primary stampede detection signal — a single key with 200 cache misses per second indicates a stampede), rebuild latency P99 by key category with an alert if rebuild latency approaches the lock timeout (because rebuild latency approaching the lock timeout signals that the lock-versus-rebuild-duration invariant is at risk of being violated under the current load), and connection pool utilization with correlation to cache miss rate (connection pool saturation correlated with elevated cache miss rate is the signature of a stampede consuming rebuild connections). Additionally, implement a synthetic stale content probe: at a regular interval (every 5 minutes for zero-staleness-tolerance categories, every 30 minutes for medium-staleness categories), a background process reads a known set of values from the live database and from the cache and alerts when the cached values differ from the database values by more than the accepted staleness window. The stale content probe is the only monitoring control that can detect a cache serving data outside the accepted staleness window without waiting for a customer to report a discrepancy.