One expired key, twelve requests hitting the database

Fictional setting, real lab evidence. Fortex in this series is a fictional setting; Tom and Ben are characters. Their conversations illustrate the reasoning. Measurements and assertions come from the linked Docker lab, not Fortex production incidents.

The Fortex journey · Chapter 5/8

The gap hidden by the hit rate

Ben added caching to Fortex's product page. It was an appealing change: little business logic to touch, fast reads when a key existed, less database work. After the inventory lesson, he was careful not to use cached stock as the authority for sales.

“Writes stay in the database. Redis takes care of the reads.”

Tom looked at the cache-aside function. “Walk me through the moment the key expires.”

“GET misses, read the origin, SET the cache. The first request reloads it.”

“And what does the next request see before the first reload finishes?”

Ben paused, then opened another window. He was getting used to placing callers side by side. This time no individual write was wrong. Each caller was doing the same reasonable work at the same time.

One expired key can create a queue at the database

Tom suggested putting the average hit rate aside. Ben chose a narrower question: for the same group of readers, how many origin operations happen when every caller sees an empty key?

A frequently read product price is cached for a minute. Most requests are cheap. At expiration, however, twelve requests can all observe a miss. Each follows the same reasonable instruction: fetch the price from the database, then cache it.

The same value is computed twelve times. If the underlying operation is expensive, duplicated work occupies database connections, increases latency, and may trigger retries. Adding connections can simply admit more redundant work. This is a cache stampede.

The comparison below keeps the requests and result unchanged. Only coordination during a miss changes.

Cache-aside does not merge concurrent misses

“I'll make the first request hold a lock,” Ben proposed.

“Try it. First draw the last caller's path,” Tom said. “When it acquires the lock, how will it know someone has already filled the cache?”

value = cache.get(key)
if value is missing:
    value = load_from_database(key)
    cache.set(key, value, ttl)
return value

This handles one request but does not decide who reloads a missing key when many requests arrive together. Sharing Redis does not automatically share the in-flight calculation.

In the lab, twelve threads first read an absent key, then meet at a barrier. The ordinary branch consequently performs twelve origin operations. It is an intentionally synchronized cold-cache schedule, not a model of every production arrival pattern.

Elect one loader and bound everyone else's wait

Ben added a cache read after acquiring the lease. “If everyone queues and then reloads, we've just moved the crowd.”

Tom added an owner paused past its lease. Ben had been about to use DEL to release it; now he had to ask whether the lock still belonged to that caller.

The coordinated branch uses a short lease:

GET product:42
if missing:
    SET lock:42 <unique-token> NX PX 30000
    winner: recheck cache → load origin → SET cache → release own lease
    others: wait briefly → recheck cache, stop at deadline

Recheck the cache after acquiring the lease. Another loader may have populated it between your earlier GET and your successful acquisition. Without that check, duplicate parallel loads can become duplicate sequential loads.

Each owner needs a unique token. Releasing a lease must atomically check ownership and delete it:

if redis.call('get', KEYS[1]) == ARGV[1] then
  return redis.call('del', KEYS[1])
end
return 0

An unconditional DEL could remove a newer owner's lease if the old owner was paused beyond its expiry. The Redis locking documentation describes token ownership and lease validity. This experiment uses one Redis instance to coalesce cache fills; it does not establish mutual exclusion under every failover scenario.

What the experiment measures

They kept the reader count and returned values the same across branches. Ben counted origin operations in PostgreSQL. “I'll report less duplicate work,” he said. “We haven't measured application p99.”

Digest-pinned PostgreSQL 17.6 and Redis 7.4.2 run in an isolated Docker network. An origin operation increments a real PostgreSQL counter and deliberately waits 250 ms. Every response carries the same fixed price=120 value. Each branch resets the counter and cache.

Both branches return twelve identical responses. The lease branch touches the origin once. An additional assertion gives the lock to new-owner, attempts release with old-owner, and confirms that the lock survives.

Read the report and raw logs, or run the downloadable source with make reproduce. This measures duplicated origin work, not QPS or p99 improvement. The driver's Docker CLI overhead is very different from a production client with persistent connections, so elapsed driver time is not an application benchmark.

A lease is allowed to expire

“With a token, only one owner can run, right?”

“The token is checked on release,” Tom said gently. “What stops the old owner while it's still running?”

Ben crossed out guarantees in his note. Preventing a stale owner from deleting a new lock did not prove that it couldn't write late data.

If a loader takes 35 seconds but its lease lasts 30, another loader can acquire a new lease while the old one is still running. Token checking prevents the old owner from deleting the new lock; it does not stop the old computation or its cache write. Versioned results or generation checks may be needed to prevent stale overwrites. For disposable cache data, occasional duplicated computation may be acceptable.

This lease is unsuitable as proof that a payment executes once. Important mutations need invariants at the owning datastore and business idempotency. Atomic updates and version checks address another layer of the problem.

Alternatives and trade-offs

Ben wanted leases for every key. Tom asked which data could be served stale, which keys were local to one process, and how many callers could reach the origin when Redis failed.

Ben reorganized his proposal around data requirements. Waiting, returning stale data, and rejecting load were choices Fortex still needed to make.

Waiters need a deadline and jittered backoff. Tight GET loops can move the stampede from the database to Redis. If Redis fails, letting every request bypass the cache may overload the origin; concurrency limits, acceptable stale results, or controlled rejection may be safer for the particular product.

Track origin operations per miss group, reload wait time, cached-value age, and timeouts. A healthy average hit rate can hide the few seconds that overwhelm an origin. A cache needs a policy for sharing work while the value is absent, as well as a policy for storing it afterward.

The page Ben takes into tomorrow

Ben ended the proposal with a condition rather than a promise: the lease coalesced work in the tested scope; expiry and failover needed separate design. His new notebook line read: “Coordinating work isn't the same as protecting a business invariant.”

He carried it into the next task: tell the warehouse when an order is created. Surely that only needed a publish call after commit.

Previous chapter: Two stock decrements, but only one takes effect · Next chapter: The database committed. Where did the event go? · Start the journey