Delete cache first or update the database first? The gap CDC must close

Fictional setting, verified lab evidence. Fortex in this narrative is fictional; Tom and Ben are characters illustrating engineering reasoning. All metrics, WAL decodings, and assertions come from the linked reproducible Docker lab, not Fortex production systems.

A Journey at Fortex · Chapter 14

Admin Price Displays $110, but Customers Still Buy at $100

At 10:15 AM sharp, Ben’s internal phone rang incessantly. On the other end was an irate VIP merchant:

“Why is our flagship headset still selling for $100 on the live storefront when our operations team raised the price to $110 over half an hour ago? The admin portal says update successful, yet customers are snapping up inventory at the old discounted rate! Who is going to compensate us for this revenue loss?”

Ben broke into a cold sweat. He hurried to his terminal to investigate:

• Direct query to PostgreSQL: SELECT price FROM products WHERE id = 1; → Returned 110.
• Query to Redis Cache: GET product:1 → Stubbornly showed 100!

Ben immediately executed DEL product:1 in Redis. Instantly, the public product page refreshed and displayed $110.

Breathing a sigh of relief, Ben opened the backend source code. He quickly pinpointed the price update handler: It executed a single UPDATE products SET price = 110 WHERE id = 1 against PostgreSQL, with absolutely zero lines touching Redis!

“Aha!” Ben smiled. He promptly created an emergency pull request: Adding a cache deletion right after the database transaction committed:

await db.query('UPDATE products SET price = $1 WHERE id = $2', [newPrice, productId]);
await db.commit();
await redis.del(`product:${productId}`); // Ben added this line

Ben rolled his chair over to show Tom: “Look Tom, I found the culprit! Whoever built the price update API forgot to invalidate the cache. I just added redis.del right after db.commit(). Now the database commits first, and then the cache is cleared. No more stale prices!”

Tom looked at Ben’s new snippet without immediate praise or criticism. He pointed his finger directly at the gap between the commit() line and the redis.del() line, asking:

“What happens if the Node.js process gets OOM-killed by Kubernetes, the container host loses power, or network connectivity to Redis drops at the exact instant between these two lines?”

Two Classic Schedules, Both Generating Stale Cache

Ben’s smile vanished.

He realized the brutal reality: The database has durably committed the price of $110, but the process dies before issuing redis.del! The stale $100 remains in Redis indefinitely, and customers continue buying at the outdated rate until the TTL expires (or forever if keys lack a TTL)!

“Then...” Ben scrambled for an alternative, “Then I'll just flip the order! I'll delete the cache first, and only then update the database!”

await redis.del(`product:${productId}`); // Delete cache first
await db.query('UPDATE products SET price = $1 WHERE id = $2', [newPrice, productId]);
await db.commit();

Ben rationalized: “Deleting the cache first is safer! If deleting succeeds but the DB update fails, the worst that happens is a cache miss where the next reader pulls the old value from the DB. No corrupted state!”

Tom shook his head and pulled out a whiteboard marker: “You're only visualizing a solitary Writer executing in isolation. Have you accounted for hundreds of concurrent Readers viewing the product at that exact moment?”

Tom drew the timeline of two concurrent actors:

Writer (Updating Price: 100 -> 110)    Reader (Customer Viewing Product)
----------------------------------     ----------------------------------
1. redis.del('product:1')  (Cache empty)
                                       2. Read Redis: Cache Miss!
                                       3. Read DB: Sees old price 100!
                                          (Writer has not committed yet)
                                       4. Writes 100 back into Redis!
5. db.query('UPDATE ... price = 110')
6. db.commit() (DB is now 110)

Ben went pale. He muttered to himself:

“That's terrifying... A reader encounters a cache miss, queries the database before the writer commits, reads the stale $100, and caches it back into Redis! Then the writer commits $110. Final state: Database holds 110, but Redis is repopulated with 100!”

WhyBit’s reproducible lab faithfully executed both concurrent interleavings:

The Illusion of “Delayed Double Deletion”

“I remember reading online about a popular trick called Delayed Double Deletion,” Ben scratched his head. “You delete cache, update DB, sleep(500ms), and delete cache a second time to purge whatever stale data a reader might have repopulated. Isn't that a solid pattern?”

Tom chuckled: “That is the quintessential ‘hope-driven engineering’ answer often given in interviews by engineers who have never operated high-throughput systems.”

Tom dissected the critical flaws of the technique:

1. Where does 500 ms come from? In production, a Garbage Collection (GC) pause, transient network congestion, or database read-replica lag can easily stretch a reader’s query to 800 ms. The reader will write stale data into Redis after the second deletion has already finished!
2. Resource Exhaustion: Calling sleep inside a synchronous request thread holds onto connection pool slots and worker threads, choking overall backend throughput.
3. Async Queues Don't Solve It: Pushing the second delete to a background queue brings back the original dilemma: If the worker crashes or the broker drops the task, who guarantees the second delete executes?

“A TTL (Time-To-Live) merely bounds the lifespan of stale data; it does not close the consistency window,” Tom concluded. “To solve this fundamentally, we need a mechanism where the invalidation intent originates directly from the exact place where data commit is guaranteed!”

Emitting Events from Committed Storage: CDC and PostgreSQL WAL

“What is the single source of truth in our architecture?” Tom asked.

“The database transaction log!” Ben answered promptly. “In PostgreSQL, that’s the Write-Ahead Log (WAL). Once a transaction commits, it is durably etched into the WAL on disk!”

Precisely. Instead of having application servers perform flawed dual-writes, the industry-standard architecture for large-scale systems is Change Data Capture (CDC):

• Applications do one thing only: Update PostgreSQL and issue COMMIT.
• PostgreSQL’s Logical Decoding engine streams committed changes directly out of the WAL.
• Dedicated connectors like Debezium tail the PostgreSQL Replication Slot and forward changes into a message broker or directly trigger cache invalidations.

The official Debezium PostgreSQL connector documentation highlights that Debezium tracks its processed WAL position (LSN - Log Sequence Number). If Debezium crashes or restarts, it resumes reading from the exact LSN where it left off, providing guaranteed At-Least-Once delivery.

“Wait Tom, At-Least-Once means events can be delivered multiple times!” Ben noted. “What if a delete cache event is received twice?”

Tom smiled: “Try running DEL product:1 twice in a row in Redis and see what happens.”

Ben ran the commands:

• First execution: Redis deletes the key and returns (integer) 1.
• Second execution: Key is already gone, Redis returns (integer) 0.

“It is completely Idempotent!” Ben cheered. “Even if re-delivered ten times, the final state in Redis remains identical: the key does not exist. The next read is guaranteed to fetch the freshest value from the database!”

The WAL Lab: Consumer Dies and Recovers Without Data Loss

In WhyBit’s reproducible lab, CDC was configured on PostgreSQL 17.6 using the logical replication slot whybit_cache and the standard test_decoding plugin:

1. Both PostgreSQL and Redis started with product price = 100.
2. The cache invalidation consumer was deliberately shut down (simulating a service outage).
3. While the consumer was offline, a writer updated the price to 110 in PostgreSQL and committed.
4. During this downtime, Redis still held the old price 100 (observable staleness window).
5. The consumer was brought back online. It immediately connected to the replication slot and drained the preserved changes directly from the WAL:

BEGIN 746
table public.products: UPDATE: id[integer]:1 price[integer]:110
COMMIT 746

6. The consumer parsed id: 1 and executed DEL product:1 in Redis.
7. Subsequent customer read requests experienced a Cache Miss, fetched 110 from PostgreSQL, and repopulated Redis. The cache converged to 110!

“That is the true value of CDC,” Tom explained. “Invalidation intent is no longer a fragile promise in the volatile memory of an app server. It is anchored to the durability of the database transaction log. Even if the consumer is down for minutes, the change sits safely in the WAL waiting to be processed!”

Operational Realities of CDC for Senior Engineers

Ben was eager: “So CDC is the silver bullet for caching?”

“There are no silver bullets in distributed systems,” Tom cautioned. “CDC buys you durability and recoverability, but imposes substantial operational responsibilities.”

Ben jotted down the operational hazards:

1. Replication Slot Disk Exhaustion: If the CDC consumer hangs or lags indefinitely, PostgreSQL must retain all unread WAL files on disk. Without strict safeguards, the database disk will reach 100% capacity and crash the entire database cluster! PostgreSQL’s documentation explicitly advises configuring max_slot_wal_keep_size and setting up alerts for retained WAL bytes (PostgreSQL streaming replication slots).
2. Out-of-Order Message Processing: When scaling consumer workers horizontally across partitions, a price update to 110 could race against a subsequent update to 120. Messages must be partitioned strictly by Entity ID or carry database LSN/version timestamps to reject stale overwrites.
3. Initial Snapshot Traps: Bootstrapping CDC on pre-existing tables with millions of records requires running an initial snapshot concurrently with WAL tailing (Debezium snapshot modes) without blocking concurrent writes.
4. Cache Stampede Upon Deletion: When CDC deletes a hot product key, thousands of concurrent read queries will hammer the database at once. Singleflight mutex leases must be combined as detailed in Chapter 5 — A Cache Key Expires, 12 Requests Slam the Database.

Lab Verification Evidence

All experimental assertions were automatically verified inside isolated Docker containers:

This report image is generated from the report.json for run 771d0fda-ed2c-4d43-8c88-78df7c125ba0: DB = 110, stale cache = 100, WAL retains transaction 746, and duplicate DEL calls return 1 then 0. It is not a production dashboard.

Report JSON · Raw log · Lab Source Code · Run 771d0fda-ed2c-4d43-8c88-78df7c125ba0

Scope & Limitations: The lab runs on a single host with PostgreSQL 17.6 and Redis 7.4.2 to isolate logical mechanics. It demonstrates WAL retention and idempotent deletion; it does not simulate a full Debezium Kafka Connect cluster or multi-datacenter network partitions.

Ben summarized the revised Fortex architecture:

• Writer services do only one job: Update PostgreSQL and commit.
• CDC pipeline tails WAL and publishes ProductChanged events.
• Invalidation workers receive events and execute DEL against Redis.
• Grafana dashboards alert on cdc_consumer_lag_seconds and postgres_wal_retained_bytes.
• Product APIs return entity version headers so frontends can detect stale reads.

Tom smiled: “Did we achieve absolute real-time strong consistency?”

“Not strong consistency,” Ben grinned. “Cache in distributed systems is always eventually consistent. But now, stale values are never orphaned forever by an unhandled server crash. And most importantly: we can measure exactly how many milliseconds behind we are!”

Fortex and dialogue are fictional narrative devices. All measurements and technical logs are from the linked reproducible lab run.

Previous: Chapter 13 — Invisible Deadlocks · Next: Chapter 15 — Kafka Hot Keys