From 2PC to Saga: when compensation arrives first

Fictional setting, verified lab evidence. Fortex in this narrative is fictional; Tom, Ben, and Hà are characters illustrating engineering reasoning. All metrics, state machine outcomes, and assertions come from the linked reproducible Docker lab, not Fortex production systems.

A Journey at Fortex · Chapter 16

Customer Refund Completed, Yet Warehouse Clutches the Inventory

At 14:30 in Fortex’s engineering office, Ben held his head in his hands, staring in disbelief at the warehouse database monitor.

A bizarre discrepancy had just occurred on order #8842:

• The Payment Service had recorded a failed credit card charge, issued a cancellation event, and verified that zero dollars were charged to the customer.
• Yet, when the fulfillment crew opened their dispatch packing software, the warehouse management system was stubbornly holding the parcel in the RESERVED state! The item had vanished from customer-facing storefronts, yet no revenue had been collected!

Ben frantically traced message broker event timestamps and discovered a bizarre paradox:

“Tom, this is madness! The ReleaseInventory compensation event arrived at the warehouse service before the original ReserveInventory forward event!”

Ben pointed to the trace logs:

1. When payment failed, the Payment Service immediately published a compensating ReleaseInventory command. Network queues were clear, so this event zipped across the wire to the warehouse in just 5 milliseconds.
2. The warehouse event handler received ReleaseInventory. It queried its database, found no record for order #8842 (because the initial order placement was delayed in a congested queue), concluded: “Nothing exists to release!”, and returned HTTP 200 OK.
3. Ten milliseconds later, the delayed ReserveInventory event finally arrived. The handler saw a new order request, cheerfully created the row, and transitioned the state to RESERVED!

“The result: The order was canceled and refunded, but the physical item is held hostage in the warehouse forever!” Ben groaned. “Distributed event-driven architectures are an operational nightmare. Shouldn't we scrap this Saga complexity and revert to classic Two-Phase Commit (2PC / XA)? PostgreSQL natively supports PREPARE TRANSACTION. We could have Payment and Warehouse prepare together: if both succeed, commit; if either fails, rollback. Clean, safe, and atomic!”

Tom smiled, grabbed a marker, and pointed to the whiteboard:

“You want to buy distributed atomicity. Fair enough. But have you calculated what 2PC truly costs in production?”

The Truth About Two-Phase Commit (2PC): Not Forbidden, but Dangerously Expensive

“In architecture interviews,” Tom began, “people often parrot a blanket slogan: ‘2PC is strictly forbidden in microservices’. That is short-sighted. 2PC is a mathematically proven distributed consensus protocol, and PostgreSQL officially supports it via PREPARE TRANSACTION (PostgreSQL two-phase transactions). It solves a genuine problem: guaranteeing all-or-nothing atomicity across multiple independent database nodes.”

“Then why do hyperscale tech companies avoid it like the plague?” Ben asked.

“Because of the unbounded uncertainty window between Phase 1 (Prepare) and Phase 2 (Commit/Rollback)!”

Tom broke down the mechanics: When a transaction executes PREPARE TRANSACTION 'tx_123', the engine persists its transaction state to disk and continues holding all Row Locks and Table Locks indefinitely, even after the client disconnects! These locks remain held until an explicit COMMIT PREPARED or ROLLBACK PREPARED command arrives (PostgreSQL PREPARE TRANSACTION).

In WhyBit’s reproducible lab:

1. Transaction 1 executes an account balance deduction and issues PREPARE TRANSACTION. The client connection terminates.
2. The prepared transaction remains visible in the system catalog pg_prepared_xacts.
3. An incoming concurrent Transaction 2 attempts to update that exact account balance.
4. Transaction 2 is completely blocked, waiting for 560 ms until it hits its configured statement_timeout and aborts!
5. Only when the administrator manually steps in with ROLLBACK PREPARED is the lock released, allowing subsequent transactions to update the balance to 80.

“Local lock contention is bad enough,” Tom warned of the deeper danger: “Far more devastating: An unresolved prepared transaction stalls PostgreSQL's VACUUM worker! Postgres cannot reclaim dead tuples older than that transaction, causing severe database bloat, disk space exhaustion, and degrading query latency across the entire cluster!”

And if the central transaction coordinator crashes during the prepare phase, participating services become stranded in an 'in-doubt' state — unable to commit, unable to abort, keeping connection pools exhausted until the system crashes!

Sagas: Exchanging Atomic Commits for Independent Local Transactions

“I see...” Ben shuddered. “2PC forces distributed nodes to hold their breath together. If one node faints, all of them suffocate!”

“Precisely,” Tom nodded. “The Saga Pattern was born to break that dependency. Instead of attempting a massive distributed transaction, a Saga breaks the business workflow into a sequence of independent Local Transactions.”

Each service completes its assigned step and immediately commits locally:

• Order Service creates the pending order → Commit!
• Payment Service charges the account → Commit!
• Warehouse Service reserves the physical stock → Commit!

If an intermediate step fails (e.g., insufficient funds), the Saga triggers a cascade of Compensating Transactions that run backward to undo the side effects of prior steps (AWS Prescriptive Guidance: Saga patterns).

Tom underlined a fundamental principle:

“A compensating transaction in a Saga is not a physical rollback (Undo)! An email cannot be unsent; it requires an apology email. Money already charged cannot vanish; it requires a new Refund transaction. Stock already allocated requires an explicit Release action. Every compensation is a brand-new forward-facing business event with explicit semantics!”

The Out-of-Order Trap: When Compensation Arrives First

“Returning to your order #8842 incident,” Tom looked at Ben. “Why did the compensation arrive before the forward action?”

“Because distributed networks and message brokers are asynchronous!” Ben answered. “The Reserve command got caught in a partition rebalance, while the Release event took a fast path.”

Tom had Ben write his original naive handler logic on the board:

release(order): if state == RESERVED then transition CANCELED; else ignore
reserve(order): if state == NEW then transition RESERVED

Tom walked through the trace:

• When delivered out of order: release arrives first → No order found → Handler silently ignores it!
• Milliseconds later, reserve arrives → Finds a fresh order → Seamlessly marks it RESERVED!
• Final state: The order is stuck as RESERVED forever!

WhyBit’s lab reproduced this failure with mathematical certainty: Given the delivery sequence compensate → forward, the naive handler ended up in the illegal RESERVED state!

Resilient State Machines: Using Tombstones to Reject Late Actions

“So how do we teach the warehouse service that once an order has been canceled, it must never reserve stock, even if the reservation command arrives later?” Ben pondered.

“We design a Finite State Machine capable of recording Cancellation Intent even before the primary entity exists!” Tom explained. “In distributed computing, this sentinel marker is known as a Tombstone or Semantic Lock.”

Ben drew the revised state transition diagram:

NEW      --compensate--> CANCELED
RESERVED --compensate--> CANCELED
CANCELED --forward-----> CANCELED  (Late forward action REJECTED!)
CANCELED --compensate--> CANCELED  (Duplicate compensation is idempotent)

The mechanics are elegant:

1. When the compensating ReleaseInventory event arrives early: The handler sees no existing order. Instead of ignoring it, it immediately creates an order entry in the CANCELED state (erecting a Tombstone)!
2. When the late ReserveInventory event finally arrives: The handler inspects the database, sees the state is already CANCELED, and flatly rejects the reservation request, keeping the state at CANCELED!
3. If network retries deliver duplicate ReleaseInventory events: The state is already CANCELED, so the handler safely treats the call as idempotent.

The lab verified this behavior: Given the exact same delivery sequence compensate → forward, the tombstone-equipped state machine finished safely at CANCELED! Warehouse inventory was protected.

Ben’s Production-Grade Saga Orchestrator

Tom stepped back, handing the marker to Ben. Ben sketched out the full architectural blueprint for Fortex's Saga Orchestrator:

               +----------------------------------+
               |        Saga Orchestrator         |
               | (State Machine + Versioning CAS) |
               +-----------------+----------------+
                                 |
        +------------------------+------------------------+
        |                                                 |
        v (Command 1)                                     v (Command 2)
+---------------+                                 +---------------+
|    Payment    |                                 |   Warehouse   |
|  Local Xact   |                                 |  Local Xact   |
+---------------+                                 +---------------+

The design rests on seven solid pillars:

1. Durable State Table saga_instance(id, state, version, deadline): State is persisted to a relational database, never held solely in ephemeral RAM.
2. Optimistic Concurrency Control (Compare-And-Set): All state mutations verify version invariants: WHERE id = $id AND version = $current_version.
3. Idempotency Guard processed_messages(message_id): Drops duplicate deliveries from message brokers.
4. Transactional Outbox: Enqueues the next command into an outbox table within the same local transaction as the state transition.
5. Durable Timeouts: Deadlines are stored in the database. A background worker periodically scans for expired sagas to trigger compensations, surviving application restarts.
6. Automated Reconciliation Jobs: Regularly cross-checks state consistency across Payment, Order, and Warehouse systems.
7. Manual Review Escalation: When compensations repeatedly fail (e.g., customer account frozen), the saga transitions to MANUAL_REVIEW for human operator intervention.

Lab Verification Evidence

All phenomena and resilience patterns were verified against an isolated PostgreSQL 17.6 container instance:

This report image is generated from run 38f739f3-429d-4c27-8ff9-94754f9109eb: one prepared transaction retains its lock, a competing update times out after 560 ms, the naive handler ends at RESERVED, and the guarded state machine ends at CANCELED.

Report JSON · Raw log · Lab Source Code · Run 38f739f3-429d-4c27-8ff9-94754f9109eb

Scope & Limitations: The lab uses a single PostgreSQL instance to isolate locking behavior and state transition logic. It proves lock retention across disconnected sessions and validates tombstone state machine convergence; it does not simulate multi-broker partitions or cluster-wide network splits.

In that afternoon's Architecture Review meeting, Ben no longer opened with trendy buzzwords or generic debates about whether 2PC or Saga was superior.

He stood before the engineering leadership and defined the core invariant:

“Once a customer order has been subjected to a compensating transaction, it must never revert to locking stock or capturing funds simply because a delayed network message arrived out of sequence!”

He then systematically presented 2PC, Choreographed Sagas, and Orchestrated Sagas as three distinct engineering trade-offs, laying bare their latency costs, resource requirements, and operational overheads.

Tom sat at the back of the room, smiling quietly. When Hà turned to ask who would lead the construction of the cross-service failure matrix for Fortex's order platform, Tom proudly pointed to Ben.

Fortex and dialogue are fictional narrative devices. All measurements and technical logs are from the linked reproducible lab run.

Previous: Chapter 15 — Kafka Hot Keys