Exactly-once processing: how do we prevent duplicate business operations?

Fortex is fictional. The logs, measurements, and results below come from the attached Docker Lab. They are not production data from a real company.

One payment, two ledger entries

At 08:17, Hà placed her laptop on Ben's desk. A merchant reconciliation showed payment-401 twice, for 590,000 VND each. Kafka contained one record.

“I commit the offset after writing to the database,” Ben said. The argument sounded solid: finish the work first, acknowledge it second, and no data should be lost.

Tom listened before asking, “How many lines of code sit between those two actions?”

Only a few. Yet the PostgreSQL COMMIT and Kafka offset commit define a failure window. If the process dies there, the money is already recorded while the broker still considers the message unprocessed. A new consumer receives the same record and repeats the business operation.

What does Kafka guarantee?

Committing an offset after processing gives a consumer at-least-once delivery behavior: a record can return, but it is not lost because it was acknowledged too early. Kafka's design documentation describes Kafka exactly-once processing as a transaction that includes output records and consumed offsets. That boundary is powerful when both input and output stay in Kafka.

It does not automatically enroll a PostgreSQL INSERT, payment gateway call, email, or HTTP request in the Kafka transaction.

Tom wrote two boundaries on the board:

1. Exactly-once delivery: assume the broker delivers only once. That is unsafe around crashes, timeouts, and rebalances.
2. Exactly-once business effect: delivery may repeat, but business state changes once.

Fortex needed the second boundary.

The missing part of Ben's answer

Ben suggested reversing the order: commit the offset first, then write the database. Tom asked what happened if the process died after the offset commit. Ben saw it immediately: the message would not return, but the ledger entry would be missing. Reordering merely exchanges duplication for data loss.

For a database effect, a transactional inbox provides a useful local boundary:

BEGIN;

INSERT INTO consumer_inbox(event_id)
VALUES ('payment-401')
ON CONFLICT DO NOTHING
RETURNING event_id;

-- Run only if the INSERT above created a row.
INSERT INTO ledger(event_id, amount)
VALUES ('payment-401', 590000);

COMMIT;

The unique constraint on event_id lets the database make one promise: the marker and ledger effect commit together, or neither does. The consumer commits its Kafka offset afterward. Following a crash, Kafka replays the record; the second insert conflicts, so the effect is skipped.

The producer must preserve a stable event ID across retries. Generating a fresh UUID for each retry prevents the consumer from recognizing the same operation.

Lab: crash inside the failure window

The lab uses Kafka 4.2.2, PostgreSQL 17.6, and confluent-kafka 2.15.1. It consumes one payment, writes the ledger, and exits before committing the offset. A consumer in the same group starts again and receives that record.

The consumer ran twice in both scenarios. Correctness lives at the sink: the inbox marker and effect share one transaction.

• Report JSON
• Raw log
• Docker Lab source
• Run ID: c3ee7d90-0c64-47c5-a907-34f04b2db2f9

Alternatives and their costs

For pipelines that read Kafka and write only to Kafka, Kafka transactions with read_committed can atomically commit output records and input offsets. An external database still remains outside that transaction.

When an external API accepts an idempotency key, pass the event ID and let the effect owner retain the first result. If it does not, the design may need a state machine, reconciliation, or an outbox at that owner. An inbox in service A cannot prevent service B from repeating an effect unless B understands the same identity.

Inbox tables also cost storage, index maintenance, retention work, and sometimes contention. Keep markers longer than every realistic replay and recovery horizon; an aggressive TTL reopens the duplicate window.

Tom left Ben a checklist: stable IDs across retries, atomic marker and effect, offset commit after the database transaction, idempotency at every external effect, sufficient inbox retention, and reconciliation for missing or extra effects.

Ben renamed the consumer from exactly-once-payment-consumer to idempotent-payment-handler.

“Less impressive,” Tom said after reading the commit. “More accurate.”

The next incident begins while Fortex copies millions of products into a search index as merchants continue changing prices. Unless the snapshot and WAL stream meet at the same boundary, a change can disappear between them.