A timeout is not a failed operation: making retries safe

Fictional setting, real lab evidence. Fortex in this series is a fictional setting; Tom and Ben are characters. Their conversations illustrate the reasoning. Measurements and assertions come from the linked Docker lab, not Fortex production incidents.

The Fortex journey · Chapter 8/8

This time Ben takes the marker

Fortex wanted users to retry a credit operation after an unreliable connection. Ben wrote retry on the board, then underlined same in “send the same request again.”

“I used to treat a timeout as a failure. But if the database already committed, a retry can credit twice. We need a key.”

Tom sat down. “Walk me through that key's lifetime.”

“The client creates a UUID when it sends the request.”

“And when the user presses retry?”

Ben looked at what he'd written. “If that path creates another UUID, the server sees new work.” He corrected the diagram and added two branches: the same intent and a new intent. He was still talking quickly, but now he was making the distinction himself.

Tom didn't need to provide the answer. He kept asking about boundaries Ben was about to take for granted.

A timeout describes what the client knows

Ben began by separating what the server had done from what the client knew. He reused the outbox question—where is the gap?—but this time it lay between commit and the caller receiving the result.

A client asks to add 100 credits. The server writes a ledger entry and commits, but the connection fails before the response arrives. The user sees a timeout and retries.

If the second transmission is treated as a new operation, the account receives 200 credits. Both database transactions are individually valid. The missing concept is that the two transmissions represent one intent.

A timeout does not prove failure. It means the waiting party did not receive a result within its deadline. Safe retries require a way to recognize an intent that has already been processed.

Keep one key across retries of the same intent

“Which account owns the key?” Tom asked.

Ben remembered RLS. “Use authenticated identity. The same key string in two accounts mustn't collide, and replay still needs authorization.” He added scope before defining the table.

Generate a random key when the intent begins, retain it for retries, and generate a new key for a genuinely new operation. Creating a fresh UUID on every retry prevents the server from recognizing duplication.

The server scopes the key to an authenticated identity and operation, and stores a meaningful input fingerprint with the result. A real API might use (account_id, operation, idempotency_key) as its unique key. The lab has one operation type, so it uses (account_id, key).

CREATE TABLE requests (
  account_id int NOT NULL,
  key text NOT NULL,
  amount int NOT NULL,
  result text NOT NULL,
  PRIMARY KEY (account_id, key)
);

The unique constraint arbitrates competing claims. A preliminary SELECT followed by INSERT still has a race. PostgreSQL's INSERT ... ON CONFLICT provides a database-enforced claim operation.

Commit the deduplication record and effect together

Tom drew a marker and a ledger as separate boxes. Ben enclosed them in one transaction, as he had with the inbox and projection.

“Save the key first, then fail before the effect, and the retry gets wrongly blocked. Apply the effect first without saving the key, and it can happen twice.” He found both gaps without Tom placing the cursor for him.

BEGIN;
WITH accepted AS (
  INSERT INTO requests VALUES (1, 'req-42', 100, 'accepted:100')
  ON CONFLICT DO NOTHING
  RETURNING account_id
)
INSERT INTO ledger(account_id, amount)
SELECT account_id, 100 FROM accepted;

SELECT result FROM requests
WHERE account_id = 1 AND key = 'req-42' AND amount = 100;
COMMIT;

A new key inserts both the request record and ledger entry. An existing key produces no accepted row, so no new ledger entry is written; the following statement retrieves the saved result. A rollback removes both the claim and effect.

The SELECT is a subsequent statement rather than part of one statement snapshot. Concurrent conflicts on a unique key require attention to isolation, waiting, and visibility. This lab tests sequential retries after a discarded response, not concurrent HTTP stress.

The fixture uses a fixed result and positive integer credit amount. A product also needs input validation, authorization, units, and numeric bounds. An account ID supplied in a request body is not authenticated identity.

The same key with a different input is a conflict

“Same key, different amount?” Tom asked.

Ben didn't choose to replay the old result just to finish the request. “That needs a conflict defined by the API contract. Treating different input as the same work can violate the caller's intent.” They reviewed which fields belonged in the fingerprint.

If the original amount is 100 and a later request uses the same key for 999, those are not equivalent requests. The lab compares the amount, performs no new ledger write, and returns no matching saved result. An HTTP implementation should turn that case into a documented validation or conflict response rather than an empty success.

For richer inputs, the fingerprint must cover the business-relevant fields using stable canonicalization. Raw JSON equality can reject equivalent payloads because of field order. Omitting currency or destination can incorrectly merge different operations.

Stripe API v1 documents saved status and response replay for idempotency keys. That is one concrete API contract, not a universal requirement that every system cache every failure identically. Define which states are retained, for how long, and how clients should retry them.

The Docker experiment

Ben proposed the cases: no deduplication, repeated key, changed amount, and the same string in another account. Tom helped build and inspect the lab.

Before writing the conclusion, Ben stated its limit himself: the driver discards the first result and retries sequentially. It isn't evidence of a real HTTP connection failure or a concurrent key-contention stress test.

PostgreSQL 17.6 runs in an isolated container. The unprotected branch inserts two ledger entries of 100. The protected branch submits req-42 twice, deliberately discarding the first result before retrying. This models response loss; there is no running HTTP service or physically interrupted network in the experiment.

Both protected calls return accepted:100, while account 1 has one ledger row. Reusing the key with amount 999 leaves its total unchanged. Account 2 can use the same key string without sharing account 1's result. The report and raw logs contain these assertions; download the source below to repeat them.

Boundaries that still need a design

Tom suggested shortening the design: “The key and unique constraint cover the main idea?”

“The main idea, but not the contract,” Ben replied. “When do keys expire? What about late retries? A provider-side effect isn't inside our transaction.”

Tom pulled the decision table closer. “You're right. Keep those questions. Go on.”

When a transaction must lead to an event, a transactional outbox preserves the publication intent, while consumers still need duplicate handling. When distinct requests update the same stock, lost-update protection protects state. These mechanisms address complementary concerns: identifying one intent, preserving valid updates, and forwarding work across systems.

An idempotency header is only the visible part of the design. The guarantee comes from a precise intent definition, durable arbitration, and a commit boundary that covers the effect being protected.

The page Ben takes into tomorrow

Ben hadn't become an engineer who never made mistakes. He still liked a compact solution and wanted to ship quickly. But the design he sent to the Fortex team now included its conditions, failure windows, lab evidence, and untested cases.

Tom read it without rewriting it for him. He left Ben's name as the presenter.

The last notebook page of this stretch didn't name a new pattern: “Before choosing a tool, ask what we're protecting, at which boundary, and what evidence would change our mind.”

Previous chapter: A missing tenant filter: can PostgreSQL RLS stop the leak? · Start the journey