Event time and processing time: which clock does the system trust?

Fortex is fictional. The two records and results below were produced in the Docker Lab. They prove one controlled schedule, not production latency.

A paid order becomes newly created

Hà showed Ben the operations page. Order F-882 was PAID, then moved back to CREATED seconds later. No message was missing; both were in Kafka.

Ben opened the consumer. The projection simply accepted the last record it processed. “But a partition preserves order.”

Tom asked where the events originated. CREATED came from an order service retrying after congestion. PAID came from the payment service and reached Kafka first. Kafka preserved the order appended to the partition. It could not infer the causal order of two producers.

Separate the three clocks

• Event time is when the event occurred according to its producer or domain.
• Ingestion time is when infrastructure received it.
• Processing time is when an operator handled it.

Processing time moves with queues, retries, pauses, rebalances, and backpressure. Event time is closer to the business event, but clocks can drift and producers can send bad timestamps.

Kafka Streams core concepts distinguish time semantics and timestamp extraction. Carrying a timestamp does not solve ordering by itself; the application must decide how older records are rejected, applied, or compensated.

Lab: two records, three policies

The lab sends records through Kafka in this order:

1. PAID, event time 10 seconds, domain version 2.
2. CREATED, event time 5 seconds, domain version 1.

• Report JSON
• Raw log
• Docker Lab source
• Run ID: f06e3fcf-3e43-47c7-8cea-05afd7b6af3d

The lab does not measure throughput. It deliberately reverses arrival and business order to test one invariant: state must not move from version 2 back to version 1.

Why is a version stronger here?

When an aggregate owner emits a monotonically increasing sequence, the sink can update conditionally:

UPDATE order_projection
SET state = :state, version = :version
WHERE order_id = :order_id
  AND version < :version;

A clock drifting by 30 seconds cannot break that condition. The version represents causal order for one aggregate. It does not establish a useful total order between unrelated orders.

If several services can mutate the same state, use an explicit state machine or choose one owner to publish normalized domain events. Taking the largest timestamp from two machines exchanges a race condition for a clock condition.

When no version exists

Event time remains useful for analytics, windows, and offline devices. Include an event ID, a precisely defined occurred_at, source and schema version, aggregate sequence when possible, ingestion timestamp, and a policy for stale events.

With timestamps alone, (occurred_at, event_id) offers a stable tie-breaker. Send unresolved conflicts to reconciliation rather than silently calling the latest arrival true.

Kafka log order is valuable when every transition for an order uses the same key and producer path. Retries from another outbox, historical imports, CDC, multiple producers, and repartitioning can still break business-time order.

Tom asked Ben to write the invariant first: for one order_id, only valid transitions are accepted and the version must increase. They added a stale_event_rejected_total metric, logged incoming and current versions, and retained rejected events for reconciliation.

Waiting in an event-time reorder buffer reduces disorder but increases latency and state. A version guard responds immediately but requires a trustworthy domain owner. A state machine catches invalid transitions but complicates migrations. Reconciliation repairs eventual mistakes while users may briefly see them.

Ben closed the incident with “arrival order was treated as causal order.” Tom changed one word: it was assumed.

For Fortex's revenue dashboard the next question was harder: how long should the system wait for late events before finalizing a number?