Windowing and late events: how long should a window remain open?

Fortex is fictional. The chart below is generated from an isolated lab report and labeled accordingly. It is neither a production dashboard nor a Kafka Streams benchmark.

Both 30 and 35 can be “correct”

Hà asked for revenue from second 0 through second 10. Ben's dashboard returned 30. A minute later it changed to 35. The daily report was right, but the realtime alert had missed its threshold.

An order with event time 4 seconds arrived after the record at 8 seconds. It belonged to the old window even though the consumer saw it later. Ben wanted to keep windows open forever so nothing would be lost.

Tom asked, “If an offline device uploads after three months, may today's dashboard still change?”

Windowing is a decision among completeness, latency, and retained state.

How a tumbling window divides time

For window size Δ, event time t belongs to the window beginning at:

$$windowStart(t)=\left\lfloor\frac{t}{\Delta}\right\rfloor\Delta$$

The lower bound is normally inclusive and the upper bound exclusive: [start, start + Δ). An event at exactly 10 seconds belongs to [10,20), not [0,10).

The Kafka Streams DSL documentation describes time windows, grace periods, and retention. Grace controls how long an event can arrive after stream time has passed a window end; retention must keep the state available for that period.

A watermark is a policy statement

This lab uses a small deterministic rule:

$$watermark=\max(eventTimeSeen)-grace$$

A record older than the watermark is too late. Real systems may calculate watermarks per partition and account for idle inputs. A fast partition can close a slow partition's window prematurely when a global maximum is used carelessly.

A watermark says that the system expects most earlier data to have arrived and accepts a defined policy for the rest. It cannot prove that no old event remains in the network.

Lab: five records, two policies

The lab sends five records through Kafka in arrival order a, b, c, d, e:

Record	Event time	Value	Meaning
a	2s	10	on time
b	8s	20	on time
c	4s	5	late but within grace
d	16s	30	advances the watermark
e	6s	7	arrives beyond the watermark

With zero grace, c and e are late and [0,10) totals 30. With five seconds of grace, c is accepted and the total becomes 35. After the 16-second event, the watermark is 11 seconds; e at 6 seconds is routed as too late.

• Report JSON
• Raw log
• Docker Lab source
• Run ID: ca5ef117-03d9-4238-b780-fef8583d7364

The consumer implements the documented semantics deterministically over records transported by Kafka. It does not use a Kafka Streams state store or RocksDB, so the report supports no throughput or memory claim about Kafka Streams.

Where do too-late events go?

“Drop” should not mean silent disappearance. Depending on the business, a record can enter a side output for audit and replay, update a corrections table, emit a revision, affect only a final report, or trigger reconciliation above a value threshold.

A dashboard should label numbers as preliminary or final. If a downstream system has already issued an invoice, correcting an old aggregate may require a compensating event instead of a silent row update.

Choosing grace from evidence

Do not choose five minutes because it sounds reasonable. Measure lateness:

$$lateness=processingTime-eventTime$$

Segment the distribution by source, partition, network, and device type. Longer grace improves completeness but retains more state, increases disk and changelog traffic, and leaves results mutable for longer. Shorter grace finalizes quickly but produces more corrections.

Production questions include p50/p95/p99 and tail lateness, idle-partition behavior, retention greater than window plus grace, storage and replay of late records, downstream support for updates or retractions, and how the UI communicates preliminary results.

Not every problem needs event-time windows. Rate limiting often cares about gateway processing time. Billing and fraud analytics often need event time and correction. Some SLAs begin at ingestion. Choose the clock from the business question.

Ben kept five seconds of grace for the lab dashboard and routed later records to corrections. More importantly, he labeled every open window “provisional.”

Hà looked at 35 and asked, “When can this number stop changing?”

This time Ben pointed to the watermark, grace, and correction policy. Tom stayed quiet. Ben had moved from writing an operator to defining what its result meant.