Kafka hot keys: one merchant can congest an entire partition

Fictional setting, verified lab evidence. Fortex in this narrative is fictional; Tom, Ben, and Hà are characters illustrating engineering reasoning. All metrics, partition end-offsets, and assertions come from the linked reproducible Docker lab, not Fortex production systems.

A Journey at Fortex · Chapter 15

Hà Sends an Alert: One Partition Is Absorbing Nearly All Traffic

At 09:00 AM sharp on Double 11 Super Sale day, the atmosphere in Fortex’s engineering war room was electric.

Hà, the Engineering Manager, pushed open the door, looking serious. An alert on the order-processing topic showed traffic heavily skewed toward one partition. This is a fictional Fortex incident; the article does not present a production screenshot or attach unmeasured CPU and lag values to it.

Hà tapped the table: “What’s worse is that it’s not just the big brand experiencing delays. Small boutique merchants are flooding customer support because their shoppers haven’t received order confirmation emails for over half an hour!”

Ben quickly pulled up the Kafka Producer configuration. He breathed a sigh of relief and turned to Hà:

“Hà, I checked our code — the partition key logic follows standard textbook best practices! Our business requires that all orders for each merchant be processed strictly in chronological order, so the producer sets the message key to merchant_id. This morning, a premier VIP cosmetic brand launched an exclusive flash campaign, generating 75% of our entire platform's order volume. Sixteen partitions are simply too few for this burst. With your approval, I’ll immediately increase the topic partitions from 16 to 64 and attach 48 new consumers to spread the load!”

Before Hà could answer, Tom pulled his chair forward and gently rested a hand on Ben’s shoulder:

“You want to widen the highway from 16 lanes to 64 lanes. But do you know which lane that massive shipping container is driving on?”

Sixty-Four Partitions, Still Exactly One Blazing Hotspot

Ben blinked in confusion: “Wait Tom, if we scale up to 64 partitions, won’t Kafka automatically distribute the traffic evenly across all 64?”

“That is true only when messages have no key,” Tom explained with a smile. “When messages carry a key, what is the general partitioning model?”

Ben remembered:

Here, p is the partition index, k is the serialized key bytes, N is the current partition count, and H is the hash selected by the client partitioner.

For Kafka's default Java producer, the concrete form is commonly written as:

This article's lab uses confluent-kafka 2.15.1 on librdkafka. Its default consistent_random partitioner uses CRC32 for nonempty keys; murmur2 is a separate configuration choice. The lab therefore verifies deterministic same-key concentration rather than a Murmur2-specific output (librdkafka configuration).

Tom sketched on the whiteboard: “Whether a client uses CRC32, Murmur2, or another compatible partitioner, the important property is determinism under one configuration. The same merchant_id = 'vip' key maps to one partition while the partition count remains unchanged.”

“Which means...” Ben’s jaw dropped.

“Which means whether you configure 16 partitions, 64 partitions, or 1,000 partitions, that entire 75% wave of orders carrying the vip key will continue to be hashed into the exact same single partition!” Tom emphasized.

And the bottleneck deepens: By core design, within a Kafka Consumer Group, each partition is assigned to exactly one consumer thread at any given time to maintain strict ordering semantics (Kafka design protocol).

“Even if you spin up 48 shiny new consumers, they will sit completely idle watching that single hot partition suffocate! Not a single one is permitted to jump in and share the burden of that specific partition!”

WhyBit’s reproducible lab demonstrated this physical reality by producing exactly 10,000 records (7,500 belonging to the VIP merchant, and 2,500 belonging to smaller merchants distributed across the catalog):

This dashboard is rendered directly from the report.json for run 85c64e3d-f809-46ab-86d5-ef731f559b24; it is not a Fortex production screenshot. The lab measures partition end-offsets, not consumer CPU, consumer lag, or P99 latency.

Reviewing the data, Ben was stunned: Increasing the partition count from 16 to 64 merely reduced the hottest partition from 7,661 down to 7,538 (the negligible difference was just small merchants who happened to hash elsewhere). The 7,500 VIP orders remained an indivisible boulder crushing a single partition!

Kafka's producer documentation further warns that scaling partitions on an active topic alters hash modulo mappings for new records, which can fracture ordering across the migration boundary (Kafka producer configs).

Ben summarized the classic Staff engineer interview takeaway in his notebook:

“Increasing partition count scales parallelism across different keys. It is completely powerless to scale parallelism within a single hot key!”

Key Salting: Trading Strict Ordering for High Throughput

“So how do we break apart that burning boulder?” Ben asked urgently.

“The most intuitive approach: If a single key is too massive, fracture it into multiple subkeys!” Tom suggested. “In distributed systems, this is known as Key Salting.”

Ben jumped into code: For regular merchants, he retained key = merchant_id. But specifically for the VIP merchant, he appended a salt derived from the order ID:

const salt = order.id % 4;
const messageKey = isVip(merchantId) 
  ? `${merchantId}:${salt}` 
  : merchantId;

The lab verification showed immediate, dramatic improvement:

• The VIP volume was evenly dispersed across 4 separate partitions.
• The peak load on a single partition plummeted from 7,661 down to 2,036 records!
• Record distribution across partitions became flatter. The lab did not run consumers, so it does not claim a measured reduction in CPU or consumer lag.

Ben cheered: “It worked, Tom! Peak load dropped nearly 4x!”

Tom paused, then asked the crucial question:

“Are Order #101 and Order #102 from that same VIP merchant still guaranteed to land on the same partition and reach downstream consumers in strict chronological order?”

Ben froze. He mentally traced the math: If Order #101 has salt = 1 landing on Partition 2, while Order #102 has salt = 2 landing on Partition 5, two independent consumer threads will process them asynchronously! Under network jitter, Order #102 could easily be processed before Order #101!

“Salting preserves ordering only within each salt bucket; it completely relinquishes global ordering across the entire merchant,” Tom analyzed. “That is the fundamental trade-off: You buy throughput at the expense of strict ordering. The real question is: What ordering granularity does Fortex's business actually require?”

Defining the Ordering Boundary: Who Truly Needs Total Ordering?

Hà joined them at the whiteboard and nodded:

“We need to look at business semantics. Product initially requested 'everything must be sequential', but digging into the workflows:

1. Checkout & Order Lifecycle: A customer only cares that events for their specific order (Created → Paid → Shipped) occur sequentially. Does Order A from Alice need to precede Order B from Bob at the same store? Absolutely not!
2. Financial Ledger & Quotas: This is the only domain requiring strict, serialized ordering of credits and debits to prevent overdrafts.”

Ben breathed a sigh of relief: “Then the boundary is straightforward!”

Tom and Ben mapped out five clear architectural strategies:

1. Partition by order_id: Ideal for checkout and fulfillment. Every event for a given order is guaranteed strict partition ordering; separate orders process 100% in parallel. The hot-key problem disappears!
2. Bounded Key Salting: Used when grouping by merchant is mandatory, but downstream consumers can re-aggregate or tolerate out-of-order execution.
3. Dedicated Topic for VIP Merchants (Fast Lanes): Route massive tenants to a dedicated topic (orders.vip).
4. Custom Partitioner: Implement custom hashing logic on producers, though this requires consistent cross-language client libraries.
5. Separate Commands from Analytics Events: Route business commands requiring sequence numbers through narrow ordered logs, while fanning out notifications and telemetry across keyless topics.

Head-of-Line Blocking and the Value of Dedicated Topics

Hà followed up: “And why were the smaller, non-sale merchants impacted?”

“That is classic Head-of-Line Blocking,” Tom explained. “When small merchants share a finite set of partitions, some may hash into the same partition as the mega-merchant. Their records then wait behind the hot key's backlog on that partition.”

Isolating high-volume tenants into Dedicated Topics solves this cleanly:

• Standard shops publish to orders.standard with standard retention and autoscaling consumer pools.
• VIP merchants publish to orders.vip with higher partition counts and dedicated, beefy consumer instances.
• The blast radius becomes smaller: the VIP backlog no longer shares a topic and consumer group with standard traffic. Real P99 still depends on the broker, consumer, and downstream services, so it needs a separate benchmark before an SLO is claimed.

Monitoring Kafka Lag Like a Senior SRE: Don’t Be Fooled by Total Lag!

Ben opened Grafana to overhaul their alerting dashboards. Tom imparted a hard-won operational lesson:

“Many on-call engineers see Total Consumer Lag = 10,000 and assume everything is fine because 10,000 divided by 20 partitions is just 500 messages per partition. But in reality, 19 partitions might have zero lag, while one single partition carries all 10,000 unread messages! That single hot partition is about to fall over!”

To detect data skew and hot keys early, production observability requires:

1. Per-Partition Lag Dashboards: Graph individual partition lag metrics rather than aggregate sums.
2. Max/Median Lag Ratio: Divide the maximum partition lag by the median lag. If this ratio exceeds 5x or 10x, severe data skew is actively occurring!
3. Key Class Throughput: Measure record rates and consumer processing times per tenant hash bucket.
4. Check Downstream Saturation: As demonstrated in Chapter 10 — Kafka Consumer Lag is 0, Yet Downstream Data is Broken, if downstream databases are bottlenecked on connection pools, adding partitions or salting only moves the congestion from Kafka to PostgreSQL!

Two useful dashboard measures can be standardized as:

Here, nᵢ is the record count or lag for partition i; Cmax measures concentration on the hottest partition, and S measures skew relative to the median partition. In the lab's 16-partition run, Cmax = 7,661 / 10,000 = 76.61% and S = 49.1.

Lab Verification Evidence

All distribution assertions were empirically verified on an official Apache Kafka 4.2.2 container deployment:

Report JSON · Raw log · Lab Source Code · Run 85c64e3d-f809-46ab-86d5-ef731f559b24

Scope & Limitations: The experiment measures actual broker partition end-offsets after acks=all producer flushes. It isolates Kafka's partition distribution algorithm; it does not simulate consumer CPU saturation benchmarks or multi-broker network partitions.

Ben updated Fortex’s operations runbook. He crossed out the naive line: “If lag spikes → increase partitions”, replacing it with an actionable diagnostic procedure:

“When lag spikes: Check the Per-Partition Lag graph to spot Hot Partitions; evaluate the Max/Median Lag ratio; check for Hot Keys; then decide: Shift the ordering boundary to Order ID, apply Key Salting, or isolate the tenant onto a Dedicated Topic.”

He submitted the design RFC to Hà, detailing every architectural trade-off: which option guarantees total ordering, and which maximizes throughput.

Tom reviewed the document and added a final note:

“This is the first time you didn't ask how many partitions the cluster had before asking what ordering boundary the business truly needed.”

Fortex and dialogue are fictional narrative devices. All measurements and technical logs are from the linked reproducible lab run.

Previous: Chapter 14 — Cache Invalidation and CDC · Next: Chapter 16 — 2PC, Saga, and Early Compensation