Ten Kafka use cases, one log

2026-08-10

Every “top ten Kafka use cases” list has the same problem: it reads like ten features, and Kafka doesn’t have ten features. It has one data structure — a partitioned, replicated, append-only log whose readers keep their own position — and the ten use cases are that one structure looked at from ten angles.

That isn’t a pedantic point. It’s the difference between someone who can operate Kafka and someone who is surprised by it. Once you can see the log, you can predict what Kafka will do in a situation nobody wrote a use case for, and you can tell instantly which of the ten you’re actually doing — because the only things that vary between them are:

1. Who writes, and who reads — one consumer group or several.
2. How long you keep it — hours, forever, or “only the latest value per key”.
3. How far back readers go — the head, or the beginning.

That’s the whole list. Everything else is naming.

The mechanism

A topic is split into partitions. A partition is an ordered, immutable sequence of records that you can only append to; each record gets a monotonically increasing offset. Consumers are not sent records — they ask for them, from a position they control.

The consequence that surprises people coming from message queues: reading does not consume. A record isn’t deleted because someone read it. It’s deleted when it ages out of retention, and that clock runs whether anyone read it or not.

Three properties fall out of that picture, and between them they explain all ten use cases:

• Replay is free. An offset is just a number. Set it backwards and you re-read. This isn’t a feature bolted on for disaster recovery; it’s what a log is.
• Fan-out is nearly free. Adding a second consumer group adds reads, not another copy of the data, and the producer neither knows nor cares how many groups exist. The asterisks: every group writes its committed offsets to an internal topic, which is trivial per group and a real load at thousands of them, and a group reading near the head is served from page cache while a group replaying from the beginning goes to disk and competes with producers for I/O.
• Slow consumers don’t back up the producer. A lagging group is behind on its own offset. It doesn’t block anyone else, and it doesn’t stop the writer — until it falls out of the retention window and starts losing data, which is why consumer lag is the metric you alert on.

Why reading doesn’t consume

This is the single most important difference from a traditional message queue, and it’s worth drawing, because almost every misunderstanding of Kafka comes from importing queue instincts.

Partitions, keys and the limits of ordering

The other half of the mechanism, and the source of the second-most-common surprise: Kafka does not give you a globally ordered topic. Order exists within a partition, and nowhere else.

Records are assigned to partitions by hashing the record key: same key → same partition → guaranteed relative order.

Records with no key don’t round-robin, whatever older write-ups say. The modern producer is sticky — it fills one partition with a batch before moving to another — and by default it also biases toward brokers that are answering faster. Ordering between unkeyed records was never something you could rely on either way, but the practical surprise is different from the one people expect: a low-volume producer can effectively pin itself to a single partition for long stretches.

One consequence of that diagram deserves more alarm than it usually gets. The mapping is hash(key) % partition_count, so changing the partition count re-maps every existing key. Add partitions to a keyed topic and records for a given key start landing in a partition holding none of that key’s history, with no ordering relationship to it. Growing a keyed topic isn’t a routine scaling operation — it’s a change to your data model, and the usual answer is to over-provision partitions at creation or to write to a new topic and migrate.

The ten, collapsed

Now the list. Grouped by what actually differs, which turns ten into five — plus one entry that isn’t a use case at all.

Moving events between services — “async messaging” and “pub-sub fan-out”. These are the same topic with a different number of consumer groups. One group: a producer hands work to a consumer without either knowing the other’s address or uptime, and a burst gets absorbed by the log instead of crushing the reader. Add a second and third group and the same order.placed record drives billing, email and analytics without the checkout service being changed, redeployed, or even informed. The reason “fan-out” doesn’t need its own machinery is that reads never consumed anything in the first place.

Collecting high-volume streams — “activity tracking”, “log aggregation” and “metrics”. Identical shape, different producer: a web app emitting clicks, a fleet of services emitting log lines, a fleet of hosts emitting gauges. Many writers, one topic, short retention, consumers near the head. Kafka is the standard answer here because its write path is close to the cheapest thing a machine can do. A partition is a directory of segment files, and the broker only ever appends to the newest one; batches arrive pre-grouped and often pre-compressed; the OS page cache serves any consumer near the head without touching a disk; and the send path can move bytes from page cache to socket without copying them through the application. Two caveats worth carrying, though: that zero-copy path is lost once TLS is in play, which is most production clusters, and the sequential-versus-random gap that made this a landslide on spinning disks is much narrower on NVMe.

Transforming in flight — “stream processing”. Read a topic, do something, write another topic. The output is a topic like any other, which is why these compose into pipelines. The important split here is stateless versus stateful, and it gets its own table below, because it decides whether your job can be restarted anywhere or has to rebuild memory first.

Treating the log as the truth — “event sourcing” and “change data capture”. These are the same idea from opposite ends. In event sourcing you decide the log is the system of record and your database is a projection you can throw away and rebuild by replaying. In CDC you already have a database, and you tail its commit log so that every change becomes an event — which is how you keep a cache, a search index and a warehouse in sync without dual writes. Dual writes are the bug this pattern exists to kill: write to the database and publish an event as two separate operations and eventually one succeeds while the other fails, leaving two systems that disagree with no record of which is right. CDC dodges it because there is only ever one write — the event is derived from the same commit log that made the write durable. The other standard escape is the transactional outbox: insert the event into an outbox table inside the same database transaction as the change, then ship that table with CDC or a poller. It costs you a table and a relay, and buys you events shaped like your domain rather than like your schema, which is usually what downstream consumers actually wanted.

Getting data into other systems — “data pipelines”. Source connectors pull from databases and SaaS APIs into topics; sink connectors push into warehouses, lakes and search indexes. The value is that N sources and M destinations wire up as N + M connectors against one log, instead of N × M bespoke integrations.

And “replay & recovery”, which is not a use case. It’s the property the other nine are built on. Replay is what makes a consumer crash survivable, a new consumer group possible, a reprocessing after a bug fix routine, and event sourcing coherent at all. Listing it as a tenth use case is like listing “having a filesystem” as a use case of a database. It belongs at the top of the article, not the bottom of the list — which is why it’s the first thing this one drew.

The ten mapped to the three knobs

↔ scroll the table sideways to see every column.

The bottom three rows are where people get hurt. Event sourcing with a seven-day retention isn’t event sourcing — it’s a queue that will quietly amputate your history. CDC without compaction grows without bound; CDC with compaction can’t be used as a full audit trail, because compaction is precisely the act of forgetting every value but the latest.

Retention: the knob that decides what you’ve actually built

↔ scroll the table sideways to see every column.

Three things worth knowing about compaction before you rely on it.

Deletion is expressed by writing a tombstone — a record with the key and a null value — and consumers must understand that convention or they’ll treat a delete as a corrupt record. Compaction is a background process, not a statement about what’s in the log right now: duplicate values for a key survive until the cleaner reaches that segment, so consumers have to tolerate seeing them.

And the one that quietly breaks projections: tombstones don’t live forever either. They’re removed by the cleaner after their own retention window — a day, by default. A consumer that bootstraps from the beginning more than that long after a key was deleted will never see the delete, and will happily rebuild its cache or index with a row that no longer exists. If you are using compaction for CDC or lookup tables, as the table above suggests, that bootstrap path is where the bug lives.

Infinite retention used to be a way of saying “and now you buy a lot of disks.” Tiered storage changes the arithmetic — brokers offload completed segments to object storage and keep only recent data locally — which is what moves log-as-system-of-record from aspirational to practical. One limitation to plan around: it does not work with compacted topics, so the cheap-forever option and the keep-latest-per-key option are, for now, mutually exclusive.

Stateless or stateful processing

The user-facing question is “what does this job need in order to restart?”, and it splits cleanly.

↔ scroll the table sideways to see every column.

The practical consequence: stateful stream jobs are databases wearing a stream-processing costume. They have working sets, recovery times and rebalance behaviour to reason about. If you have ever had a “simple” windowed aggregation take twenty minutes to come back after a deploy, that was the changelog being replayed from scratch — and the first fix is almost never a bigger machine. It’s to stop throwing the state directory away: the runtime checkpoints how far the local store has consumed its changelog, so a restart that still has the directory replays only the tail. Persist it across restarts, then add standby replicas so another node already holds a warm copy when a task moves, and the twenty minutes becomes seconds.

Delivery semantics, stated honestly

↔ scroll the table sideways to see every column.

Two notes on that table before the caveat. The idempotent producer is already on — it has been the default for years, along with acks=all, so every stock producer already avoids duplicates caused by its own retries within a session; transactions are what you add on top, and they’re where the cost lives. And the consumer side of exactly-once is the half people forget: isolation.level defaults to read_uncommitted, so a pipeline with perfectly configured transactions will still deliver aborted records to a consumer left on defaults, and nothing will look broken until the numbers disagree.

The caveat that matters more than the table: exactly-once covers Kafka-to-Kafka work. A transaction can make “consume from topic A, produce to topic B, commit the offset” atomic, because all three are Kafka operations it controls. It cannot make “consume from topic A and charge a credit card” atomic, because the card network is not in the transaction. The moment your side effect leaves Kafka, you are back to at-least-once plus an idempotency key on the receiving side — the same conclusion any distributed system reaches.

So the honest default is: at-least-once delivery, idempotent consumers. Design every consumer so that seeing the same record twice is boring.

When Kafka is the wrong answer

The most useful thing to know about a tool is where it stops.

The first two rows used to be flat statements that Kafka couldn’t do this, and that is no longer true — which is worth spelling out, because it’s the part of Kafka’s shape that changed most recently.

In a consumer group, there is no per-message acknowledgement. The group commits an offset, meaning “everything up to here is handled”, so a single record your code can’t process sits at the front of its partition and stops the line until you fix it, skip it, or divert it to a dead-letter topic. That is still the model behind every use case above.

Share groups, production-ready since Kafka 4.2, are a second way of reading the same log, built for exactly the workload consumer groups handle badly. Records are acquired under a lock rather than assigned by position, each one is acknowledged, released or rejected individually, and a per-record delivery counter retires a record that keeps failing instead of letting it block anything. Two consequences follow. The partition-count ceiling doesn’t apply — more consumers than partitions is the point — and ordering goes away, since records from one partition are handed to several consumers at once and redeliveries arrive out of order.

Which is the honest way to read this whole section: Kafka is no longer the wrong shape for task-queue workloads, it just makes you give up the ordering guarantee to get queue behaviour. Everything else in the table still stands — there is still no delay, no TTL, no priority and no per-message routing, in either model.

TL;DR

TL;DR: There aren’t ten Kafka use cases. There’s one partitioned, append-only log whose readers hold their own offsets, and ten names for using it. The only real variables are how many consumer groups read it, how long you retain it, and how far back readers go.

Three consequences explain the whole list. Reading doesn’t consume — records die on the retention clock, not on delivery — so fan-out is free and “async messaging” and “pub-sub” are the same topic with a different number of groups. Replay is free, because an offset is a number you can move, which is why “replay & recovery” isn’t a use case but the property the other nine stand on. And ordering is per partition, never global: records with the same key land on the same partition and stay in order, which is the guarantee you actually want. That makes partition count a decision to get right up front — it caps how many consumers a consumer group can use, and because the mapping is a hash modulo the partition count, changing it later re-maps every existing key and breaks the per-key ordering you were relying on.

Get retention right or you’ve built something other than what you named: event sourcing on a seven-day window is a queue that eats your history, and a compacted CDC topic keeps only the latest value per key, so it can never be your audit trail. Know that stateful stream jobs are databases in costume, with a restart time equal to replaying their changelog. Assume at-least-once and make consumers idempotent — exactly-once is real, but only for read-process-write inside Kafka; the moment a side effect leaves it, you need an idempotency key like everyone else.

And know where it stops. In a consumer group there’s no per-message ack — the group commits a position, so one poison record blocks its partition until you divert it. Share groups are a newer second way to read the same log that fixes exactly that, with per-record acknowledgement, individual redelivery and a delivery counter that retires a record which keeps failing — at the price of the ordering guarantee. What neither model gives you, and what still sends people to RabbitMQ or SQS, is per-message delay, TTL, priority or routing.

kafka