Redis Streams vs Kafka: Choosing for Event Systems

Event-driven architecture looks clean on a whiteboard. Events flow through a message bus, consumers react asynchronously, and services stay decoupled. The problem starts when you need to pick the actual technology. Two systems appear constantly: Redis Streams and Apache Kafka. They both move events. They are not interchangeable.

The confusion is understandable. Both support producer-consumer patterns. Both offer consumer groups. Both have strong community support. But Kafka is a distributed commit log designed for durable, high-throughput, replayable event streams at serious scale. Redis Streams is a data type inside Redis. It is powerful and useful, but architecturally closer to a lightweight queue than a durable event log.

Picking Kafka because it sounds serious, or picking Redis Streams because it is already in the stack, without understanding the actual tradeoffs, is how event system bugs become production incidents.

What each system is#

Redis Streams#

Redis Streams is a data type introduced in Redis 5.0. It is a log-like structure stored in Redis memory, with entries identified by time-based IDs. Streams support consumer groups. Multiple consumers can share a stream and each message is delivered to one consumer in the group, tracked by acknowledgment (Redis Streams documentation). Consumers that have not acknowledged a message have it tracked as "pending" until it is acknowledged or re-delivered.

Retention is bounded. By default a stream grows until memory limits are reached. You control retention with MAXLEN trimming, either exact (MAXLEN N) or approximate (MAXLEN ~ N, which is faster). This means Redis Streams is not a durable log by default. It is a bounded, in-memory queue-like structure with consumer group mechanics layered on top. Redis persistence options (RDB snapshots, AOF logs) add some durability, but they are designed for Redis's primary use as a cache and are not equivalent to Kafka's durable, replicated log.

Apache Kafka#

Apache Kafka is a distributed event streaming platform. The current release is Kafka 4.3 (kafka.apache.org). Events are written to named topics, partitioned for parallelism, and replicated across brokers for fault tolerance. Retention is time-based or size-based but can be set to indefinite. Events are not deleted after consumption. Multiple independent consumer groups can read the same topic from any offset, including the beginning.

Kafka was designed at LinkedIn for high-throughput, durable activity log data before becoming an open-source Apache project. Its core properties are described in the Apache Kafka documentation: it functions as a replacement for traditional message brokers with superior throughput, partitioning, replication, and fault-tolerance. It also serves as the backbone for log aggregation, stream processing, event sourcing, and metrics pipelines.

Kafka's operational surface is significant. You need brokers, ZooKeeper (or KRaft in newer versions), topic configuration, partition sizing, replication factors, and consumer group offset management. Managed services like Confluent Cloud, Amazon MSK, and Upstash Kafka reduce this complexity substantially.

At a glance comparison#

Dimension by dimension analysis#

Retention and replay#

This is the most important dimension for the decision. Kafka's retention is a first-class design property. Events stay on disk until you decide to delete them, whether that is one day or one year. Any consumer group can seek to any offset (the beginning, a specific timestamp, or a specific position) and replay events from there. This makes Kafka suitable for use cases where events are the authoritative record: event sourcing, audit logs, rebuilding derived state, and feeding new consumer services with historical data.

Redis Streams retention is memory-bound. You set MAXLEN to keep the last N entries, which is appropriate for a queue-like system but not for an event log you need to replay. Once events are trimmed or the Redis instance is restarted without persistence, they are gone. Using Redis Streams for replay requires building that retention layer yourself, usually by persisting events to another store. At that point you have rebuilt part of Kafka.

Consumer groups and fan-out#

Both systems support consumer groups, but the semantics differ in an important way. In Redis Streams, each message in a stream is delivered to one consumer within a group. Two different consumer groups can both read the stream, but each group's consumers compete for messages within that group. If you want Service A and Service B to each receive every event, you need either two consumer groups or two separate streams.

Kafka handles this natively. Multiple consumer groups read the same topic independently. Service A's consumer group and Service B's consumer group each get all messages at their own position in the log. The producer does not need to know how many consumers exist. This fan-out pattern is core to how Kafka enables service decoupling at scale.

Throughput and latency#

Redis Streams are fast. Sub-millisecond latency is achievable because Redis operates primarily in memory. For workloads with moderate event volume where latency is the primary concern, Redis Streams has an edge for the smallest payloads.

Kafka's throughput at scale is extraordinary. Sequential disk writes combined with consumer batching and partition parallelism allow Kafka to handle millions of events per second across a cluster. For high-volume event streams (clickstreams, IoT sensor data, payment events at large scale), Kafka's disk-based architecture is an advantage, not a limitation. Memory capacity does not constrain event volume.

Operational complexity#

Redis Streams add almost no operational complexity if Redis is already in your stack. You are using a data type you already run. Consumer group management, pending message handling, and trimming strategy require learning, but the infrastructure layer is already there.

Kafka requires dedicated infrastructure. A self-hosted Kafka cluster involves brokers, partition sizing, replication factor decisions, log retention configuration, and consumer group offset management. The KRaft mode (available since Kafka 3.3, default in Kafka 4.x) eliminates ZooKeeper from the operational picture, which is a meaningful simplification, but Kafka still requires more operational knowledge than Redis Streams.

Managed Kafka services like Confluent Cloud, Amazon MSK, and Upstash Kafka significantly reduce this burden. For teams that do not want to manage brokers, a managed Kafka service is often the right trade. You get Kafka semantics without the self-hosting complexity.

Reliability and at-least-once delivery#

Redis Streams tracks unacknowledged messages in a pending entries list (PEL). If a consumer crashes before acknowledging a message, the message remains in the PEL and can be re-delivered via XAUTOCLAIM or XCLAIM. This provides at-least-once delivery semantics within the bounds of Redis persistence.

Kafka provides at-least-once delivery by default, and exactly-once semantics are available through the transactional producer API and idempotent consumers. Kafka's replication across brokers means that a single broker failure does not lose committed messages. For systems where every event must be processed and data loss is unacceptable, Kafka's durability model is stronger and more operationally explicit.

Real-world scenarios#

Common wrong choices#

Choosing Kafka because it is "enterprise." Kafka earns its complexity for specific use cases: high-throughput durable logs, multi-consumer fan-out, long-term replay. Adding Kafka to a product that sends a few hundred events per minute and only has one consuming service is operational overhead without engineering payoff. A job queue or Redis Streams handles those workloads with far less complexity.

Choosing Redis Streams when replay is a requirement. If any part of the system design assumes that a new service can read historical events, or that a consumer can reprocess past events after a bug fix, that requirement cannot be met with Redis Streams' bounded retention. This assumption is sometimes buried in product requirements ("we want to be able to rerun analytics") and only surfaces after events have already been trimmed.

Treating consumer group semantics as equivalent across both systems. Redis consumer groups route each message to one consumer within the group. Kafka consumer groups each receive all messages independently. Misunderstanding this leads to fan-out bugs where services only see a fraction of events instead of all of them.

Skipping managed Kafka out of cost concern. Self-hosting Kafka is operationally expensive. Managed services have per-message costs but eliminate broker operations, replication management, and upgrade cycles. For teams without dedicated infrastructure engineers, managed Kafka is usually cheaper in total cost than self-hosting.

Recommendation framework#

The recommendation comes down to two properties: do you need replay, and do you need independent fan-out. The decision tree below captures the path most teams should follow.

The flow itself is simple. A producer writes events, the broker holds them, and consumers read at their own pace. What differs is whether the broker keeps history (Kafka does) or trims it (Redis Streams does).

Final decision guide#

Start by answering two questions: Does any consumer need to replay events from the past? Do multiple independent services need to receive the same events independently?

If either answer is yes, Kafka is the right choice. The replay guarantee and independent consumer group model are exactly what Kafka was built for. Use a managed service if self-hosting Kafka is not feasible for your team.

If both answers are no, meaning events are consumed once, retention is short, and infrastructure simplicity matters, Redis Streams is the better choice, especially if Redis is already in the stack. For pure job queuing with retry logic, a library like BullMQ on top of Redis is often even more appropriate than raw Redis Streams.

The failure mode to avoid is choosing Kafka for status reasons and then spending weeks configuring brokers, partitions, and consumer groups for a workload that needed a queue. The other failure mode is choosing Redis Streams and discovering six months later that a compliance requirement needs event history you never retained.

For teams building AI products with event pipelines for model inference, feedback loops, or real-time personalization, see our post on system design for AI products. Event system design for AI workloads often has specific replay and retention requirements that map clearly to Kafka.

Key Takeaways#

• Redis Streams is an in-memory, bounded event log. Retention is controlled by MAXLEN trimming. It does not provide durable replay guarantees without additional persistence infrastructure.
• Apache Kafka is a durable distributed commit log. The current release is Kafka 4.3 (kafka.apache.org). Events are retained on disk until explicitly expired, and any consumer group can replay from any offset.
• Consumer group semantics differ: Redis Streams routes each message to one consumer in a group. Kafka's consumer groups each receive all messages independently, which is true fan-out for multiple services.
• For throughput at scale, Kafka's sequential disk I/O handles higher sustained event rates than Redis's memory-bound model.
• Operational complexity is higher for Kafka. Managed services (Confluent Cloud, MSK, Upstash Kafka) significantly reduce the self-hosting burden.
• Replay and long-term retention are the clearest signals for Kafka. Simplicity and existing Redis infrastructure are the clearest signals for Redis Streams.
• Do not choose Kafka for status. Choose it when replay, fan-out, or durable retention justify the operational weight.

FAQ#

Can Redis Streams persist events to disk?

Redis has two persistence mechanisms: RDB snapshots and AOF (Append-Only File) logging. These add durability to Redis data including Streams, but they are designed for Redis's primary role as a cache. They do not provide Kafka-style guaranteed, replication-based durability. An RDB snapshot may lag the current state by minutes, and AOF can be configured for fsync-per-write but at a performance cost. For event systems where durability is a hard requirement, Kafka's replication model is more appropriate than Redis persistence.

What is the throughput difference in practice?

Both systems can handle high event rates, but the bottleneck differs. Redis Streams is memory-bound, so event retention is limited by available RAM. Kafka is disk-bound and benefits from sequential I/O, which modern SSDs handle very efficiently. In practice, a Kafka cluster can sustain millions of events per second across partitions. Redis Streams can handle tens to hundreds of thousands of events per second on well-provisioned instances, which is sufficient for most product workloads.

Is Kafka overkill for a startup?

Often, yes. A startup processing thousands of events per day does not need Kafka's throughput or operational surface. Redis Streams, a managed queue (SQS, CloudTasks), or a simple BullMQ setup handles the workload with less engineering investment. Kafka becomes justified when replay is a requirement, when multiple teams are consuming the same event stream independently, or when event volume genuinely strains simpler systems.

Can I use both in the same system?

Yes. A common pattern is using Redis Streams or BullMQ for low-latency task queuing (send email, trigger webhook), while using Kafka for the durable event log that downstream analytics, audit, and platform services consume. The two serve different purposes and complement each other. The cost is operating two systems, which is justified when the use cases are genuinely different.

What about managed Kafka versus self-hosted?

For most product teams, managed Kafka is the right choice. Services like Confluent Cloud, Amazon MSK, and Upstash Kafka handle broker management, replication, patching, and scaling. The per-message cost is higher than self-hosting on equivalent hardware, but the engineering time saved on infrastructure is almost always worth it unless you have dedicated platform engineers. Self-hosted Kafka on KRaft (Kafka 4.x, no ZooKeeper) is simpler than it used to be but still requires meaningful operational investment.

Does Redis Streams support exactly-once delivery?

No. Redis Streams provides at-least-once delivery through the pending entries list. Messages that are not acknowledged can be re-claimed and re-delivered. Exactly-once semantics require idempotent consumers or external deduplication. Kafka supports exactly-once semantics through its transactional producer API and idempotent consumer patterns, which are more fully developed and documented for high-stakes event processing.