Apache Kafka is the streaming standard. It powers event pipelines at LinkedIn (where it was created), Netflix, Uber, Airbnb, and basically every modern SaaS at scale. Understanding how Kafka actually works under the hood is what separates “I use Kafka” from “I run Kafka well.”
The Core Abstraction: Distributed, Partitioned, Replicated Log
Kafka is fundamentally a commit log: an ordered, immutable sequence of records. The clever part is that this log is distributed, partitioned, and replicated for scale and fault tolerance.
📚
Topic
A named stream. Logs of related events. "user-events", "orders", "payments-completed".
🔪
Partition
A topic is split into N partitions. Each partition is an ordered, append-only log file. Parallelism unit.
🔄
Replica
Each partition has multiple copies on different brokers. One leader, others are followers.
🏢
Broker
A Kafka server. A cluster has many brokers. Each broker hosts some partition leaders and some followers.
A producer publishes messages to topics. It chooses the partition based on:
🎯
Key-based partitioning
Messages with the same key always go to the same partition. Default Kafka behavior. Perfect for "all events for user X are ordered."
🎰
Round-robin
No key → distribute evenly across partitions. Maximum throughput, no per-key ordering.
🛠️
Custom partitioner
Plug in your own logic. Multi-tenant SaaS often partitions by tenant ID for isolation.
Producer acks
How many replicas must acknowledge before the producer considers the write successful?
acks=0
Fire and forget
No acknowledgment. Fastest. Possible message loss. Almost never used.
acks=1
Leader ack only
Producer waits for leader to write. If leader crashes before replication, message lost. Default in older Kafka.
acks=all (or -1)
Wait for in-sync replicas
Producer waits for all in-sync replicas (ISR) to ack. No data loss as long as one replica survives. The right default for production.
min.insync.replicas
Minimum ISR for writes
Together with acks=all, defines durability. Setting min.insync.replicas=2 with replication factor 3 means writes succeed only if at least 2 replicas are healthy.
Idempotent producer
When enabled (enable.idempotence=true), the producer attaches a sequence number to each message. The broker rejects duplicates from the same producer. Combined with acks=all, you get exactly-once writes within Kafka.
Consumers
A consumer reads messages from topics. The key concepts:
👥
Consumer group
A logical group of consumers sharing the work. Each partition is assigned to exactly one consumer in the group at a time.
📍
Offset
Consumer's current position in a partition. Stored in __consumer_offsets topic. Survives consumer restarts.
⚖️
Rebalance
When consumers join or leave a group, partitions are reassigned. Pause-the-world by default; cooperative rebalancing reduces this.
🔢
Parallelism cap
A consumer group can have at most N consumers, where N = number of partitions. More consumers than partitions = idle consumers.
Consumer group example
Topic: "orders" (4 partitions)Consumer Group "order-processor": - Consumer A → Partition 0, 1 - Consumer B → Partition 2, 3Consumer Group "analytics": - Consumer X → Partition 0, 1, 2, 3 (one consumer reads everything)Both groups receive all messages independently.
This is what makes Kafka different from a queue: multiple consumer groups read the same data independently. Each group tracks its own position. Want to add a new pipeline tomorrow? Spin up a new consumer group; it reads from the beginning (or wherever you want).
Replication and Fault Tolerance
For each partition, one broker is the leader; others are followers. All reads and writes go through the leader. Followers replicate from the leader.
In-sync replicas (ISR)
A follower is “in-sync” if it has caught up with the leader within a configurable threshold. Only ISRs are eligible for leadership.
Healthy cluster
ISR includes all replicas
All followers caught up. Any can become leader on failure. Maximum fault tolerance.
Degraded cluster
ISR shrinks
Some followers fall behind (network issues, disk slow). They drop out of ISR. Cluster keeps running but with reduced fault tolerance.
Leader election
When a leader fails, a follower from ISR is promoted. With unclean.leader.election.enable=false (recommended), only ISRs can be elected — preventing data loss at the cost of availability.
Storage
Kafka writes data to disk. The trick: sequential disk writes are nearly as fast as memory writes because of OS page cache and disk read-ahead.
💾
Log segments
Each partition is written as a sequence of segment files. New writes append to the active segment.
🗑️
Retention
Old segments are deleted by time (default: 7 days) or size. Configurable per topic.
🧹
Log compaction
Alternative to time-based retention. Keeps the latest message per key, deletes older ones. Used for "current state" topics.
🚀
Zero-copy
Linux sendfile(2) lets Kafka transfer log data from disk to network without copying through user space. Massive throughput gains.
Kafka Streams and KSQL
Kafka isn’t just a transport — it has built-in stream processing:
Kafka Streams
Java/Scala library
Embedded in your app. Stateful operators (joins, aggregations, windows). State stored in local RocksDB + replicated via changelog topics.
ksqlDB
SQL-like processing
Run SQL queries against Kafka topics. CREATE STREAM, CREATE TABLE, JOIN, GROUP BY. For analysts and SQL-friendly teams.
For more sophisticated stream processing (or polyglot environments), use Flink with Kafka as input/output.
Kafka Connect
Pre-built connectors for getting data in and out of Kafka:
⬇️
Source connectors
Pull data from external systems into Kafka. Debezium for CDC from databases. JDBC, MongoDB, S3 connectors. Hundreds available.
⬆️
Sink connectors
Push data from Kafka to external systems. Snowflake, BigQuery, Elasticsearch, S3. Most have exactly-once semantics built in.
Connect handles offset management, error handling, exactly-once. You write configuration; the connectors handle the rest.