Kafka: a Distributed Messaging System for Log Processing
đź“ś Abstract
Log processing has become a critical component of the data pipeline for consumer internet companies. We introduce Kafka, a distributed messaging system that we developed for collecting and delivering high volumes of log data with low latency. Our system incorporates ideas from existing log aggregators and messaging systems, and is suitable for both offline and online message consumption. We made quite a few unconventional yet practical design choices in Kafka to make our system efficient and scalable. Our experimental results show that Kafka has superior performance when compared to two popular messaging systems. We have been using Kafka in production for some time and it is processing hundreds of gigabytes of new data each day.
✨ Summary
The paper presents Kafka as a distributed messaging system designed specifically for high-volume log collection and delivery. Its central abstraction is a partitioned, append-only log: producers publish records to topics, brokers store topic partitions as sequential segment files, and consumers retrieve records using offsets. Consumers maintain their own positions, enabling replay, independent consumption by multiple consumer groups, and efficient handling of both real-time and batch workloads. The design emphasizes sequential I/O, batching, operating-system page caching, zero-copy transfer through sendfile, and minimal broker-side delivery state. Partitioning provides scalability and parallelism while preserving ordering within each partition. The system provides at-least-once delivery and per-partition ordering, but the original design lacks built-in replication and can lose unconsumed data when a broker fails. In the reported experiments, Kafka achieved substantially higher producer and consumer throughput than ActiveMQ and RabbitMQ under the tested configurations. The LinkedIn deployment unified online services, offline analytics, Hadoop loading, and data-warehouse ingestion around a common event-streaming infrastructure. (notes.stephenholiday.com)
The paper’s documented influence is visible in the subsequent development of Apache Kafka. Apache’s documentation retains the paper’s core model of partitioned logs, consumer-controlled offsets, retention, replay, and partition-based parallelism, while later Kafka versions add replication and stronger fault-tolerance mechanisms. (kafka.apache.org) Apache’s official publications list also identifies subsequent research and engineering work on replicated Kafka logging, LinkedIn’s real-time activity-data pipeline, Kafka and Samza, and nearline/offline data integration. (kafka.apache.org) Industry adoption extended the design to ETL, monitoring, log collection, event processing, and large-scale data pipelines; a USENIX industry tutorial documented Kafka use at companies including LinkedIn, Twitter, and Netflix. (usenix.org)