Kafka principle and architecture analysis

 

Kafka is a distributed messaging system developed by LinkedIn, written in Scala, which is widely used for horizontal scalability and high throughput. Kafka is a distributed, publish/subscribe based messaging system. The main design objectives are as follows:

  1. Provides message persistence in a time complexity of O(1), ensuring constant time-complex access performance even for data above terabytes.
  2. High throughput. Even on very cheap commercial machines, it is possible to support the transmission of more than 100K messages per second.
  3. Supports message partitioning between Kafka servers and distributed consumption while ensuring sequential transmission of messages within each Partition.
  4. Both offline data processing and real-time data processing are supported.
  5. Scale out: Supports online horizontal expansion.

The Kafka architecture is as follows:

Kafka_principle_and_architecture_0.png

A typical Kafka cluster contains several Producers, several brokers, several Consumer Groups, and a Zookeeper cluster. Kafka manages the cluster configuration through Zookeeper, elects the leader, and rebalances when the Consumer Group changes. Producer uses push mode to publish messages to the broker, Consumer uses pull mode to subscribe to and consume messages from the broker, and Consumer uses ZooKeeper to record partition consumption offsets.

Why use a messaging system?

  1. Decoupling
    • The message format is equivalent to the interface design in the synchronous call. On the basis of ensuring the interface is unchanged, the producer and the consumer can independently expand and modify.
    • The broker is responsible for managing the mapping between producers and consumers, and the increase and decrease of consumers does not require any changes by the producer.
  2. Redundancy: Messages are kept in the message system until data is lost.
  3. Peak clipping/buffering: When the producer sends a message and the consumer has limited processing power, the message queue can act as a buffer to ensure that the producer does not block the message, the consumer is not crushed, and the data is not throw.
  4. Order guarantee: Kafka guarantees the order of messages within a Partition.

How does Kafka achieve high performance?

  1. The message sequence is written to the disk: when the mechanical disk reads and writes data, it is divided into three steps: seek, rotation delay, and data transmission. Every time you want to write, the rotation delay is random, and the performance is much lower than the sequential write.
  2. Page Cache: A large number of small I/O operations that are too frequent can cause disk bottlenecks. Therefore, the data in Kafka is written to the operating system’s cache and then flushed to disk in batches.
  3. Zero copy: Use the zero copy method sendfile provided by the system to send data directly from the file to the network card cache, reducing system calls and memory copies.
  4. Batch compression: When network bandwidth is a bottleneck, data can be compressed. A single data compression ratio may be poor, and multiple data compressions together may have a better compression ratio.
  5. Partition expansion: A topic’s data is stored in partitions according to the partition when it is stored. Consumers are single-threaded to pull data when they consume partitions. When pulling data into a consumption bottleneck, you can increase the consumption speed by increasing the number of partitions.

data


  1. Kafka Design Analysis (1): Introduction to Kafka Background and Architecture

  2. How Kafka can publish millions of messages in 1 second

The article was last published on: 2019-11-03 11:45:50

Orignal link:https://blog.csdn.net/hustspy1990/article/details/102880822