半兽人 发表于: 2015-03-10   最后更新时间: 2016-10-27  
  •   40 订阅,3528 游览
We designed Kafka to be able to act as a unified platform for handling all the real-time data feeds a large company might have. To do this we had to think through a fairly broad set of use cases.


It would have to have high-throughput to support high volume event streams such as real-time log aggregation.


It would need to deal gracefully with large data backlogs to be able to support periodic data loads from offline systems.


It also meant the system would have to handle low-latency delivery to handle more traditional messaging use-cases.


We wanted to support partitioned, distributed, real-time processing of these feeds to create new, derived feeds. This motivated our partitioning and consumer model.


Finally in cases where the stream is fed into other data systems for serving we new the system would have to be able to guarantee fault-tolerance in the presence of machine failures.


Supporting these uses led use to a design with a number of unique elements, more akin to a database log then a traditional messaging system. We will outline some elements of the design in the following sections.


发表于: 1年前   最后更新时间: 2月前   游览量:3528
上一条: New Producer配置
下一条: kafka持久化

  • 评论…
    • in this conversation