kafka设计动机

原创
半兽人 发表于: 2015-03-10   最后更新时间: 2020-11-23 13:59:33  
{{totalSubscript}} 订阅, 45,924 游览

动机(Motivation)

We designed Kafka to be able to act as a unified platform for handling all the real-time data feeds a large company might have. To do this we had to think through a fairly broad set of use cases.
我们设计kafka是必须能够作为一个统一的平台,来处理一家大公司可能有的所有实时数据。要做到这一点,我们不得不思考一个相当广泛的用例。

It would have to have high-throughput to support high volume event streams such as real-time log aggregation.
它必须具有高吞吐量,以支持大容量的事件流,如实时日志聚集。

It would need to deal gracefully with large data backlogs to be able to support periodic data loads from offline systems.
它将需要优雅地处理大型数据积压,要能够支持从脱机系统的定期数据装载。

It also meant the system would have to handle low-latency delivery to handle more traditional messaging use-cases.
这也意味着该系统将不得不处理低延迟交付,以处理更传统的消息传递的用例。

We wanted to support partitioned, distributed, real-time processing of these feeds to create new, derived feeds. This motivated our partitioning and consumer model.
我们想要支持以分区,分布式的,实时处理来这些创建新派生的feeds,促使我们的分区和消费模式。

Finally in cases where the stream is fed into other data systems for serving we new the system would have to be able to guarantee fault-tolerance in the presence of machine failures.
最后,在其中该流被送入其他数据系统用于服务的情况下,我们的新的系统必须能够保证在机器故障存在的容错。

Supporting these uses led use to a design with a number of unique elements, more akin to a database log then a traditional messaging system. We will outline some elements of the design in the following sections.
为了支持这些用途,需要设计一些独特的元素。比起传统的消息系统,更像数据库日志,我们将在下一节介绍。

更新于 2020-11-23

自由如风 4年前

We wanted to support partitioned, distributed, real-time processing of these feeds to create new, derived feeds.
我们想要它支持分区的、分布的、实时处理的这些信息流,用来产生新的信息流

宇过天晴 7年前

It would need to deal gracefully with large data backlogs to be able to support periodic data loads from offline systems.
它将需要能优雅的处理大量积压的数据,即需要支持从离线系统中周期性的加载数据

查看kafka更多相关的文章或提一个关于kafka的问题,也可以与我们一起分享文章