kafka持久化

半兽人 发表于: 2015-03-10   最后更新时间: 2016-10-27  
  •   53 订阅,4526 游览

不要害怕文件系统!

Kafka relies heavily on the filesystem for storing and caching messages. There is a general perception that "disks are slow" which makes people skeptical that a persistent structure can offer competitive performance. In fact disks are both much slower and much faster than people expect depending on how they are used; and a properly designed disk structure can often be as fast as the network.

Kafka高度依赖文件系统来存储和缓存消息。一般的人都认为“磁盘是缓慢的”,这使得人们对“持久化结构提供具有竞争性的性能”这样的结论持有怀疑态度。实际上,磁盘比人们预想的快很多也慢很多,这取决于它们如何被使用;一个好的磁盘结构设计可以使之跟网络速度一样快。


The key fact about disk performance is that the throughput of hard drives has been diverging from the latency of a disk seek for the last decade. As a result the performance of linear writes on a JBOD configuration with six 7200rpm SATA RAID-5 array is about 600MB/sec but the performance of random writes is only about 100k/sec—a difference of over 6000X. These linear reads and writes are the most predictable of all usage patterns, and are heavily optimized by the operating system. A modern operating system provides read-ahead and write-behind techniques that prefetch data in large block multiples and group smaller logical writes into large physical writes. A further discussion of this issue can be found in this ACM Queue article; they actually find that sequential disk access can in some cases be faster than random memory access!

一个有关磁盘性能的关键事实是:在过去的十年,磁盘驱动器的吞吐量跟寻道延迟是相背离的。结果就是:在6个7200rpm SATA RAID-5 的磁盘阵列上线性写的速度大概是600M/秒,但是随机写的速度只有100K/秒,两者相差将近6000倍。线性读写在大多数应用场景下是可以预测的,因此,操作系统利用read-ahead和write-behind技术来从大的数据块中预取数据,或者将多个逻辑上的写操作组合成一个大写物理写操作中。更多的讨论可以在ACM Queue Artical中找到,他们发现,对磁盘的线性读在有些情况下可以比内存的随机访问要更快。


To compensate for this performance divergence modern operating systems have become increasingly aggressive in their use of main memory for disk caching. A modern OS will happily divert all free memory to disk caching with little performance penalty when the memory is reclaimed. All disk reads and writes will go through this unified cache. This feature cannot easily be turned off without using direct I/O, so even if a process maintains an in-process cache of the data, this data will likely be duplicated in OS pagecache, effectively storing everything twice.

为了补偿这个性能上的分歧,现代操作系统用内存做磁盘缓存变得非常的aggressive(积极)。现在操作系统会非常开心地将所有空闲的内存用于磁盘缓冲,尽管在内存回收的时候会有一点性能上的代价。所有的磁盘读写会通过这个统一的缓存。这个功能不能轻易关掉,用直接IO的方法,所以尽管 一个进程维护着一个进程内的数据缓存,这些数据还是会在OS的页缓存中被复制,实际上就是所有的数据都保存了两次。


Furthermore we are building on top of the JVM, and anyone who has spent any time with Java memory usage knows two things:

此外,我们是在JVM的基础上构建的,熟悉java内存应用管理的人应该清楚以下两件事情:

  1. The memory overhead of objects is very high, often doubling the size of the data stored (or worse).
    一个对象的内存消耗是非常高的,经常是所存数据的两倍或者更多
  2. Java garbage collection becomes increasingly fiddly and slow as the in-heap data increases.
    随着堆内数据的增多,Java的垃圾回收变得越来越繁琐而缓慢。

As a result of these factors using the filesystem and relying on pagecache is superior to maintaining an in-memory cache or other structure—we at least double the available cache by having automatic access to all free memory, and likely double again by storing a compact byte structure rather than individual objects. Doing so will result in a cache of up to 28-30GB on a 32GB machine without GC penalties. Furthermore this cache will stay warm even if the service is restarted, whereas the in-process cache will need to be rebuilt in memory (which for a 10GB cache may take 10 minutes) or else it will need to start with a completely cold cache (which likely means terrible initial performance). This also greatly simplifies the code as all logic for maintaining coherency between the cache and filesystem is now in the OS, which tends to do so more efficiently and more correctly than one-off in-process attempts. If your disk usage favors linear reads then read-ahead is effectively pre-populating this cache with useful data on each disk read.

This suggests a design which is very simple: rather than maintain as much as possible in-memory and flush it all out to the filesystem in a panic when we run out of space, we invert that. All data is immediately written to a persistent log on the filesystem without necessarily flushing to disk. In effect this just means that it is transferred into the kernel's pagecache.

This style of pagecache-centric design is described in an article on the design of Varnish here (along with a healthy dose of arrogance).

基于这些事实,利用文件系统并且依靠页缓存比维护一个内存缓存或者其他结构要好——我们至少要使得可用的缓存加倍,通过自动访问可用内存,并且通过存储更紧凑的字节结构而不是一个对象,这将有可能再次加倍。这么做的结果就是在一台32GB的机器上,如果不考虑GC惩罚,将最多有28-30GB的缓存。此外,这些缓存将会一直存在即使服务重启,然而进程内缓存需要在内存中重构(10GB缓存需要花费10分钟)或者它需要一个完全冷缓存启动(非常差的初始化性能)。它同时也简化了代码,因为现在所有的维护缓存和文件系统之间内聚的逻辑都在操作系统内部了,这使得这样做比one-off in-process attempts更加高效与准确。如果你的磁盘应用更加倾向于顺序读取,那么read-ahead在每次磁盘读取中实际上获取到缓存中的是有用的数据。


以上这些建议了一个简单的设计:不同于维护尽可能多的内存缓存并且在需要的时候刷新到文件系统中,我们换一种思路。所有的数据不需要调用刷新程序,而是立刻将它写到一个持久化的日志中。事实上,这仅仅意味着,数据将被传输到内核页缓存中并稍后被刷新。我们可以增加一个配置项以让系统的用户来控制数据在什么时候被刷新到物理硬盘上。


常数时间就足够了 (Constant Time Suffices)

The persistent data structure used in messaging systems are often a per-consumer queue with an associated BTree or other general-purpose random access data structures to maintain metadata about messages. BTrees are the most versatile data structure available, and make it possible to support a wide variety of transactional and non-transactional semantics in the messaging system. They do come with a fairly high cost, though: Btree operations are O(log N). Normally O(log N) is considered essentially equivalent to constant time, but this is not true for disk operations. Disk seeks come at 10 ms a pop, and each disk can do only one seek at a time so parallelism is limited. Hence even a handful of disk seeks leads to very high overhead. Since storage systems mix very fast cached operations with very slow physical disk operations, the observed performance of tree structures is often superlinear as data increases with fixed cache--i.e. doubling your data makes things much worse then twice as slow.

消息系统元数据的持久化数据结果经常是一个B树。B树是一个很好的结构,可以用在事务型与非事务型的语义中。但是它需要一个很高的花费。B树的操作需要 O(logN)。通常情况下,这被认为与常数时间等价,但这对磁盘操作来说是不对的。磁盘寻道一次需要10ms,并且一次只能寻一个,因此并行化是受限 的。


Intuitively a persistent queue could be built on simple reads and appends to files as is commonly the case with logging solutions. This structure has the advantage that all operations are O(1) and reads do not block writes or each other. This has obvious performance advantages since the performance is completely decoupled from the data size—one server can now take full advantage of a number of cheap, low-rotational speed 1+TB SATA drives. Though they have poor seek performance, these drives have acceptable performance for large reads and writes and come at 1/3 the price and 3x the capacity.

直觉上来讲,一个持久化的队列可以构建在对一个文件的读和追加上,就像一般情况下的日志解决方案。尽管和B树相比,这种结构不能支持丰富的语义,但是它有一个优点,所有的操作都是常数时间,读数据不会阻塞写数据。


Having access to virtually unlimited disk space without any performance penalty means that we can provide some features not usually found in a messaging system. For example, in Kafka, instead of attempting to deleting messages as soon as they are consumed, we can retain messages for a relative long period (say a week). This leads to a great deal of flexibility for consumers, as we will describe.

事实上几乎无限制的磁盘空间的访问,没有任何性能损失。这意味着我们可以提供一般消息系统无法提供的特性。比如说,消息被消费后不是立马被删除,我们可以将这些消息保留一段相对比较长的时间(比如一个星期)。






发表于: 1年前   最后更新时间: 4月前   游览量:4526
上一条: kafka设计动机
下一条: kafka效率
评论…

  • 评论…
    • in this conversation
      提问