kafka新生产监控

半兽人 发表于: 2015-03-10   最后更新时间: 2015-06-23  
  •   40 订阅,2424 游览

New producer monitoring

The following metrics are available on new producer instances.
Metric/Attribute name Description Mbean name
waiting-threads The number of user threads blocked waiting for buffer memory to enqueue their records kafka.producer:type=producer-metrics,client-id=([-.\w]+)
buffer-total-bytes The maximum amount of buffer memory the client can use (whether or not it is currently used). kafka.producer:type=producer-metrics,client-id=([-.\w]+)
buffer-available-bytes The total amount of buffer memory that is not being used (either unallocated or in the free list). kafka.producer:type=producer-metrics,client-id=([-.\w]+)
bufferpool-wait-time The fraction of time an appender waits for space allocation. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
batch-size-avg The average number of bytes sent per partition per-request. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
batch-size-max The max number of bytes sent per partition per-request. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
compression-rate-avg The average compression rate of record batches. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
record-queue-time-avg The average time in ms record batches spent in the record accumulator. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
record-queue-time-max The maximum time in ms record batches spent in the record accumulator kafka.producer:type=producer-metrics,client-id=([-.\w]+)
request-latency-avg The average request latency in ms kafka.producer:type=producer-metrics,client-id=([-.\w]+)
request-latency-max The maximum request latency in ms kafka.producer:type=producer-metrics,client-id=([-.\w]+)
record-send-rate The average number of records sent per second. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
records-per-request-avg The average number of records per request. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
record-retry-rate The average per-second number of retried record sends kafka.producer:type=producer-metrics,client-id=([-.\w]+)
record-error-rate The average per-second number of record sends that resulted in errors kafka.producer:type=producer-metrics,client-id=([-.\w]+)
record-size-max The maximum record size kafka.producer:type=producer-metrics,client-id=([-.\w]+)
record-size-avg The average record size kafka.producer:type=producer-metrics,client-id=([-.\w]+)
requests-in-flight The current number of in-flight requests awaiting a response. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
metadata-age The age in seconds of the current producer metadata being used. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
connection-close-rate Connections closed per second in the window. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
connection-creation-rate New connections established per second in the window. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
network-io-rate The average number of network operations (reads or writes) on all connections per second. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
outgoing-byte-rate The average number of outgoing bytes sent per second to all servers. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
request-rate The average number of requests sent per second. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
request-size-avg The average size of all requests in the window. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
request-size-max The maximum size of any request sent in the window. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
incoming-byte-rate Bytes/second read off all sockets kafka.producer:type=producer-metrics,client-id=([-.\w]+)
response-rate Responses received sent per second. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
select-rate Number of times the I/O layer checked for new I/O to perform per second kafka.producer:type=producer-metrics,client-id=([-.\w]+)
io-wait-time-ns-avg The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
io-wait-ratio The fraction of time the I/O thread spent waiting. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
io-time-ns-avg The average length of time for I/O per select call in nanoseconds. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
io-ratio The fraction of time the I/O thread spent doing I/O kafka.producer:type=producer-metrics,client-id=([-.\w]+)
connection-count The current number of active connections. kafka.producer:type=producer-metrics,client-id=([-.\w]+)
outgoing-byte-rate The average number of outgoing bytes sent per second for a node. kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)
request-rate The average number of requests sent per second for a node. kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)
request-size-avg The average size of all requests in the window for a node. kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)
request-size-max The maximum size of any request sent in the window for a node. kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)
incoming-byte-rate The average number of responses received per second for a node. kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)
request-latency-avg The average request latency in ms for a node. kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)
request-latency-max The maximum request latency in ms for a node. kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)
response-rate Responses received sent per second for a node. kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)
record-send-rate The average number of records sent per second for a topic. kafka.producer:type=producer-topic-metrics,client-id=([-.\w]+),topic=([-.\w]+)
byte-rate The average number of bytes sent per second for a topic. kafka.producer:type=producer-topic-metrics,client-id=([-.\w]+),topic=([-.\w]+)
compression-rate The average compression rate of record batches for a topic. kafka.producer:type=producer-topic-metrics,client-id=([-.\w]+),topic=([-.\w]+)
record-retry-rate The average per-second number of retried record sends for a topic kafka.producer:type=producer-topic-metrics,client-id=([-.\w]+),topic=([-.\w]+)
record-error-rate The average per-second number of record sends that resulted in errors for a topic. kafka.producer:type=producer-topic-metrics,client-id=([-.\w]+),topic=([-.\w]+)
We recommend monitor GC time and other stats and various server stats such as CPU utilization, I/O service time, etc. On the client side, we recommend monitor the message/byte rate (global and per topic), request rate/size/time, and on the consumer side, max lag in messages among all partitions and min fetch request rate. For a consumer to keep up, max lag needs to be less than a threshold and min fetch rate needs to be larger than 0.





发表于: 1年前   最后更新时间: 1年前   游览量:2424
上一条: kafka监控
下一条: kafka审核
评论…

  • 评论…
    • in this conversation
      提问