Kafka 2.7.1, 大量topic offset标记不更新,部分topic的lag出现负值,但是topic的生产和消费目前状态正常,日志kafkaServer.out无异常

silence 发表于: 2022-05-18   最后更新时间: 2022-05-19 17:27:49   1,573 游览

topic的生产和消费目前状态正常,日志kafkaServer.out无异常,但是使用kafka-consumer-groups.sh观察后发现。

  1. 大量topic offset标记不更新,
  2. 部分topic的lag出现负值

kafka 2.7.1,三节点集群

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group facade-order-api-consumer-group --describe | grep facade.order.apply.shipping.methods.reply

facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   8          138             37              -101            consumer-2-632e60c7-3c7c-46d3-a109-cb02da3510db /10.77.29.190   consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   7          127             159             32              consumer-2-632e60c7-3c7c-46d3-a109-cb02da3510db /10.77.29.190   consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   4          137             170             33              consumer-2-632e60c7-3c7c-46d3-a109-cb02da3510db /10.77.29.190   consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   2          122             33              -89             consumer-2-632e60c7-3c7c-46d3-a109-cb02da3510db /10.77.29.190   consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   0          131             165             34              consumer-2-632e60c7-3c7c-46d3-a109-cb02da3510db /10.77.29.190   consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   6          139             172             33              consumer-2-632e60c7-3c7c-46d3-a109-cb02da3510db /10.77.29.190   consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   3          143             180             37              consumer-2-632e60c7-3c7c-46d3-a109-cb02da3510db /10.77.29.190   consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   5          136             37              -99             consumer-2-632e60c7-3c7c-46d3-a109-cb02da3510db /10.77.29.190   consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   9          144             181             37              consumer-2-632e60c7-3c7c-46d3-a109-cb02da3510db /10.77.29.190   consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   1          131             162             31              consumer-2-632e60c7-3c7c-46d3-a109-cb02da3510db /10.77.29.190   consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   16         136             166             30              consumer-2-b7a37f0c-65dd-43be-ba10-ac23ad4288a0 /10.77.29.65    consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   11         136             33              -103            consumer-2-b7a37f0c-65dd-43be-ba10-ac23ad4288a0 /10.77.29.65    consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   15         140             176             36              consumer-2-b7a37f0c-65dd-43be-ba10-ac23ad4288a0 /10.77.29.65    consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   10         134             168             34              consumer-2-b7a37f0c-65dd-43be-ba10-ac23ad4288a0 /10.77.29.65    consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   19         133             165             32              consumer-2-b7a37f0c-65dd-43be-ba10-ac23ad4288a0 /10.77.29.65    consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   14         135             33              -102            consumer-2-b7a37f0c-65dd-43be-ba10-ac23ad4288a0 /10.77.29.65    consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   18         130             163             33              consumer-2-b7a37f0c-65dd-43be-ba10-ac23ad4288a0 /10.77.29.65    consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   13         154             191             37              consumer-2-b7a37f0c-65dd-43be-ba10-ac23ad4288a0 /10.77.29.65    consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   12         153             190             37              consumer-2-b7a37f0c-65dd-43be-ba10-ac23ad4288a0 /10.77.29.65    consumer-2
facade-order-api-consumer-group facade.order.apply.shipping.methods.reply   17         133             32              -101            consumer-2-b7a37f0c-65dd-43be-ba10-ac23ad4288a0 /10.77.29.65    consumer-2

因未影响上下游生产和消费,未尝试任何操作进行干预

发表于 2022-05-18
添加评论

查看topic那些负数的分区是否是正常的。

bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic foo --describe

如果都正常,lag负数的分区offset也一直在增长,可查看lag为负数的原因:kafka consumer lag 为负数

你的答案

查看kafka相关的其他问题或提一个您自己的问题