Broker日志提示大量警告
已查询该文章但未找到解决办法:https://www.orchome.com/1066
2020-04-30 16:50:37,010 WARN kafka.server.epoch.LeaderEpochFileCache: Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:16, offset:339490219592}, Current: {epoch:498, offset335305734105} for Partition: __consumer_offsets-47
2020-04-30 16:50:37,011 WARN kafka.server.epoch.LeaderEpochFileCache: Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:16, offset:339490219599}, Current: {epoch:498, offset335305734105} for Partition: __consumer_offsets-47
2020-04-30 16:50:37,011 WARN kafka.server.epoch.LeaderEpochFileCache: Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:14, offset:105463605}, Current: {epoch:336, offset104508994} for Partition: appDataToK3_test-12
2020-04-30 16:50:37,011 WARN kafka.server.epoch.LeaderEpochFileCache: Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:17, offset:92001836731}, Current: {epoch:479, offset86819324213} for Partition: __consumer_offsets-4
2020-04-30 16:50:37,012 WARN kafka.server.epoch.LeaderEpochFileCache: Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:16, offset:339490219606}, Current: {epoch:498, offset335305734105} for Partition: __consumer_offsets-47
描述
kafka日常使用没有特殊操作
曾怀疑是follower 跟不上 leader 导致提示异常,查看topic ISR ,发现上面日志中
__consumer_offsets-47
有报警告,但是下面查看topic 分区47的__consumer_offsets
IRS是正常Topic: __consumer_offsets Partition: 38 Leader: 153 Replicas: 153,154,150 Isr: 154,153 Topic: __consumer_offsets Partition: 39 Leader: 154 Replicas: 154,150,151 Isr: 154,151,150 Topic: __consumer_offsets Partition: 40 Leader: 150 Replicas: 150,152,153 Isr: 152,150,153 Topic: __consumer_offsets Partition: 41 Leader: 151 Replicas: 151,153,154 Isr: 151,154,153 Topic: __consumer_offsets Partition: 42 Leader: 152 Replicas: 152,154,150 Isr: 152,154,150 Topic: __consumer_offsets Partition: 43 Leader: 153 Replicas: 153,150,151 Isr: 151,153 Topic: __consumer_offsets Partition: 44 Leader: 154 Replicas: 154,151,152 Isr: 154,152,151 Topic: __consumer_offsets Partition: 45 Leader: 150 Replicas: 150,153,154 Isr: 154,150,153 Topic: __consumer_offsets Partition: 46 Leader: 151 Replicas: 151,154,150 Isr: 151,154,150 Topic: __consumer_offsets Partition: 47 Leader: 152 Replicas: 152,150,151 Isr: 152,150,151 Topic: __consumer_offsets Partition: 48 Leader: 153 Replicas: 153,151,152 Isr: 151,152,153 Topic: __consumer_offsets Partition: 49 Leader: 154 Replicas: 154,152,153 Isr: 154,152,153
下图是监控 JMX kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent 的值
问题:
- 什么问题导致该警告频繁打印,怎么能消除
- 如何解决 follower 跟不上leader问题,可以调整什么参数吗?
num.replica.fetchers 这个可以调大试试
一般配置默认就可以的;出现较多、较长时间ISR不同步问题,检查一下Kafka集群性能吧。
这个一般是kafka一些版本的bug:
重新分配一下告警的分区,警告就会没有了,可参考。
https://www.orchome.com/454#item-7
好的,感谢回答,还有个问题,就是kafka正常使用的情况下,如何解决 follower 跟不上leader问题,可以调整什么参数吗?
节点满足这2个“同步”条件,以区分“活着”还是“故障”。leader跟踪“同步”节点。如果一个follower死掉,卡住,或落后,leader将从同步副本列表中移除它。落后是通过
replica.lag.max.messages
配置控制,卡住是通过replica.lag.time.max.ms
配置控制的。你的答案