kafkaServer.out文件有大量以下告警,严重影响存储空间
[2023-11-22 11:23:44,497] WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:0, offset:232304816}, Current: {epoch:10, offset232258847} for Partition: loginfoTopic-17 (kafka.server.epoch.LeaderEpochFileCache)
[2023-11-22 11:23:44,621] WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:0, offset:232304771}, Current: {epoch:10, offset232258814} for Partition: loginfoTopic-11 (kafka.server.epoch.LeaderEpochFileCache)
[2023-11-22 11:23:44,622] WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:0, offset:232304795}, Current: {epoch:10, offset232258833} for Partition: loginfoTopic-8 (kafka.server.epoch.LeaderEpochFileCache)
[2023-11-22 11:23:44,650] WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:0, offset:232304838}, Current: {epoch:10, offset232258887} for Partition: loginfoTopic-2 (kafka.server.epoch.LeaderEpochFileCache)
topic情况
[root@abc kafka]# bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic loginfoTopic
Topic:loginfoTopic PartitionCount:20 ReplicationFactor:1 Configs:
Topic: loginfoTopic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 3 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 4 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 5 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 6 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 7 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 8 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 9 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 10 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 11 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 12 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 13 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 14 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 15 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 16 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 17 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 18 Leader: 0 Replicas: 0 Isr: 0
Topic: loginfoTopic Partition: 19 Leader: 0 Replicas: 0 Isr: 0
kafka的部分配置是
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
系统的日常数据采集是正常的,就是这个告警会不断重复出现,消耗大量的存储空间,导致系统无法正常运行。想通过更改kafkaServer.out的日志记录策略来减少日志的记录量,没找到可以设置的地方。
只是一个运维小白,希望各位大佬能详说,感谢
参考一下:
Kafka Broker 日志警告无法消除 : Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch
重新分配分区的要求是怎么样的,我不太理解
broker只有1个的话,replica能是1吗
是kafka的bug导致的,你就1个节点,是不能的。
那是不是需要先多创建一个节点还是两个
broker.id=0,1
然后再进行分区?分区前要先创建副本吗
这三个参数需要改吗
offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1
我完全不会kafka,请大佬见谅QAQ
1、kafka版本有bug,你需要先升级。
2、你现在只有一个kafka节点是
0
,所以你没地方可迁移呀。请问需要升级到哪个版本才没有这个bug呀
当前版本为
除了
其他版本都可以。
好的。感谢大佬耐心解答!
我把版本更新到了1.0.1还是有这个问题,请问还有什么可能性呀
1.0.1是什么版本?我没见过这个版本。
我在官网下载的安装包是Scala 2.11 - kafka_2.11-1.0.1.tgz (asc, sha512)
下这个吧
要大于以上的小版本。
好嘞,感谢感谢
升级到该版本还是存在这个问题TAT
磁盘清理了吗?是新集群,还是用的老数据?
没有清理磁盘,需要清理哪些?
用的老数据
kafka数据:
在
server.properties
中,找到log.dirs
,全删,然后重启集群。zookeeper数据:
/var/lib/zookeeper
问题已解决,感谢大佬耐心指点和解答!:)
你的答案