25 Jan 2021 02:25:09,183 INFO [PollableSourceRunner-KafkaSource-kafka74_01] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commitforgroup ka
doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
25 Jan 202102:25:13,512 INFO [PollableSourceRunner-KafkaSource-kafka74] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commitforgroup kadoo
p_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
25 Jan 202102:25:14,285 INFO [PollableSourceRunner-KafkaSource-kafka74_02] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commitforgroup ka
doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
25 Jan 202102:25:14,285 INFO [PollableSourceRunner-KafkaSource-kafka74_01] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commitforgroup ka
doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
25 Jan 202102:35:12,738 INFO [PollableSourceRunner-KafkaSource-kafka74_02] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commitforgroup ka
doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
25 Jan 202102:35:12,809 INFO [PollableSourceRunner-KafkaSource-kafka74_04] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commitforgroup ka
doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
25 Jan 202102:45:12,334 INFO [PollableSourceRunner-KafkaSource-kafka74_01] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commitforgroup ka
doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
25 Jan 202102:45:12,378 INFO [PollableSourceRunner-KafkaSource-kafka74] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commitforgroup kadoo
p_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
25 Jan 202102:45:12,396 INFO [PollableSourceRunner-KafkaSource-kafka74_02] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commitforgroup ka
doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
这个和消费者有什么关系吗。别的topic也被影响了
25 Jan 2021 02:25:09,183 INFO [PollableSourceRunner-KafkaSource-kafka74_01] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commit for group ka doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 25 Jan 2021 02:25:13,512 INFO [PollableSourceRunner-KafkaSource-kafka74] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commit for group kadoo p_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 25 Jan 2021 02:25:14,285 INFO [PollableSourceRunner-KafkaSource-kafka74_02] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commit for group ka doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 25 Jan 2021 02:25:14,285 INFO [PollableSourceRunner-KafkaSource-kafka74_01] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commit for group ka doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 25 Jan 2021 02:35:12,738 INFO [PollableSourceRunner-KafkaSource-kafka74_02] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commit for group ka doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 25 Jan 2021 02:35:12,809 INFO [PollableSourceRunner-KafkaSource-kafka74_04] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commit for group ka doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 25 Jan 2021 02:45:12,334 INFO [PollableSourceRunner-KafkaSource-kafka74_01] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commit for group ka doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 25 Jan 2021 02:45:12,378 INFO [PollableSourceRunner-KafkaSource-kafka74] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commit for group kadoo p_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 25 Jan 2021 02:45:12,396 INFO [PollableSourceRunner-KafkaSource-kafka74_02] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542) - Offset commit for group ka doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
你的带宽可能被平衡给占用了。
可以考虑限流:https://www.orchome.com/510
如果不是带宽的问题,会不会是我在均衡的时候,消费者组中有一个消费者没有找到leader,然后触发了消费者rebalance,所以引起了消费延迟
你之前描述的有歧义呀,原来是消费者rebanlance... 是会影响的。
我需要知道你怎么操作的,rebanlance出现的频次。
过程是这样子的,我扩容了一台,然后做了其中一个topic的rebalance,然后发现别的topic消费堆积
你是做了分区迁移是吧?
1、已经迁移成功率吗?
2、你查看一下该topic的描述,贴一下出来。
3、如果还在迁移,就是之前说的,可能是这个把流量占用了。
我的json文件没指定分区,直接写的是 topic
{ "topics": [ {"topic": "ods_be_monitor_item_detail"} ], "version":1 }
{"version":1,"partitions":[{"topic":"ods_be_monitor_item_detail","partition":4,"replicas":[156,157,111]},{"topic":"ods_be_monitor_item_detail","partition":2,"replicas":[154,155,156]},{"topic":"ods_be_monitor_item_detail","partition":14,"replicas":[154,157,111]},{"topic":"ods_be_monitor_item_detail","partition":1,"replicas":[153,154,155]},{"topic":"ods_be_monitor_item_detail","partition":5,"replicas":[157,111,153]},{"topic":"ods_be_monitor_item_detail","partition":9,"replicas":[155,157,111]},{"topic":"ods_be_monitor_item_detail","partition":12,"replicas":[111,155,156]},{"topic":"ods_be_monitor_item_detail","partition":8,"replicas":[154,156,157]},{"topic":"ods_be_monitor_item_detail","partition":7,"replicas":[153,155,156]},{"topic":"ods_be_monitor_item_detail","partition":0,"replicas":[111,153,154]},{"topic":"ods_be_monitor_item_detail","partition":3,"replicas":[155,156,157]},{"topic":"ods_be_monitor_item_detail","partition":6,"replicas":[111,154,155]},{"topic":"ods_be_monitor_item_detail","partition":11,"replicas":[157,153,154]},{"topic":"ods_be_monitor_item_detail","partition":13,"replicas":[153,156,157]},{"topic":"ods_be_monitor_item_detail","partition":10,"replicas":[156,111,153]}]}
现在迁移完成了吗?
3个副本,如果消息量大的话,内部占用流量不小的。
迁移完成了大概花了2个小时的时间,一个分区大概45g数据,15个分区,三副本,是因为流量导致影响到集群了吗,具体什么原因能帮忙分析一下吗
现在集群已经恢复正常了吗?
恢复了,昨天迁移差不多完成就恢复了
因为昨天只迁移了一个topic,还有好几个大的,现在都不敢动了,一动就影响集群
之前就是因为迁移会占用资源,影响正常业务,所以才出了限流。
好的,谢谢
我每次最后重启的那台broker,都没有leader落在上面,造成资源利用率不高,是需要重新选举leader吗
比如这种,明明我有五台,然后leader都重复,现在生产消息有时候超时,设置了5s的超时时间
[root@prd-ic-kafka-06 kafka]# /opt/kafka_2.10-0.10.2.2/bin/kafka-topics.sh --zookeeper 172.19.39.156:2181 --topic ods_meituan_crawler_flow_analysis --describe Topic:ods_meituan_crawler_flow_analysis PartitionCount:5 ReplicationFactor:3 Configs: Topic: ods_meituan_crawler_flow_analysis Partition: 0 Leader: 154 Replicas: 157,154,155 Isr: 155,154,157 Topic: ods_meituan_crawler_flow_analysis Partition: 1 Leader: 155 Replicas: 153,155,156 Isr: 155,153,156 Topic: ods_meituan_crawler_flow_analysis Partition: 2 Leader: 154 Replicas: 154,156,157 Isr: 154,157,156 Topic: ods_meituan_crawler_flow_analysis Partition: 3 Leader: 155 Replicas: 155,157,153 Isr: 155,153,157 Topic: ods_meituan_crawler_flow_analysis Partition: 4 Leader: 153 Replicas: 156,153,154 Isr: 154,153,156
还有一个疑问,如果leader发生了变化,对消费者或者消费者组有什么影响。。这样问答不方便,所以多问了几个问题
https://www.orchome.com/33
auto.leader.rebalance.enable=true 这个配置生产环境是不是一般都会开的,会有什么影响吗
测试了一下,修改这个配置,重启发现leader也没有转移
500s,扫描一次
你的答案