1、在线上业务spark-streaming中,出现了OffsetOutOfRangeException这个错误,经初步分析是由于zk中保存的offset小于broker中的offset。这个问题,版主有什么高见?
2、在验证上面的问题中,有几个困惑。在kafka0.10中,kafka自己保存consumergroup的信息,图片传不上来,直接粘贴吧
[root@kvdb08 lmy]# kafka-topics --zookeeper kvdb08:2181/kafka --list
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
__consumer_offsets
上面__consumer_offsets应该是保存consumergroup的信息吧?还是跟以前版本一样,consumergroup信息保存在zk中?但是我用命令
kafka-run-class kafka.tools.ConsumerOffsetChecker --broker-info kvdb08:9092 --zookeeper kvdb08:2181/kafka --topic consumer-count --group consumer-count
查看的结果是可以查看的
Group Topic Pid Offset logSize Lag Owner
consumer-count consumer-count 0 9614 11216 1602 none
consumer-count consumer-count 1 9846 11487 1641 none
consumer-count consumer-count 2 10030 11707 1677 none
consumer-count consumer-count 3 9778 11408 1630 none
consumer-count consumer-count 4 10330 12054 1724 none
consumer-count consumer-count 5 10334 12061 1727 none
但在zookeeper-client中没有找到consumer-count这个消费组的信息
[zk: localhost:2181(CONNECTED) 0] ls /kafka/consumers
[console-consumer-17136, console-consumer-7906, console-consumer-36419, console-consumer-60409, console-consumer-36808, schema-registry-kvdb08-8081, console-consumer-73372, console-consumer-86852]
所以consumerGroup还是存在于kafka-broker中?
kafka-consumer-group.sh --new-consumer --bootstrap-server .... --list
你的答案