kafka客户端消费消息出现这种以下情况,

Perfect Go 发表于: 2018-10-19   最后更新时间: 2018-10-19 23:20:44   13,736 游览
2018-10-19 12:23:13.454 [Thread-13] DEBUG org.apache.kafka.clients.NetworkClient - Node -1 disconnected.
2018-10-19 12:23:13.455 [Thread-13] WARN  org.apache.kafka.clients.NetworkClient - Bootstrap broker 120.77.254.103:3811 disconnected
2018-10-19 12:23:13.455 [Thread-13] DEBUG o.a.k.c.consumer.internals.ConsumerNetworkClient - Cancelled GROUP_COORDINATOR request ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@4a09c1dd, request=RequestSend(header={api_key=10,api_version=0,correlation_id=0,client_id=consumer-1}, body={group_id=}), createdTimeMs=1539922993332, sendTimeMs=1539922993441) with correlation id 0 due to node -1 being disconnected
2018-10-19 12:23:13.456 [Thread-13] DEBUG org.apache.kafka.clients.NetworkClient - Give up sending metadata request since no node is available
2018-10-19 12:23:13.556 [Thread-13] DEBUG org.apache.kafka.clients.NetworkClient - Initialize connection to node -1 for sending metadata request
2018-10-19 12:23:13.556 [Thread-13] DEBUG org.apache.kafka.clients.NetworkClient - Initiating connection to node -1 at 120.77.254.103:3811.
2018-10-19 12:23:14.970 [Thread-13] DEBUG org.apache.kafka.common.network.Selector - Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
2018-10-19 12:23:14.970 [Thread-13] DEBUG org.apache.kafka.clients.NetworkClient - Completed connection to node -1
2018-10-19 12:23:15.067 [Thread-13] DEBUG org.apache.kafka.clients.NetworkClient - Sending metadata request {topics=[trade-topic]} to node -1
2018-10-19 12:23:15.075 [Thread-13] DEBUG org.apache.kafka.common.network.Selector - Connection with /120.77.254.103 disconnected
java.io.EOFException: null
    at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:83)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:343)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:291)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:148)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:136)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:197)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:387)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.refreshCommittedOffsetsIfNeeded(ConsumerCoordinator.java:369)
    at org.apache.kafka.clients.consumer.KafkaConsumer.updateFetchPositions(KafkaConsumer.java:1531)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1018)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
    at com.exchange.message.job.ConsumerListener.run(ConsumerListener.java:26)
    at java.lang.Thread.run(Thread.java:748)
发表于 2018-10-19
添加评论

debug日志可以忽视的。

Perfect Go -> 半兽人 5年前

你好,我单独写了支持消费和生产消息的jar包。单独运行这个jar,可以正常产生消息和消费消息。当将这个jar被依赖到另一个项目,便不停出现这样的问题。

2018-10-19 14:32:27.087 [restartedMain] DEBUG org.apache.kafka.clients.NetworkClient - Node -1 disconnected.
2018-10-19 14:32:27.087 [restartedMain] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker 120.77.254.103:3811 disconnected
2018-10-19 14:32:27.087 [restartedMain] DEBUG org.apache.kafka.clients.NetworkClient - Give up sending metadata request since no node is available
2018-10-19 14:32:27.188 [restartedMain] DEBUG org.apache.kafka.clients.NetworkClient - Initialize connection to node -1 for sending metadata request
2018-10-19 14:32:27.188 [restartedMain] DEBUG org.apache.kafka.clients.NetworkClient - Initiating connection to node -1 at 120.77.254.103:3811.
2018-10-19 14:32:27.196 [restartedMain] DEBUG org.apache.kafka.common.network.Selector - Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
2018-10-19 14:32:27.196 [restartedMain] DEBUG org.apache.kafka.clients.NetworkClient - Completed connection to node -1
2018-10-19 14:32:27.289 [restartedMain] DEBUG org.apache.kafka.clients.NetworkClient - Sending metadata request {topics=[trade-topic]} to node -1
2018-10-19 14:32:27.297 [restartedMain] DEBUG org.apache.kafka.common.network.Selector - Connection with /120.77.254.103 disconnected

由上先是断开于node-1的连接,后又开始发送请求连接node-1,完成连接后。开始请求主题trade-topic的消息,现在这个topic一直有新消息在发送。后有不知道什么情况又断开连接,然后开始打印下面的错误

java.io.EOFException: null
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:83)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:343)
at org.apache.kafka.common.network.Selector.poll(Selector.java:291)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209)

后台一直不停的重复打印。

半兽人 -> Perfect Go 5年前

让引用你项目的业务,排查下依赖关系。是不是jar包冲突了。

Perfect Go -> 半兽人 5年前

发现是两个项目都有kafka依赖包,一个是1.0.2,一个是0.9.1。去掉0.9.1后,可以正常消费了。

半兽人 -> Perfect Go 5年前

ok,解决就好

你的答案

查看kafka相关的其他问题或提一个您自己的问题