kafka 2.11-2.1.1 java.lang.OutOfMemoryError: Direct buffer memory 如何处理

giant 发表于: 2019-07-30   最后更新时间: 2019-07-30 15:17:40   5,604 游览

提问说明

 java.lang.OutOfMemoryError: Direct buffer memory
  1. 已改为：ExplicitGCInvokesConcurrent：
KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true"

2： 

KAFKA_HEAP_OPTS="-Xmx2048M -Xms2048M"

3： product.propertis

buffer.memory=1024M

Direct buffer memory

kafka
发表于 2019-07-30
giant
添加评论 
total               used           free           shared         buff/cache       available
Mem:          16047        9413         534              3               6099               6264
Swap:             0           0                 0
回复
回答于 5年前
giant
半兽人 -> giant 5年前

内存的单位是啥？M？

0 0 回复
giant -> 半兽人 5年前

是的 ,M。 我在前面参数的基础上， 在kafka-run-class.sh文件加了个-XX:MaxDirectMemorySize=512M, 启动 报错：

[2019-07-30 14:59:43,134] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-35 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 14:59:43,134] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-0 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 14:59:43,135] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-6 in 1 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 14:59:43,135] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-12 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 14:59:43,135] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-18 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 14:59:43,136] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-24 in 1 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 14:59:43,139] ERROR [GroupMetadataManager brokerId=0] Error loading offsets from __consumer_offsets-30 (kafka.coordinator.group.GroupMetadataManager)
org.apache.kafka.common.KafkaException: Unknown group metadata version 3
    at kafka.coordinator.group.GroupMetadataManager$.schemaForGroupValue(GroupMetadataManager.scala:1077)
    at kafka.coordinator.group.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:1309)
    at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2$$anonfun$apply$13.apply(GroupMetadataManager.scala:602)
    at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2$$anonfun$apply$13.apply(GroupMetadataManager.scala:574)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2.apply(GroupMetadataManager.scala:574)
    at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2.apply(GroupMetadataManager.scala:555)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at kafka.coordinator.group.GroupMetadataManager.doLoadGroupsAndOffsets(GroupMetadataManager.scala:555)
    at kafka.coordinator.group.GroupMetadataManager.loadGroupsAndOffsets(GroupMetadataManager.scala:500)
    at kafka.coordinator.group.GroupMetadataManager$$anonfun$scheduleLoadGroupAndOffsets$1.apply$mcV$sp(GroupMetadataManager.scala:491)
    at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:114)
    at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
0 0 回复
半兽人 -> giant 5年前

这个看着你的存储文件已经不正确了，清理一下。

0 0 回复
giant -> 半兽人 5年前

哪个是存储文件？

0 0 回复
半兽人 -> giant 5年前

log.dir

0 0 回复
giant -> 半兽人 5年前

第一个解决了，l谢谢您， 您是从哪里看出存储文件已经不正确了？

0 0 回复
giant -> 半兽人 5年前

删掉 log.dirs 下的 文件，重启zk和kafka 后， 第一次使用，可以， 几分钟后就不行了。

2019-07-30 17:53:58.539  WARN 19940 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient 
  : [Consumer clientId=consumer-1, groupId=ADDRESS.PROCESSOR] 
Connection to node -1 could not be established. Broker may not be available.

[2019-07-30 17:17:21,588] INFO Shutting down the log cleaner. (kafka.log.LogCleaner)
[2019-07-30 17:17:21,589] INFO [kafka-log-cleaner-thread-0]: Shutting down (kafka.log.LogCleaner)
[2019-07-30 17:17:21,589] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner)
[2019-07-30 17:17:21,589] INFO [kafka-log-cleaner-thread-0]: Shutdown completed (kafka.log.LogCleaner)


[2019-07-30 17:17:21,640] INFO Processed session termination for sessionid: 0x10157e437850000 (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-30 17:17:21,643] INFO Closed socket connection for client / which had sessionid 0x10157e437850000 
(org.apache.zookeeper.server.NIOServerCnxn)

[2019-07-30 17:07:00,588] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)
[2019-07-30 17:12:00,588] TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
[2019-07-30 17:12:00,590] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)
[2019-07-30 17:17:00,590] TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
[2019-07-30 17:17:00,591] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)
[2019-07-30 17:17:20,985] INFO [Controller id=0] Shutting down broker 0 (kafka.controller.KafkaController)
[2019-07-30 17:17:21,630] INFO [ControllerEventThread controllerId=0] Shutting down (kafka.controller.ControllerEventManager$ControllerEventThread)
[2019-07-30 17:17:21,630] INFO [ControllerEventThread controllerId=0] Stopped (kafka.controller.ControllerEventManager$ControllerEventThread)
[2019-07-30 17:17:21,630] INFO [ControllerEventThread controllerId=0] Shutdown completed (kafka.controller.ControllerEventManager$ControllerEventThread)
[2019-07-30 17:17:21,633] INFO [PartitionStateMachine controllerId=0] Stopped partition state machine (kafka.controller.PartitionStateMachine)
[2019-07-30 17:17:21,633] INFO [ReplicaStateMachine controllerId=0] Stopped replica state machine (kafka.controller.ReplicaStateMachine)
[2019-07-30 17:17:21,636] INFO [RequestSendThread controllerId=0] Shutting down (kafka.controller.RequestSendThread)
[2019-07-30 17:17:21,636] INFO [RequestSendThread controllerId=0] Stopped (kafka.controller.RequestSendThread)
[2019-07-30 17:17:21,636] INFO [RequestSendThread controllerId=0] Shutdown completed (kafka.controller.RequestSendThread)
[2019-07-30 17:17:21,638] INFO [Controller id=0] Resigned (kafka.controller.KafkaController)

server.log

 Terminating process due to signal SIGHUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2019-07-30 17:17:20,969] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)
[2019-07-30 17:17:20,969] INFO [KafkaServer id=0] Starting controlled shutdown (kafka.server.KafkaServer)
[2019-07-30 17:17:20,996] INFO [KafkaServer id=0] Controlled shutdown succeeded (kafka.server.KafkaServer)
[2019-07-30 17:17:20,999] INFO [/config/changes-event-process-thread]: Shutting down (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2019-07-30 17:17:21,000] INFO [/config/changes-event-process-thread]: Shutdown completed (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2019-07-30 17:17:21,000] INFO [/config/changes-event-process-thread]: Stopped (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2019-07-30 17:17:21,000] INFO [SocketServer brokerId=0] Stopping socket server request processors (kafka.network.SocketServer)
[2019-07-30 17:17:21,011] INFO [SocketServer brokerId=0] Stopped socket server request processors (kafka.network.SocketServer)
[2019-07-30 17:17:21,012] INFO [Kafka Request Handler on Broker 0], shutting down (kafka.server.KafkaRequestHandlerPool)
[2019-07-30 17:17:21,014] INFO [Kafka Request Handler on Broker 0], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2019-07-30 17:17:21,017] INFO [KafkaApi-0] Shutdown complete. (kafka.server.KafkaApis)
[2019-07-30 17:17:21,017] INFO [ExpirationReaper-0-topic]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,095] INFO [ExpirationReaper-0-topic]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,096] INFO [ExpirationReaper-0-topic]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,098] INFO [TransactionCoordinator id=0] Shutting down. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-07-30 17:17:21,098] INFO [ProducerId Manager 0]: Shutdown complete: last producerId assigned 159000 (kafka.coordinator.transaction.ProducerIdManager)
[2019-07-30 17:17:21,099] INFO [Transaction State Manager 0]: Shutdown complete (kafka.coordinator.transaction.TransactionStateManager)
[2019-07-30 17:17:21,099] INFO [Transaction Marker Channel Manager 0]: Shutting down (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2019-07-30 17:17:21,099] INFO [Transaction Marker Channel Manager 0]: Stopped (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2019-07-30 17:17:21,100] INFO [Transaction Marker Channel Manager 0]: Shutdown completed (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2019-07-30 17:17:21,100] INFO [TransactionCoordinator id=0] Shutdown complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-07-30 17:17:21,100] INFO [GroupCoordinator 0]: Shutting down. (kafka.coordinator.group.GroupCoordinator)
[2019-07-30 17:17:21,101] INFO [ExpirationReaper-0-Heartbeat]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,298] INFO [ExpirationReaper-0-Heartbeat]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,298] INFO [ExpirationReaper-0-Heartbeat]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,299] INFO [ExpirationReaper-0-Rebalance]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,301] INFO [ExpirationReaper-0-Rebalance]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,301] INFO [ExpirationReaper-0-Rebalance]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,302] INFO [GroupCoordinator 0]: Shutdown complete. (kafka.coordinator.group.GroupCoordinator)
[2019-07-30 17:17:21,303] INFO [ReplicaManager broker=0] Shutting down (kafka.server.ReplicaManager)
[2019-07-30 17:17:21,303] INFO [LogDirFailureHandler]: Shutting down (kafka.server.ReplicaManager$LogDirFailureHandler)
[2019-07-30 17:17:21,303] INFO [LogDirFailureHandler]: Stopped (kafka.server.ReplicaManager$LogDirFailureHandler)
[2019-07-30 17:17:21,303] INFO [LogDirFailureHandler]: Shutdown completed (kafka.server.ReplicaManager$LogDirFailureHandler)
[2019-07-30 17:17:21,304] INFO [ReplicaFetcherManager on broker 0] shutting down (kafka.server.ReplicaFetcherManager)
[2019-07-30 17:17:21,305] INFO [ReplicaFetcherManager on broker 0] shutdown completed (kafka.server.ReplicaFetcherManager)
[2019-07-30 17:17:21,305] INFO [ReplicaAlterLogDirsManager on broker 0] shutting down (kafka.server.ReplicaAlterLogDirsManager)
[2019-07-30 17:17:21,305] INFO [ReplicaAlterLogDirsManager on broker 0] shutdown completed (kafka.server.ReplicaAlterLogDirsManager)
[2019-07-30 17:17:21,305] INFO [ExpirationReaper-0-Fetch]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,344] INFO [ExpirationReaper-0-Fetch]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,344] INFO [ExpirationReaper-0-Fetch]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,345] INFO [ExpirationReaper-0-Produce]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,381] INFO [ExpirationReaper-0-Produce]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,381] INFO [ExpirationReaper-0-Produce]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,381] INFO [ExpirationReaper-0-DeleteRecords]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,582] INFO [ExpirationReaper-0-DeleteRecords]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,583] INFO [ExpirationReaper-0-DeleteRecords]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-30 17:17:21,586] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
[2019-07-30 17:17:21,587] INFO Shutting down. (kafka.log.LogManager)
[2019-07-30 17:17:21,605] INFO [ProducerStateManager partition=__consumer_offsets-21] Writing producer snapshot at offset 425 (kafka.log.ProducerStateManager)
[2019-07-30 17:17:21,607] INFO [ProducerStateManager partition=D_GENERATE_ADDRESS_TOPIC-0] Writing producer snapshot at offset 2 (kafka.log.ProducerStateManager)
[2019-07-30 17:17:21,615] INFO [ProducerStateManager partition=__consumer_offsets-44] Writing producer snapshot at offset 171 (kafka.log.ProducerStateManager)
[2019-07-30 17:17:21,630] INFO Shutdown complete. (kafka.log.LogManager)
[2019-07-30 17:17:21,639] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
[2019-07-30 17:17:21,642] INFO Session: 0x10157e437850000 closed (org.apache.zookeeper.ZooKeeper)
[2019-07-30 17:17:21,643] INFO EventThread shut down for session: 0x10157e437850000 (org.apache.zookeeper.ClientCnxn)
[2019-07-30 17:17:21,644] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
[2019-07-30 17:17:21,644] INFO [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:23,793] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:23,793] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:23,794] INFO [SocketServer brokerId=0] Shutting down socket server (kafka.network.SocketServer)
[2019-07-30 17:17:23,813] INFO [SocketServer brokerId=0] Shutdown completed (kafka.network.SocketServer)
[2019-07-30 17:17:23,817] INFO [KafkaServer id=0] shut down completed (kafka.server.KafkaServer)
0 0 回复

仍然报这个错， 哪位朋友帮看看

回复
回答于 5年前
giant
半兽人 -> giant 5年前

找到/binkafka-run-class.sh，加大-XX:MaxDirectMemorySize，该参数默认是64M，可以根据需求调大
并且检查JVM参数里面有无：-XX:+DisableExplicitGC，如果有就去掉.

0 0 回复
giant -> 半兽人 5年前

-XX:MaxDirectMemorySize=512M ,-XX:+DisableExplicitGC之前就没有，
KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true"

0 0 回复
半兽人 -> giant 5年前

先解决第一个报错。

0 0 回复 
[2019-07-30 14:59:43,139] ERROR [GroupMetadataManager brokerId=0] Error loading offsets from __consumer_offsets-30 (kafka.coordinator.group.GroupMetadataManager)
org.apache.kafka.common.KafkaException: Unknown group metadata version 3
    at kafka.coordinator.group.GroupMetadataManager$.schemaForGroupValue(GroupMetadataManager.scala:1077)
    at kafka.coordinator.group.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:1309)
    at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2$$anonfun$apply$13.apply(GroupMetadataManager.scala:602)
    at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2$$anonfun$apply$13.apply(GroupMetadataManager.scala:574)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2.apply(GroupMetadataManager.scala:574)
    at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2.apply(GroupMetadataManager.scala:555)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at kafka.coordinator.group.GroupMetadataManager.doLoadGroupsAndOffsets(GroupMetadataManager.scala:555)
    at kafka.coordinator.group.GroupMetadataManager.loadGroupsAndOffsets(GroupMetadataManager.scala:500)
    at kafka.coordinator.group.GroupMetadataManager$$anonfun$scheduleLoadGroupAndOffsets$1.apply$mcV$sp(GroupMetadataManager.scala:491)
    at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:114)
    at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[2019-07-30 14:59:43,140] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-36 in 1 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 14:59:43,140] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-42 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 14:59:43,140] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-48 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 14:59:43,279] INFO [GroupCoordinator 0]: Preparing to rebalance group CALLBACK.PROCESSOR in state PreparingRebalance with old generation 0 (__consumer_offsets-21) (reason: Adding new member consumer-1-86dbbaa7-ed76-495b-af22-17186d8e284b) (kafka.coordinator.group.GroupCoordinator)
[2019-07-30 14:59:43,287] INFO [GroupCoordinator 0]: Stabilized group CALLBACK.PROCESSOR generation 1 (__consumer_offsets-21) (kafka.coordinator.group.GroupCoordinator)
[2019-07-30 14:59:43,297] INFO [GroupCoordinator 0]: Assignment received from leader for group CALLBACK.PROCESSOR for generation 1 (kafka.coordinator.group.GroupCoordinator)
kill -9 29786[2019-07-30 15:09:42,389] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 2 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 15:19:42,387] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 15:29:42,387] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 15:39:42,387] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-07-30 15:48:25,840] ERROR Processor got uncaught exception. (kafka.network.Processor)
java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:695)
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
    at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
    at sun.nio.ch.IOUtil.read(IOUtil.java:195)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
    at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:103)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:117)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:381)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:342)
    at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:609)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:541)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
    at kafka.network.Processor.poll(SocketServer.scala:689)
    at kafka.network.Processor.run(SocketServer.scala:594)
    at java.lang.Thread.run(Thread.java:748)
[2019-07-30 15:48:26,412] ERROR Processor got uncaught exception. (kafka.network.Processor)
java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:695)
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
    at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
    at sun.nio.ch.IOUtil.read(IOUtil.java:195)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
    at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:103)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:117)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:381)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:342)
    at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:609)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:541)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
    at kafka.network.Processor.poll(SocketServer.scala:689)
    at kafka.network.Processor.run(SocketServer.scala:594)
    at java.lang.Thread.run(Thread.java:748)
[2019-07-30 15:49:42,387] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
回复
回答于 5年前
giant

你的留言太混乱了，现在到哪一步了。

回复
回答于 5年前
半兽人
半兽人 -> 半兽人 5年前

你集群有几台，重新安装后，都干了些什么。

0 0 回复
giant -> 半兽人 5年前

没集群，就一台，我半路来的，不清楚安装后干了啥， 现在是启动可以， 但是后面会报如下错误。

[2019-07-30 17:17:21,586] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
[2019-07-30 17:17:21,587] INFO Shutting down. (kafka.log.LogManager)
[2019-07-30 17:17:21,605] INFO [ProducerStateManager partition=__consumer_offsets-21] Writing producer snapshot at offset 425 (kafka.log.ProducerStateManager)
[2019-07-30 17:17:21,607] INFO [ProducerStateManager partition=D_GENERATE_ADDRESS_TOPIC-0] Writing producer snapshot at offset 2 (kafka.log.ProducerStateManager)
[2019-07-30 17:17:21,615] INFO [ProducerStateManager partition=__consumer_offsets-44] Writing producer snapshot at offset 171 (kafka.log.ProducerStateManager)
[2019-07-30 17:17:21,630] INFO Shutdown complete. (kafka.log.LogManager)
[2019-07-30 17:17:21,639] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
[2019-07-30 17:17:21,642] INFO Session: 0x10157e437850000 closed (org.apache.zookeeper.ZooKeeper)
[2019-07-30 17:17:21,643] INFO EventThread shut down for session: 0x10157e437850000 (org.apache.zookeeper.ClientCnxn)
[2019-07-30 17:17:21,644] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
[2019-07-30 17:17:21,644] INFO [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:23,793] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:23,793] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-07-30 17:17:23,794] INFO [SocketServer brokerId=0] Shutting down socket server (kafka.network.SocketServer)
[2019-07-30 17:17:23,813] INFO [SocketServer brokerId=0] Shutdown completed (kafka.network.SocketServer)
[2019-07-30 17:17:23,817] INFO [KafkaServer id=0] shut down completed (kafka.server.KafkaServer)
0 0 回复
半兽人 -> giant 5年前

你这些日志都是没问题的，你用的什么命令启动的？

0 0 回复
你的答案

查看kafka相关的其他问题或提一个您自己的问题。
提问

找不到想要的答案？提一个您自己的问题。