giant

0 声望

这家伙太懒,什么都没留下

个人动态
  • 半兽人 回复 giantkafka 2.11-2.1.1 java.lang.OutOfMemoryError: Direct buffer memory 如何处理 中 :

    你这些日志都是没问题的,你用的什么命令启动的?

    5年前
  • giant 回复 半兽人kafka 2.11-2.1.1 java.lang.OutOfMemoryError: Direct buffer memory 如何处理 中 :

    没集群,就一台,我半路来的,不清楚安装后干了啥, 现在是启动可以, 但是后面会报如下错误。

    [2019-07-30 17:17:21,586] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
    [2019-07-30 17:17:21,587] INFO Shutting down. (kafka.log.LogManager)
    [2019-07-30 17:17:21,605] INFO [ProducerStateManager partition=__consumer_offsets-21] Writing producer snapshot at offset 425 (kafka.log.ProducerStateManager)
    [2019-07-30 17:17:21,607] INFO [ProducerStateManager partition=D_GENERATE_ADDRESS_TOPIC-0] Writing producer snapshot at offset 2 (kafka.log.ProducerStateManager)
    [2019-07-30 17:17:21,615] INFO [ProducerStateManager partition=__consumer_offsets-44] Writing producer snapshot at offset 171 (kafka.log.ProducerStateManager)
    [2019-07-30 17:17:21,630] INFO Shutdown complete. (kafka.log.LogManager)
    [2019-07-30 17:17:21,639] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
    [2019-07-30 17:17:21,642] INFO Session: 0x10157e437850000 closed (org.apache.zookeeper.ZooKeeper)
    [2019-07-30 17:17:21,643] INFO EventThread shut down for session: 0x10157e437850000 (org.apache.zookeeper.ClientCnxn)
    [2019-07-30 17:17:21,644] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
    [2019-07-30 17:17:21,644] INFO [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:23,793] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:23,793] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:23,794] INFO [SocketServer brokerId=0] Shutting down socket server (kafka.network.SocketServer)
    [2019-07-30 17:17:23,813] INFO [SocketServer brokerId=0] Shutdown completed (kafka.network.SocketServer)
    [2019-07-30 17:17:23,817] INFO [KafkaServer id=0] shut down completed (kafka.server.KafkaServer)
    
    5年前
  • giant 回复 半兽人kafka 2.11-2.1.1 java.lang.OutOfMemoryError: Direct buffer memory 如何处理 中 :

    删掉 log.dirs 下的 文件,重启zk和kafka 后, 第一次使用,可以, 几分钟后就不行了。

    2019-07-30 17:53:58.539  WARN 19940 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient 
      : [Consumer clientId=consumer-1, groupId=ADDRESS.PROCESSOR] 
    Connection to node -1 could not be established. Broker may not be available.
    
    [2019-07-30 17:17:21,588] INFO Shutting down the log cleaner. (kafka.log.LogCleaner)
    [2019-07-30 17:17:21,589] INFO [kafka-log-cleaner-thread-0]: Shutting down (kafka.log.LogCleaner)
    [2019-07-30 17:17:21,589] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner)
    [2019-07-30 17:17:21,589] INFO [kafka-log-cleaner-thread-0]: Shutdown completed (kafka.log.LogCleaner)
    
    
    [2019-07-30 17:17:21,640] INFO Processed session termination for sessionid: 0x10157e437850000 (org.apache.zookeeper.server.PrepRequestProcessor)
    [2019-07-30 17:17:21,643] INFO Closed socket connection for client / which had sessionid 0x10157e437850000 
    (org.apache.zookeeper.server.NIOServerCnxn)
    
    [2019-07-30 17:07:00,588] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)
    [2019-07-30 17:12:00,588] TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
    [2019-07-30 17:12:00,590] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)
    [2019-07-30 17:17:00,590] TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
    [2019-07-30 17:17:00,591] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)
    [2019-07-30 17:17:20,985] INFO [Controller id=0] Shutting down broker 0 (kafka.controller.KafkaController)
    [2019-07-30 17:17:21,630] INFO [ControllerEventThread controllerId=0] Shutting down (kafka.controller.ControllerEventManager$ControllerEventThread)
    [2019-07-30 17:17:21,630] INFO [ControllerEventThread controllerId=0] Stopped (kafka.controller.ControllerEventManager$ControllerEventThread)
    [2019-07-30 17:17:21,630] INFO [ControllerEventThread controllerId=0] Shutdown completed (kafka.controller.ControllerEventManager$ControllerEventThread)
    [2019-07-30 17:17:21,633] INFO [PartitionStateMachine controllerId=0] Stopped partition state machine (kafka.controller.PartitionStateMachine)
    [2019-07-30 17:17:21,633] INFO [ReplicaStateMachine controllerId=0] Stopped replica state machine (kafka.controller.ReplicaStateMachine)
    [2019-07-30 17:17:21,636] INFO [RequestSendThread controllerId=0] Shutting down (kafka.controller.RequestSendThread)
    [2019-07-30 17:17:21,636] INFO [RequestSendThread controllerId=0] Stopped (kafka.controller.RequestSendThread)
    [2019-07-30 17:17:21,636] INFO [RequestSendThread controllerId=0] Shutdown completed (kafka.controller.RequestSendThread)
    [2019-07-30 17:17:21,638] INFO [Controller id=0] Resigned (kafka.controller.KafkaController)
    

    server.log

     Terminating process due to signal SIGHUP (org.apache.kafka.common.utils.LoggingSignalHandler)
    [2019-07-30 17:17:20,969] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)
    [2019-07-30 17:17:20,969] INFO [KafkaServer id=0] Starting controlled shutdown (kafka.server.KafkaServer)
    [2019-07-30 17:17:20,996] INFO [KafkaServer id=0] Controlled shutdown succeeded (kafka.server.KafkaServer)
    [2019-07-30 17:17:20,999] INFO [/config/changes-event-process-thread]: Shutting down (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
    [2019-07-30 17:17:21,000] INFO [/config/changes-event-process-thread]: Shutdown completed (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
    [2019-07-30 17:17:21,000] INFO [/config/changes-event-process-thread]: Stopped (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
    [2019-07-30 17:17:21,000] INFO [SocketServer brokerId=0] Stopping socket server request processors (kafka.network.SocketServer)
    [2019-07-30 17:17:21,011] INFO [SocketServer brokerId=0] Stopped socket server request processors (kafka.network.SocketServer)
    [2019-07-30 17:17:21,012] INFO [Kafka Request Handler on Broker 0], shutting down (kafka.server.KafkaRequestHandlerPool)
    [2019-07-30 17:17:21,014] INFO [Kafka Request Handler on Broker 0], shut down completely (kafka.server.KafkaRequestHandlerPool)
    [2019-07-30 17:17:21,017] INFO [KafkaApi-0] Shutdown complete. (kafka.server.KafkaApis)
    [2019-07-30 17:17:21,017] INFO [ExpirationReaper-0-topic]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,095] INFO [ExpirationReaper-0-topic]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,096] INFO [ExpirationReaper-0-topic]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,098] INFO [TransactionCoordinator id=0] Shutting down. (kafka.coordinator.transaction.TransactionCoordinator)
    [2019-07-30 17:17:21,098] INFO [ProducerId Manager 0]: Shutdown complete: last producerId assigned 159000 (kafka.coordinator.transaction.ProducerIdManager)
    [2019-07-30 17:17:21,099] INFO [Transaction State Manager 0]: Shutdown complete (kafka.coordinator.transaction.TransactionStateManager)
    [2019-07-30 17:17:21,099] INFO [Transaction Marker Channel Manager 0]: Shutting down (kafka.coordinator.transaction.TransactionMarkerChannelManager)
    [2019-07-30 17:17:21,099] INFO [Transaction Marker Channel Manager 0]: Stopped (kafka.coordinator.transaction.TransactionMarkerChannelManager)
    [2019-07-30 17:17:21,100] INFO [Transaction Marker Channel Manager 0]: Shutdown completed (kafka.coordinator.transaction.TransactionMarkerChannelManager)
    [2019-07-30 17:17:21,100] INFO [TransactionCoordinator id=0] Shutdown complete. (kafka.coordinator.transaction.TransactionCoordinator)
    [2019-07-30 17:17:21,100] INFO [GroupCoordinator 0]: Shutting down. (kafka.coordinator.group.GroupCoordinator)
    [2019-07-30 17:17:21,101] INFO [ExpirationReaper-0-Heartbeat]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,298] INFO [ExpirationReaper-0-Heartbeat]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,298] INFO [ExpirationReaper-0-Heartbeat]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,299] INFO [ExpirationReaper-0-Rebalance]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,301] INFO [ExpirationReaper-0-Rebalance]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,301] INFO [ExpirationReaper-0-Rebalance]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,302] INFO [GroupCoordinator 0]: Shutdown complete. (kafka.coordinator.group.GroupCoordinator)
    [2019-07-30 17:17:21,303] INFO [ReplicaManager broker=0] Shutting down (kafka.server.ReplicaManager)
    [2019-07-30 17:17:21,303] INFO [LogDirFailureHandler]: Shutting down (kafka.server.ReplicaManager$LogDirFailureHandler)
    [2019-07-30 17:17:21,303] INFO [LogDirFailureHandler]: Stopped (kafka.server.ReplicaManager$LogDirFailureHandler)
    [2019-07-30 17:17:21,303] INFO [LogDirFailureHandler]: Shutdown completed (kafka.server.ReplicaManager$LogDirFailureHandler)
    [2019-07-30 17:17:21,304] INFO [ReplicaFetcherManager on broker 0] shutting down (kafka.server.ReplicaFetcherManager)
    [2019-07-30 17:17:21,305] INFO [ReplicaFetcherManager on broker 0] shutdown completed (kafka.server.ReplicaFetcherManager)
    [2019-07-30 17:17:21,305] INFO [ReplicaAlterLogDirsManager on broker 0] shutting down (kafka.server.ReplicaAlterLogDirsManager)
    [2019-07-30 17:17:21,305] INFO [ReplicaAlterLogDirsManager on broker 0] shutdown completed (kafka.server.ReplicaAlterLogDirsManager)
    [2019-07-30 17:17:21,305] INFO [ExpirationReaper-0-Fetch]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,344] INFO [ExpirationReaper-0-Fetch]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,344] INFO [ExpirationReaper-0-Fetch]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,345] INFO [ExpirationReaper-0-Produce]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,381] INFO [ExpirationReaper-0-Produce]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,381] INFO [ExpirationReaper-0-Produce]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,381] INFO [ExpirationReaper-0-DeleteRecords]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,582] INFO [ExpirationReaper-0-DeleteRecords]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,583] INFO [ExpirationReaper-0-DeleteRecords]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
    [2019-07-30 17:17:21,586] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
    [2019-07-30 17:17:21,587] INFO Shutting down. (kafka.log.LogManager)
    [2019-07-30 17:17:21,605] INFO [ProducerStateManager partition=__consumer_offsets-21] Writing producer snapshot at offset 425 (kafka.log.ProducerStateManager)
    [2019-07-30 17:17:21,607] INFO [ProducerStateManager partition=D_GENERATE_ADDRESS_TOPIC-0] Writing producer snapshot at offset 2 (kafka.log.ProducerStateManager)
    [2019-07-30 17:17:21,615] INFO [ProducerStateManager partition=__consumer_offsets-44] Writing producer snapshot at offset 171 (kafka.log.ProducerStateManager)
    [2019-07-30 17:17:21,630] INFO Shutdown complete. (kafka.log.LogManager)
    [2019-07-30 17:17:21,639] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
    [2019-07-30 17:17:21,642] INFO Session: 0x10157e437850000 closed (org.apache.zookeeper.ZooKeeper)
    [2019-07-30 17:17:21,643] INFO EventThread shut down for session: 0x10157e437850000 (org.apache.zookeeper.ClientCnxn)
    [2019-07-30 17:17:21,644] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
    [2019-07-30 17:17:21,644] INFO [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:23,793] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:23,793] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    [2019-07-30 17:17:23,794] INFO [SocketServer brokerId=0] Shutting down socket server (kafka.network.SocketServer)
    [2019-07-30 17:17:23,813] INFO [SocketServer brokerId=0] Shutdown completed (kafka.network.SocketServer)
    [2019-07-30 17:17:23,817] INFO [KafkaServer id=0] shut down completed (kafka.server.KafkaServer)
    
    5年前
  • giant 回复 半兽人kafka 2.11-2.1.1 java.lang.OutOfMemoryError: Direct buffer memory 如何处理 中 :

    第一个解决了,l谢谢您, 您是从哪里看出存储文件已经不正确了?

    5年前