Kafka生产者并发创建消息

Running 发表于: 2017-03-22   最后更新时间: 2018-04-27 09:35:44   11,148 游览

前提:

一台broker服务器、一台生产者、无消费者

设置生产者的线程池为20 ,生产10000条数据,生产到8000条左右的时候就提示如下内容。
请问这个到底是哪个地方的连接数不够了呢?broker服务器上对如下都进行了设置 ,仍然没有解决下面的问题

#### I/O线程等待队列中的最大的请求数,超过这个数量,network线程就不会再接收一个新的请求
queued.max.requests= 500

##### The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=104857600

#### The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=104857600

#### The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600

max.connections.per.ip=1000

异常

Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): bind
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:463)
    at sun.nio.ch.Net.bind(Net.java:455)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
    at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:91)
    ... 17 more
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:338)
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:188)
    at com.baozun.demo.util.KafkaSendUtil.sendMessage(KafkaSendUtil.java:41)
    at com.baozun.demo.producer.KafkaProducer$1.run(KafkaProducer.java:58)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.KafkaException: java.io.IOException: Unable to establish loopback connection
    at org.apache.kafka.common.network.Selector.<init>(Selector.java:124)
    at org.apache.kafka.common.network.Selector.<init>(Selector.java:145)
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:303)
    ... 6 more
Caused by: java.io.IOException: Unable to establish loopback connection
    at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:125)
    at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:69)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.nio.ch.PipeImpl.<init>(PipeImpl.java:141)
    at sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50)
    at java.nio.channels.Pipe.open(Pipe.java:150)
    at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:127)
    at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44)
    at java.nio.channels.Selector.open(Selector.java:227)
    at org.apache.kafka.common.network.Selector.<init>(Selector.java:122)
    ... 8 more
Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): bind
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:463)
    at sun.nio.ch.Net.bind(Net.java:455)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
    at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:91)
    ... 17 more
发表于 2017-03-22
添加评论

你的怎么解决的,是在本地运行的吗

除kafka自身的配置外,如果你是linux系统,那就得设置limits.conf。
其次排查程序问题,是20个producer共用一个连接 还是20个producer有20个连接。

你好,能详细说下怎么排查程序问题,以及windows系统需要改哪些配置

你的答案

查看kafka相关的其他问题或提一个您自己的问题