kafka报错打开文件数过多导致kafka关闭？

服务器信息：

1. kafka版本：kafka_2.11-0.10.1.0
2. jvm版本：java version "1.7.0_67"
3. kafka最大文件数配置为: 65535

kafka配置如下

broker.id=23
listeners=PLAINTEXT://192.168.1.100:9092
num.network.threads=33
num.io.threads=64
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.dirs=/data/disk01/,/data/disk02/,/data/disk03/,/data/disk04/,/data/disk05/,/data/disk06/,/data/disk07/,/data/disk08/,/data/disk09/,/data/disk10/,/data/disk11/,/data/disk12/
num.partitions=20
num.recovery.threads.per.data.dir=10
log.flush.interval.messages=20000
log.flush.interval.ms=10000
log.flush.scheduler.interval.ms=2000
log.retention.hours=24
log.roll.hours=24
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleanup.interval.mins=10
log.cleaner.enable=false
message.max.bytes=20000000
auto.create.topics.enable=false
log.index.interval.bytes=4096
log.index.size.max.bytes=10485760
replica.fetch.max.bytes=104857600
replica.fetch.wait.max.ms=500
replica.high.watermark.checkpoint.interval.ms=5000
replica.socket.timeout.ms=30000
replica.socket.receive.buffer.bytes=65536
replica.lag.time.max.ms=10000
auto.leader.rebalance.enable=true
delete.topic.enable=true
log.message.format.version=0.8.2
zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181,192.168.1.103:2181,192.168.1.104:2181,192.168.1.105:2181
zookeeper.connection.timeout.ms=6000
zookeeper.sync.time.ms=2000

问题描述：

近期生产环境kafka集群经常发生由于打开文件数过多导致kafka关闭的情况。
每次出现问题的kafka数量为1个，每次出现问题的kafka不固定,日志中有大量报错：

1. 打开文件过多报错前有大量如下info信息，kafka isr由多个变成出错设备自己一个

    Shrinking ISR for partition [some_topic,13] from 23,17 to 17
    Shrinking ISR for partition [some_topic2,12] from 22，23,17 to 17
    Shrinking ISR for partition [some_topic3,11] from 20，23,17 to 17
    Shrinking ISR for partition [some_topic4,12] from 19，25,17 to 17
   

2. 打开文件过多报错前有大量

    [2018-08-20 07:57:09,602] ERROR [ReplicaFetcherThread-0-18], Error for partition [some_topic2,18] to broker 18:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
    [2018-08-20 07:57:09,602] ERROR [ReplicaFetcherThread-0-18], Error for partition [some_topic1,5] to broker 18:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
    [2018-08-20 07:57:09,602] ERROR [ReplicaFetcherThread-0-18], Error for partition [some_topic19,9] to broker 18:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
    [2018-08-20 07:57:09,602] ERROR [ReplicaFetcherThread-0-18], Error for partition [some_topic1,9] to broker 18:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
   

3. 大量打开文件数过多报错

    [2018-08-20 07:57:10,355] ERROR Error while accepting connection (kafka.network.Acceptor)
    java.io.IOException: 打开的文件过多
            at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
            at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
            at kafka.network.Acceptor.accept(SocketServer.scala:326)
            at kafka.network.Acceptor.run(SocketServer.scala:269)
            at java.lang.Thread.run(Thread.java:745)
    [2018-08-20 07:57:10,355] ERROR Error while accepting connection (kafka.network.Acceptor)
    java.io.IOException: 打开的文件过多
            at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
            at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
            at kafka.network.Acceptor.accept(SocketServer.scala:326)
            at kafka.network.Acceptor.run(SocketServer.scala:269)
            at java.lang.Thread.run(Thread.java:745)
    [2018-08-20 07:57:10,355] ERROR Error while accepting connection (kafka.network.Acceptor)
    java.io.IOException: 打开的文件过多
   

问题初步分析：

1. smartacl检查了文件系统，文件系统正常
2. 检控kafka打开文件数，发现是大量close-wait状态socket连接，有1000个左右close-wait状态连接的ip，每个ip有500以上的连接，其中有1000以上连接有50个左右