kafka3.3.1通过nginx代理后,生产数据报错NOT_LEADER_OR_FOLLOWER

Jovi 发表于: 2023-01-11   最后更新时间: 2023-01-11 20:53:31   2,219 游览

服务器说明:

*.*.57.243 kafka1
*.*.57.244 kafka2 
*.*.57.245 kafka3 
*.*.60.240 nginx

配置:

kafka1 配置:
listeners=OUTSIDE://center.kafka1:9090,PLAINTEXT://center.kafka1:9092,CONTROLLER://*.*.57.243:9093
kafka2 配置:
listeners=OUTSIDE://center.kafka2:9091,PLAINTEXT://center.kafka2:9092,CONTROLLER://*.*.57.244:9093
kafka3 配置:
listeners=OUTSIDE://center.kafka3:19092,PLAINTEXT://center.kafka3:9092,CONTROLLER://*.*.57.245:9093

host配置(kafka三台服务器及nginx)

*.*.57.243 center.kafka1
*.*.57.244 center.kafka2
*.*.57.245 center.kafka3

转发配置nginx

stream {
   server{
        listen 9090;
        proxy_pass center.kafka1:9092;
    }
    server{
        listen 9091;
        proxy_pass center.kafka2:9092;
    }
    server{
        listen 9092;
        proxy_pass center.kafka3:9092;
    }
}

kafka生产端:

host配置
*.*.60.240 center.kafka1
*.*.60.240 center.kafka2
*.*.60.240 center.kafka3

执行:

./kafka-console-producer.sh --bootstrap-server center.kafka1:9090,center.kafka2:9091,center.kafka3:9092 --topic test2
>sdfsdf
[2023-01-11 17:54:45,344] WARN [Producer clientId=console-producer] Got error produce response with correlation id 7 on topic-partition test2-0, retrying (2 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2023-01-11 17:54:45,345] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test2-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-01-11 17:54:45,457] WARN [Producer clientId=console-producer] Got error produce response with correlation id 9 on topic-partition test2-0, retrying (1 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2023-01-11 17:54:45,457] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test2-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-01-11 17:54:45,569] WARN [Producer clientId=console-producer] Got error produce response with correlation id 11 on topic-partition test2-0, retrying (0 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2023-01-11 17:54:45,569] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test2-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-01-11 17:54:45,685] ERROR Error when sending message to topic test2 with key: null, value: 6 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.
[2023-01-11 17:54:45,686] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test2-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, thi

出错信息:

  1. 内网连接没有这个问题,通过nginx代理会出现这个问题
  2. 这个问题的出现条件为偶现,就是相同的topic开启多次生产后会出现并且出现次数很频繁。
发表于 2023-01-11
添加评论

org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.

意思是当前的发送的topic分区不是leader,所以我是怀疑你的映射错乱了,导致的。

另外中间加一个nginx做代理会影响效率的,最后你的配置还是太少,可以参考 如何从外部访问私网上的kafka集群? 丰富你的参数,尝试解决你的问题。

你的答案

查看kafka相关的其他问题或提一个您自己的问题