服务器说明:
*.*.57.243 kafka1
*.*.57.244 kafka2
*.*.57.245 kafka3
*.*.60.240 nginx
配置:
kafka1 配置:
listeners=OUTSIDE://center.kafka1:9090,PLAINTEXT://center.kafka1:9092,CONTROLLER://*.*.57.243:9093
kafka2 配置:
listeners=OUTSIDE://center.kafka2:9091,PLAINTEXT://center.kafka2:9092,CONTROLLER://*.*.57.244:9093
kafka3 配置:
listeners=OUTSIDE://center.kafka3:19092,PLAINTEXT://center.kafka3:9092,CONTROLLER://*.*.57.245:9093
host配置(kafka三台服务器及nginx)
*.*.57.243 center.kafka1
*.*.57.244 center.kafka2
*.*.57.245 center.kafka3
转发配置nginx
stream {
server{
listen 9090;
proxy_pass center.kafka1:9092;
}
server{
listen 9091;
proxy_pass center.kafka2:9092;
}
server{
listen 9092;
proxy_pass center.kafka3:9092;
}
}
kafka生产端:
host配置
*.*.60.240 center.kafka1
*.*.60.240 center.kafka2
*.*.60.240 center.kafka3
执行:
./kafka-console-producer.sh --bootstrap-server center.kafka1:9090,center.kafka2:9091,center.kafka3:9092 --topic test2
>sdfsdf
[2023-01-11 17:54:45,344] WARN [Producer clientId=console-producer] Got error produce response with correlation id 7 on topic-partition test2-0, retrying (2 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2023-01-11 17:54:45,345] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test2-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-01-11 17:54:45,457] WARN [Producer clientId=console-producer] Got error produce response with correlation id 9 on topic-partition test2-0, retrying (1 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2023-01-11 17:54:45,457] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test2-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-01-11 17:54:45,569] WARN [Producer clientId=console-producer] Got error produce response with correlation id 11 on topic-partition test2-0, retrying (0 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2023-01-11 17:54:45,569] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test2-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-01-11 17:54:45,685] ERROR Error when sending message to topic test2 with key: null, value: 6 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.
[2023-01-11 17:54:45,686] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test2-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, thi
出错信息:
- 内网连接没有这个问题,通过nginx代理会出现这个问题
- 这个问题的出现条件为偶现,就是相同的topic开启多次生产后会出现并且出现次数很频繁。
意思是当前的发送的topic分区不是leader,所以我是怀疑你的映射错乱了,导致的。
另外中间加一个nginx做代理会影响效率的,最后你的配置还是太少,可以参考 如何从外部访问私网上的kafka集群? 丰富你的参数,尝试解决你的问题。
你的答案