kafka开启kerberos启动时报错：Controller 0's connection to broker kafka-node1:9092 (id: 0 rack: null) was unsuccessful

陨落星辰、 发表于: 2020-07-07   最后更新时间: 2020-07-07 14:16:52   7,954 游览

1.kafka开启kerberos启动controller.log日志报错连接不上broker

2.搭建环境:操作系统centos7.5，jdk1.8.0_111.tar.gz，apache-zookeeper-3.6.1-bin.tar.gz，kafka_2.12-2.5.0.tgz。
说明：zookeeper未开启kerberos。kafka开启kerberos

3.搭建过程修改参数
（1）server.properties添加部分参数

#其他参数默认未修改
......
broker.id=0
......
zookeeper.connect=kafka-node1:2181,kafka-node2:2181,kafka-node3:2181
zookeeper.connection.timeout.ms=18000

.....

listeners=SASL_PLAINTEXT://kafka-node1:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka

ssl.client.auth=none
ssl.key.password=
ssl.keystore.location=
ssl.keystore.password=
ssl.truststore.location=
ssl.truststore.password=

（2）kafka_server_jaas.conf文件

[root@kafka-node1 kafka_2.12-2.5.0]# pwd
/home/kafka_2.12-2.5.0
[root@kafka-node1 kafka_2.12-2.5.0]# cat config/kafka_server_jaas.conf 
KafkaServer {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/etc/security/kerberos/kafka-node1.keytab"
    principal="kafka/kafka-node1@EXAMPLE.COM";
};

[root@kafka-node1 kafka_2.12-2.5.0]#

(3)kafka-run-class.sh添加启动参数

......
if [ -z "$KAFKA_OPTS" ]; then
  KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/home/kafka_2.12-2.5.0/config/kafka_server_jaas.conf"
fi
......

4.启动后查看controller.log日志有大量警告日志如下：

[2020-07-07 13:04:07,566] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker kafka-node1:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
org.apache.kafka.common.errors.SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Ticket expired (32) - PROCESS_TGS)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Ticket expired (32) - PROCESS_TGS)]
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:483)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:483)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:390)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:296)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:237)
    at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:177)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)
    at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:74)
    at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:288)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:242)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Caused by: GSSException: No valid credentials provided (Mechanism level: Ticket expired (32) - PROCESS_TGS)
    at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
    ... 15 more
Caused by: KrbException: Ticket expired (32) - PROCESS_TGS
    at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
    at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:259)
    at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:270)
    at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302)
    at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120)
    at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
    at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
    ... 18 more
Caused by: KrbException: Identifier doesn't match expected value (906)
    at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
    at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
    at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
    at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
    ... 24 more
[2020-07-07 13:04:07,670] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker kafka-node1:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to kafka-node1:9092 (id: 0 rack: null) failed.
    at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
    at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:288)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:242)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
[2020-07-07 13:04:07,772] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker kafka-node1:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to kafka-node1:9092 (id: 0 rack: null) failed.
    at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
    at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:288)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:242)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

5.验证kinit过程成功

[root@kafka-node1 kafka_2.12-2.5.0]# klist
klist: Credentials cache keyring 'persistent:0:0' not found
[root@kafka-node1 kafka_2.12-2.5.0]# kinit  -kt /etc/security/kerberos/kafka-node1.keytab kafka/kafka-node1
[root@kafka-node1 kafka_2.12-2.5.0]# klist
Ticket cache: KEYRING:persistent:0:0
Default principal: kafka/kafka-node1@EXAMPLE.COM

Valid starting       Expires              Service principal
2020-07-07T12:55:37  2020-07-08T12:55:37  krbtgt/EXAMPLE.COM@EXAMPLE.COM
[root@kafka-node1 kafka_2.12-2.5.0]#

6.在kafka-run-class.sh添加kerberosdebug日志参数:

    -Dsun.security.krb5.debug=true
if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then
  KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Dsun.security.krb5.debug=true"
fi

7.重新启动controller.log日志报错依旧，输出日志发现票据过期

[2020-07-07 13:04:07,156] INFO [Controller id=0, targetBrokerId=0] Failed authentication with kafka-node1/10.10.10.101 (An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Ticket expired (32) - PROCESS_TGS)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.) (org.apache.kafka.common.network.Selector)
[2020-07-07 13:04:07,157] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (kafka-node1/10.10.10.101:9092) failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Ticket expired (32) - PROCESS_TGS)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state. (org.apache.kafka.clients.NetworkClient)
Found KeyTab /etc/security/kerberos/kafka-node1.keytab for kafka/kafka-node1@EXAMPLE.COM
Found ticket for kafka/kafka-node1@EXAMPLE.COM to go to krbtgt/EXAMPLE.COM@EXAMPLE.COM expiring on Sun Feb 07 14:28:15 CST 2106
Found ticket for kafka/kafka-node1@EXAMPLE.COM to go to krbtgt/EXAMPLE.COM@EXAMPLE.COM expiring on Sun Feb 07 14:28:15 CST 2106
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for kafka/kafka-node1@EXAMPLE.COM to go to krbtgt/EXAMPLE.COM@EXAMPLE.COM expiring on Sun Feb 07 14:28:15 CST 2106
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
>>> KrbKdcReq send: kdc=kafka-node1 UDP:88, timeout=30000, number of retries =3, #bytes=648
>>> KDCCommunication: kdc=kafka-node1 UDP:88, timeout=30000,Attempt =1, #bytes=648
>>> KrbKdcReq send: #bytes read=176
>>> KdcAccessibility: remove kafka-node1
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
     cTime is Sat Jul 03 12:52:23 CST 2027 1814590343000
     sTime is Tue Jul 07 13:04:07 CST 2020 1594098247000
     suSec is 25565
     error code is 32
     error Message is Ticket expired
     cname is kafka/kafka-node1@EXAMPLE.COM
     sname is kafka/kafka-node1@EXAMPLE.COM
     msgType is 30
KrbException: Ticket expired (32) - PROCESS_TGS
    at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
    at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:259)
    at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:270)
    at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302)
    at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120)
    at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
    at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:483)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:483)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:390)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:296)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:237)
    at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:177)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)
    at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:74)
    at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:288)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:242)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Caused by: KrbException: Identifier doesn't match expected value (906)
    at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
    at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
    at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
    at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
    ... 24 more

cotroller日志报错连接不到broker,kerberos的DEBUG日志，一直报错票据过期,直接kinit验证又正常，找不到排查思路了。求助！！！

kafka
发表于 2020-07-07
陨落星辰、
添加评论

参考的这个吗？
https://www.orchome.com/500

回复
回答于 4年前
半兽人

是的，我zookeeper集群是单独搭的一个未开启kerberos，所以kafka_server_jaas.conf文件里面只加了KafkaServer配置

0 0 回复

zookeeper连接kafka也需要配置。

0 0 回复
你的答案

查看kafka相关的其他问题或提一个您自己的问题。
提问

找不到想要的答案？提一个您自己的问题。