求救:java 连接 kafka kerberos 问题

按照博主大大的笔记,安装了kerbros认证,但是用java连接kafka时,报错:

Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner  authentication information from the user

详细日志:

Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:717)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:597)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:579)
    at kafka.test.KbConsumer.main(KbConsumer.java:30)
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner  authentication information from the user
    at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:94)
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:93)
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:51)
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:84)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:657)
    ... 3 more
Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner  authentication information from the user
    at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:940)
    at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
    at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
    at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:58)
    at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:109)
    at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:55)
    at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:89)
    at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
    ... 7 more

代码以及配置:
Consumer.java

 public class KbConsumer {
    public static void main(String[] args) {
        System.setProperty("java.security.krb5.conf",System.getProperty("user.dir") + "\\krb5.conf");
        System.setProperty("java.security.auth.login.config", System.getProperty("user.dir") + "\\kafka_client_jaas.conf");
        Properties props = new Properties();
        props.put(BOOTSTRAP_SERVERS_CONFIG, "10.1.2.46:1234");
        props.put(ENABLE_AUTO_COMMIT_CONFIG, "true");
        props.put(GROUP_ID_CONFIG, "test_consumer_group");
        props.put(AUTO_COMMIT_INTERVAL_MS_CONFIG, 1000);
        props.put(AUTO_OFFSET_RESET_CONFIG, "earliest");
        props.put("sasl.kerberos.service.name", "kafka");
        props.put(KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
        props.put(VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
        props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singleton("wwxx"));
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(100);
            for (ConsumerRecord<String, String> record : records)
                System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());

        }
    }
}

kafka_client_jaas.conf

KafkaClient {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        useTicketCache=true
        keyTab="/etc/security/keytabs/kafka.keytab"
        principal="clients@EX.COM";
};

krb5.conf

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = EX.COM
 dns_lookup_realm = false
 dns_lookup_kdc = true
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true

[realms]
 EX.COM = {
  kdc = 10.1.2.46
  admin_server = 10.1.2.46
 }

[domain_realm]
kafka = EX.COM
host = EX.COM
zookeeper = EX.COM
127.0.0.1 = EX.COM
10.1.2.46 = EX.COM
bd005 = EX.COM

kafka.keytab

4 03/13/18 14:27:39 zookeeper/10.1.2.46@EX.COM (aes128-cts-hmac-sha1-96) 
   4 03/13/18 14:27:39 zookeeper/10.1.2.46@EX.COM (des-hmac-sha1) 
   4 03/13/18 14:27:39 zookeeper/10.1.2.46@EX.COM (arcfour-hmac) 
   4 03/13/18 14:27:39 zookeeper/10.1.2.46@EX.COM (des-cbc-md5) 
   4 03/13/18 14:27:39 zookeeper/10.1.2.46@EX.COM (des3-cbc-sha1) 
   2 03/13/18 14:28:25 kafka/10.1.2.46@EX.COM (aes128-cts-hmac-sha1-96) 
   2 03/13/18 14:28:25 kafka/10.1.2.46@EX.COM (des3-cbc-sha1) 
   2 03/13/18 14:28:25 kafka/10.1.2.46@EX.COM (arcfour-hmac) 
   2 03/13/18 14:28:25 kafka/10.1.2.46@EX.COM (des-hmac-sha1) 
   2 03/13/18 14:28:25 kafka/10.1.2.46@EX.COM (des-cbc-md5) 
   4 03/13/18 14:27:39 zookeeper/10.1.2.46@EX.COM (aes256-cts-hmac-sha1-96) 
   2 03/13/18 10:26:01 kafka/127.0.0.1@EX.COM (aes256-cts-hmac-sha1-96) 
   2 03/13/18 10:26:01 kafka/127.0.0.1@EX.COM (aes128-cts-hmac-sha1-96) 
   2 03/13/18 10:26:01 kafka/127.0.0.1@EX.COM (des3-cbc-sha1) 
   2 03/13/18 10:26:01 kafka/127.0.0.1@EX.COM (arcfour-hmac) 
   2 03/13/18 10:26:01 kafka/127.0.0.1@EX.COM (des-hmac-sha1) 
   2 03/13/18 10:26:01 kafka/127.0.0.1@EX.COM (des-cbc-md5) 
   2 03/13/18 10:26:34 zookeeper/127.0.0.1@EX.COM (aes256-cts-hmac-sha1-96) 
   2 03/13/18 10:26:34 zookeeper/127.0.0.1@EX.COM (aes128-cts-hmac-sha1-96) 
   2 03/13/18 10:26:34 zookeeper/127.0.0.1@EX.COM (des3-cbc-sha1) 
   2 03/13/18 10:26:34 zookeeper/127.0.0.1@EX.COM (arcfour-hmac) 
   2 03/13/18 10:26:34 zookeeper/127.0.0.1@EX.COM (des-hmac-sha1) 
   2 03/13/18 10:26:34 zookeeper/127.0.0.1@EX.COM (des-cbc-md5) 
   2 03/13/18 10:28:19 kafka/bd005@EX.COM (aes256-cts-hmac-sha1-96) 
   2 03/13/18 10:28:19 kafka/bd005@EX.COM (aes128-cts-hmac-sha1-96) 
   2 03/13/18 10:28:19 kafka/bd005@EX.COM (des3-cbc-sha1) 
   2 03/13/18 10:28:19 kafka/bd005@EX.COM (arcfour-hmac) 
   2 03/13/18 10:28:19 kafka/bd005@EX.COM (des-hmac-sha1) 
   2 03/13/18 10:28:19 kafka/bd005@EX.COM (des-cbc-md5) 
   2 03/13/18 10:31:45 clients@EX.COM (aes256-cts-hmac-sha1-96) 
   2 03/13/18 10:31:45 clients@EX.COM (aes128-cts-hmac-sha1-96) 
   2 03/13/18 10:31:45 clients@EX.COM (des3-cbc-sha1) 
   2 03/13/18 10:31:45 clients@EX.COM (arcfour-hmac) 
   2 03/13/18 10:31:45 clients@EX.COM (des-hmac-sha1) 
   2 03/13/18 10:31:45 clients@EX.COM (des-cbc-md5) 
   2 03/13/18 14:28:25 kafka/10.1.2.46@EX.COM (aes256-cts-hmac-sha1-96)

principal

   K/M@EX.COM
admin/admin@EX.COM
clients@EX.COM
host/10.1.2.46@EX.COM
host/bd005@EX.COM
kadmin/admin@EX.COM
kadmin/bd005@EX.COM
kadmin/changepw@EX.COM
kafka/10.1.2.46@EX.COM
kafka/127.0.0.1@EX.COM
kafka/bd005@EX.COM
krbtgt/EX.COM@EX.COM
test/10.1.2.46@EX.COM
test/bd005@EX.COM
zookeeper/10.1.2.46@EX.COM
zookeeper/127.0.0.1@EX.COM

kafka可以正常启动,也可以自主生产消费。consumer的代码可能有误,新人刚接触kafka,还望高人指点。
此外还有一个问题,对topic添加指定权限后,在其他服务器上也无法对topic进行读写。
使用的命令:

bin/kafka-acls.sh --authorizer-properties zookeeper.connect={zookeeper-host} --add --allow-principal User:* --allow-host *   --operation all --topic wwxx





发表于: 8月前   最后更新时间: 8月前   游览量:2667
上一条: 到头了!
下一条: 已经是最后了!

评论…


  • 这个错误是说kafka客户端不支持获取认证信息,把认证加到jvm环境中。
    • 嗯,现在解决了,问题出在jaas.conf文件的  keyTab="D:/iSpace/kafka/kafka.keytab",这个我之前指定的路径是服务器上的,改成本地就可以了。不过还有一个问题,就是我新创建了一个topic,没有分配权限,但是可以生产和消费……
        • 最后那个分配权限的命令是根据之前做SASL_PLAIN认证写的,意思是想允许来自任何IP的任何用户都可以对topic进行读写。但是,我新创建一个了topic,不分配任何权限也可以进行读写,不知道是不是SASL没生效的问题……
            • 是的,刚才又跑了遍代码,又出现那个问题了,上午的时候把
              -Djava.security.krb5.conf=D:/iSpace/kafka/krb5.conf
              -Djava.security.auth.login.config=D:/iSpace/kafka/kafka_client_jaas.conf 
              添加到 jvm 里,把 jaas.conf 的 keyTab="D:/iSpace/kafka/kafka.keytab"也改成本地的了,确实跑通了啊,见鬼了……
                • 我在server.properties中添加了

                  authorizer.class.name = kafka.security.auth.SimpleAclAuthorizer
                  super.users=User:kafka/bd005@EX.COM

                  然后给 pruducer 和 consumer 分配权限,这是分配后的信息

                  Current ACLs for resource `Topic:wwxx1`: 
                    User:clients has Allow permission for operations: Read from hosts: 10.1.2.46
                   User:clients@EX.COM has Allow permission for operations: Write from hosts: 10.1.2.46
                   User:clients@EX.COM has Allow permission for operations: Read from hosts: 10.1.2.46
                   User:clients has Allow permission for operations: Write from hosts: 10.1.2.46 

                  Current ACLs for resource `Group:news-consumer-group`: 
                    User:clients has Allow permission for operations: Read from hosts: 10.1.2.46
                   User:clients@EX.COM has Allow permission for operations: Read from hosts: 10.1.2.46 

                  通过命令生产和消费,分别出现了:
                  生产:WARN Bootstrap broker 10.1.2.46:9988 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
                  消费:WARN Error while fetching metadata with correlation id 3 : {wwxx1=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
                  能看出来是哪个地方出现了问题吗?
                    • kerberos用户认证通了,但是ACL有问题,给topic分配权限后,使用命令测试就出现上面的错,客户端也是,可以连上kafka但是发送/消费不了。http://orchome.com/500 我是按照这个教程做的, 也看了 http://www.orchome.com/378 这个问题,但是没理解……
                      • 评论…
                        • in this conversation