kafka实战kerberos(笔记)

半兽人 发表于: 2017-01-10   最后更新时间: 2017-11-14  
  •   259 订阅,14557 游览

1 安装配置kerberos

more /etc/krb5.conf

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = EXAMPLE.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true

[realms]
 EXAMPLE.COM = {
  kdc = 10.211.55.5
  admin_server = 10.211.55.5
 }

[domain_realm]
kafka = EXAMPLE.COM
zookeeper = EXAMPLE.COM
weiwei = EXAMPLE.COM
10.211.55.5 = EXAMPLE.COM
127.0.0.1 = EXAMPLE.COM

kadmin.local

Authenticating as principal kafka/admin@EXAMPLE.COM with password.
kadmin.local:  listprincs
K/M@EXAMPLE.COM
clients@EXAMPLE.COM
host/10.211.55.5@EXAMPLE.COM
host/weiwei@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/weiwei@EXAMPLE.COM
kafka/10.211.55.5@EXAMPLE.COM
kafka/127.0.0.1@EXAMPLE.COM
kafka/localhost@EXAMPLE.COM
kafka/weiwei@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
root/admin@EXAMPLE.COM
zookeeper/10.211.55.5@EXAMPLE.COM
zookeeper/127.0.0.1@EXAMPLE.COM

klist -t -e -k /var/kerberos/krb5kdc/kafka.keytab

Keytab name: FILE:/var/kerberos/krb5kdc/kafka.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   3 07/24/16 00:58:30 kafka/10.211.55.5@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   3 07/24/16 00:58:30 kafka/10.211.55.5@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
   3 07/24/16 00:58:30 kafka/10.211.55.5@EXAMPLE.COM (des3-cbc-sha1)
   3 07/24/16 00:58:30 kafka/10.211.55.5@EXAMPLE.COM (arcfour-hmac)
   3 07/24/16 00:58:30 kafka/10.211.55.5@EXAMPLE.COM (des-hmac-sha1)
   3 07/24/16 00:58:30 kafka/10.211.55.5@EXAMPLE.COM (des-cbc-md5)
   2 07/24/16 12:23:18 zookeeper/10.211.55.5@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   2 07/24/16 12:23:18 zookeeper/10.211.55.5@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
   2 07/24/16 12:23:18 zookeeper/10.211.55.5@EXAMPLE.COM (des3-cbc-sha1)
   2 07/24/16 12:23:18 zookeeper/10.211.55.5@EXAMPLE.COM (arcfour-hmac)
   2 07/24/16 12:23:18 zookeeper/10.211.55.5@EXAMPLE.COM (des-hmac-sha1)
   2 07/24/16 12:23:18 zookeeper/10.211.55.5@EXAMPLE.COM (des-cbc-md5)
   2 07/25/16 11:31:37 kafka/127.0.0.1@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   2 07/25/16 11:31:37 kafka/127.0.0.1@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
   2 07/25/16 11:31:37 kafka/127.0.0.1@EXAMPLE.COM (des3-cbc-sha1)
   2 07/25/16 11:31:37 kafka/127.0.0.1@EXAMPLE.COM (arcfour-hmac)
   2 07/25/16 11:31:37 kafka/127.0.0.1@EXAMPLE.COM (des-hmac-sha1)
   2 07/25/16 11:31:37 kafka/127.0.0.1@EXAMPLE.COM (des-cbc-md5)
   3 07/25/16 13:13:31 kafka/weiwei@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   3 07/25/16 13:13:31 kafka/weiwei@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
   3 07/25/16 13:13:31 kafka/weiwei@EXAMPLE.COM (des3-cbc-sha1)
   3 07/25/16 13:13:31 kafka/weiwei@EXAMPLE.COM (arcfour-hmac)
   3 07/25/16 13:13:31 kafka/weiwei@EXAMPLE.COM (des-hmac-sha1)
   3 07/25/16 13:13:31 kafka/weiwei@EXAMPLE.COM (des-cbc-md5)
   2 07/25/16 15:07:58 zookeeper/127.0.0.1@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   2 07/25/16 15:07:58 zookeeper/127.0.0.1@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
   2 07/25/16 15:07:58 zookeeper/127.0.0.1@EXAMPLE.COM (des3-cbc-sha1)
   2 07/25/16 15:07:58 zookeeper/127.0.0.1@EXAMPLE.COM (arcfour-hmac)
   2 07/25/16 15:07:58 zookeeper/127.0.0.1@EXAMPLE.COM (des-hmac-sha1)
   2 07/25/16 15:07:58 zookeeper/127.0.0.1@EXAMPLE.COM (des-cbc-md5)
   2 07/25/16 18:47:55 clients@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   2 07/25/16 18:47:55 clients@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
   2 07/25/16 18:47:55 clients@EXAMPLE.COM (des3-cbc-sha1)
   2 07/25/16 18:47:55 clients@EXAMPLE.COM (arcfour-hmac)
   2 07/25/16 18:47:55 clients@EXAMPLE.COM (des-hmac-sha1)
   2 07/25/16 18:47:55 clients@EXAMPLE.COM (des-cbc-md5)

more zookeeper_jaas.conf

Server{
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    useTicketCache=false
    keyTab="/var/kerberos/krb5kdc/kafka.keytab"
    principal="zookeeper/10.211.55.5@EXAMPLE.COM";
};

more kafka_server_jaas.conf

KafkaServer {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        keyTab="/var/kerberos/krb5kdc/kafka.keytab"
    principal="kafka/weiwei@EXAMPLE.COM";
};

Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/var/kerberos/krb5kdc/kafka.keytab"
    principal="kafka/weiwei@EXAMPLE.COM";
};

more config/server.properties

advertised.host.name=10.211.55.5
advertised.listeners=SASL_PLAINTEXT://10.211.55.5:9093
listeners=SASL_PLAINTEXT://10.211.55.5:9093
#listeners=PLAINTEXT://127.0.0.1:9093
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI

sasl.kerberos.service.name=kafka

more start-zk-and-kafka

#!/bin/bash
export KAFKA_HEAP_OPTS='-Xmx256M'
export KAFKA_OPTS='-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.confi
g=/etc/kafka/zookeeper_jaas.conf'
bin/zookeeper-server-start.sh config/zookeeper.properties &

sleep 5

export KAFKA_OPTS='-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.confi
g=/etc/kafka/kafka_server_jaas.conf'
bin/kafka-server-start.sh config/server.properties

more config/zookeeper.properties

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000

more config/producer.properties/consumer.properties

security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka

more producer2

export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.confi
g=/etc/kafka/kafka_client_jaas.conf"

bin/kafka-console-producer.sh --broker-list 10.211.55.5:9093 --topic test --producer.config
config/producer.properties

more consumer2

export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.confi
g=/etc/kafka/kafka_client_jaas.conf"

bin/kafka-console-consumer.sh --bootstrap-server 10.211.55.5:9093 --topic test --new-consume
r --from-beginning --consumer.config config/consumer.properties






发表于: 1年前   最后更新时间: 7月前   游览量:14557
上一条: 获取kafka版本
下一条: Kafka Stream演示程序

评论…


  • hi, 按照文档配置,遇到  could not login:the client is being asked for a password, bug kafka client code does not currently support,求大神解答
    按照上面的配置,Kafka单机没有问题,启用多台Kafka服务器,broker报错如下:
    [2018-05-03 14:52:13,975] ERROR [Controller id=1, targetBrokerId=0] Connection to node 0 failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state. (org.apache.kafka.clients.NetworkClient)
    我会遇到这个问题: 是不是哪里配置不多
    [2018-04-27 07:00:54,558] WARN SASL configuration failed: javax.security.auth.login.LoginException: Cannot locate KDC Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
    [2018-04-27 07:00:54,560] INFO Opening socket connection to server 192.168.182.17/192.168.182.17:10363 (org.apache.zookeeper.ClientCnxn)
    [2018-04-27 07:00:54,561] INFO zookeeper state changed (AuthFailed) (org.I0Itec.zkclient.ZkClient)
    [2018-04-27 07:00:54,561] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
    [2018-04-27 07:00:54,565] INFO Socket connection established to 192.168.182.17/192.168.182.17:10363, initiating session (org.apache.zookeeper.ClientCnxn)
    [2018-04-27 07:00:54,572] INFO Session establishment complete on server 192.168.182.17/192.168.182.17:10363, sessionid = 0x10130ca09830001, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
    [2018-04-27 07:00:54,574] INFO Session: 0x10130ca09830001 closed (org.apache.zookeeper.ZooKeeper)
    [2018-04-27 07:00:54,575] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
    org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure
     at org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.java:947)
     at org.I0Itec.zkclient.ZkClient.waitUntilConnected(ZkClient.java:924)
     at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1231)
     at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157)
     at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131)
     at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:115)
     at kafka.utils.ZkUtils$.withMetrics(ZkUtils.scala:92)
     at kafka.server.KafkaServer.initZk(KafkaServer.scala:346)
     at kafka.server.KafkaServer.startup(KafkaServer.scala:194)
     at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
     at kafka.Kafka$.main(Kafka.scala:92)
     at kafka.Kafka.main(Kafka.scala)
    [2018-04-27 07:00:54,576] INFO EventThread shut down for session: 0x10130ca09830001 (org.apache.zookeeper.ClientCnxn)
    [2018-04-27 07:00:54,578] INFO shutting down (kafka.server.KafkaServer)
    [2018-04-27 07:00:54,583] INFO shut down completed (kafka.server.KafkaServer)
    [2018-04-27 07:00:54,583] FATAL Exiting Kafka. (kafka.server.KafkaServerStartable)
    [2018-04-27 07:00:54,585] INFO shutting down (kafka.server.KafkaServer)

    你好! 
         在kafka配置文件server.properties中添加参数zookeeper.set.acl=true后,运行kafka.topics.sh创建topic时报错如下:
    Error while executing topic command : org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/topics
    [2018-04-26 16:30:43,027] ERROR org.I0Itec.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/topics
     at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:68)
     at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:1001)
    问一下kafka.topics.sh具体怎么配置?

    • export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.confi
      g=/etc/kafka/kafka_client_jaas.conf"

      bin/kafka-console-consumer.sh --bootstrap-server 10.211.55.5:9093 --topic test --new-consume
      r --from-beginning --consumer.config config/consumer.properties
        我启动之后显示认证失败,这是什么问题?
        ERROR An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state. (org.apache.zookeeper.client.ZooKeeperSaslClient)
        ERROR SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
        INFO zookeeper state changed (AuthFailed) (org.I0Itec.zkclient.ZkClient)
        INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
        FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
        org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure

        你好,export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.confi
        g=/etc/kafka/kafka_client_jaas.conf"这个配置是要添加在kafka-console-producer.sh和kafka-console-consumer.sh这两个脚本里面是吧?

        • 按步骤走的,启动kafka时一直警告无效证书,有什么方法可以看下认证过程,或者定位到问题
          [2017-11-15 14:49:29,820] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
          [2017-11-15 14:49:29,915] WARN Connection to node 0 terminated during authentication. This may indicate that authentication failed due to invalid credentials. (org.apache.kafka.clients.NetworkClient)
          • 评论…
            • in this conversation
              提问