在使用kerberos的时候,遇到了以下的错误:
zookeeper.out:
2018-01-21 20:01:43,434 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /192.168.137.98:43432
2018-01-21 20:01:43,463 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /192.168.137.98:43432
2018-01-21 20:01:43,482 [myid:] - INFO [SyncThread:0:ZooKeeperServer@687] - Established session 0x1611883cc1d0004 with negotiated timeout 6000 for client /192.168.137.98:43432
2018-01-21 20:01:43,523 [myid:] - ERROR [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1055] - cnxn.saslServer is null: cnxn object did not initialize its saslServer properly.
2018-01-21 20:01:43,877 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@368] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1611883cc1d0004, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
at java.lang.Thread.run(Thread.java:748)
2018-01-21 20:01:43,878 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /192.168.137.98:43432 which had sessionid 0x1611883cc1d0004
kafka log:
[2018-01-21 20:01:42,624] INFO KafkaConfig values:
advertised.host.name = null
advertised.listeners = null
advertised.port = null
alter.config.policy.class.name = null
authorizer.class.name =
auto.create.topics.enable = true
auto.leader.rebalance.enable = true
background.threads = 10
broker.id = 0
broker.id.generation.enable = true
broker.rack = null
compression.type = producer
connections.max.idle.ms = 600000
controlled.shutdown.enable = true
controlled.shutdown.max.retries = 3
controlled.shutdown.retry.backoff.ms = 5000
controller.socket.timeout.ms = 30000
create.topic.policy.class.name = null
default.replication.factor = 1
delete.records.purgatory.purge.interval.requests = 1
delete.topic.enable = true
fetch.purgatory.purge.interval.requests = 1000
group.initial.rebalance.delay.ms = 0
group.max.session.timeout.ms = 300000
group.min.session.timeout.ms = 6000
host.name =
inter.broker.listener.name = null
inter.broker.protocol.version = 1.0-IV0
leader.imbalance.check.interval.seconds = 300
leader.imbalance.per.broker.percentage = 10
listener.security.protocol.map = PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
listeners = SASL_PLAINTEXT://192.168.137.98:9092
log.cleaner.backoff.ms = 15000
log.cleaner.dedupe.buffer.size = 134217728
log.cleaner.delete.retention.ms = 86400000
log.cleaner.enable = true
log.cleaner.io.buffer.load.factor = 0.9
log.cleaner.io.buffer.size = 524288
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
log.cleaner.min.cleanable.ratio = 0.5
log.cleaner.min.compaction.lag.ms = 0
log.cleaner.threads = 1
log.cleanup.policy = [delete]
log.dir = /tmp/kafka-logs
log.dirs = /usr/local/kafka/kafka-logs
log.flush.interval.messages = 9223372036854775807
log.flush.interval.ms = null
log.flush.offset.checkpoint.interval.ms = 60000
log.flush.scheduler.interval.ms = 9223372036854775807
log.flush.start.offset.checkpoint.interval.ms = 60000
log.index.interval.bytes = 4096
log.index.size.max.bytes = 10485760
log.message.format.version = 1.0-IV0
log.message.timestamp.difference.max.ms = 9223372036854775807
log.message.timestamp.type = CreateTime
log.preallocate = false
log.retention.bytes = -1
log.retention.check.interval.ms = 300000
log.retention.hours = 168
log.retention.minutes = null
log.retention.ms = null
log.roll.hours = 168
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 1073741824
log.segment.delete.delay.ms = 60000
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
message.max.bytes = 1000012
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
min.insync.replicas = 1
num.io.threads = 8
num.network.threads = 3
num.partitions = 1
num.recovery.threads.per.data.dir = 1
num.replica.fetchers = 1
offset.metadata.max.bytes = 4096
offsets.commit.required.acks = -1
offsets.commit.timeout.ms = 5000
offsets.load.buffer.size = 5242880
offsets.retention.check.interval.ms = 600000
offsets.retention.minutes = 1440
offsets.topic.compression.codec = 0
offsets.topic.num.partitions = 50
offsets.topic.replication.factor = 1
offsets.topic.segment.bytes = 104857600
port = 9092
principal.builder.class = null
producer.purgatory.purge.interval.requests = 1000
queued.max.request.bytes = -1
queued.max.requests = 500
quota.consumer.default = 9223372036854775807
quota.producer.default = 9223372036854775807
quota.window.num = 11
quota.window.size.seconds = 1
replica.fetch.backoff.ms = 1000
replica.fetch.max.bytes = 1048576
replica.fetch.min.bytes = 1
replica.fetch.response.max.bytes = 10485760
replica.fetch.wait.max.ms = 500
replica.high.watermark.checkpoint.interval.ms = 5000
replica.lag.time.max.ms = 10000
replica.socket.receive.buffer.bytes = 65536
replica.socket.timeout.ms = 30000
replication.quota.window.num = 11
replication.quota.window.size.seconds = 1
request.timeout.ms = 30000
reserved.broker.max.id = 1000
sasl.enabled.mechanisms = [GSSAPI]
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.principal.to.local.rules = [DEFAULT]
sasl.kerberos.service.name = kafka
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism.inter.broker.protocol = GSSAPI
security.inter.broker.protocol = SASL_PLAINTEXT
socket.receive.buffer.bytes = 102400
socket.request.max.bytes = 104857600
socket.send.buffer.bytes = 102400
ssl.cipher.suites = null
ssl.client.auth = none
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.abort.timed.out.transaction.cleanup.interval.ms = 60000
transaction.max.timeout.ms = 900000
transaction.remove.expired.transaction.cleanup.interval.ms = 3600000
transaction.state.log.load.buffer.size = 5242880
transaction.state.log.min.isr = 1
transaction.state.log.num.partitions = 50
transaction.state.log.replication.factor = 1
transaction.state.log.segment.bytes = 104857600
transactional.id.expiration.ms = 604800000
unclean.leader.election.enable = false
zookeeper.connect = 192.168.137.98:2181
zookeeper.connection.timeout.ms = 6000
zookeeper.session.timeout.ms = 6000
zookeeper.set.acl = false
zookeeper.sync.time.ms = 2000
(kafka.server.KafkaConfig)
[2018-01-21 20:01:42,773] INFO starting (kafka.server.KafkaServer)
[2018-01-21 20:01:42,775] INFO Connecting to zookeeper on 192.168.137.98:2181 (kafka.server.KafkaServer)
[2018-01-21 20:01:42,802] INFO JAAS File name: /usr/local/kafka/config/kafka_server_jaas.conf (org.I0Itec.zkclient.ZkClient)
[2018-01-21 20:01:42,823] INFO Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,823] INFO Client environment:host.name=rh74v1.sample1.com (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:java.version=1.8.0_151 (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:java.home=/usr/java/jdk1.8.0_151/jre (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:java.class.path=.:/usr/java/jdk1.8.0_151/lib:/usr/java/jdk1.8.0_151/jre/lib::/usr/local/kafka/bin/../libs/aopalliance-repackaged-2.5.0-b32.jar:/usr/local/kafka/bin/../libs/argparse4j-0.7.0.jar:/usr/local/kafka/bin/../libs/commons-lang3-3.5.jar:/usr/local/kafka/bin/../libs/connect-api-1.0.0.jar:/usr/local/kafka/bin/../libs/connect-file-1.0.0.jar:/usr/local/kafka/bin/../libs/connect-json-1.0.0.jar:/usr/local/kafka/bin/../libs/connect-runtime-1.0.0.jar:/usr/local/kafka/bin/../libs/connect-transforms-1.0.0.jar:/usr/local/kafka/bin/../libs/guava-20.0.jar:/usr/local/kafka/bin/../libs/hk2-api-2.5.0-b32.jar:/usr/local/kafka/bin/../libs/hk2-locator-2.5.0-b32.jar:/usr/local/kafka/bin/../libs/hk2-utils-2.5.0-b32.jar:/usr/local/kafka/bin/../libs/jackson-annotations-2.9.1.jar:/usr/local/kafka/bin/../libs/jackson-core-2.9.1.jar:/usr/local/kafka/bin/../libs/jackson-databind-2.9.1.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-base-2.9.1.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-json-provider-2.9.1.jar:/usr/local/kafka/bin/../libs/jackson-module-jaxb-annotations-2.9.1.jar:/usr/local/kafka/bin/../libs/javassist-3.20.0-GA.jar:/usr/local/kafka/bin/../libs/javassist-3.21.0-GA.jar:/usr/local/kafka/bin/../libs/javax.annotation-api-1.2.jar:/usr/local/kafka/bin/../libs/javax.inject-1.jar:/usr/local/kafka/bin/../libs/javax.inject-2.5.0-b32.jar:/usr/local/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/local/kafka/bin/../libs/javax.ws.rs-api-2.0.1.jar:/usr/local/kafka/bin/../libs/jersey-client-2.25.1.jar:/usr/local/kafka/bin/../libs/jersey-common-2.25.1.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-2.25.1.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-core-2.25.1.jar:/usr/local/kafka/bin/../libs/jersey-guava-2.25.1.jar:/usr/local/kafka/bin/../libs/jersey-media-jaxb-2.25.1.jar:/usr/local/kafka/bin/../libs/jersey-server-2.25.1.jar:/usr/local/kafka/bin/../libs/jetty-continuation-9.2.22.v20170606.jar:/usr/local/kafka/bin/../libs/jetty-http-9.2.22.v20170606.jar:/usr/local/kafka/bin/../libs/jetty-io-9.2.22.v20170606.jar:/usr/local/kafka/bin/../libs/jetty-security-9.2.22.v20170606.jar:/usr/local/kafka/bin/../libs/jetty-server-9.2.22.v20170606.jar:/usr/local/kafka/bin/../libs/jetty-servlet-9.2.22.v20170606.jar:/usr/local/kafka/bin/../libs/jetty-servlets-9.2.22.v20170606.jar:/usr/local/kafka/bin/../libs/jetty-util-9.2.22.v20170606.jar:/usr/local/kafka/bin/../libs/jopt-simple-5.0.4.jar:/usr/local/kafka/bin/../libs/kafka_2.11-1.0.0.jar:/usr/local/kafka/bin/../libs/kafka_2.11-1.0.0-sources.jar:/usr/local/kafka/bin/../libs/kafka_2.11-1.0.0-test-sources.jar:/usr/local/kafka/bin/../libs/kafka-clients-1.0.0.jar:/usr/local/kafka/bin/../libs/kafka-log4j-appender-1.0.0.jar:/usr/local/kafka/bin/../libs/kafka-streams-1.0.0.jar:/usr/local/kafka/bin/../libs/kafka-streams-examples-1.0.0.jar:/usr/local/kafka/bin/../libs/kafka-tools-1.0.0.jar:/usr/local/kafka/bin/../libs/log4j-1.2.17.jar:/usr/local/kafka/bin/../libs/lz4-java-1.4.jar:/usr/local/kafka/bin/../libs/maven-artifact-3.5.0.jar:/usr/local/kafka/bin/../libs/metrics-core-2.2.0.jar:/usr/local/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/kafka/bin/../libs/plexus-utils-3.0.24.jar:/usr/local/kafka/bin/../libs/reflections-0.9.11.jar:/usr/local/kafka/bin/../libs/rocksdbjni-5.7.3.jar:/usr/local/kafka/bin/../libs/scala-library-2.11.11.jar:/usr/local/kafka/bin/../libs/slf4j-api-1.7.25.jar:/usr/local/kafka/bin/../libs/slf4j-log4j12-1.7.25.jar:/usr/local/kafka/bin/../libs/snappy-java-1.1.4.jar:/usr/local/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/usr/local/kafka/bin/../libs/zkclient-0.10.jar:/usr/local/kafka/bin/../libs/zookeeper-3.4.10.jar (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:os.version=3.10.0-693.el7.x86_64 (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:user.name=elkuser (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:user.home=/home/elkuser (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,824] INFO Client environment:user.dir=/usr/local/kafka/bin (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,825] INFO Initiating client connection, connectString=192.168.137.98:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@79924b (org.apache.zookeeper.ZooKeeper)
[2018-01-21 20:01:42,830] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2018-01-21 20:01:42,898] INFO Waiting for keeper state SaslAuthenticated (org.I0Itec.zkclient.ZkClient)
[2018-01-21 20:01:43,386] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2018-01-21 20:01:43,405] INFO Client will use GSSAPI as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2018-01-21 20:01:43,416] INFO TGT refresh thread started. (org.apache.zookeeper.Login)
[2018-01-21 20:01:43,425] INFO Opening socket connection to server 192.168.137.98/192.168.137.98:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2018-01-21 20:01:43,429] INFO Socket connection established to 192.168.137.98/192.168.137.98:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-01-21 20:01:43,453] INFO TGT valid starting at: Sun Jan 21 20:01:43 CST 2018 (org.apache.zookeeper.Login)
[2018-01-21 20:01:43,453] INFO TGT expires: Mon Jan 22 20:01:43 CST 2018 (org.apache.zookeeper.Login)
[2018-01-21 20:01:43,453] INFO TGT refresh sleeping until: Mon Jan 22 16:05:10 CST 2018 (org.apache.zookeeper.Login)
[2018-01-21 20:01:43,485] INFO Session establishment complete on server 192.168.137.98/192.168.137.98:2181, sessionid = 0x1611883cc1d0004, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2018-01-21 20:01:43,497] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2018-01-21 20:01:43,532] ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2018-01-21 20:01:43,532] INFO zookeeper state changed (AuthFailed) (org.I0Itec.zkclient.ZkClient)
[2018-01-21 20:01:43,533] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2018-01-21 20:01:43,538] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure
at org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.java:947)
at org.I0Itec.zkclient.ZkClient.waitUntilConnected(ZkClient.java:924)
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1231)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131)
at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:115)
at kafka.utils.ZkUtils$.withMetrics(ZkUtils.scala:92)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:346)
at kafka.server.KafkaServer.startup(KafkaServer.scala:194)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:92)
at kafka.Kafka.main(Kafka.scala)
[2018-01-21 20:01:43,541] INFO shutting down (kafka.server.KafkaServer)
[2018-01-21 20:01:43,546] INFO shut down completed (kafka.server.KafkaServer)
[2018-01-21 20:01:43,550] FATAL Exiting Kafka. (kafka.server.KafkaServerStartable)
[2018-01-21 20:01:43,556] INFO shutting down (kafka.server.KafkaServer)
krb5kdc.log:
Jan 21 20:01:43 rh74v1.sample1.com krb5kdc[46018](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.137.98: ISSUE: authtime 1516536103, etypes {rep=18 tkt=18 ses=18}, kafka/192.168.137.98@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Jan 21 20:01:43 rh74v1.sample1.com krb5kdc[46018](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.137.98: ISSUE: authtime 1516536103, etypes {rep=18 tkt=18 ses=18}, kafka/192.168.137.98@EXAMPLE.COM for zookeeper/192.168.137.98@EXAMPLE.COM
My configuration :
/etc/krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
default_realm = EXAMPLE.COM
[realms]
EXAMPLE.COM = {
kdc = 192.168.137.98
admin_server = 192.168.137.98
}
[domain_realm]
kafka = EXAMPLE.COM
zookeeper = EXAMPLE.COM
rh74v1 = EXAMPLE.COM
rh65v1 = EXAMPLE.COM
sample1.com =EXAMPLE.COM
.sample1.com =EXAMPLE.COM
192.168.137.98 = EXAMPLE.COM
192.168.137.99 = EXAMPLE.COM
127.0.0.1 = EXAMPLE.COM
kdc.conf :
[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88
[realms]
EXAMPLE.COM = {
master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
}
kafka.keytab:
Keytab name: FILE:/var/kerberos/krb5kdc/kafka.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
2 01/21/2018 19:37:17 kafka/192.168.137.98@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
2 01/21/2018 19:37:17 kafka/192.168.137.98@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
2 01/21/2018 19:37:17 kafka/192.168.137.98@EXAMPLE.COM (des3-cbc-sha1)
2 01/21/2018 19:37:17 kafka/192.168.137.98@EXAMPLE.COM (arcfour-hmac)
2 01/21/2018 19:37:17 kafka/192.168.137.98@EXAMPLE.COM (camellia256-cts-cmac)
2 01/21/2018 19:37:18 kafka/192.168.137.98@EXAMPLE.COM (camellia128-cts-cmac)
2 01/21/2018 19:37:18 kafka/192.168.137.98@EXAMPLE.COM (des-hmac-sha1)
2 01/21/2018 19:37:18 kafka/192.168.137.98@EXAMPLE.COM (des-cbc-md5)
2 01/21/2018 19:37:36 zookeeper/192.168.137.98@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
2 01/21/2018 19:37:36 zookeeper/192.168.137.98@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
2 01/21/2018 19:37:36 zookeeper/192.168.137.98@EXAMPLE.COM (des3-cbc-sha1)
2 01/21/2018 19:37:36 zookeeper/192.168.137.98@EXAMPLE.COM (arcfour-hmac)
2 01/21/2018 19:37:36 zookeeper/192.168.137.98@EXAMPLE.COM (camellia256-cts-cmac)
2 01/21/2018 19:37:36 zookeeper/192.168.137.98@EXAMPLE.COM (camellia128-cts-cmac)
2 01/21/2018 19:37:36 zookeeper/192.168.137.98@EXAMPLE.COM (des-hmac-sha1)
2 01/21/2018 19:37:36 zookeeper/192.168.137.98@EXAMPLE.COM (des-cbc-md5)
kadmin.local: listprincs
K/M@EXAMPLE.COM
admin/admin@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/rh74v1.sample1.com@EXAMPLE.COM
kafka/192.168.137.98@EXAMPLE.COM
kiprop/rh74v1.sample1.com@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
root/admin@EXAMPLE.COM
zookeeper/192.168.137.98@EXAMPLE.COM
zookeeper_jaas.conf
Server{
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/var/kerberos/krb5kdc/kafka.keytab"
principal="zookeeper/192.168.137.98@EXAMPLE.COM";
};
zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=5
clientPort=2181
dataDir=/usr/local/zookeeper/data
dataLogDir=/usr/local/zookeeper/zkdatalog
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000
kafka_server_jaas.conf
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/var/kerberos/krb5kdc/kafka.keytab"
principal="kafka/192.168.137.98@EXAMPLE.COM";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/var/kerberos/krb5kdc/kafka.keytab"
principal="kafka/192.168.137.98@EXAMPLE.COM";
};
Kafka server.properties
broker.id=0
listeners=SASL_PLAINTEXT://192.168.137.98:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/usr/local/kafka/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.137.98:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
环境:
$ java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
已经替换了JCE文件:
/usr/java/jdk1.8.0_151/jre/lib/security
total 176
-rw-r--r--. 1 root root 4054 Sep 6 10:29 blacklist
-rw-r--r--. 1 root root 1273 Sep 6 10:29 blacklisted.certs
-rw-r--r--. 1 root root 113367 Sep 6 10:29 cacerts
-rw-r--r--. 1 root root 2466 Sep 6 10:29 java.policy
-rw-r--r--. 1 root root 38239 Sep 6 10:29 java.security
-rw-r--r--. 1 root root 98 Sep 6 10:29 javaws.policy
-rw-r--r--. 1 root root 3035 Jan 21 19:17 local_policy.jar
drwxr-xr-x. 4 root root 38 Dec 3 21:23 policy
-rw-r--r--. 1 root root 0 Sep 6 10:29 trusted.libraries
-rw-r--r--. 1 root root 3023 Jan 21 19:17 US_export_policy.jar
这个版本的Java的security下缺省没有jar文件,在policy下有limit和unlimit两个目录,我的理解是在security下有jar文件就会用。
$ uname -a
Linux rh74v1.sample1.com 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.137.98 netmask 255.255.255.0 broadcast 192.168.137.255
inet6 fe80::6ee3:ba85:baeb:f050 prefixlen 64 scopeid 0x20<link>
ether 00:15:5d:0b:73:0d txqueuelen 1000 (Ethernet)
RX packets 47721 bytes 3794876 (3.6 MiB)
RX errors 0 dropped 3 overruns 0 frame 0
TX packets 23106 bytes 5160724 (4.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::10b1:9ee:c270:39b1 prefixlen 64 scopeid 0x20<link>
ether 00:15:5d:0b:73:0e txqueuelen 1000 (Ethernet)
RX packets 31282 bytes 7576136 (7.2 MiB)
RX errors 0 dropped 8 overruns 0 frame 0
TX packets 35 bytes 6828 (6.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 743 bytes 111345 (108.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 743 bytes 111345 (108.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:07:bc:1d txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
start-zookeeper.sh:
#!/bin/bash
export KAFKA_HEAP_OPTS='-Xmx256M'
export KAFKA_OPTS='-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/usr/local/zookeeper/conf/zookeeper_jaas.conf'
/usr/local/zookeeper/bin/zkServer.sh start /usr/local/zookeeper/conf/zoo.cfg
start-kafka.sh
#!/bin/bash
export KAFKA_HEAP_OPTS='-Xmx256M'
export KAFKA_OPTS='-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/usr/local/kafka/config/kafka_server_jaas.conf'
/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
看问题信息和https://www.orchome.com/325 非常类似,但JCE文件替换掉还是有问题,请教大神们,看看问题可能还出在哪儿? 感谢!
替换JCE版本对了么。
https://www.orchome.com/500 这是我的安装笔记,对比一下。
感觉完全一样,替换版本也应该没有问题啊。。。
https://www.orchome.com/171
排查各类日志中的细节异常。是否每一环境都是正确的。往往某个环境出错 而导致全盘出错。
例如:
一旦你启动broker,你应该就能在server.log看到
with addresses: PLAINTEXT -> EndPoint(192.168.64.1,9092,PLAINTEXT),SSL -> EndPoint(192.168.64.1,9093,SSL)
用以下命令,快速验证服务器的keystore和truststore设置是否正确:
openssl s_client -debug -connect localhost:9093 -tls1
(注意: TLSv1 应列出 ssl.enabled.protocols)
在命令的输出中,你应该能看到服务器的证书:
-----BEGIN CERTIFICATE-----
{variable sized random bytes}
-----END CERTIFICATE-----
subject=/C=US/ST=CA/L=Santa Clara/O=org/OU=org/CN=Sriharsha Chintalapani
issuer=/C=US/ST=CA/L=Santa Clara/O=org/OU=org/CN=kafka/emailAddress=test@test.com
如果证书没有出现或者有任何其他错误信息,那么你的keystore设置不正确。
https://www.orchome.com/171说的是kafka使用SSL加密和认证,我的测试是用SASL/Kerberos认证,没没用SSL。
从zookeeper和Kafka的log可以看到Session已经建立了,但zookeeper之后发了个这个信息:
2018-01-21 20:01:43,523 [myid:] - ERROR [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1055] - cnxn.saslServer is null: cnxn object did not initialize its saslServer properly.
Kafka出信息:
[2018-01-21 20:01:43,532] ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient)
中断Session,启动失败。。
在KRB5KDC的log里有如下两条认证信息,不知道是对还不是不对:
Jan 21 20:01:43 rh74v1.sample1.com krb5kdc46018: AS_REQ (4 etypes {18 17 16 23}) 192.168.137.98: ISSUE: authtime 1516536103, etypes {rep=18 tkt=18 ses=18}, kafka/192.168.137.98@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Jan 21 20:01:43 rh74v1.sample1.com krb5kdc46018: TGS_REQ (4 etypes {18 17 16 23}) 192.168.137.98: ISSUE: authtime 1516536103, etypes {rep=18 tkt=18 ses=18}, kafka/192.168.137.98@EXAMPLE.COM for zookeeper/192.168.137.98@EXAMPLE.COM
大哥 你的这个问题解决了吗,跪求解决方案
实战笔记参考:
3种常见的错误:
总有一款适合你。
大神,kafka启动时提示无法使用kafka_server_jass中的client信息去连接zookeeper,出现错误,Authentication failure。是不是zookeeper开启kerberos的方法不对,有没有相关文档,您发的这几个都不是我想要的
你们问题解决了,我这边也遇到了相同的问题
已经解决了,由于时间较长,不记得如何解决的这个问题。大致告诉你一下开启zookeeper的kerberos的步骤:
1、修改conf/zookeeper.properties 添加如下内容:
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
2、新建zookeeper的认证配置文件 vi conf/zookeeper_server_jass.conf
Server{
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/security/keytabs/zookeeper.keytab"
principal="zookeeper/xxx@EXAMPLE.COM";
};
3、修改启动脚本zookeeper-server-start.sh 添加如下内容:
export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/DATA/kafka/config/zookeeper_jaas.conf"
kafka的kerberos启动步骤如下:
1、修改配置文件vi conf/server.properties,添加或修改如下内容:
host=xxx.xxx.xxx.xxx
port=9092
listeners=SASL_PLAINTEXT://xxx.xxx.xxx.xxx:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka
advertised.host=xxx.xxx.xxx.xxx
advertised.port=9092
advertised.listeners=SASL_PLAINTEXT://xxx.xxx.xxx.xxx:9092
2、建立kafka的认证配置文件kafka_server_jaas.conf
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/krb.keytab"
principal="kafka/xxxxxxxx@EXAMPLE.COM";
};
// Zookeeper client authentication
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/zookeeperclient.keytab"
principal="zookeeperclient/xxxxxxxx@EXAMPLE.COM";
};
3、修改启动脚本kafka-server-start.sh,添加如下内容:
export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/DATA/kafka/config/kafka_server_jaas.conf"
祝你好运
谢谢,我看配置基本都一样,zookeeper报以下错误,cnxn.saslServer is null: cnxn object did not initialize its saslServer properly,不知道是不是需要对zookeeper做什么操作
2020-01-09 10:08:34,162 [myid:] - ERROR [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:ZooKeeperServer@968] - cnxn.saslServer is null: cnxn object did not initialize its saslServer properly.
2020-01-09 10:08:34,583 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@360] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x16f880d50a70001, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:231)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:748)
你的答案