2023年10月10日，Kafka 3.6.0发布。
以下是Kafka 3.6.0版本中解决JIRA问题的摘要，有关该版本的完整文档，入门指南以及关于该项目的信息，请参考Kafka官方文档，或kafka中文教程。
有关升级的注意事项：在升级你的集群之前，请仔细阅读此版本的升级文档。升级有关不兼容性和破坏性的变更，性能变化以及可能影响Kakfa生产的任何其他变化。
新功能
[KAFKA-7739] - Kafka分层存储
[KAFKA-14305] - KRaft 元数据交易
[KAFKA-14627] - Modernize Connect plugin discovery
[KAFKA-15030] - 添加 connect-plugin-path 命令行工具
[KAFKA-15031] - 增加 plugin.discovery 工作者配置
[KAFKA-15228] - 为 connect-plugin-path 工具添加 sync-manifests 子命令
改进
[KAFKA-4107] - Support offset reset capability in Kafka Connect
[KAFKA-8982] - Admin.deleteRecords should retry when failing to fetch metadata
[KAFKA-12261] - Splitting partition causes message loss for consumers with auto.offset.reset=latest
[KAFKA-13299] - Accept listeners that have the same port but use IPv4 vs IPv6
[KAFKA-13431] - Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)
[KAFKA-13504] - Retry connect internal topics' creation in case of InvalidReplicationFactorException
[KAFKA-13875] - update docs to include topoicId for kafka-topics.sh --describe output
[KAFKA-14038] - Optimize calculation of size for log in remote tier
[KAFKA-14539] - Simplify StreamsMetadataState by replacing the Cluster metadata with partition info map
[KAFKA-14661] - Upgrade Zookeeper to 3.8.2
[KAFKA-14669] - Include MirrorMaker connector configurations in docs
[KAFKA-14709] - Move content in connect/mirror/README.md to the docs
[KAFKA-14735] - Improve KRaft metadata image change performance at high topic counts
[KAFKA-14752] - improve kafka examples under examples package
[KAFKA-14766] - Improve performance of VarInt encoding/decoding
[KAFKA-14791] - Create a builder class for PartitionRegistration
[KAFKA-14828] - Remove R/W lock from StandardAuthorizer
[KAFKA-14866] - When broker shutdown, the controller module needs to remove its metrics
[KAFKA-14868] - Remove some forgotten metrics when the replicaManager is closed
[KAFKA-14926] - Remove metrics on Log Cleaner shutdown
[KAFKA-14936] - Add Grace Period To Stream Table Join
[KAFKA-14937] - Refactoring for client code to reduce boilerplate
[KAFKA-14944] - Reduce CompletedFetch#parseRecord() memory copy
[KAFKA-14982] - Improve the kafka-metadata-quorum output
[KAFKA-14988] - Upgrade scalaCollectionCompact to v2.9 for CVE-2022-36944
[KAFKA-14991] - Improving Producer's record timestamp validation
[KAFKA-14993] - Improve TransactionIndex instance handling while copying to and fetching from RSM.
[KAFKA-15034] - Improvement of ReplaceField performance for long list
[KAFKA-15036] - Kraft leader change fails when invoking getFinalizedFeatures
[KAFKA-15039] - Reduce logging level to trace in PartitionChangeBuilder.tryElection()
[KAFKA-15076] - KRaft should prefer snapshots when listeners are at the start of the log
[KAFKA-15078] - When fetching offset 0 the KRaft leader should response with SnapshotId
[KAFKA-15085] - Make Timer.java implement AutoCloseable
[KAFKA-15107] - Additional custom metadata for remote log segment
[KAFKA-15121] - FileStreamSourceConnector and FileStreamSinkConnector should implement KIP-875 APIs (alterOffsets)
[KAFKA-15123] - Add tests for ChunkedBytesStream
[KAFKA-15126] - Change range queries to accept null lower and upper bounds
[KAFKA-15130] - Delete remote segments when delete a topic
[KAFKA-15131] - Improve RemoteStorageManager exception handling documentation
[KAFKA-15139] - Optimize the performance of `Set.removeAll(List)` in `MirrorCheckpointConnector`
[KAFKA-15141] - High CPU usage with log4j2
[KAFKA-15153] - Use Python `is` instead of `==` to compare for None
[KAFKA-15155] - Follow PEP 8 best practice in Python to check if a container is empty
[KAFKA-15159] - Update minor dependencies in preparation for 3.5.1
[KAFKA-15177] - MirrorMaker 2 should implement the alterOffsets KIP-875 API
[KAFKA-15182] - Normalize offsets before invoking SourceConnector::alterOffsets
[KAFKA-15183] - Add more controller, loader, snapshot emitter metrics
[KAFKA-15213] - Provide the exact offset to QuorumController.replay
[KAFKA-15219] - Support delegation tokens in KRaft
[KAFKA-15222] - Upgrade zinc scala incremental compiler plugin version to a latests stable fit version (1.9.2)
[KAFKA-15245] - Improve Tiered Storage Metrics
[KAFKA-15291] - Implement Versioned interfaces in common Connect plugins
[KAFKA-15336] - Connect plugin Javadocs should mention serviceloader manifests
Bug
[KAFKA-8690] - Flakey test ConnectWorkerIntegrationTest#testAddAndRemoveWorke
[KAFKA-9926] - Flaky test PlaintextAdminIntegrationTest.testCreatePartitions
[KAFKA-10337] - Wait for pending async commits in commitSync() even if no offsets are specified
[KAFKA-10579] - Flaky test connect.integration.InternalTopicsIntegrationTest.testStartWhenInternalTopicsCreatedManuallyWithCompactForBrokersDefaultCleanupPolicy
[KAFKA-12525] - Inaccurate task status due to status record interleaving in fast rebalances in Connect
[KAFKA-12842] - Failing test: org.apache.kafka.connect.integration.ConnectWorkerIntegrationTest.testSourceTaskNotBlockedOnShutdownWithNonExistentTopic
[KAFKA-13197] - KStream-GlobalKTable join semantics don't match documentation
[KAFKA-13337] - Scanning for Connect plugins can fail with AccessDeniedException
[KAFKA-13668] - Failed cluster authorization should not be fatal for producer
[KAFKA-14273] - Kafka doesn't start with KRaft on Windows
[KAFKA-14654] - Connectors have incorrect Thread Context Classloader during initialization
[KAFKA-14662] - ACL listings in documentation are out of date
[KAFKA-14694] - RPCProducerIdManager should not wait for a new block
[KAFKA-14712] - Confusing error when writing downgraded FeatureImage
[KAFKA-14831] - Illegal state errors should be fatal in transactional producer
[KAFKA-14863] - Plugins which do not have a valid no-args constructor are visible in the REST API
[KAFKA-14938] - Flaky test org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest#testConnectorBoundary
[KAFKA-14962] - Whitespace in ACL configuration causes Kafka startup to fail
[KAFKA-14967] - MockAdminClient throws NullPointerException in CreateTopicsResult
[KAFKA-14978] - ExactlyOnceWorkerSourceTask does not remove parent metrics
[KAFKA-14997] - JmxToolTest failing with initializationError
[KAFKA-15012] - JsonConverter fails when there are leading Zeros in a field
[KAFKA-15016] - LICENSE-binary file contains dependencies not included anymore
[KAFKA-15021] - KRaft controller increases leader epoch when shrinking ISR
[KAFKA-15053] - Regression for security.protocol validation starting from 3.3.0
[KAFKA-15059] - Exactly-once source tasks fail to start during pending rebalances
[KAFKA-15077] - FileTokenRetriever doesn't trim the token before returning it.
[KAFKA-15080] - Fetcher's lag never set when partition is idle
[KAFKA-15091] - Javadocs for SourceTask::commit are incorrect
[KAFKA-15096] - CVE 2023-34455 - Vulnerability identified with Apache kafka
[KAFKA-15098] - KRaft migration does not proceed and broker dies if authorizer.class.name is set
[KAFKA-15100] - Unsafe to call tryCompleteFetchResponse on request timeout
[KAFKA-15102] - Mirror Maker 2 - KIP690 backward compatibility
[KAFKA-15106] - AbstractStickyAssignor may stuck in 3.5
[KAFKA-15109] - ISR shrink/expand issues on ZK brokers during migration
[KAFKA-15114] - StorageTool help specifies user as parameter not name
[KAFKA-15135] - RLM listener configurations passed but ignored by RLMM
[KAFKA-15137] - Don't log the entire request in KRaftControllerChannelManager
[KAFKA-15145] - AbstractWorkerSourceTask re-processes records filtered out by SMTs on retriable exceptions
[KAFKA-15162] - Reflective plugin scanning misses plugins which are in parent classloaders but not classpath
[KAFKA-15189] - Do not initialize RemoteStorage related metrics when disabled at cluster
[KAFKA-15212] - Remove unneeded classgraph license file
[KAFKA-15216] - InternalSinkRecord::newRecord method ignores the headers argument
[KAFKA-15218] - NPE will be thrown while deleting topic and fetch from follower concurrently
[KAFKA-15220] - KRaftMetadataCache returns fenced brokers from getAliveBrokerNode
[KAFKA-15235] - No test coverage reports for Java due to settings for Jacoco being incompatible with Gradle 8.x
[KAFKA-15238] - Connect workers can be disabled by DLQ-related blocking admin client calls
[KAFKA-15243] - User creation mismatch
[KAFKA-15244] - Connect PluginType.from(Class) result is incorrect when subclassing multiple plugin interfaces
[KAFKA-15263] - KRaftMigrationDriver can run the migration twice
[KAFKA-15312] - FileRawSnapshotWriter must flush before atomic move
[KAFKA-15319] - Upgrade rocksdb to fix CVE-2022-37434
[KAFKA-15338] - The metric group documentation for metrics added in KAFKA-13945 is incorrect
[KAFKA-15345] - KRaft leader should notify the listener only when it has read up to the leader's epoch
[KAFKA-15353] - Empty ISR returned from controller after AlterPartition request
[KAFKA-15374] - ZK migration fails on configs for default broker resource
[KAFKA-15375] - When running in KRaft mode, LogManager may creates CleanShutdown file by mistake
[KAFKA-15377] - GET /connectors/{connector}/tasks-config endpoint exposes externalized secret values
[KAFKA-15389] - MetadataLoader may publish an empty image on first start
[KAFKA-15391] - Delete topic may lead to directory offline
[KAFKA-15404] - Failing Test DynamicBrokerReconfigurationTest#testThreadPoolResize
[KAFKA-15414] - remote logs get deleted after partition reassignment
[KAFKA-15429] - Kafka Streams attempts to commit on a closed producer when shutting down after an exception when running with EOS
[KAFKA-15435] - KRaft migration record counts in log message are incorrect
[KAFKA-15441] - Broker sessions can time out during ZK migration
[KAFKA-15450] - Disable ZK migration when JBOD configured
[KAFKA-15473] - Connect connector-plugins endpoint shows duplicate plugins
[KAFKA-15487] - CVE-2023-40167, CVE-2023-36479 - Upgrade jetty to 9.4.52, 10.0.16, 11.0.16, 12.0.1
[KAFKA-15498] - Upgrade Snappy-Java to 1.1.10.4
[KAFKA-15503] - CVE-2023-40167, CVE-2023-36479 - Upgrade jetty to 9.4.52, 10.0.16, 11.0.16, 12.0.1
任务
[KAFKA-14559] - Handle object name with wildcards in the Jmx tool
[KAFKA-14759] - Move test-only connectors from connect-runtime to test-specific module
[KAFKA-14760] - Move ThroughputThrottler, break connect-runtime dependency on tools
[KAFKA-14933] - Document Kafka Connect's log level REST APIs added in KIP-495
[KAFKA-14950] - Implement assign() and assignment()
[KAFKA-14966] - Extract reusable logic from OffsetFetcher
[KAFKA-14974] - Restore backward compatibility in KafkaBasedLog
[KAFKA-15069] - Refactor scanning hierarchy out of DelegatingClassLoader
[KAFKA-15087] - Move InterBrokerSendThread to server-commons module
[KAFKA-15150] - Add ServiceLoaderScanner implementation
[KAFKA-15194] - Rename local tiered storage segment with offset as prefix for easy navigation
[KAFKA-15233] - Add public documentation for plugin.discovery migration steps
[KAFKA-15272] - Fix the logic which finds candidate log segments to upload it to tiered storage
[KAFKA-15286] - Migrate ApiVersion related code to kraft
[KAFKA-15400] - Fix flaky RemoteIndexCache test
[KAFKA-15421] - Enable DynamicBrokerReconfigurationTest#testThreadPoolResize test
[KAFKA-15422] - Update documentation for Delegation Tokens in Kafka with KRaft
测试
[KAFKA-12384] - Flaky Test ListOffsetsRequestTest.testResponseIncludesLeaderEpoch
[KAFKA-14682] - Unused stubbings are not reported by Mockito during CI builds
[KAFKA-14718] - Flaky DedicatedMirrorIntegrationTest test suite
[KAFKA-14905] - Failing tests in MM2 ForwardingAdmin test since KIP-894
[KAFKA-15052] - Fix flaky test QuorumControllerTest.testBalancePartitionLeaders()
[KAFKA-15148] - Some integration tests are running as unit tests
[KAFKA-15180] - Generalize integration tests to change use of KafkaConsumer to Consumer
[KAFKA-15211] - DistributedConfigTest#shouldFailWithInvalidKeySize fails when run after TestSslUtils#generate
[KAFKA-15226] - System tests for plugin.discovery worker configuration
[KAFKA-15239] - producerPerformance system test for old client failed after v3.5.0
[KAFKA-15251] - Upgrade system test to use 3.5.1
[KAFKA-15393] - MirrorMaker2 integration tests are shutting down uncleanly
[KAFKA-15416] - Flaky test TopicAdminTest::retryEndOffsetsShouldRetryWhenTopicNotFound
[KAFKA-15425] - Compatibility break in Admin.listOffsets() (2)
[KAFKA-15439] - Add transaction tests enabled with tiered storage
[KAFKA-15453] - Enable `testFencingOnTransactionExpiration` in TransactionsWithTieredStoreTest
[KAFKA-15499] - Fix the flaky DeleteSegmentsDueToLogStartOffsetBreachTest
子任务
[KAFKA-9564] - Integration Test framework for Tiered Storage
[KAFKA-9579] - Remote consumer fetch implementation by adding respective purgatory
[KAFKA-12969] - Add cluster or broker level config for topic level tiered storage confgs.
[KAFKA-13187] - Replace EasyMock and PowerMock with Mockito for DistributedHerderTest
[KAFKA-14059] - Replace EasyMock and PowerMock with Mockito in WorkerSourceTaskTest
[KAFKA-14278] - Convert INVALID_PRODUCER_EPOCH into PRODUCER_FENCED TxnOffsetCommit
[KAFKA-14368] - Implement connector offset write REST API
[KAFKA-14462] - New Group Coordinator State Machine
[KAFKA-14500] - Implement JoinGroup/SyncGroup APIs
[KAFKA-14501] - Implement Heartbeat API
[KAFKA-14514] - Implement range broker side assignor
[KAFKA-14518] - Rebalance on topic/partition metadata changes
[KAFKA-14522] - Move RemoteIndexCache to the storage module
[KAFKA-14561] - Improve transactions experience for older clients by ensuring ongoing transaction
[KAFKA-14583] - Move ReplicaVerificationTool to tools
[KAFKA-14584] - Deprecate StateChangeLogMerger tool
[KAFKA-14591] - Move DeleteRecordsCommand to tools
[KAFKA-14592] - Move FeatureCommand to tools
[KAFKA-14594] - Move LogDirsCommand to tools
[KAFKA-14632] - Compression optimization: Remove unnecessary intermediate buffers
[KAFKA-14633] - Compression optimization: Use BufferSupplier to allocate the intermediate decompressed buffer
[KAFKA-14647] - Move TopicFilter shared class
[KAFKA-14702] - Extend server side assignor to support rack aware replica placement
[KAFKA-14734] - Use CommandDefaultOptions in StreamsResetter
[KAFKA-14737] - Move kafka.utils.json to server-common
[KAFKA-14755] - improve java-producer-consumer-demo
[KAFKA-14756] - improve exactly-once-demo example and ExactlyOnceMessageProcessor
[KAFKA-14784] - Implement connector offset reset REST API
[KAFKA-14851] - Move StreamResetterTest to tools
[KAFKA-14884] - Include check transaction is still ongoing right before append
[KAFKA-14888] - RemoteLogManager - deleting expired/size breached log segments to remote storage implementation
[KAFKA-14920] - Address timeouts and out of order sequences
[KAFKA-14930] - Public documentation for new Kafka Connect offset management REST APIs
[KAFKA-14953] - Add metrics for tiered storage
[KAFKA-15023] - Get rack information for source topic partitions for a task
[KAFKA-15024] - Add cost function for task/client
[KAFKA-15025] - Implement min-cost flow without balancing tasks for same subtopology
[KAFKA-15027] - Implement rack aware assignment for standby tasks
[KAFKA-15028] - AddPartitionsToTxnManager metrics
[KAFKA-15037] - initialize unifiedLog with remoteStorageSystemEnable correctly
[KAFKA-15040] - segment copy to remote storage won't work in KRaft mode
[KAFKA-15054] - Add configs and logic to decide if rack aware assignment should be enabled
[KAFKA-15066] - passing listener name config into TopicBasedRemoteLogMetadataManagerConfig
[KAFKA-15083] - Passing "remote.log.metadata.*" configs into RLMM
[KAFKA-15084] - Remove lock contention in RemoteIndexCache
[KAFKA-15157] - Print startup time for RemoteIndexCache
[KAFKA-15167] - Tiered Storage Test Harness Framework
[KAFKA-15168] - Handle overlapping remote log segments in RemoteLogMetadata cache
[KAFKA-15176] - Add missing tests for remote storage metrics
[KAFKA-15181] - Race condition on partition assigned to TopicBasedRemoteLogMetadataManager
[KAFKA-15199] - remove leading and trailing spaces from user input in release.py
[KAFKA-15210] - Mention vote should be open for at atleast 72 hours
[KAFKA-15232] - Move ToolsUtils to tools
[KAFKA-15236] - Rename Remote Storage metrics to remove ambiguity
[KAFKA-15246] - CoordinatorContext should be protected by a lock
[KAFKA-15256] - Add code reviewers to contributors list in release email
[KAFKA-15260] - RLM Task should wait until RLMM is initialized before copying segments to remote
[KAFKA-15261] - ReplicaFetcher thread should not block if RLMM is not initialized
[KAFKA-15267] - Cluster-wide disablement of Tiered Storage
[KAFKA-15287] - Change NodeApiVersions.create() to contains both apis of zk and kraft broker
[KAFKA-15288] - Change BrokerApiVersionsCommandTest to support kraft mode
[KAFKA-15289] - Support KRaft mode in RequestQuotaTest
[KAFKA-15290] - Add support to onboard existing topics to tiered storage
[KAFKA-15293] - Update metrics doc to add tiered storage metrics
[KAFKA-15294] - Make remote storage related configs as public (i.e. non-internal)
[KAFKA-15295] - Add config validation when remote storage is enabled on a topic
[KAFKA-15329] - Make default `remote.log.metadata.manager.class.name` as topic based RLMM
[KAFKA-15351] - Update log-start-offset after leader election for topics enabled with remote storage
[KAFKA-15352] - Ensure consistency while deleting the remote log segments
[KAFKA-15380] - Try complete actions after callback
[KAFKA-15399] - Enable OffloadAndConsumeFromLeader test
[KAFKA-15410] - Add functional integration tests with tiered storage
[KAFKA-15427] - Integration tests in TS test harness detect resource leaks
[KAFKA-15442] - add document to introduce tiered storage feature and the usage
[KAFKA-15459] - Convert coordinator retriable errors to a known producer response error.
