group.coordinator.cached.buffer.max.bytes and share.coordinator.cached.buffer.max.bytes. They allow the respective coordinators to set the maximum buffer size retained for reuse. For further details, please refer to KIP-1196.remote.log.metadata.topic.min.isr with 2 as default value. You can correct the min.insync.replicas for the existed __remote_log_metadata topic via kafka-configs.sh if needed. For further details, please refer to KIP-1235.remote.log.metadata.admin. has been introduced. It allows independent configuration of the admin client used by TopicBasedRemoteLogMetadataManager. For further details, please refer to KIP-1208.--max-partition-memory-bytes option in kafka-console-producer is deprecated and will be removed in Kafka 5.0. Please use --batch-size instead.org.apache.kafka.common.header.internals.RecordHeader class has been updated to be read thread-safe. See KIP-1205 for details. In other words, each individual Header object within a ConsumerRecord's headers can now be safely read from multiple threads concurrently.org.apache.kafka.disallowed.login.modules config was deprecated. Please use the org.apache.kafka.allowed.login.modules instead.remote.log.manager.thread.pool.size config was deprecated. Please use the remote.log.manager.follower.thread.pool.size instead.KafkaPrincipalBuilder now extends KafkaPrincipalSerde. Force developer to implement KafkaPrincipalSerde interface for custom KafkaPrincipalBuilder. For further details, please refer to KIP-1157.org.apache.kafka.streams.KafkaStreams#removeStreamThread has been changed. The consumer has no longer remove once removeStreamThread finished. Instead, consumer would be kicked off from the group after org.apache.kafka.streams.processor.internals.StreamThread completes its run function.kafka_mx4jenable system property, was deprecated and will be removed in Kafka 5.0.PARTITIONER_ADPATIVE_PARTITIONING_ENABLE_CONFIG in ProducerConfig was deprecated and will be removed in Kafka 5.0. Please use the PARTITIONER_ADAPTIVE_PARTITIONING_ENABLE_CONFIG instead.ConsumerPerformance command line tool has a new --include option that is an alternative to the --topic option. This new option allows to pass a regular expression specifying a list of topics to include for consumption, which is useful to test consumer performance across multiple topics or dynamically matching topic sets.kafka-producer-perf-test.sh, --bootstrap-server and --reporting-interval option are added.kafka-console-consumer.sh and kafka-console-share-consumer.sh, the option --property which is used to specify the properties for the formatter is deprecated in favor of --formatter-property.kafka-console-producer.sh, the option --property which is used to specify properties in the form key=value to the message reader is deprecated in favor of --reader-property.kafka-consumer-perf-test.sh and kafka-share-consumer-perf-test.sh, the option --messages is deprecated in favor of --num-records to bring all performance testing tools in line.--command-property is used for all command-line tools which accept configuration properties directly on the command line. The tools affected are:kafka-console-consumer.sh, kafka-console-producer.sh and kafka-console-share-consumer.sh (--consumer-property and --producer-property are deprecated in favor of --command-property)kafka-producer-perf-test.sh (--producer-props is deprecated in favor of --command-property)kafka-consumer-perf-test.sh and kafka-share-consumer-perf-test.sh gain the --command-property option to bring all performance testing tools in line--command-config is used for all command-line tools which accept a file of configuration properties. The tools affected are:kafka-cluster.sh (--config is deprecated in favor of --command-config)kafka-console-consumer.sh, kafka-console-producer.sh and kafka-console-share-consumer.sh (--consumer.config and --producer.config are deprecated in favor of --command-config)kafka-consumer-perf-test.sh, kafka-producer-perf-test.sh and kafka-share-consumer-perf-test.sh (--consumer.config and --producer.config are deprecated in favor of --command-config)kafka-verifiable-consumer.sh and kafka-verifiable-producer.sh (--consumer.config and --producer.config are deprecated in favor of --command-config)kafka-leader-election.sh (--admin.config is deprecated in favor of --command-config)kafka-streams-application-reset.sh (--config-file is deprecated in favor of --command-config)num.replica.fetchers config has a new lower bound of 1.cleanup.policy now supports empty values, which means infinite retention. This is equivalent to setting retention.ms=-1 and retention.bytes=-1cleanup.policy is empty and remote.storage.enable is set to true, the local log segments will be cleaned based on the values of log.local.retention.bytes and log.local.retention.ms.cleanup.policy is empty and remote.storage.enable is set to false, local log segments will not be deleted automatically. However, records can still be deleted explicitly through deleteRecords API calls, which will advance the log start offset and remove the corresponding log segments.controller.quorum.auto.join.enable has been added to QuorumConfig, enabling KRaft controllers to automatically join the cluster's voter set, and defaults to false. If the configuration is set to true the controller must be shutdown before removing the controller from the voter set to avoid the removed controller to automatically join again. For further details, please refer to KIP-853.[name=start-time-ms, group=app-info, description=Metric indicating start-time-ms, tags={}][name=commit-id, group=app-info, description=Metric indicating commit-id, tags={}][name=version, group=app-info, description=Metric indicating version, tags={}] In addition, the client-id will be added to the tags of these metrics. The new metric names will be:[name=start-time-ms, group=app-info, description=Metric indicating start-time-ms, tags={client-id=...}][name=commit-id, group=app-info, description=Metric indicating commit-id, tags={client-id=...}][name=version, group=app-info, description=Metric indicating version, tags={client-id=...}] For further details, please refer to KIP-1120.org.apache.kafka.server:type=AssignmentsManager.QueuedReplicaToDirAssignments, org.apache.kafka.storage.internals.log:type=RemoteStorageThreadPool.RemoteLogReaderTaskQueueSize, and org.apache.kafka.storage.internals.log:type=RemoteStorageThreadPool.RemoteLogReaderAvgIdlePercent have been deprecated and will be removed in Kafka 5.0. As replacements, the following metrics have been introduced, which report the same information: kafka.server:type=AssignmentsManager.QueuedReplicaToDirAssignments, kafka.log.remote:type=RemoteStorageThreadPool.RemoteLogReaderTaskQueueSize, and kafka.log.remote:type=RemoteStorageThreadPool.RemoteLogReaderAvgIdlePercent. For further details, please refer to KIP-1100.AvgIdleRatio has been added to the ControllerEventManager and MetadataLoader groups. These metrics measure the average idle ratio of their respective event queue threads, providing visibility into how much time each component spends waiting for events versus processing them. The metric value ranges from 0.0 (always busy) to 1.0 (always idle).org.apache.kafka.streams.KafkaStreams$CloseOptions and its related methods, such as KafkaStreams#close(org.apache.kafka.streams.KafkaStreams$CloseOptions). As a replacement, please use org.apache.kafka.streams.CloseOptions and KafkaStreams#close(org.apache.kafka.streams.CloseOptions). For further details, please refer to KIP-1153.ConnectorClientConfigOverridePolicy, AllowlistConnectorClientConfigOverridePolicy, has been added. This enables specifying the configurations that connectors can override via connector.client.config.override.allowlist. From Kafka 5.0.0, this will be the default connector.client.config.override.policy policy. The PrincipalConnectorClientConfigOverridePolicy policy is now deprecated and will be removed in Kafka 5.0.0. For further details, please refer to KIP-1188.--node-id flag to the FeatureCommand command. It specifies the node to describe. If not provided, an arbitrary node is used.Note: Kafka Streams 4.1.0 contains a critical memory leak bug (KAFKA-19748) that affects users of range scans and certain DSL operators (session windows, sliding windows, stream-stream joins, foreign-key joins). Users running Kafka Streams should consider upgrading directly to 4.1.1, which includes the fix for it.
kafka-features.sh tool to upgrade to share.version=1. For more information, please read the release notes.kafka.log.LogCleaner to org.apache.kafka.storage.internals.log.LogCleaner in the log4j2.yaml configuration file. Added loggers for org.apache.kafka.storage.internals.log.LogCleaner$CleanerThread and org.apache.kafka.storage.internals.log.Cleaner classes to CleanerAppender.state-change.log files has been updated from stage-change.log.[date] to state-change.log.[date] in the log4j2.yaml configuration file.log.cleaner.enable is deprecated. Users should no longer set it to false to prepare for future removal. After the removal, log.cleaner.threads will also have a lower bound of 1. For further details, please refer to KIP-1148.min.insync.replicas value at the broker-level config will be removed. Please set at the cluster-level if necessary. For further details, please refer to here.flush method now detects potential deadlocks and prohibits its use inside a callback. This change prevents unintended blocking behavior, which was a known risk in earlier versions.force option of ConfigCommand has been removed, as it has been non-operational since version 0.10.1.0.listConsumerGroups() and listConsumerGroups(ListConsumerGroupsOptions) methods in Admin are deprecated, and will be removed in the next major version. Use Admin.listGroups(ListGroupsOptions.forConsumerGroups()) instead.For a rolling upgrade:
Note: Apache Kafka 4.0 only supports KRaft mode - ZooKeeper mode has been removed. As such, broker upgrades to 4.0.0 (and higher) require KRaft mode and the software and metadata versions must be at least 3.3.x (the first version when KRaft mode was deemed production ready). For clusters in KRaft mode with versions older than 3.3.x, we recommend upgrading to 3.9.x before upgrading to 4.0.x. Clusters in ZooKeeper mode have to be migrated to KRaft mode before they can be upgraded to 4.0.x.
For a rolling upgrade:
bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --release-version 4.0IBP_4_0_IV1(23, "4.0", "IV1", true) means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.state-change.log files has been updated from stage-change.log.[date] to state-change.log.[date] in the log4j2.yaml configuration file. See KAFKA-19576 for details.KStreams#transformValues() (remove with 4.0.0 release) to KStreams#processValues(). For more details, see the migration guide.group.coordinator.Eligible Leader Replicas(ELR). For more information check here.org.apache.kafka.sasl.oauthbearer.allowed.urls) to set the allowed URLs as SASL OAUTHBEARER token or jwks endpoints. By default, the value is an empty list. Users should explicitly set the allowed list if necessary.metrics.jmx.blacklist and metrics.jmx.whitelist configurations were removed from the org.apache.kafka.common.metrics.JmxReporter Please use metrics.jmx.exclude and metrics.jmx.include respectively instead.auto.include.jmx.reporter configuration was removed. The metric.reporters configuration is now set to org.apache.kafka.common.metrics.JmxReporter by default.org.apache.kafka.common.metrics.JmxReporter with string argument was removed. See KIP-606 for details.bufferpool-wait-time-total, io-waittime-total, and iotime-total metrics were removed. Please use bufferpool-wait-time-ns-total, io-wait-time-ns-total, and io-time-ns-total metrics as replacements, respectively.kafka.common.requests.DescribeLogDirsResponse.LogDirInfo class was removed. Please use the kafka.clients.admin.DescribeLogDirsResult.descriptions() class and kafka.clients.admin.DescribeLogDirsResult.allDescriptions() instead.kafka.common.requests.DescribeLogDirsResponse.ReplicaInfo class was removed. Please use the kafka.clients.admin.DescribeLogDirsResult.descriptions() class and kafka.clients.admin.DescribeLogDirsResult.allDescriptions() instead.org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler class was removed. Please use the org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler class instead.org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerValidatorCallbackHandler class was removed. Please use the org.apache.kafka.common.security.oauthbearer.OAuthBearerValidatorCallbackHandler class instead.org.apache.kafka.common.errors.NotLeaderForPartitionException class was removed. The org.apache.kafka.common.errors.NotLeaderOrFollowerException is returned if a request could not be processed because the broker is not the leader or follower for a topic partition.org.apache.kafka.clients.producer.internals.DefaultPartitioner and org.apache.kafka.clients.producer.UniformStickyPartitioner classes were removed.log.message.format.version and message.format.version configs were removed.onNewBatch in org.apache.kafka.clients.producer.Partitioner class was removed.config/kraft directory since Zookeeper has been removed. These files have been consolidated with other configuration files. Now all configuration files are in config directory.--bootstrap-server only supports comma-separated value, such as host1:port1,host2:port2,.... Providing other formats, like space-separated bootstrap servers (e.g., host1:port1 host2:port2 host3:port3), will result in an exception, even though this was allowed in Apache Kafka versions prior to 4.0.delegation.token.master.key configuration was removed. Please use delegation.token.secret.key instead.offsets.commit.required.acks configuration was removed. See KIP-1041 for details.log.message.timestamp.difference.max.ms configuration was removed. Please use log.message.timestamp.before.max.ms and log.message.timestamp.after.max.ms instead. See KIP-937 for details.remote.log.manager.copier.thread.pool.size configuration default value was changed to 10 from -1. Values of -1 are no longer valid; a minimum value of 1 or higher is required. See KIP-1030remote.log.manager.expiration.thread.pool.size configuration default value was changed to 10 from -1. Values of -1 are no longer valid; a minimum value of 1 or higher is required. See KIP-1030remote.log.manager.thread.pool.size configuration default value was changed to 2 from 10. See KIP-1030segment.bytes/log.segment.bytes has changed from 14 bytes to 1MB. See KIP-1030use.incremental.alter.configs configuration was removed from MirrorSourceConnector. The modified behavior is now identical to the previous required configuration, therefore users should ensure that brokers in the target cluster are at least running 2.3.0.add.source.alias.to.metrics configuration was removed from MirrorSourceConnector. The source cluster alias is now always added to the metrics.config.properties.blacklist was removed from the org.apache.kafka.connect.mirror.MirrorSourceConfig Please use config.properties.exclude instead.topics.blacklist was removed from the org.apache.kafka.connect.mirror.MirrorSourceConfig Please use topics.exclude instead.groups.blacklist was removed from the org.apache.kafka.connect.mirror.MirrorSourceConfig Please use groups.exclude instead.kafka.common.MessageReader class was removed. Please use the org.apache.kafka.tools.api.RecordReader interface to build custom readers for the kafka-console-producer tool.kafka.tools.DefaultMessageFormatter class was removed. Please use the org.apache.kafka.tools.consumer.DefaultMessageFormatter class instead.kafka.tools.LoggingMessageFormatter class was removed. Please use the org.apache.kafka.tools.consumer.LoggingMessageFormatter class instead.kafka.tools.NoOpMessageFormatter class was removed. Please use the org.apache.kafka.tools.consumer.NoOpMessageFormatter class instead.--whitelist option was removed from the kafka-console-consumer command line tool. Please use --include instead.kafka.admin.FeatureCommand, kafka.tools.ClusterTool, kafka.tools.EndToEndLatency, kafka.tools.StateChangeLogMerger, kafka.tools.StreamsResetter, kafka.tools.JmxTool.--authorizer, --authorizer-properties, and --zk-tls-config-file options were removed from the kafka-acls command line tool. Please use --bootstrap-server or --bootstrap-controller instead.kafka.serializer.Decoder trait was removed, please use the org.apache.kafka.tools.api.Decoder interface to build custom decoders for the kafka-dump-log tool.kafka.coordinator.group.OffsetsMessageFormatter class was removed. Please use the org.apache.kafka.tools.consumer.OffsetsMessageFormatter class instead.kafka.coordinator.group.GroupMetadataMessageFormatter class was removed. Please use the org.apache.kafka.tools.consumer.GroupMetadataMessageFormatter class instead.kafka.coordinator.transaction.TransactionLogMessageFormatter class was removed. Please use the org.apache.kafka.tools.consumer.TransactionLogMessageFormatter class instead.--topic-white-list option was removed from the kafka-replica-verification command line tool. Please use --topics-include instead.--broker-list option was removed from the kafka-verifiable-consumer command line tool. Please use --bootstrap-server instead.kafka-configs.sh now uses incrementalAlterConfigs API to alter broker configurations instead of the deprecated alterConfigs API, and it will fall directly if the broker doesn't support incrementalAlterConfigs API, which means the broker version is prior to 2.3.x. See KIP-1011 for more details.kafka.admin.ZkSecurityMigrator tool was removed.whitelist and blacklist configurations were removed from the org.apache.kafka.connect.transforms.ReplaceField transformation. Please use include and exclude respectively instead.onPartitionsRevoked(Collection<TopicPartition>) and onPartitionsAssigned(Collection<TopicPartition>) methods were removed from SinkTask.commitRecord(SourceRecord) method was removed from SourceTask.poll(long) method was removed from the consumer. Please use poll(Duration) instead. Note that there is a difference in behavior between the two methods. The poll(Duration) method does not block beyond the timeout awaiting partition assignment, whereas the earlier poll(long) method used to wait beyond the timeout.committed(TopicPartition) and committed(TopicPartition, Duration) methods were removed from the consumer. Please use committed(Set<TopicPartition>) and committed(Set<TopicPartition>, Duration) instead.setException(KafkaException) method was removed from the org.apache.kafka.clients.consumer.MockConsumer. Please use setPollException(KafkaException) instead.enable.idempotence configuration will no longer automatically fall back when the max.in.flight.requests.per.connection value exceeds 5.sendOffsetsToTransaction(Map<TopicPartition, OffsetAndMetadata>, String) method has been removed from the Producer API.linger.ms changed from 0 to 5 in Apache Kafka 4.0 as the efficiency gains from larger batches typically result in similar or lower producer latency despite the increased linger.alterConfigs method was removed from the org.apache.kafka.clients.admin.Admin. Please use incrementalAlterConfigs instead.org.apache.kafka.common.ConsumerGroupState enumeration and related methods have been deprecated. Please use GroupState instead which applies to all types of group.Admin.describeConsumerGroups method used to return a ConsumerGroupDescription in state DEAD if the group ID was not found. In Apache Kafka 4.0, the GroupIdNotFoundException is thrown instead as part of the support for new types of group.org.apache.kafka.clients.admin.DeleteTopicsResult.values() method was removed. Please use org.apache.kafka.clients.admin.DeleteTopicsResult.topicNameValues() instead.org.apache.kafka.clients.admin.TopicListing.TopicListing(String, boolean) method was removed. Please use org.apache.kafka.clients.admin.TopicListing.TopicListing(String, Uuid, boolean) instead.org.apache.kafka.clients.admin.ListConsumerGroupOffsetsOptions.topicPartitions(List<TopicPartition>) method was removed. Please use org.apache.kafka.clients.admin.Admin.listConsumerGroupOffsets(Map<String, ListConsumerGroupOffsetsSpec>, ListConsumerGroupOffsetsOptions) instead.dryRun methods were removed from the org.apache.kafka.clients.admin.UpdateFeaturesOptions. Please use validateOnly instead.org.apache.kafka.clients.admin.FeatureUpdate with short and boolean arguments was removed. Please use the constructor that accepts short and the specified UpgradeType enum instead.allowDowngrade method was removed from the org.apache.kafka.clients.admin.FeatureUpdate.org.apache.kafka.clients.admin.DescribeTopicsResult.DescribeTopicsResult(Map<String, KafkaFuture<TopicDescription>>) method was removed. Please use org.apache.kafka.clients.admin.DescribeTopicsResult.DescribeTopicsResult(Map<Uuid, KafkaFuture<TopicDescription>>, Map<String, KafkaFuture<TopicDescription>>) instead.values() method was removed from the org.apache.kafka.clients.admin.DescribeTopicsResult. Please use topicNameValues() instead.all() method was removed from the org.apache.kafka.clients.admin.DescribeTopicsResult. Please use allTopicNames() instead.JoinWindows.of() and JoinWindows#grace(). See KAFKA-17531 for details.KStream#transformValues() which was removed with Apache Kafka 4.0.0 release, and you need to rewrite your program to use KStreams#processValues() instead, pay close attention to the migration guide.--delete-config option in the kafka-topics command line tool has been deprecated.nextSegmentWithTxnIndex is introduced in RLMM to allow the implementation to return the next segment metadata with a transaction index. This API is used when the consumers are enabled with isolation level as READ_COMMITTED. See KIP-1058 for more details.num.recovery.threads.per.data.dir has been changed from 1 to 2. The impact of this is faster recovery post unclean shutdown at the expense of extra IO cycles. See KIP-1030message.timestamp.after.max.ms has been changed from Long.Max to 1 hour. The impact of this messages with a timestamp of more than 1 hour in the future will be rejected when message.timestamp.type=CreateTime is set. See KIP-1030TransactionAbortableException enhances error handling within transactional operations by clearly indicating scenarios where transactions should be aborted due to errors. It is important for applications to properly manage both TimeoutException and TransactionAbortableException when working with transaction producers.state-change.log files incorrectly rotates to stage-change.log.[date] (changing state to stage). This issue is corrected in 4.0.1. See KAFKA-19576 for details.See Upgrading From Previous Versions in the 3.9 documentation.