| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to You under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| |
| <script id="configuration-template" type="text/x-handlebars-template"> |
| Kafka uses key-value pairs in the <a href="http://en.wikipedia.org/wiki/.properties">property file format</a> for configuration. These values can be supplied either from a file or programmatically. |
| |
| <h3><a id="brokerconfigs" href="#brokerconfigs">3.1 Broker Configs</a></h3> |
| |
| The essential configurations are the following: |
| <ul> |
| <li><code>broker.id</code> |
| <li><code>log.dirs</code> |
| <li><code>zookeeper.connect</code> |
| </ul> |
| |
| Topic-level configurations and defaults are discussed in more detail <a href="#topicconfigs">below</a>. |
| |
| <!--#include virtual="generated/kafka_config.html" --> |
| |
| <p>More details about broker configuration can be found in the scala class <code>kafka.server.KafkaConfig</code>.</p> |
| |
| <h4><a id="dynamicbrokerconfigs" href="#dynamicbrokerconfigs">3.1.1 Updating Broker Configs</a></h4> |
| From Kafka version 1.1 onwards, some of the broker configs can be updated without restarting the broker. See the |
| <code>Dynamic Update Mode</code> column in <a href="#brokerconfigs">Broker Configs</a> for the update mode of each broker config. |
| <ul> |
| <li><code>read-only</code>: Requires a broker restart for update</li> |
| <li><code>per-broker</code>: May be updated dynamically for each broker</li> |
| <li><code>cluster-wide</code>: May be updated dynamically as a cluster-wide default. May also be updated as a per-broker value for testing.</li> |
| </ul> |
| |
| To alter the current broker configs for broker id 0 (for example, the number of log cleaner threads): |
| <pre class="line-numbers"><code class="language-bash"> > bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --add-config log.cleaner.threads=2 |
| </code></pre> |
| |
| To describe the current dynamic broker configs for broker id 0: |
| <pre class="line-numbers"><code class="language-bash"> > bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --describe |
| </code></pre> |
| |
| To delete a config override and revert to the statically configured or default value for broker id 0 (for example, |
| the number of log cleaner threads): |
| <pre class="line-numbers"><code class="language-bash"> > bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --delete-config log.cleaner.threads |
| </code></pre> |
| |
| Some configs may be configured as a cluster-wide default to maintain consistent values across the whole cluster. All brokers |
| in the cluster will process the cluster default update. For example, to update log cleaner threads on all brokers: |
| <pre class="line-numbers"><code class="language-bash"> > bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config log.cleaner.threads=2 |
| </code></pre> |
| |
| To describe the currently configured dynamic cluster-wide default configs: |
| <pre class="line-numbers"><code class="language-bash"> > bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --describe |
| </code></pre> |
| |
| All configs that are configurable at cluster level may also be configured at per-broker level (e.g. for testing). |
| If a config value is defined at different levels, the following order of precedence is used: |
| <ul> |
| <li>Dynamic per-broker config stored in ZooKeeper</li> |
| <li>Dynamic cluster-wide default config stored in ZooKeeper</li> |
| <li>Static broker config from <code>server.properties</code></li> |
| <li>Kafka default, see <a href="#brokerconfigs">broker configs</a></li> |
| </ul> |
| |
| <h5>Updating Password Configs Dynamically</h5> |
| <p>Password config values that are dynamically updated are encrypted before storing in ZooKeeper. The broker config |
| <code>password.encoder.secret</code> must be configured in <code>server.properties</code> to enable dynamic update |
| of password configs. The secret may be different on different brokers.</p> |
| <p>The secret used for password encoding may be rotated with a rolling restart of brokers. The old secret used for encoding |
| passwords currently in ZooKeeper must be provided in the static broker config <code>password.encoder.old.secret</code> and |
| the new secret must be provided in <code>password.encoder.secret</code>. All dynamic password configs stored in ZooKeeper |
| will be re-encoded with the new secret when the broker starts up.</p> |
| <p>In Kafka 1.1.x, all dynamically updated password configs must be provided in every alter request when updating configs |
| using <code>kafka-configs.sh</code> even if the password config is not being altered. This constraint will be removed in |
| a future release.</p> |
| |
| <h5>Updating Password Configs in ZooKeeper Before Starting Brokers</h5> |
| |
| From Kafka 2.0.0 onwards, <code>kafka-configs.sh</code> enables dynamic broker configs to be updated using ZooKeeper before |
| starting brokers for bootstrapping. This enables all password configs to be stored in encrypted form, avoiding the need for |
| clear passwords in <code>server.properties</code>. The broker config <code>password.encoder.secret</code> must also be specified |
| if any password configs are included in the alter command. Additional encryption parameters may also be specified. Password |
| encoder configs will not be persisted in ZooKeeper. For example, to store SSL key password for listener <code>INTERNAL</code> |
| on broker 0: |
| |
| <pre class="line-numbers"><code class="language-bash"> > bin/kafka-configs.sh --zookeeper localhost:2181 --entity-type brokers --entity-name 0 --alter --add-config |
| 'listener.name.internal.ssl.key.password=key-password,password.encoder.secret=secret,password.encoder.iterations=8192' |
| </code></pre> |
| |
| The configuration <code>listener.name.internal.ssl.key.password</code> will be persisted in ZooKeeper in encrypted |
| form using the provided encoder configs. The encoder secret and iterations are not persisted in ZooKeeper. |
| |
| <h5>Updating SSL Keystore of an Existing Listener</h5> |
| Brokers may be configured with SSL keystores with short validity periods to reduce the risk of compromised certificates. |
| Keystores may be updated dynamically without restarting the broker. The config name must be prefixed with the listener prefix |
| <code>listener.name.{listenerName}.</code> so that only the keystore config of a specific listener is updated. |
| The following configs may be updated in a single alter request at per-broker level: |
| <ul> |
| <li><code>ssl.keystore.type</code></li> |
| <li><code>ssl.keystore.location</code></li> |
| <li><code>ssl.keystore.password</code></li> |
| <li><code>ssl.key.password</code></li> |
| </ul> |
| If the listener is the inter-broker listener, the update is allowed only if the new keystore is trusted by the truststore |
| configured for that listener. For other listeners, no trust validation is performed on the keystore by the broker. Certificates |
| must be signed by the same certificate authority that signed the old certificate to avoid any client authentication failures. |
| |
| <h5>Updating SSL Truststore of an Existing Listener</h5> |
| Broker truststores may be updated dynamically without restarting the broker to add or remove certificates. |
| Updated truststore will be used to authenticate new client connections. The config name must be prefixed with the |
| listener prefix <code>listener.name.{listenerName}.</code> so that only the truststore config of a specific listener |
| is updated. The following configs may be updated in a single alter request at per-broker level: |
| <ul> |
| <li><code>ssl.truststore.type</code></li> |
| <li><code>ssl.truststore.location</code></li> |
| <li><code>ssl.truststore.password</code></li> |
| </ul> |
| If the listener is the inter-broker listener, the update is allowed only if the existing keystore for that listener is trusted by |
| the new truststore. For other listeners, no trust validation is performed by the broker before the update. Removal of CA certificates |
| used to sign client certificates from the new truststore can lead to client authentication failures. |
| |
| <h5>Updating Default Topic Configuration</h5> |
| Default topic configuration options used by brokers may be updated without broker restart. The configs are applied to topics |
| without a topic config override for the equivalent per-topic config. One or more of these configs may be overridden at |
| cluster-default level used by all brokers. |
| <ul> |
| <li><code>log.segment.bytes</code></li> |
| <li><code>log.roll.ms</code></li> |
| <li><code>log.roll.hours</code></li> |
| <li><code>log.roll.jitter.ms</code></li> |
| <li><code>log.roll.jitter.hours</code></li> |
| <li><code>log.index.size.max.bytes</code></li> |
| <li><code>log.flush.interval.messages</code></li> |
| <li><code>log.flush.interval.ms</code></li> |
| <li><code>log.retention.bytes</code></li> |
| <li><code>log.retention.ms</code></li> |
| <li><code>log.retention.minutes</code></li> |
| <li><code>log.retention.hours</code></li> |
| <li><code>log.index.interval.bytes</code></li> |
| <li><code>log.cleaner.delete.retention.ms</code></li> |
| <li><code>log.cleaner.min.compaction.lag.ms</code></li> |
| <li><code>log.cleaner.min.cleanable.ratio</code></li> |
| <li><code>log.cleanup.policy</code></li> |
| <li><code>log.segment.delete.delay.ms</code></li> |
| <li><code>unclean.leader.election.enable</code></li> |
| <li><code>min.insync.replicas</code></li> |
| <li><code>max.message.bytes</code></li> |
| <li><code>compression.type</code></li> |
| <li><code>log.preallocate</code></li> |
| <li><code>log.message.timestamp.type</code></li> |
| <li><code>log.message.timestamp.difference.max.ms</code></li> |
| </ul> |
| |
| From Kafka version 2.0.0 onwards, unclean leader election is automatically enabled by the controller when the config |
| <code>unclean.leader.election.enable</code> is dynamically updated. |
| In Kafka version 1.1.x, changes to <code>unclean.leader.election.enable</code> take effect only when a new controller is elected. |
| Controller re-election may be forced by running: |
| |
| <pre class="line-numbers"><code class="language-bash"> > bin/zookeeper-shell.sh localhost |
| rmr /controller |
| </code></pre> |
| |
| <h5>Updating Log Cleaner Configs</h5> |
| Log cleaner configs may be updated dynamically at cluster-default level used by all brokers. The changes take effect |
| on the next iteration of log cleaning. One or more of these configs may be updated: |
| <ul> |
| <li><code>log.cleaner.threads</code></li> |
| <li><code>log.cleaner.io.max.bytes.per.second</code></li> |
| <li><code>log.cleaner.dedupe.buffer.size</code></li> |
| <li><code>log.cleaner.io.buffer.size</code></li> |
| <li><code>log.cleaner.io.buffer.load.factor</code></li> |
| <li><code>log.cleaner.backoff.ms</code></li> |
| </ul> |
| |
| <h5>Updating Thread Configs</h5> |
| The size of various thread pools used by the broker may be updated dynamically at cluster-default level used by all brokers. |
| Updates are restricted to the range <code>currentSize / 2</code> to <code>currentSize * 2</code> to ensure that config updates are |
| handled gracefully. |
| <ul> |
| <li><code>num.network.threads</code></li> |
| <li><code>num.io.threads</code></li> |
| <li><code>num.replica.fetchers</code></li> |
| <li><code>num.recovery.threads.per.data.dir</code></li> |
| <li><code>log.cleaner.threads</code></li> |
| <li><code>background.threads</code></li> |
| </ul> |
| |
| <h5>Updating ConnectionQuota Configs</h5> |
| The maximum number of connections allowed for a given IP/host by the broker may be updated dynamically at cluster-default level used by all brokers. |
| The changes will apply for new connection creations and the existing connections count will be taken into account by the new limits. |
| <ul> |
| <li><code>max.connections.per.ip</code></li> |
| <li><code>max.connections.per.ip.overrides</code></li> |
| </ul> |
| |
| <h5>Adding and Removing Listeners</h5> |
| <p>Listeners may be added or removed dynamically. When a new listener is added, security configs of the listener must be provided |
| as listener configs with the listener prefix <code>listener.name.{listenerName}.</code>. If the new listener uses SASL, |
| the JAAS configuration of the listener must be provided using the JAAS configuration property <code>sasl.jaas.config</code> |
| with the listener and mechanism prefix. See <a href="#security_jaas_broker">JAAS configuration for Kafka brokers</a> for details.</p> |
| |
| <p>In Kafka version 1.1.x, the listener used by the inter-broker listener may not be updated dynamically. To update the inter-broker |
| listener to a new listener, the new listener may be added on all brokers without restarting the broker. A rolling restart is then |
| required to update <code>inter.broker.listener.name</code>.</p> |
| |
| In addition to all the security configs of new listeners, the following configs may be updated dynamically at per-broker level: |
| <ul> |
| <li><code>listeners</code></li> |
| <li><code>advertised.listeners</code></li> |
| <li><code>listener.security.protocol.map</code></li> |
| </ul> |
| Inter-broker listener must be configured using the static broker configuration <code>inter.broker.listener.name</code> |
| or <code>inter.broker.security.protocol</code>. |
| |
| <h3><a id="topicconfigs" href="#topicconfigs">3.2 Topic-Level Configs</a></h3> |
| |
| Configurations pertinent to topics have both a server default as well an optional per-topic override. If no per-topic configuration is given the server default is used. The override can be set at topic creation time by giving one or more <code>--config</code> options. This example creates a topic named <i>my-topic</i> with a custom max message size and flush rate: |
| <pre class="line-numbers"><code class="language-bash"> > bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic my-topic --partitions 1 |
| --replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1 |
| </code></pre> |
| Overrides can also be changed or set later using the alter configs command. This example updates the max message size for <i>my-topic</i>: |
| <pre class="line-numbers"><code class="language-bash"> > bin/kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --entity-name my-topic |
| --alter --add-config max.message.bytes=128000 |
| </code></pre> |
| |
| To check overrides set on the topic you can do |
| <pre class="line-numbers"><code class="language-bash"> > bin/kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --entity-name my-topic --describe |
| </code></pre> |
| |
| To remove an override you can do |
| <pre class="line-numbers"><code class="language-bash"> > bin/kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --entity-name my-topic |
| --alter --delete-config max.message.bytes |
| </code></pre> |
| |
| The following are the topic-level configurations. The server's default configuration for this property is given under the Server Default Property heading. A given server default config value only applies to a topic if it does not have an explicit topic config override. |
| |
| <!--#include virtual="generated/topic_config.html" --> |
| |
| <h3><a id="producerconfigs" href="#producerconfigs">3.3 Producer Configs</a></h3> |
| |
| Below is the configuration of the producer: |
| <!--#include virtual="generated/producer_config.html" --> |
| |
| <h3><a id="consumerconfigs" href="#consumerconfigs">3.4 Consumer Configs</a></h3> |
| |
| Below is the configuration for the consumer: |
| <!--#include virtual="generated/consumer_config.html" --> |
| |
| <h3><a id="connectconfigs" href="#connectconfigs">3.5 Kafka Connect Configs</a></h3> |
| Below is the configuration of the Kafka Connect framework. |
| <!--#include virtual="generated/connect_config.html" --> |
| |
| <h4><a id="sourceconnectconfigs" href="#sourceconnectconfigs">3.5.1 Source Connector Configs</a></h4> |
| Below is the configuration of a source connector. |
| <!--#include virtual="generated/source_connector_config.html" --> |
| |
| <h4><a id="sinkconnectconfigs" href="#sinkconnectconfigs">3.5.2 Sink Connector Configs</a></h4> |
| Below is the configuration of a sink connector. |
| <!--#include virtual="generated/sink_connector_config.html" --> |
| |
| <h3><a id="streamsconfigs" href="#streamsconfigs">3.6 Kafka Streams Configs</a></h3> |
| Below is the configuration of the Kafka Streams client library. |
| <!--#include virtual="generated/streams_config.html" --> |
| |
| <h3><a id="adminclientconfigs" href="#adminclientconfigs">3.7 AdminClient Configs</a></h3> |
| Below is the configuration of the Kafka Admin client library. |
| <!--#include virtual="generated/admin_client_config.html" --> |
| </script> |
| |
| <div class="p-configuration"></div> |