tree: b126abd997bf3a1220a2fe5527f768987b254bc5 [path history] [tgz]
  1. configuration/
  2. package/
  3. src/
  4. appConfig-default.json
  8. metainfo-default.xml
  9. pom.xml
  11. resources-default.json

Kafka On YARN (KOYA)


  • Use capabilities of YARN for Kafka broker management
  • Automate broker recovery
  • Make it easy to deploy, configure and monitor Kafka clusters
  • Simplify management tasks (alternative to Kafka command line utilities)


Kafka as YARN application using Slider


Checkout Slider code (

git clone
cd incubator-slider
git checkout -b develop remotes/origin/develop

Download Kafka binary package (, e.g.

wget -O /tmp/kafka_2.10-

Build Kafka app package

mvn clean package -DskipTests -Pkafka-app-package -pkg.src=/tmp


  • Slider application package: app-packages/kafka/target/


####Install Slider

To use the archive with embedded Slider, copy it to the machine from which you launch YARN applications (Hadoop client, gateway or edge node). Extract the file and configure Slider:

If the environment variables HADOOP_CONF_DIR or JAVA_HOME are not already defined through your Hadoop installation, you can export them in slider-0.90.0-incubating/conf/


export HADOOP_CONF_DIR=/etc/hadoop/conf
export JAVA_HOME=/usr/jdk64/jdk1.8.0_60

If the registry ZooKeeper quorum was not already configured through Hadoop, modify slider-0.90.0-incubating/conf/slider-client.xml:


Above steps are not required with HDP 2.2

More information regarding Slider client configuration refer to

Configure KOYA application package

Before the Kafka cluster can be launched, the brokers need to be defined. Currently Slider does not support configuration properties at instance level, therefore each broker has to be configured as a component.

If you use the full archive, the configuration file templates are already in your working directory. Otherwise extract them from the Slider package.


Extract the packaged configuration files you are going to customize:

unzip appConfig.json resources.json

Adjust following properties in the global section:

    "application.def": "",
    "": "256m",
    "": "128m",
    "": "${ZK_HOST}"

Above will be used to configure and launch the Kafka server(s). All properties prefixed with will be set in the file supplied with the Kafka archive. Only non-default settings need to be defined here.


Configure the number of servers and other resource requirements:

  "components" : {
    "broker" : {
      "yarn.role.priority" : "1",
      "yarn.component.instances" : "5",
      "yarn.memory" : "768",
      "yarn.vcores" : "1",

More information about the application configuration can be found here.

Deploy KOYA Cluster

The Slider application package needs to be copied to the HDFS location that was specified as application.def in appConfig.json:

hdfs dfs -copyFromLocal /path/in/appConfig

Now the KOYA cluster can be deployed and launched:

slider-0.90.0-incubating/bin/slider create koya --template ~/koya/appConfig.json  --resources ~/koya/resources.json