Apache Hadoop Docker image with Kerberos enabled

Required versions

JDK8
Hadoop 2.10.2

Default Environment Variables

Name	Value	Description
`KRB_REALM`	`EXAMPLE.COM`	The Kerberos Realm, more information here
`DOMAIN_REALM`	`example.com`	The Kerberos Domain Realm, more information here
`KERB_MASTER_KEY`	`masterkey`	The Kerberos master database password, more information here
`KERBEROS_ADMIN_USER`	`admin/admin`	The KDC admin user
`KERBEROS_ADMIN_PASSWORD`	`admin`	The KDC admin password

You can simply define these variables in the docker-compose.yml.

Run image

cd flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster
wget -O hadoop/hadoop.tar.gz https://archive.apache.org/dist/hadoop/common/hadoop-2.10.2/hadoop-2.10.2.tar.gz
docker compose build
docker compose up

Usage

Get the container name with docker ps and login to the container with

docker exec -it <container-name> /bin/bash

To obtain a Kerberos ticket, execute

kinit -kt /home/hadoop-user/hadoop-user.keytab hadoop-user

Afterwards you can use hdfs CLI like

hdfs dfs -ls /

Known issues

Unable to obtain Kerberos password

Error

docker compose up fails for the first time with the error

Login failure for nn/hadoop.docker.com@EXAMPLE.COM from keytab /etc/security/keytabs/nn.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

Solution

Stop the containers with docker compose down and start again with docker compose up -d.

Java Keystore

If the Keystore has been expired, then create a new keystore.jks:

create private key

openssl genrsa -des3 -out server.key 1024

create csr

openssl req -new -key server.key -out server.csr`

remove passphrase in key

cp server.key server.key.org
openssl rsa -in server.key.org -out server.key

create self-signed cert

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

create JKS and import certificate

keytool -import -keystore keystore.jks -alias CARoot -file server.crt`