This page provides instructions on how to enable SSL for the network communication between different Flink components.
SSL can be enabled for all network communication between Flink components. SSL keystores and truststore has to be deployed on each Flink node and configured (conf/flink-conf.yaml) using keys in the security.ssl.* namespace (Please see the configuration page for details). SSL can be selectively enabled/disabled for different transports using the following flags. These flags are only applicable when security.ssl.enabled is set to true.
You need to have a Java Keystore generated and copied to each node in the Flink cluster. The common name or subject alternative names in the certificate should match the node's hostname and IP address. Keystores and truststores can be generated using the keytool utility. All Flink components should have read access to the keystore and truststore files.
Execute the following keytool commands to create a truststore with a self signed CA.
{% highlight bash %} keytool -genkeypair -alias ca -keystore ca.keystore -dname “CN=Sample CA” -storepass password -keypass password -keyalg RSA -ext bc=ca:true keytool -keystore ca.keystore -storepass password -alias ca -exportcert > ca.cer keytool -importcert -keystore ca.truststore -alias ca -storepass password -noprompt -file ca.cer {% endhighlight %}
Now create keystores for each node with certificates signed by the above CA. Let node1.company.org and node2.company.org be the hostnames with IPs 192.168.1.1 and 192.168.1.2 respectively
{% highlight bash %} keytool -genkeypair -alias node1 -keystore node1.keystore -dname “CN=node1.company.org” -ext SAN=dns:node1.company.org,ip:192.168.1.1 -storepass password -keypass password -keyalg RSA keytool -certreq -keystore node1.keystore -storepass password -alias node1 -file node1.csr keytool -gencert -keystore ca.keystore -storepass password -alias ca -ext SAN=dns:node1.company.org,ip:192.168.1.1 -infile node1.csr -outfile node1.cer keytool -importcert -keystore node1.keystore -storepass password -file ca.cer -alias ca -noprompt keytool -importcert -keystore node1.keystore -storepass password -file node1.cer -alias node1 -noprompt {% endhighlight %}
{% highlight bash %} keytool -genkeypair -alias node2 -keystore node2.keystore -dname “CN=node2.company.org” -ext SAN=dns:node2.company.org,ip:192.168.1.2 -storepass password -keypass password -keyalg RSA keytool -certreq -keystore node2.keystore -storepass password -alias node2 -file node2.csr keytool -gencert -keystore ca.keystore -storepass password -alias ca -ext SAN=dns:node2.company.org,ip:192.168.1.2 -infile node2.csr -outfile node2.cer keytool -importcert -keystore node2.keystore -storepass password -file ca.cer -alias ca -noprompt keytool -importcert -keystore node2.keystore -storepass password -file node2.cer -alias node2 -noprompt {% endhighlight %}
Configure each node in the standalone cluster to pick up the keystore and truststore files present in the local file system.
{% highlight yaml %} security.ssl.enabled: true security.ssl.keystore: /usr/local/node1.keystore security.ssl.keystore-password: password security.ssl.key-password: password security.ssl.truststore: /usr/local/ca.truststore security.ssl.truststore-password: password {% endhighlight %}
{% highlight yaml %} security.ssl.enabled: true security.ssl.keystore: /usr/local/node2.keystore security.ssl.keystore-password: password security.ssl.key-password: password security.ssl.truststore: /usr/local/ca.truststore security.ssl.truststore-password: password {% endhighlight %}
The keystores and truststore can be deployed in a YARN setup in multiple ways depending on the cluster setup. Following are two ways to achieve this.
The keystores and truststore should be generated and deployed on all nodes in the YARN setup where Flink components can potentially be executed. The same Flink config file from the Flink YARN client is used for all the Flink components running in the YARN cluster. Therefore we need to ensure the keystore is deployed and accessible using the same filepath in all the YARN nodes.
{% highlight yaml %} security.ssl.enabled: true security.ssl.keystore: /usr/local/node.keystore security.ssl.keystore-password: password security.ssl.key-password: password security.ssl.truststore: /usr/local/ca.truststore security.ssl.truststore-password: password {% endhighlight %}
Now you can start the YARN session from the CLI like you would normally do.
We can use the YARN client's ship files option (-yt) to distribute the keystores and truststore. Since the same keystore will be deployed at all nodes, we need to ensure a single certificate in the keystore can be served for all nodes. This can be done by either using the Subject Alternative Name (SAN) extension in the certificate and setting it to cover all nodes (hostname and ip addresses) in the cluster or by using wildcard subdomain names (if the cluster is setup accordingly).
{% highlight yaml %} security.ssl.enabled: true security.ssl.keystore: deploy-keys/node.keystore security.ssl.keystore-password: password security.ssl.key-password: password security.ssl.truststore: deploy-keys/ca.truststore security.ssl.truststore-password: password {% endhighlight %}
{% highlight bash %} flink run -m yarn-cluster -yt deploy-keys/ TestJob.jar {% endhighlight %}
When deployed using YARN, Flink‘s web dashboard is accessible through YARN proxy’s Tracking URL. To ensure that the YARN proxy is able to access Flink‘s https url you need to configure YARN proxy to accept Flink’s SSL certificates. Add the custom CA certificate into Java's default truststore on the YARN Proxy node.
{% top %}