If you'd prefer to install Heron on Kubernetes without using the Helm package manager, see the Heron on Kubernetes by hand document.
Helm is an open source package manager for Kubernetes that enables you to quickly and easily install even the most complex software systems on Kubernetes. Heron has a Helm chart that you can use to install Heron on Kubernetes using just a few commands. The chart can be used to install Heron on the following platforms:
In order to install Heron on Kubernetes using Helm, you'll need to have an existing Kubernetes cluster on one of the supported platforms (which includes bare metal installations).
In order to get started, you need to install Helm on your machine. Installation instructions for macOS and Linux are below.
You can install Helm on macOS using Homebrew:
$ brew install kubernetes-helm
You can install Helm on Linux using a simple installation script:
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 $ chmod 700 get_helm.sh $ ./get_helm.sh
To use Helm with Kubernetes, you need to first make sure that kubectl is using the right configuration context for your cluster. To check which context is being used:
$ kubectl config current-context
Once you've installed the Helm client on your machine and gotten Helm pointing to your Kubernetes cluster, you need to make your client aware of the heron-charts Helm repository, which houses the chart for Heron:
$ helm repo add heron-charts https://storage.googleapis.com/heron-charts "heron-charts" has been added to your repositories
Create a namespace to install into:
$ kubectl create namespace heron
Now you can install the Heron package:
$ helm install heron-charts/heron -g -n heron
This will install Heron and provide the installation in the heron namespace (-n) with a random name (-g) like jazzy-anaconda. To provide the installation with a name, such as heron-kube:
$ helm install heron-kube heron-charts/heron \ --namespace heron
The default platform for running Heron on Kubernetes is Minikube. To specify a different platform, you can use the --set platform=PLATFORM flag. Here's an example:
$ helm install heron-kube heron-charts/heron \ --namespace heron \ --set platform=gke
The available platforms are:
| Platform | Tag |
|---|---|
| Minikube | minikube |
| Google Kubernetes Engine | gke |
| Amazon Web Services | aws |
| Bare metal | baremetal |
To run Heron on Minikube, you need to first install Minikube. Once Minikube is installed, you can start it by running minikube start. Please note, however, that Heron currently requires the following resources:
To start up Minikube with the minimum necessary resources:
$ minikube start \ --memory=7168 \ --cpus=5 \ --disk-size=20g
Once Minikube is running, you can then install Heron in one of two ways:
Create a namespace to install into:
$ kubectl create namespace heron
# Use the Minikube default $ helm install heron-kube heron-charts/heron \ --namespace heron # Explicitly select Minikube $ helm install heron-kube heron-charts/heron \ --namespace heron \ --set platform=minikube
The resources required to run Heron on Google Kubernetes Engine vary based on your use case. To run a basic Heron cluster intended for development and experimentation, you'll need at least:
To create a cluster with those resources using the gcloud tool:
$ gcloud container clusters create heron-gke-dev-cluster \ --num-nodes=3 \ --machine-type=n1-standard-2
For a production-ready cluster you'll want a larger cluster with:
To create such a cluster:
$ gcloud container clusters create heron-gke-prod-cluster \ --num-nodes=8 \ --machine-type=n1-standard-8
Once the cluster has been successfully created, you‘ll need to install that cluster’s credentials locally so that they can be used by kubectl. You can do this in just one command:
$ gcloud container clusters get-credentials heron-gke-dev-cluster # or heron-gke-prod-cluster
Create a namespace to install into:
$ kubectl create namespace heron
Now you can install Heron:
$ helm install heron-kube heron-charts/heron \ --namespace heron \ --set platform=gke
Helm enables you to supply sets of variables via YAML files. There are currently a handful of different resource configurations that can be applied to your Heron on GKE cluster upon installation:
| Configuration | Description |
|---|---|
small.yaml | Smaller Heron cluster intended for basic testing, development, and experimentation |
medium.yaml | Closer geared for production usage |
To apply the small configuration, for example:
$ helm install heron-kube heron-charts/heron \ --namespace heron \ --set platform=gke \ --values https://raw.githubusercontent.com/apache/incubator-heron/master/deploy/kubernetes/gcp/small.yaml
To run Heron on Kubernetes on Amazon Web Services (AWS), you'll need to
$ helm install heron-kube heron-charts/heron \ --namespace heron \ --set platform=aws
You can make Heron to use S3 to distribute the user topologies. First you need to set up a S3 bucket and configure an IAM user with enough permissions over it. Get access keys for the user. Then you can deploy Heron like this:
$ helm install heron-kube heron-charts/heron \ --namespace heron \ --set platform=aws \ --set uploader.class=s3 \ --set uploader.s3Bucket=heron \ --set uploader.s3PathPrefix=topologies \ --set uploader.s3AccessKey=XXXXXXXXXXXXXXXXXXXX \ --set uploader.s3SecretKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \ --set uploader.s3Region=us-west-1
To run Heron on a bare metal Kubernetes cluster:
$ helm install heron-kube heron-charts/heron \ --namespace heron \ --set platform=baremetal
When setting the
heronCLI configuration, make sure that the cluster name matches the name of the Helm installation. This can be either the name auto-generated by Helm or the name you supplied via the--nameflag upon installation (in some of the examples above, theheron-kubernetesname was used). Make sure to adjust the name accordingly if necessary.
Once all of the components have been successfully started up, you need to open up a proxy port to your Kubernetes cluster using the kubectl proxy command:
$ kubectl proxy -p 8001
Note: All of the following Kubernetes specific urls are valid with the Kubernetes 1.10.0 release.
Now, verify that the Heron API server running on Minikube is available using curl:
$ curl http://localhost:8001/api/v1/namespaces/default/services/heron-kubernetes-apiserver:9000/proxy/api/v1/version
You should get a JSON response like this:
{ "heron.build.git.revision" : "ddbb98bbf173fb082c6fd575caaa35205abe34df", "heron.build.git.status" : "Clean", "heron.build.host" : "ci-server-01", "heron.build.time" : "Sat Mar 31 09:27:19 UTC 2018", "heron.build.timestamp" : "1522488439000", "heron.build.user" : "release-agent", "heron.build.version" : "0.17.8" }
Once you have a Heron cluster up and running on Kubernetes via Helm, you can use the heron CLI tool like normal if you set the proper URL for the Heron API server. When running Heron on Kubernetes, that URL is:
$ http://localhost:8001/api/v1/namespaces/default/services/heron-kubernetes-apiserver:9000/proxy
To set that URL:
$ heron config heron-kubernetes set service_url \ http://localhost:8001/api/v1/namespaces/default/services/heron-kubernetes-apiserver:9000/proxy
To test your cluster, you can submit an example topology:
$ heron submit heron-kubernetes \ ~/.heron/examples/heron-streamlet-examples.jar \ org.apache.heron.examples.streamlet.WindowedWordCountTopology \ WindowedWordCount