If you'd prefer to install Heron on Kubernetes without using the Helm package manager, see the Heron on Kubernetes by hand document.
Helm is an open source package manager for Kubernetes that enables you to quickly and easily install even the most complex software systems on Kubernetes. Heron has a Helm chart that you can use to install Heron on Kubernetes using just a few commands. The chart can be used to install Heron on the following platforms:
In order to install Heron on Kubernetes using Helm, you'll need to have an existing Kubernetes cluster on one of the supported platforms (which includes bare metal installations).
In order to get started, you need to install Helm on your machine. Installation instructions for macOS and Linux are below.
You can install Helm on macOS using Homebrew:
$ brew install kubernetes-helm
You can install Helm on Linux using a simple installation script:
$ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > install-helm.sh $ chmod 700 install-helm.sh $ ./install-helm.sh
To run Helm on Kubernetes, you need to first make sure that kubectl is using the right configuration context for your cluster. To check which context is being used:
$ kubectl config current-context
If the context is correct, then you can get Helm running using just one command:
$ helm init
If the output of that command includes Happy Helming!
then Helm is ready to go.
Once you've installed the Helm client on your machine and gotten Helm running in your Kubernetes cluster, you need to make your client aware of the heron-charts
Helm repository, which houses the chart for Heron:
$ helm repo add heron-charts https://storage.googleapis.com/heron-charts "heron-charts" has been added to your repositories
Now you can install the Heron package:
$ helm install heron-charts/heron
This will install Heron and provide the installation with a random name like jazzy-anaconda
. To provide the installation with a name, such as heron-kubernetes
:
$ helm install heron-charts/heron \ --name heron-kubernetes
The default platform for running Heron on Kubernetes is Minikube. To specify a different platform, you can use the --set platform=PLATFORM
flag. Here's an example:
$ helm install heron-charts/heron \ --set platform=gke
The available platforms are:
Platform | Tag |
---|---|
Minikube | minikube |
Google Kubernetes Engine | gke |
Amazon Web Services | aws |
Bare metal | baremetal |
To run Heron on Minikube, you need to first install Minikube. Once Minikube is installed, you can start it by running minikube start
. Please note, however, that Heron currently requires the following resources:
To start up Minikube with the minimum necessary resources:
$ minikube start \ --memory=7168 \ --cpus=5 \ --disk-size=20g
Once Minikube is running, you can then install Heron in one of two ways:
# Use the Minikube default $ helm install heron-charts/heron # Explicitly select Minikube $ helm install heron-charts/heron \ --set platform=minikube
The resources required to run Heron on Google Kubernetes Engine vary based on your use case. To run a basic Heron cluster intended for development and experimentation, you'll need at least:
To create a cluster with those resources using the gcloud tool:
$ gcloud container clusters create heron-gke-dev-cluster \ --num-nodes=3 \ --machine-type=n1-standard-2
For a production-ready cluster you'll want a larger cluster with:
To create such a cluster:
$ gcloud container clusters create heron-gke-prod-cluster \ --num-nodes=8 \ --machine-type=n1-standard-8
Once the cluster has been successfully created, you‘ll need to install that cluster’s credentials locally so that they can be used by kubectl. You can do this in just one command:
$ gcloud container clusters get-credentials heron-gke-dev-cluster # or heron-gke-prod-cluster
Once, the cluster is running (that could take a few minutes), you can initialize Helm on the cluster:
$ helm init
Then, you'll need to adjust some RBAC permissions for your cluster:
$ kubectl create serviceaccount tiller \ --namespace kube-system \ $ kubectl create clusterrolebinding tiller-cluster-rule \ --clusterrole cluster-admin \ --serviceaccount kube-system:tiller $ kubectl patch deploy tiller-deploy \ --namespace kube-system \ --patch '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
Finally, you can install Heron:
$ helm install heron-charts/heron \ --set platform=gke
Helm enables you to supply sets of variables via YAML files. There are currently a handful of different resource configurations that can be applied to your Heron on GKE cluster upon installation:
Configuration | Description |
---|---|
small.yaml | Smaller Heron cluster intended for basic testing, development, and experimentation |
medium.yaml | Closer geared for production usage |
To apply the small
configuration, for example:
$ helm install heron-charts/heron \ --set platform=gke \ --values https://raw.githubusercontent.com/apache/incubator-heron/master/deploy/kubernetes/gcp/small.yaml
To run Heron on Kubernetes on Amazon Web Services (AWS), you'll need to
$ helm install heron-charts/heron \ --set platform=aws
You can make Heron to use S3 to distribute the user topologies. First you need to set up a S3 bucket and configure an IAM user with enough permissions over it. Get access keys for the user. Then you can deploy Heron like this:
$ helm install heron-charts/heron \ --set platform=aws \ --set uploader.class=s3 \ --set uploader.s3Bucket=heron \ --set uploader.s3PathPrefix=topologies \ --set uploader.s3AccessKey=XXXXXXXXXXXXXXXXXXXX \ --set uploader.s3SecretKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \ --set uploader.s3Region=us-west-1
To run Heron on a bare metal Kubernetes cluster:
$ helm install heron-charts/heron \ --set platform=baremetal
When setting the
heron
CLI configuration, make sure that the cluster name matches the name of the Helm installation. This can be either the name auto-generated by Helm or the name you supplied via the--name
flag upon installation (in some of the examples above, theheron-kubernetes
name was used). Make sure to adjust the name accordingly if necessary.
Once all of the components have been successfully started up, you need to open up a proxy port to your Kubernetes cluster using the kubectl proxy
command:
$ kubectl proxy -p 8001
Note: All of the following Kubernetes specific urls are valid with the Kubernetes 1.10.0 release.
Now, verify that the Heron API server running on Minikube is available using curl:
$ curl http://localhost:8001/api/v1/namespaces/default/services/heron-kubernetes-apiserver:9000/proxy/api/v1/version
You should get a JSON response like this:
{ "heron.build.git.revision" : "ddbb98bbf173fb082c6fd575caaa35205abe34df", "heron.build.git.status" : "Clean", "heron.build.host" : "ci-server-01", "heron.build.time" : "Sat Mar 31 09:27:19 UTC 2018", "heron.build.timestamp" : "1522488439000", "heron.build.user" : "release-agent", "heron.build.version" : "0.17.8" }
Once you have a Heron cluster up and running on Kubernetes via Helm, you can use the heron
CLI tool like normal if you set the proper URL for the Heron API server. When running Heron on Kubernetes, that URL is:
$ http://localhost:8001/api/v1/namespaces/default/services/heron-kubernetes-apiserver:9000/proxy
To set that URL:
$ heron config heron-kubernetes set service_url \ http://localhost:8001/api/v1/namespaces/default/services/heron-kubernetes-apiserver:9000/proxy
To test your cluster, you can submit an example topology:
$ heron submit heron-kubernetes \ ~/.heron/examples/heron-streamlet-examples.jar \ org.apache.heron.examples.streamlet.WindowedWordCountTopology \ WindowedWordCount