Heron Kubernetes

Requirements:

Kubernetes cluster with 3 or more nodes
kubectl
heron client

Start Components:

Start zookeeper:

$ kubectl create -f https://raw.githubusercontent.com/apache/incubator-heron/master/deploy/kubernetes/general/zookeeper.yaml

$ kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
zk-0                                  1/1       Running   0          1m

# wait until zk-0 STATUS is Running before proceeding to the next steps

Start bookkeeper:

$ kubectl create -f https://raw.githubusercontent.com/apache/incubator-heron/master/deploy/kubernetes/general/bookkeeper.yaml

This deploys bookkeeper in a DaemonSet and requires the ability of exposing hostPort for pods communication. In some environments like K8S on DC/OS, hostPort is not well supported. You can consider deploying bookkeeper in a StatefulSet with Persistent Volumes as below. Please see Persistent Volumes for more details.

$ kubectl create -f https://raw.githubusercontent.com/apache/incubator-heron/master/deploy/kubernetes/general/bookkeeper.statefulset.yaml

Start heron tools:

$ kubectl create -f https://raw.githubusercontent.com/apache/incubator-heron/master/deploy/kubernetes/general/tools.yaml

Start heron API server:

$ kubectl create -f https://raw.githubusercontent.com/apache/incubator-heron/master/deploy/kubernetes/general/apiserver.yaml

Deploy via heron API server

We will start a proxy to the cluster and then construct a proxy url to access the API server.

Start kubectl proxy:

$ kubectl proxy -p 8001

Verify we can access the API server:

$ curl http://localhost:8001/api/v1/namespaces/default/services/heron-apiserver:9000/proxy/api/v1/version
{
   "heron.build.git.revision" : "bf9fe93f76b895825d8852e010dffd5342e1f860",
   "heron.build.git.status" : "Clean",
   "heron.build.host" : "ci-server-01",
   "heron.build.time" : "Sun Oct  1 20:42:18 UTC 2017",
   "heron.build.timestamp" : "1506890538000",
   "heron.build.user" : "release-agent1",
   "heron.build.version" : "0.16.2"
}

Set service_url:

$ heron config kubernetes \
set service_url http://localhost:8001/api/v1/namespaces/default/services/heron-apiserver:9000/proxy \
org.apache.heron.examples.api.AckingTopology acking

Submit an example topology:

$ heron submit kubernetes ~/.heron/examples/heron-api-examples.jar \
org.apache.heron.examples.api.AckingTopology acking

View heron ui:

http://localhost:8001/api/v1/namespaces/default/services/heron-ui:8889/proxy