Document deploying OpenWhisk on kind (#535)
The kind project is the replacement for the retired Kubernetes
dind-cluster project. Kind uses Docker-in-Docker virtualization to
create a virtual multi-node Kubernetes cluster on a single host.
This commit contains multiple pieces:
1. Documents usage of kind to deploy OpenWhisk
2. Switches TravisCI from using dind-cluster to kind
3. Removes minikube from top-level README in favor
of steering people to kind.
Switching to kind for TravisCI is required to be able to track
Kubernetes versions (dind-cluster stops at 1.15).
One unfortunate aspect of this switch is that kind uses containerd as
the underlying container engine, therefore it is no longer possible to
test the DockerContainerFactory in the deploy-kube TravisCI runs. This
is a symptom of the broader Kubernetes ecosystem moving away from
Docker in favor of containerd.
diff --git a/.travis.yml b/.travis.yml
index fde5f8d..b27a506 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -23,11 +23,10 @@
- secure: d7CuMXbhT83W2x78qiLwgogX1+3aPicd1PlTwwNNDN6QSkImbxareyKThnsqlHIiNj3o5l5DBuiYjy7wrF/xD1g8BQMmTwm99DRx5q3CI3Im3VCi/ZK8SaNjuOy24d7cf5k2tB/87Gk7zmKsMDYm+fpCl+GpgUmIEeIwthiAxuXSDWZ8eQPIptmxj56DeFRNouvXG+dEUtBfWiwN27UPxNKExCixFnegmdtffLbz6hhst7BHr5Ry9acbycre98PCwWZcu9lxFs+SJ1kvnzX2iue4otmDkF1WkJjxaOFPJVs/D3YItg+neLCSxjwBskPed+Fct8bOjcM/uVROJPNIq5icBmaPX2isH0lvtxOeVw/dmioWYXYPN9ygBOe4eO/vtPllN0bcAUo5xl9jXev8ciAozYrYpHVh9Fplfd81rcYTeYzALmRJBdoiWoc3KQGzwGc9sB1ffmy+KWgG9T0zbnS4fALSR4PSzyNlKSLXw9vuvdNP0OBYtO+6yTJXavIxqmDoj64Lb93n+uGPatnaIGPTKEEBMJTSjsgYVEfxzzZuxUT9Ldkf2lzqvN2PCllGoMqxkgsdb8i4v4QgRaMWBDbKa5Va4k0O4dnhRmtdbJavOSKN6fECJbyfoJlV1VvJGxk5znVLRznBmUPBKbNccyPZJULugKD3QIh4q8Q5jBU=
- secure: CJtnU94HTDqd4A6uvhFl8IpnmU+wTdlzb8bPBFUl/lI/VKXiRrYpgJdKUro5xEoxFKuqMprLhbyf66niyWLTIeogjUAEu/h/o2dBVeGgSGKoqC0hQgqvnxKFeGlzFJ0XuEs3vbStJGRnQszGsfnnDrscJtR0x9X+1w4aBKI7iPyyuFtVkDD1UsmBbSi+M8FTeq7G7A0reMDaey7uog3CFCpIMl4geshcohQEcKEGbnXQZoLPFpb7cBOE83VXBJ7Y7Dgf/U4keiLovvnuJThGKZm/SVV2KlELmBmtmbx3rMT6Vb5k9ChSdRWapromNnnzmJBIQ5Scc2mwV3A93/SMha1F3IlYpDKs5djfTw8jZfVnuiou7HhTaRjHkmmcwP12/k30gLe2kw0Vezg1TCY4zgtOpcmCxc8RHEy0ceA74rKvRi8LbexTCwX+iAMQFn/pSrh/OqAq/50JbLyczcoO1zXWS38txUQNLW8i+XllhCg9pjkjyfBeGjOOcWiVIz9rWJd2XufjSXDcj6xoZHtkh1XDt1CnVkpsYKtyyZucQnhUM9ebmaWqbSW2+bpqC/2hI+G+kOyyCesGdB1q+VmN1augMMs6RgWjk4yw5dyLAshATSoUlE8KH2cDcJL19r4ECaQ99PSLwxoB89yfPoJiNc42vwxRdsLmB1BMNyPa81Y=
matrix:
- - TRAVIS_KUBE_VERSION=1.13 OW_INCLUDE_SYSTEM_TESTS=true OW_CONTAINER_FACTORY=docker
- - TRAVIS_KUBE_VERSION=1.14 OW_INCLUDE_SYSTEM_TESTS=true OW_CONTAINER_FACTORY=docker
- - TRAVIS_KUBE_VERSION=1.14 OW_INCLUDE_SYSTEM_TESTS=false OW_CONTAINER_FACTORY=kubernetes
- - TRAVIS_KUBE_VERSION=1.14 OW_INCLUDE_SYSTEM_TESTS=false OW_CONTAINER_FACTORY=kubernetes OW_LEAN_MODE=true
- - TRAVIS_KUBE_VERSION=1.14 OW_INCLUDE_SYSTEM_TESTS=true OW_CONTAINER_FACTORY=docker OW_LEAN_MODE=true
+ - TRAVIS_KUBE_VERSION=v1.13.10 OW_INCLUDE_SYSTEM_TESTS=false OW_CONTAINER_FACTORY=kubernetes
+ - TRAVIS_KUBE_VERSION=v1.14.6 OW_INCLUDE_SYSTEM_TESTS=false OW_CONTAINER_FACTORY=kubernetes
+ - TRAVIS_KUBE_VERSION=v1.14.6 OW_INCLUDE_SYSTEM_TESTS=false OW_CONTAINER_FACTORY=kubernetes OW_LEAN_MODE=true
+ - TRAVIS_KUBE_VERSION=v1.15.3 OW_INCLUDE_SYSTEM_TESTS=false OW_CONTAINER_FACTORY=kubernetes
services:
- docker
@@ -42,7 +41,7 @@
before_install:
- ./tools/travis/setupscan.sh
- ./tools/travis/scancode.sh
- - ./tools/travis/start-kubeadm-dind.sh
+ - ./tools/travis/start-kind.sh
script:
- ./tools/travis/deploy-chart.sh
diff --git a/README.md b/README.md
index 891ad6a..523f90d 100644
--- a/README.md
+++ b/README.md
@@ -82,31 +82,14 @@
version 18.06 or later. Please follow our
[setup instructions](docs/k8s-docker-for-mac.md) to initially create
your cluster.
-2. Linux: Use kubeadm-dind-cluster, but carefully follow our
-[setup instructions](docs/k8s-dind-cluster.md) because the default
-setup of kubeadm-dind-cluster does *not* meet the requirements for
-running OpenWhisk.
+2. Linux: Use [kind](https://github.com/kubernetes-sigs/kind).
+Please follow our [setup instructions](docs/k8s-kind.md)
+to initially create your cluster.
3. Windows: Use the built-in Kubernetes support in Docker for Windows
version 18.06 or later. Please follow our
[setup instructions](docs/k8s-docker-for-windows.md) to initially create
your cluster.
-### Using Minikube
-
-Minikube provides a Kubernetes cluster running inside a virtual
-machine (for example VirtualBox). It can be used on MacOS, Linux, or
-Windows to run OpenWhisk, but is somewhat less flexible than the
-docker-in-docker options described above. Configuring the Minikube VM
-with 4GB of memory and 2 virtual CPUs is sufficient for the default
-settings of OpenWhisk. For details on setting up
-Minikube, see these [setup instructions](docs/k8s-minikube.md).
-
-### Using Minishift
-
-Minishift is the OpenShift equivalent to Minikube. We would welcome
-documentation on the process of running OpenWhisk on Minishift.
-
-
### Using a Kubernetes cluster from a cloud provider
You can also provision a Kubernetes cluster from a cloud provider,
@@ -231,8 +214,8 @@
OpenWhisk to your cluster. For details, see the documentation
appropriate to your Kubernetes cluster:
* [Docker for Mac](docs/k8s-docker-for-mac.md#configuring-openwhisk)
-* [kubeadm-dind-cluster](docs/k8s-dind-cluster.md#configuring-openwhisk)
-* [Minikube](docs/k8s-minikube.md#configuring-openwhisk)
+* [Docker for Windows](docs/k8s-docker-for-windows.md#configuring-openwhisk)
+* [kind](docs/k8s-kind.md#configuring-openwhisk)
* [IBM Kubernetes Service (IKS)](docs/k8s-ibm-public.md#configuring-openwhisk)
* [IBM Cloud Private (ICP)](docs/k8s-ibm-private.md#configuring-openwhisk)
* [Google (GKE)](docs/k8s-google.md#configuring-openwhisk)
diff --git a/docs/k8s-kind.md b/docs/k8s-kind.md
new file mode 100644
index 0000000..5b752dc
--- /dev/null
+++ b/docs/k8s-kind.md
@@ -0,0 +1,127 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+
+# Deploying OpenWhisk on kind
+
+## Overview
+
+You can run Kubernetes on top of Docker on Linux, MacOS, and Windows using the
+[kind](https://github.com/kubernetes-sigs/kind) project.
+Based on using Docker-in-Docker (DIND) virtualization and
+`kubeadm`, kind can be used to create a virtual multi-node
+Kubernetes cluster that is suitable for deploying
+OpenWhisk for development and testing. For detailed instructions on kind, we
+refer you to that project's [github repository](https://github.com/kubernetes-sigs/kind).
+Here we will only cover the basic operations needed to create and
+operate a default cluster with two virtual worker nodes.
+
+## Initial setup
+
+Download the latest stable release of `kind` for your platform from
+https://github.com/kubernetes-sigs/kind/releases. Our TravisCI testing
+currently uses kind v0.5.1 on an ubuntu 18.04 host.
+
+### Creating the Kubernetes Cluster
+
+On Linux, make sure your userid is in the `docker` group on the host
+machine. This will enable you to run `kind` without
+requiring `sudo` to gain `root` privileges.
+
+Create a kind-cluster.yaml to configure your cluster.
+```yaml
+kind: Cluster
+apiVersion: kind.sigs.k8s.io/v1alpha3
+nodes:
+- role: control-plane
+- role: worker
+ extraPortMappings:
+ - hostPort: 31001
+ containerPort: 31001
+- role: worker
+```
+The extraPortMappings stanza enables port forwarding
+from the localhost to the in-cluster network.
+This is required on MacOS, but to simplify the instructions
+we use the same setup for all platforms.
+
+Now create your cluster with the command:
+```shell
+kind create cluster --config kind-cluster.yaml
+```
+
+Next, configure `kubectl` by executing
+```shell
+KUBECONFIG="$(kind get kubeconfig-path)"
+```
+
+Then label the two worker nodes so that one is reserved for the invoker
+and the other will be used to run the rest of the OpenWhisk system.
+```shell
+kubectl label node kind-worker openwhisk-role=core
+kubectl label node kind-worker2 openwhisk-role=invoker
+```
+
+### Configuring OpenWhisk
+
+To configure OpenWhisk, you first need to define a mycluster.yaml
+that specifies the "inside the cluster" ingress information and
+other system configuration. First, determine the internalIP of
+a worker node with the command:
+```
+kubectl describe node kind-worker | grep InternalIP: | awk '{print $2}'
+```
+A mycluster.yaml for a standard deployment of OpenWhisk would look
+like the below, replacing <InternalIP> with its actual value:
+```yaml
+whisk:
+ ingress:
+ type: NodePort
+ apiHostName: <INTERNAL_IP>
+ apiHostPort: 31001
+
+invoker:
+ containerFactory:
+ impl: "kubernetes"
+
+nginx:
+ httpsNodePort: 31001
+```
+Note that you must use the KubernetesContainerFactory when running
+OpenWhisk on `kind` because it is configured to use `containerd`
+as the underlying container engine.
+
+External to the Kubernetes cluster, for example when using the `wsk` cli,
+we will use the port forwarding configured by the `extraPortMappings`
+in kind-cluster.yaml to allow the OpenWhisk apihost property
+to be set to localhost:31001
+
+## Limitations
+
+Using kind is only appropriate for development and testing purposes.
+It is not recommended for production deployments of OpenWhisk.
+
+TLS termination will be handled by OpenWhisk's `nginx` service and
+will use self-signed certificates. You will need to invoke `wsk` with
+the `-i` command line argument to bypass certificate checking.
+
+Unlike using Kubernetes with Docker for Mac 18.06 and later, only the
+virtual master/worker nodes are visible to Docker on the host system. The
+individual pods running the OpenWhisk system are only visible using
+`kubectl` and not directly via host Docker commands.
diff --git a/docs/k8s-minikube.md b/docs/k8s-minikube.md
index 20decf4..362a073 100644
--- a/docs/k8s-minikube.md
+++ b/docs/k8s-minikube.md
@@ -21,6 +21,10 @@
## Overview
+WARNING: Consider using kind instead of Minikube. We no longer
+test these instructions and are considering removing them
+entirely.
+
Minikube runs a single node Kubernetes cluster inside of a VM
(virtual machine) running on your host machine. Depending on your host
machine, you will have different choices for the VM.
diff --git a/tools/travis/collect-logs.sh b/tools/travis/collect-logs.sh
index ed3f4a3..ba636ba 100755
--- a/tools/travis/collect-logs.sh
+++ b/tools/travis/collect-logs.sh
@@ -23,6 +23,9 @@
cd $ROOTDIR
+# kind puts config file in non-standard place; must set KUBECONFIG
+export KUBECONFIG="$(kind get kubeconfig-path)"
+
echo "Gathering logs to upload to https://app.box.com/v/openwhisk-travis-logs"
mkdir logs
@@ -41,7 +44,4 @@
kubectl -n openwhisk logs -lname=ow4travis-alarmprovider >& logs/kafkaprovider.log
kubectl -n openwhisk logs -lname=ow4travis-cloudantprovider >& logs/cloudantprovider.log
kubectl -n openwhisk logs -lname=ow4travis-kafkaprovider >& logs/kafkaprovider.log
-kubectl get pods --all-namespaces -o wide --show-all >& logs/all-pods.txt
-
-# System level logs from kubernetes cluster
-$HOME/dind-cluster.sh dump >& logs/dind-cluster-dump.txt
+kubectl get pods --all-namespaces -o wide >& logs/all-pods.txt
diff --git a/tools/travis/deploy-chart.sh b/tools/travis/deploy-chart.sh
index f290abe..eab00a7 100755
--- a/tools/travis/deploy-chart.sh
+++ b/tools/travis/deploy-chart.sh
@@ -166,23 +166,26 @@
OW_CONTAINER_FACTORY=${OW_CONTAINER_FACTORY:="docker"}
# Default to not including system tests in helm test suite
-OW_INCLUDE_SYSTEM_TESTS=${$OW_INCLUDE_SYSTEM_TESTS:="false"}
+OW_INCLUDE_SYSTEM_TESTS=${OW_INCLUDE_SYSTEM_TESTS:="false"}
# Default timeout limit to 60 steps
TIMEOUT_STEP_LIMIT=${TIMEOUT_STEP_LIMIT:=60}
+# kind puts config file in non-standard place; must set KUBECONFIG
+export KUBECONFIG="$(kind get kubeconfig-path)"
+
# Label nodes for affinity.
# For DockerContainerFactory, at least one must be labeled as an invoker.
echo "Labeling nodes with openwhisk-role assignments"
-kubectl label nodes kube-node-1 openwhisk-role=core
-kubectl label nodes kube-node-2 openwhisk-role=invoker
+kubectl label nodes kind-worker openwhisk-role=core
+kubectl label nodes kind-worker2 openwhisk-role=invoker
-# Configure a NodePort Ingress assuming kubeadm-dind-cluster conventions.
-# Use kube-node-1 as the ingress, since we labeled it as our core node above.
-# (But using kube-node-2 would also work because Kubernetes
+# Configure a NodePort Ingress assuming kind conventions.
+# Use kind-worker as the ingress, since we labeled it as our core node above.
+# (But using kind-worker2 would also work because Kubernetes
# exposes the same NodePort service on all worker nodes.)
WSK_PORT=31001
-WSK_HOST=$(kubectl describe node kube-node-1 | grep InternalIP: | awk '{print $2}')
+WSK_HOST=$(kubectl describe node kind-worker | grep InternalIP: | awk '{print $2}')
if [ -z "$WSK_HOST" ]; then
echo "FAILED! Could not determine value for WSK_HOST"
exit 1
@@ -201,14 +204,8 @@
testing:
includeSystemTests: $OW_INCLUDE_SYSTEM_TESTS
-# TODO: instead document how to enable dynamic volume provisioning for dind
-k8s:
- persistence:
- enabled: false
-
invoker:
containerFactory:
- dind: true
impl: $OW_CONTAINER_FACTORY
kubernetes:
agent:
diff --git a/tools/travis/run-tests.sh b/tools/travis/run-tests.sh
index 0254f19..17557fb 100755
--- a/tools/travis/run-tests.sh
+++ b/tools/travis/run-tests.sh
@@ -16,6 +16,8 @@
# limitations under the License.
#
+export KUBECONFIG="$(kind get kubeconfig-path)"
+
###
# Now run the tests provided in the Chart to verify the deployment
###
diff --git a/tools/travis/start-kind.sh b/tools/travis/start-kind.sh
new file mode 100755
index 0000000..3cc6206
--- /dev/null
+++ b/tools/travis/start-kind.sh
@@ -0,0 +1,86 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -x
+
+# Install kubectl
+curl -Lo ./kubectl https://storage.googleapis.com/kubernetes-release/release/v1.16.1/bin/linux/amd64/kubectl
+chmod +x kubectl
+sudo cp kubectl /usr/local/bin/kubectl
+
+# Install kind
+curl -Lo ./kind https://github.com/kubernetes-sigs/kind/releases/download/v0.5.1/kind-linux-amd64
+chmod +x kind
+sudo cp kind /usr/local/bin/kind
+
+# Boot kind
+cat > mycluster.yaml <<EOF
+kind: Cluster
+apiVersion: kind.sigs.k8s.io/v1alpha3
+nodes:
+- role: control-plane
+- role: worker
+- role: worker
+EOF
+
+kind create cluster --config mycluster.yaml --name kind --image kindest/node:${TRAVIS_KUBE_VERSION} --wait 300s
+
+export KUBECONFIG="$(kind get kubeconfig-path)"
+
+echo "Kubernetes cluster is deployed and reachable"
+kubectl describe nodes
+
+# Download and install misc packages and utilities
+pushd /tmp
+ # download and install the wsk cli
+ wget -q https://github.com/apache/openwhisk-cli/releases/download/latest/OpenWhisk_CLI-latest-linux-amd64.tgz
+ tar xzf OpenWhisk_CLI-latest-linux-amd64.tgz
+ sudo cp wsk /usr/local/bin/wsk
+
+ # Download and install helm
+ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh && chmod +x get_helm.sh && ./get_helm.sh
+popd
+
+# Pods running in kube-system namespace should have cluster-admin role
+kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
+
+# Install tiller into the cluster
+/usr/local/bin/helm init --service-account default
+
+# Wait for tiller to be ready
+TIMEOUT=0
+TIMEOUT_COUNT=60
+until [ $TIMEOUT -eq $TIMEOUT_COUNT ]; do
+ TILLER_STATUS=$(kubectl -n kube-system get pods -o wide | grep tiller-deploy | awk '{print $3}')
+ TILLER_READY_COUNT=$(kubectl -n kube-system get pods -o wide | grep tiller-deploy | awk '{print $2}')
+ if [[ "$TILLER_STATUS" == "Running" ]] && [[ "$TILLER_READY_COUNT" == "1/1" ]]; then
+ break
+ fi
+ echo "Waiting for tiller to be ready"
+ kubectl -n kube-system get pods -o wide
+ let TIMEOUT=TIMEOUT+1
+ sleep 5
+done
+
+if [ $TIMEOUT -eq $TIMEOUT_COUNT ]; then
+ echo "Failed to install tiller"
+
+ # Dump diagnostic info to see why tiller failed
+ kubectl -n kube-system describe pods
+ exit 1
+fi
