StreamPipes k8s - The Operator's Dream

StreamPipes k8s is a helm chart to deploy StreamPipes on Kubernetes.

Current version: 0.97.0-SNAPSHOT

We provide two helm chart templates to get you going:

• default: a lightweight template with few pipeline elements, needs less memory
• full: contains more pipeline elements, requires >16 GB RAM (recommended)

Prerequisite

Requires Helm (https://helm.sh/) and an active connection to a kubernetes cluster with a running tiller server.

Tested with:

• K8s v1.19.3
• Helm v3.1.2
• Minikube v1.15.1 (recommended for local testing)

NOTE: We experienced some problems with using host path volumes in Docker Desktop environments for persistent storage. Therefore, we suggest to use minikube for local testing.

Local testing

We recommend using minikube for local testing. Follow instructions in their docs to setup test environment

Once installed, start local minikube node with a mapped host volume:

minikube start --mount-string ${HOME}/streampipes-k8s:/streampipes-k8s --mount --memory=4g --cpus=4

Start the a helm chart template by running the following command from the root of this folder:

NOTE: Starting might take a while since we also initially pull all Docker images from Dockerhub.

helm install streampipes ./

# full template only recommend if you have sufficient resources
# helm install streampipes ./ --set deployment=full

After a while, all containers should successfully started, indicated by the Running status.

kubectl get pods
NAME                                           READY   STATUS    RESTARTS   AGE
backend-76ddc486c8-nswpc                       1/1     Running   0          3m27s
couchdb-77db98cf7b-xnnvb                       1/1     Running   0          3m27s
influxdb-b95b6479-r8wh8                        1/1     Running   0          3m27s
extensions-all-jvm-79c445dbd9-m8xcs     1/1     Running   0          3m27s
ui-b94bd9766-rm6zb                             2/2     Running   0          3m27s

For minikube users:

NOTE: If you're running Docker Desktop or Minikube with a local k8s cluster, the above step to use your host IP might not work. Luckily, you can port-forward a service port to your localhost using the following command to be able to access the UI either via http://localhost or http://<HOST_IP> (you require sudo to run this command in order to bind to a privileged port).

kubectl port-forward svc/ui --address=0.0.0.0 8088:8088

Deleting the current helm chart deployment:

helm del streampipes

We retain the created persistent volume. You need to manually delete it:

rm -rf ${HOME}/streampipes-k8s

Parameters

Common parameters

StreamPipes common parameters

Extensions common parameters

External common parameters

Couchdb common parameters

Influxdb common parameters

Nats common parameters

Kafka common parameters

|

Zookeeper common parameters

Pulsar common parameters

Monitoring common parameters

Monitoring - Prometheus

Monitoring - Grafana

Bugs and Feature Requests

If you‘ve found a bug or have a feature that you’d love to see in StreamPipes, feel free to create an issue on GitHub.

Get help

If you have any problems during the installation or questions around StreamPipes, you'll get help through one of our community channels:

• Slack
• Mailing Lists

And don't forget to follow us on Twitter!

Contribute

We welcome contributions to StreamPipes. If you are interested in contributing to StreamPipes, let us know! You'll get to know an open-minded and motivated team working together to build the next IIoT analytics toolbox.

Here are some first steps in case you want to contribute:

• Subscribe to our dev mailing list dev-subscribe@streampipes.apache.org
• Send an email, tell us about your interests and which parts of StreamPipes you'd like to contribute (e.g., core or UI)!
• Ask for a mentor who helps you to understand the code base and guides you through the first setup steps
• Find an issue on GitHub which is tagged with a good first issue tag
• Have a look at our developer wiki at https://cwiki.apache.org/confluence/display/STREAMPIPES to learn more about StreamPipes development.

Have fun!

Feedback

We'd love to hear your feedback! Subscribe to users@streampipes.apache.org

License

Apache License 2.0