tree: 019b77ebc17593cc7453e39b1055fb752e7125de [path history] [tgz]
  1. templates/
  2. Chart.yaml
  3. README.md
  4. values.yaml
installer/k8s/README.md

StreamPipes k8s - The Operator's Dream

StreamPipes k8s is a helm chart to deploy StreamPipes on Kubernetes.

Current version: 0.97.0-SNAPSHOT

We provide two helm chart templates to get you going:

  • default: a lightweight template with few pipeline elements, needs less memory
  • full: contains more pipeline elements, requires >16 GB RAM (recommended)

Prerequisite

Requires Helm (https://helm.sh/) and an active connection to a kubernetes cluster with a running tiller server.

Tested with:

  • K8s v1.19.3
  • Helm v3.1.2
  • Minikube v1.15.1 (recommended for local testing)

NOTE: We experienced some problems with using host path volumes in Docker Desktop environments for persistent storage. Therefore, we suggest to use minikube for local testing.

Local testing

We recommend using minikube for local testing. Follow instructions in their docs to setup test environment

Once installed, start local minikube node with a mapped host volume:

minikube start --mount-string ${HOME}/streampipes-k8s:/streampipes-k8s --mount --memory=4g --cpus=4

Start the a helm chart template by running the following command from the root of this folder:

NOTE: Starting might take a while since we also initially pull all Docker images from Dockerhub.

helm install streampipes ./

# full template only recommend if you have sufficient resources
# helm install streampipes ./ --set deployment=full

After a while, all containers should successfully started, indicated by the Running status.

kubectl get pods
NAME                                           READY   STATUS    RESTARTS   AGE
backend-76ddc486c8-nswpc                       1/1     Running   0          3m27s
couchdb-77db98cf7b-xnnvb                       1/1     Running   0          3m27s
influxdb-b95b6479-r8wh8                        1/1     Running   0          3m27s
extensions-all-jvm-79c445dbd9-m8xcs     1/1     Running   0          3m27s
ui-b94bd9766-rm6zb                             2/2     Running   0          3m27s

For minikube users:

NOTE: If you're running Docker Desktop or Minikube with a local k8s cluster, the above step to use your host IP might not work. Luckily, you can port-forward a service port to your localhost using the following command to be able to access the UI either via http://localhost:8088 or http://<HOST_IP>:8088 (If you want to use privileged ports such as 80, you need to run this command with sudo to bind to the privileged port).

kubectl port-forward svc/ui --address=0.0.0.0 8088:8088

Deleting the current helm chart deployment:

helm del streampipes

We retain the created persistent volume. You need to manually delete it:

rm -rf ${HOME}/streampipes-k8s

Parameters

Common parameters

Parameter NameDescriptionValue
deploymentDeployment type (lite or full)lite
preferredBrokerPreferred broker for deployment“nats”
monitoringSystemEnable monitoring system (true/false)false
pullPolicyImage pull policy“Always”
restartPolicyRestart policy for the containerAlways
persistentVolumeReclaimPolicyReclaim policy for persistent volumes“Delete”
persistentVolumeAccessModesAccess mode for persistent volumes“ReadWriteOnce”
initialDelaySecondsInitial delay for liveness and readiness probes60
periodSecondsInterval between liveness and readiness probes30
failureThresholdNumber of consecutive failures for readiness probes30
hostPathHost path for the application""

StreamPipes common parameters

Parameter NameDescriptionValue
streampipes.versionStreamPipes version“0.97.0-SNAPSHOT”
streampipes.registryStreamPipes registry URL“apachestreampipes”
streampipes.auth.secretNameThe secret name for storing secrets“sp-secrets”
streampipes.auth.users.admin.userThe initial admin useradmin@streampipes.apache.org
streampipes.auth.users.admin.passwordThe initial admin password (leave empty for autogen)“admin”
streampipes.auth.users.service.userThe initial service account user“sp-service-client”
streampipes.auth.users.service.secretThe initial service account secretempty (auto-generated)
streampipes.auth.encryption.passcodePasscode for value encryptionempty (auto-generated)
streampipes.core.appNameStreamPipes backend application name“backend”
streampipes.core.portStreamPipes backend port8030
streampipes.core.persistence.storageClassNameStorage class name for backend PVs“hostpath”
streampipes.core.persistence.storageSizeSize of the backend PV“1Gi”
streampipes.core.persistence.claimNameName of the backend PersistentVolumeClaim“backend-pvc”
streampipes.core.persistence.pvNameName of the backend PersistentVolume“backend-pv”
streampipes.core.service.nameName of the backend service“backend”
streampipes.core.service.portTargetPort of the StreamPipes backend service8030
streampipes.ui.appNameStreamPipes UI application name“ui”
streampipes.ui.resolverActiveFlag for enabling DNS resolver for Nginx proxytrue
streampipes.ui.portStreamPipes UI port8088
streampipes.ui.resolverDNS resolver for Nginx proxy“kube-dns.kube-system.svc.cluster.local”
streampipes.ui.service.nameName of the UI service“ui”
streampipes.ui.service.typeType of the UI service“ClusterIP”
streampipes.ui.service.nodePortNode port for the UI service8088
streampipes.ui.service.portTargetPort of the StreamPipes UI service8088
streampipes.ingress.activeFlag for enabling Ingress for StreamPipesfalse
streampipes.ingress.annotationsAnnotations for Ingress{}
streampipes.ingress.hostHostname for Ingress""
streampipes.ingressroute.activeFlag for enabling IngressRoute for StreamPipestrue
streampipes.ingressroute.annotationsAnnotations for IngressRoute{}
streampipes.ingressroute.entryPointsEntry points for IngressRoute[“web”, “websecure”]
streampipes.ingressroute.hostHostname for IngressRoute""
streampipes.ingressroute.certResolverActiveFlag for enabling certificate resolver for IngressRoutetrue
streampipes.ingressroute.certResolverCertificate resolver for IngressRoute""

Extensions common parameters

Parameter NameDescriptionValue
extensions.iiot.appNameIIoT extensions application nameextensions-all-iiot
extensions.iiot.imageNameIIoT extensions image nameextensions-all-jvm
extensions.iiot.portPort for the IIoT extensions application8090
extensions.iiot.service.nameName of the IIoT extensions serviceextensions-all-iiot
extensions.iiot.service.portTargetPort of the IIoT extensions service8090

External common parameters

Couchdb common parameters

Parameter NameDescriptionValue
external.couchdb.appNameCouchDB application name“couchdb”
external.couchdb.versionCouchDB version3.3.1
external.couchdb.userCouchDB admin username“admin”
external.couchdb.passwordCouchDB admin passwordempty (auto-generated)
external.couchdb.portPort for the CouchDB service5984
external.couchdb.service.nameName of the CouchDB service“couchdb”
external.couchdb.service.portTargetPort of the CouchDB service5984
external.couchdb.persistence.storageClassNameStorage class name for CouchDB PVs“hostpath”
external.couchdb.persistence.storageSizeSize of the CouchDB PV“1Gi”
external.couchdb.persistence.claimNameName of the CouchDB PersistentVolumeClaim“couchdb-pvc”
external.couchdb.persistence.pvNameName of the CouchDB PersistentVolume“couchdb-pv”

Influxdb common parameters

Parameter NameDescriptionValue
external.influxdb.appNameInfluxDB application name“influxdb”
external.influxdb.versionInfluxDB version2.6
external.influxdb.usernameInfluxDB admin username“admin”
external.influxdb.passwordInfluxDB admin passwordempty (auto-generated)
external.influxdb.adminTokenInfluxDB admin tokenempty (auto-generated)
external.influxdb.initOrgInfluxDB initial organization“sp”
external.influxdb.initBucketInfluxDB initial bucket“sp”
external.influxdb.initModeInfluxDB initialization mode“setup”
external.influxdb.apiPortPort number for the InfluxDB service (API)8083
external.influxdb.httpPortPort number for the InfluxDB service (HTTP)8086
external.influxdb.grpcPortPort number for the InfluxDB service (gRPC)8090
external.influxdb.service.nameName of the InfluxDB service“influxdb”
external.influxdb.service.apiPortTargetPort of the InfluxDB service for API8083
external.influxdb.service.httpPortTargetPort of the InfluxDB service for HTTP8086
external.influxdb.service.grpcPortTargetPort of the InfluxDB service for gRPC8090
external.influxdb.persistence.storageClassNameStorage class name for InfluxDB PVs“hostpath”
external.influxdb.persistence.storageSizeSize of the InfluxDB PV“1Gi”
external.influxdb.persistence.storageSizeV1Size of the InfluxDB PV for v1 databases“1Gi”
external.influxdb.persistence.claimNameName of the InfluxDBv2 PersistentVolumeClaim“influxdb2-pvc”
external.influxdb.persistence.claimNameV1Name of the InfluxDBv1 PersistentVolumeClaim“influxdb-pvc”
external.influxdb.persistence.pvNameName of the InfluxDBv2 PersistentVolume“influxdb2-pv”
external.influxdb.persistence.pvNameV1Name of the InfluxDBv1 PersistentVolume“influxdb-pv”

Nats common parameters

Parameter NameDescriptionValue
external.nats.appNameNATS application name“nats”
external.nats.portPort for the NATS service4222
external.nats.versionNATS version
external.nats.service.typeType of the NATS service“NodePort”
external.nats.service.externalTrafficPolicyExternal traffic policy for the NATS service“Local”
external.nats.service.nameName of the NATS service“nats”
external.nats.service.portTargetPort of the NATS service4222

Kafka common parameters

Parameter NameDescriptionValue
external.kafka.appNameKafka application name“kafka”
external.kafka.versionKafka version2.2.0
external.kafka.portPort for the Kafka service9092
external.kafka.external.hostnameName which will be advertised to external clients. Clients which use (default) port 9094“localhost”
external.kafka.service.nameName of the Kafka service“kafka”
external.kafka.service.portTargetPort of the Kafka service9092
external.kafka.service.portOutsidePort for Kafka client outside of the cluster9094
external.kafka.persistence.storageClassNameStorage class name for Kafka PVs“hostpath”
external.kafka.persistence.storageSizeSize of the Kafka PV“1Gi”
external.kafka.persistence.claimNameName of the Kafka PersistentVolumeClaim“kafka-pvc”
external.kafka.persistence.pvNameName of the Kafka PersistentVolume“kafka-pv”

|

Zookeeper common parameters

Parameter NameDescriptionValue
external.zookeeper.appNameZooKeeper application name“zookeeper”
external.zookeeper.versionZooKeeper version3.4.13
external.zookeeper.portPort for the ZooKeeper service2181
external.zookeeper.service.nameName of the ZooKeeper service“zookeeper”
external.zookeeper.service.portTargetPort of the ZooKeeper service2181
external.zookeeper.persistence.storageClassNameStorage class name for ZooKeeper PVs“hostpath”
external.zookeeper.persistence.storageSizeSize of the ZooKeeper PV“1Gi”
external.zookeeper.persistence.claimNameName of the ZooKeeper PersistentVolumeClaim“zookeeper-pvc”
external.zookeeper.persistence.pvNameName of the ZooKeeper PersistentVolume“zookeeper-pv”

Pulsar common parameters

Parameter NameDescriptionValue
external.pulsar.appNamepulsar application name“pulsar”
external.pulsar.versionpulsar version3.0.0
external.pulsar.portPort for the pulsar service6650
external.pulsar.service.nameName of the pulsar service“pulsar”
external.pulsar.service.portTargetPort of the pulsar service6650
external.pulsar.persistence.storageClassNameStorage class name for pulsar PVs“hostpath”
external.pulsar.persistence.storageSizeSize of the pulsar PV“1Gi”
external.pulsar.persistence.claimNameName of the pulsar PersistentVolumeClaim“pulsar-pvc”
external.pulsar.persistence.pvNameName of the pulsar PersistentVolume“pulsar-pv”

Monitoring common parameters

Monitoring - Prometheus

Parameter NameDescriptionValue
prometheus.appNamePrometheus application name“prometheus”
prometheus.versionPrometheus version2.45.0
prometheus.portPrometheus port9090
prometheus.service.namePrometheus service name“prometheus”
prometheus.service.portPrometheus service port9090
prometheus.persistence.storageClassNamePrometheus storage class name“hostpath”
prometheus.persistence.storageSizePrometheus storage size“2Gi”
prometheus.persistence.claimNamePrometheus PVC claim name“prometheus-pvc”
prometheus.persistence.pvNamePrometheus PV name“prometheus-pv”
prometheus.persistence.tokenStorageSizePrometheus token storage size“16Ki”
prometheus.config.scrapeIntervalPrometheus scrape interval10s
prometheus.config.evaluationIntervalPrometheus evaluation interval15s
prometheus.config.backendJobNamePrometheus backend job name“backend”
prometheus.config.extensionsNamePrometheus extensions job name“extensions-all-iiot”
prometheus.config.tokenFileNamePrometheus token file name“token”
prometheus.config.tokenFileDirPrometheus token file directory“/opt/data”

Monitoring - Grafana

Parameter NameDescriptionValue
grafana.appNameGrafana application name“grafana”
grafana.versionGrafana version10.1.2
grafana.portGrafana port3000
grafana.service.nameGrafana service name“grafana”
grafana.service.portGrafana service port3000
grafana.persistence.storageClassNameGrafana storage class name“hostpath”
grafana.persistence.storageSizeGrafana storage size“1Gi”
grafana.persistence.claimNameGrafana PVC claim name“grafana-pvc”
grafana.persistence.pvNameGrafana PV name“grafana-pv”

Bugs and Feature Requests

If you‘ve found a bug or have a feature that you’d love to see in StreamPipes, feel free to create an issue on GitHub.

Get help

If you have any problems during the installation or questions around StreamPipes, you'll get help through one of our community channels:

And don't forget to follow us on Twitter!

Contribute

We welcome contributions to StreamPipes. If you are interested in contributing to StreamPipes, let us know! You'll get to know an open-minded and motivated team working together to build the next IIoT analytics toolbox.

Here are some first steps in case you want to contribute:

  • Subscribe to our dev mailing list dev-subscribe@streampipes.apache.org
  • Send an email, tell us about your interests and which parts of StreamPipes you'd like to contribute (e.g., core or UI)!
  • Ask for a mentor who helps you to understand the code base and guides you through the first setup steps
  • Find an issue on GitHub which is tagged with a good first issue tag
  • Have a look at our developer wiki at https://cwiki.apache.org/confluence/display/STREAMPIPES to learn more about StreamPipes development.

Have fun!

Feedback

We'd love to hear your feedback! Subscribe to users@streampipes.apache.org

License

Apache License 2.0