| <!-- |
| ~ Licensed to the Apache Software Foundation (ASF) under one or more |
| ~ contributor license agreements. See the NOTICE file distributed with |
| ~ this work for additional information regarding copyright ownership. |
| ~ The ASF licenses this file to You under the Apache License, Version 2.0 |
| ~ (the "License"); you may not use this file except in compliance with |
| ~ the License. You may obtain a copy of the License at |
| ~ |
| ~ http://www.apache.org/licenses/LICENSE-2.0 |
| ~ |
| ~ Unless required by applicable law or agreed to in writing, software |
| ~ distributed under the License is distributed on an "AS IS" BASIS, |
| ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| ~ See the License for the specific language governing permissions and |
| ~ limitations under the License. |
| ~ |
| --> |
| |
| # StreamPipes k8s - The Operator's Dream |
| |
| StreamPipes k8s is a helm chart to deploy StreamPipes on Kubernetes. |
| |
| <!-- BEGIN do not edit: set via ../upgrade_versions.sh --> |
| **Current version:** 0.98.0-SNAPSHOT |
| <!-- END do not edit --> |
| |
| We provide two helm chart templates to get you going: |
| |
| - **default**: a lightweight template with few pipeline elements, needs less memory |
| - **full**: contains more pipeline elements, requires **>16 GB RAM** (recommended) |
| |
| ## Prerequisite |
| |
| Requires Helm (https://helm.sh/) and an active connection to a kubernetes cluster with a running tiller server. |
| |
| Tested with: |
| |
| * K8s v1.19.3 |
| * Helm v3.1.2 |
| * Minikube v1.15.1 (recommended for local testing) |
| |
| > **NOTE**: We experienced some problems with using host path volumes in Docker Desktop environments for persistent |
| > storage. Therefore, we suggest to use minikube for local testing. |
| |
| ## Local testing |
| |
| We recommend using [minikube](https://minikube.sigs.k8s.io/docs/) for local testing. Follow instructions in their docs |
| to setup test environment |
| |
| Once installed, start local minikube node with a mapped host volume: |
| |
| ```bash |
| minikube start --mount-string ${HOME}/streampipes-k8s:/streampipes-k8s --mount --memory=4g --cpus=4 |
| ``` |
| |
| **Start** the a helm chart template by running the following command from the root of this folder: |
| > **NOTE**: Starting might take a while since we also initially pull all Docker images from Dockerhub. |
| |
| ```bash |
| helm install streampipes ./ |
| |
| # full template only recommend if you have sufficient resources |
| # helm install streampipes ./ --set deployment=full |
| ``` |
| |
| After a while, all containers should successfully started, indicated by the `Running` status. |
| |
| ```bash |
| kubectl get pods |
| NAME READY STATUS RESTARTS AGE |
| backend-76ddc486c8-nswpc 1/1 Running 0 3m27s |
| couchdb-77db98cf7b-xnnvb 1/1 Running 0 3m27s |
| influxdb-b95b6479-r8wh8 1/1 Running 0 3m27s |
| extensions-all-jvm-79c445dbd9-m8xcs 1/1 Running 0 3m27s |
| ui-b94bd9766-rm6zb 2/2 Running 0 3m27s |
| ``` |
| |
| For **minikube users**: |
| > **NOTE**: If you're running Docker Desktop or Minikube with a local k8s cluster, the above step to use your host IP |
| > might not work. Luckily, you can port-forward a service port to your localhost using the following command to be able to |
| > access the UI either via `http://localhost:8088` or `http://<HOST_IP>:8088` (If you want to use privileged ports such as 80, you need to run this command with sudo to bind to the privileged port). |
| |
| ```bash |
| kubectl port-forward svc/ui --address=0.0.0.0 8088:8088 |
| ``` |
| |
| **Deleting** the current helm chart deployment: |
| |
| ```bash |
| helm del streampipes |
| ``` |
| |
| We retain the created persistent volume. You need to manually delete it: |
| |
| ```bash |
| rm -rf ${HOME}/streampipes-k8s |
| ``` |
| |
| ## Parameters |
| |
| ### Common parameters |
| |
| | Parameter Name | Description | Value | |
| |-------------------------------|-----------------------------------------------------|-----------------| |
| | deployment | Deployment type (lite or full) | lite | |
| | preferredBroker | Preferred broker for deployment | "nats" | |
| | monitoringSystem | Enable monitoring system (true/false) | false | |
| | pullPolicy | Image pull policy | "Always" | |
| | restartPolicy | Restart policy for the container | Always | |
| | persistentVolumeReclaimPolicy | Reclaim policy for persistent volumes | "Delete" | |
| | persistentVolumeAccessModes | Access mode for persistent volumes | "ReadWriteOnce" | |
| | initialDelaySeconds | Initial delay for liveness and readiness probes | 60 | |
| | periodSeconds | Interval between liveness and readiness probes | 30 | |
| | failureThreshold | Number of consecutive failures for readiness probes | 30 | |
| | hostPath | Host path for the application | "" | |
| |
| ### StreamPipes common parameters |
| |
| | Parameter Name | Description | Value | |
| |-----------------------------------------------|---------------------------------------------------------|------------------------------------------| |
| | streampipes.version | StreamPipes version | "0.98.0-SNAPSHOT" | |
| | streampipes.registry | StreamPipes registry URL | "apachestreampipes" | |
| | streampipes.auth.secretName | The secret name for storing secrets | "sp-secrets" | |
| | streampipes.auth.users.admin.user | The initial admin user | "admin@streampipes.apache.org" | |
| | streampipes.auth.users.admin.password | The initial admin password (leave empty for autogen) | "admin" | |
| | streampipes.auth.users.service.user | The initial service account user | "sp-service-client" | |
| | streampipes.auth.users.service.secret | The initial service account secret | empty (auto-generated) | |
| | streampipes.auth.encryption.passcode | Passcode for value encryption | empty (auto-generated) | |
| | streampipes.core.appName | StreamPipes backend application name | "backend" | |
| | streampipes.core.port | StreamPipes backend port | 8030 | |
| | streampipes.core.persistence.storageClassName | Storage class name for backend PVs | "hostpath" | |
| | streampipes.core.persistence.storageSize | Size of the backend PV | "1Gi" | |
| | streampipes.core.persistence.claimName | Name of the backend PersistentVolumeClaim | "backend-pvc" | |
| | streampipes.core.persistence.pvName | Name of the backend PersistentVolume | "backend-pv" | |
| | streampipes.core.service.name | Name of the backend service | "backend" | |
| | streampipes.core.service.port | TargetPort of the StreamPipes backend service | 8030 | |
| | streampipes.ui.appName | StreamPipes UI application name | "ui" | |
| | streampipes.ui.resolverActive | Flag for enabling DNS resolver for Nginx proxy | true | |
| | streampipes.ui.port | StreamPipes UI port | 8088 | |
| | streampipes.ui.resolver | DNS resolver for Nginx proxy | "kube-dns.kube-system.svc.cluster.local" | |
| | streampipes.ui.service.name | Name of the UI service | "ui" | |
| | streampipes.ui.service.type | Type of the UI service | "ClusterIP" | |
| | streampipes.ui.service.nodePort | Node port for the UI service | 8088 | |
| | streampipes.ui.service.port | TargetPort of the StreamPipes UI service | 8088 | |
| | streampipes.ingress.active | Flag for enabling Ingress for StreamPipes | false | |
| | streampipes.ingress.annotations | Annotations for Ingress | {} | |
| | streampipes.ingress.host | Hostname for Ingress | "" | |
| | streampipes.ingressroute.active | Flag for enabling IngressRoute for StreamPipes | true | |
| | streampipes.ingressroute.annotations | Annotations for IngressRoute | {} | |
| | streampipes.ingressroute.entryPoints | Entry points for IngressRoute | ["web", "websecure"] | |
| | streampipes.ingressroute.host | Hostname for IngressRoute | "" | |
| | streampipes.ingressroute.certResolverActive | Flag for enabling certificate resolver for IngressRoute | true | |
| | streampipes.ingressroute.certResolver | Certificate resolver for IngressRoute | "" | |
| |
| ### Extensions common parameters |
| |
| | Parameter Name | Description | Value | |
| |------------------------------|-------------------------------------------|---------------------| |
| | extensions.iiot.appName | IIoT extensions application name | extensions-all-iiot | |
| | extensions.iiot.imageName | IIoT extensions image name | extensions-all-jvm | |
| | extensions.iiot.port | Port for the IIoT extensions application | 8090 | |
| | extensions.iiot.service.name | Name of the IIoT extensions service | extensions-all-iiot | |
| | extensions.iiot.service.port | TargetPort of the IIoT extensions service | 8090 | |
| |
| ### External common parameters |
| |
| #### Couchdb common parameters |
| |
| | Parameter Name | Description | Value | |
| |-----------------------------------------------|-------------------------------------------|------------------------| |
| | external.couchdb.appName | CouchDB application name | "couchdb" | |
| | external.couchdb.version | CouchDB version | 3.3.1 | |
| | external.couchdb.user | CouchDB admin username | "admin" | |
| | external.couchdb.password | CouchDB admin password | empty (auto-generated) | |
| | external.couchdb.port | Port for the CouchDB service | 5984 | |
| | external.couchdb.service.name | Name of the CouchDB service | "couchdb" | |
| | external.couchdb.service.port | TargetPort of the CouchDB service | 5984 | |
| | external.couchdb.persistence.storageClassName | Storage class name for CouchDB PVs | "hostpath" | |
| | external.couchdb.persistence.storageSize | Size of the CouchDB PV | "1Gi" | |
| | external.couchdb.persistence.claimName | Name of the CouchDB PersistentVolumeClaim | "couchdb-pvc" | |
| | external.couchdb.persistence.pvName | Name of the CouchDB PersistentVolume | "couchdb-pv" | |
| |
| |
| #### Influxdb common parameters |
| |
| | Parameter Name | Description | Value | |
| |------------------------------------------------|----------------------------------------------|------------------------| |
| | external.influxdb.appName | InfluxDB application name | "influxdb" | |
| | external.influxdb.version | InfluxDB version | 2.6 | |
| | external.influxdb.username | InfluxDB admin username | "admin" | |
| | external.influxdb.password | InfluxDB admin password | empty (auto-generated) | |
| | external.influxdb.adminToken | InfluxDB admin token | empty (auto-generated) | |
| | external.influxdb.initOrg | InfluxDB initial organization | "sp" | |
| | external.influxdb.initBucket | InfluxDB initial bucket | "sp" | |
| | external.influxdb.initMode | InfluxDB initialization mode | "setup" | |
| | external.influxdb.apiPort | Port number for the InfluxDB service (API) | 8083 | |
| | external.influxdb.httpPort | Port number for the InfluxDB service (HTTP) | 8086 | |
| | external.influxdb.grpcPort | Port number for the InfluxDB service (gRPC) | 8090 | |
| | external.influxdb.service.name | Name of the InfluxDB service | "influxdb" | |
| | external.influxdb.service.apiPort | TargetPort of the InfluxDB service for API | 8083 | |
| | external.influxdb.service.httpPort | TargetPort of the InfluxDB service for HTTP | 8086 | |
| | external.influxdb.service.grpcPort | TargetPort of the InfluxDB service for gRPC | 8090 | |
| | external.influxdb.persistence.storageClassName | Storage class name for InfluxDB PVs | "hostpath" | |
| | external.influxdb.persistence.storageSize | Size of the InfluxDB PV | "1Gi" | |
| | external.influxdb.persistence.storageSizeV1 | Size of the InfluxDB PV for v1 databases | "1Gi" | |
| | external.influxdb.persistence.claimName | Name of the InfluxDBv2 PersistentVolumeClaim | "influxdb2-pvc" | |
| | external.influxdb.persistence.claimNameV1 | Name of the InfluxDBv1 PersistentVolumeClaim | "influxdb-pvc" | |
| | external.influxdb.persistence.pvName | Name of the InfluxDBv2 PersistentVolume | "influxdb2-pv" | |
| | external.influxdb.persistence.pvNameV1 | Name of the InfluxDBv1 PersistentVolume | "influxdb-pv" | |
| |
| #### Nats common parameters |
| |
| | Parameter Name | Description | Value | |
| |---------------------------------------------|----------------------------------------------|------------| |
| | external.nats.appName | NATS application name | "nats" | |
| | external.nats.port | Port for the NATS service | 4222 | |
| | external.nats.version | NATS version | | |
| | external.nats.service.type | Type of the NATS service | "NodePort" | |
| | external.nats.service.externalTrafficPolicy | External traffic policy for the NATS service | "Local" | |
| | external.nats.service.name | Name of the NATS service | "nats" | |
| | external.nats.service.port | TargetPort of the NATS service | 4222 | |
| |
| #### Kafka common parameters |
| |
| | Parameter Name | Description | Value | |
| |---------------------------------------------|------------------------------------------------------------------------------------------|-------------| |
| | external.kafka.appName | Kafka application name | "kafka" | |
| | external.kafka.version | Kafka version | 3.9.0 | |
| | external.kafka.port | Port for the Kafka service | 9092 | |
| | external.kafka.external.hostname | Name which will be advertised to external clients. Clients which use (default) port 9094 | "localhost" | |
| | external.kafka.service.name | Name of the Kafka service | "kafka" | |
| | external.kafka.service.port | TargetPort of the Kafka service | 9092 | |
| | external.kafka.service.portOutside | Port for Kafka client outside of the cluster | 9094 | |
| | external.kafka.persistence.storageClassName | Storage class name for Kafka PVs | "hostpath" | |
| | external.kafka.persistence.storageSize | Size of the Kafka PV | "1Gi" | |
| | external.kafka.persistence.claimName | Name of the Kafka PersistentVolumeClaim | "kafka-pvc" | |
| | external.kafka.persistence.pvName | Name of the Kafka PersistentVolume | "kafka-pv" | |
| | |
| |
| #### Pulsar common parameters |
| |
| | Parameter Name | Description | Value | |
| |----------------------------------------------|------------------------------------------|--------------| |
| | external.pulsar.appName | pulsar application name | "pulsar" | |
| | external.pulsar.version | pulsar version | 3.0.0 | |
| | external.pulsar.port | Port for the pulsar service | 6650 | |
| | external.pulsar.service.name | Name of the pulsar service | "pulsar" | |
| | external.pulsar.service.port | TargetPort of the pulsar service | 6650 | |
| | external.pulsar.persistence.storageClassName | Storage class name for pulsar PVs | "hostpath" | |
| | external.pulsar.persistence.storageSize | Size of the pulsar PV | "1Gi" | |
| | external.pulsar.persistence.claimName | Name of the pulsar PersistentVolumeClaim | "pulsar-pvc" | |
| | external.pulsar.persistence.pvName | Name of the pulsar PersistentVolume | "pulsar-pv" | |
| |
| ### Monitoring common parameters |
| |
| #### Monitoring - Prometheus |
| |
| | Parameter Name | Description | Value | |
| |-----------------------------------------|---------------------------------|-----------------------| |
| | prometheus.appName | Prometheus application name | "prometheus" | |
| | prometheus.version | Prometheus version | 2.45.0 | |
| | prometheus.port | Prometheus port | 9090 | |
| | prometheus.service.name | Prometheus service name | "prometheus" | |
| | prometheus.service.port | Prometheus service port | 9090 | |
| | prometheus.persistence.storageClassName | Prometheus storage class name | "hostpath" | |
| | prometheus.persistence.storageSize | Prometheus storage size | "2Gi" | |
| | prometheus.persistence.claimName | Prometheus PVC claim name | "prometheus-pvc" | |
| | prometheus.persistence.pvName | Prometheus PV name | "prometheus-pv" | |
| | prometheus.persistence.tokenStorageSize | Prometheus token storage size | "16Ki" | |
| | prometheus.config.scrapeInterval | Prometheus scrape interval | 10s | |
| | prometheus.config.evaluationInterval | Prometheus evaluation interval | 15s | |
| | prometheus.config.backendJobName | Prometheus backend job name | "backend" | |
| | prometheus.config.extensionsName | Prometheus extensions job name | "extensions-all-iiot" | |
| | prometheus.config.tokenFileName | Prometheus token file name | "token" | |
| | prometheus.config.tokenFileDir | Prometheus token file directory | "/opt/data" | |
| |
| #### Monitoring - Grafana |
| |
| | Parameter Name | Description | Value | |
| |--------------------------------------|----------------------------|---------------| |
| | grafana.appName | Grafana application name | "grafana" | |
| | grafana.version | Grafana version | 10.1.2 | |
| | grafana.port | Grafana port | 3000 | |
| | grafana.service.name | Grafana service name | "grafana" | |
| | grafana.service.port | Grafana service port | 3000 | |
| | grafana.persistence.storageClassName | Grafana storage class name | "hostpath" | |
| | grafana.persistence.storageSize | Grafana storage size | "1Gi" | |
| | grafana.persistence.claimName | Grafana PVC claim name | "grafana-pvc" | |
| | grafana.persistence.pvName | Grafana PV name | "grafana-pv" | |
| |
| ## Bugs and Feature Requests |
| |
| If you've found a bug or have a feature that you'd love to see in StreamPipes, feel free to create an issue |
| on [GitHub](https://github.com/apache/streampipes/issues). |
| |
| ## Get help |
| |
| If you have any problems during the installation or questions around StreamPipes, you'll get help through one of our |
| community channels: |
| |
| - [Slack](https://slack.streampipes.org) |
| - [Mailing Lists](https://streampipes.apache.org/community/mailing-lists/) |
| |
| And don't forget to follow us on [Twitter](https://twitter.com/streampipes)! |
| |
| ## Contribute |
| |
| We welcome contributions to StreamPipes. If you are interested in contributing to StreamPipes, let us know! You'll |
| get to know an open-minded and motivated team working together to build the next IIoT analytics toolbox. |
| |
| Here are some first steps in case you want to contribute: |
| |
| * Subscribe to our dev mailing list [dev-subscribe@streampipes.apache.org](dev-subscribe@streampipes.apache.org) |
| * Send an email, tell us about your interests and which parts of StreamPipes you'd like to contribute (e.g., core or |
| UI)! |
| * Ask for a mentor who helps you to understand the code base and guides you through the first setup steps |
| * Find an issue on [GitHub](https://github.com/apache/streampipes/issues) which is tagged with a _good first issue_ tag |
| * Have a look at our developer wiki |
| at [https://cwiki.apache.org/confluence/display/STREAMPIPES](https://cwiki.apache.org/confluence/display/STREAMPIPES) |
| to learn more about StreamPipes development. |
| |
| Have fun! |
| |
| ## Feedback |
| |
| We'd love to hear your feedback! Subscribe to [users@streampipes.apache.org](mailto:users@streampipes.apache.org) |
| |
| ## License |
| |
| [Apache License 2.0](../LICENSE) |
