The StreamPipes command-line interface (CLI) is focused on developers in order to provide an easy entrypoint to set up a suitable dev environment, either planning on developing
Current version: 0.97.0-SNAPSHOT
streampipes env --list [INFO] Available StreamPipes environment templates: pipeline-element ... streampipes env --set pipeline-element streampipes up -d
NOTE: use
./installer/cli/streampipes
if you haven't add it to the PATH and sourced it (see section “Runstreampipes
from anywhere?”).
The CLI is basically a wrapper around multiple docker
and docker-compose
commands plus some additional sugar.
Tested on: (macOS, Linux, Windows)
NOTE: If you‘re using Windows the CLI only works in combination with GitBash - CMD, PowerShell won’t work.
StreamPipes CLI - Manage your StreamPipes environment with ease Usage: streampipes COMMAND [OPTIONS] Options: --help, -h show help --version, -v show version Commands: add Add new StreamPipes service (e.g., adapter, processor, sink) clean Remove StreamPipes data volumes, dangling images and network down Stop and remove StreamPipes containers env Inspect and select StreamPipes environments info Get information logs Get container logs for specific container ps List all StreamPipes container for running environment pull Download latest images from Dockerhub restart Restart StreamPipes container start Start StreamPipes container stop Stop StreamPipes container up Create and start StreamPipes container environment Run 'streampipes COMMAND --help' for more info on a command.
List available environment templates.
streampipes env --list
Inspect services in an available environment to know what kind of services it is composed of.
streampipes env --inspect pipeline-element
Set environment, e.g. pipeline-element
, if you want to write a new pipeline element.
streampipes env --set pipeline-element
Start environment ( default: dev
mode). Here the service definition in the selected environment is used to start the multi-container landscape.
NOTE:
dev
mode is enabled by default since we rely on open ports to core service such ascouchdb
,kafka
etc. to reach from the IDE when developing. If you don't want to map ports (except the UI port), then use the--no-ports
flag.
streampipes up -d # start in production mode with unmapped ports # streampipes up -d --no-ports
Now you're good to go to write your new pipeline element :tada: :tada: :tada:
HINT for extensions: Use our Maven archetypes to setup a project skeleton and use your IDE of choice for development. However, we do recommend using IntelliJ.
HINT for core: To work on
backend
orui
features you need to set the template tobackend
and clone the core repository streampipes - check the prerequisites there for more information.
Stop environment and remove docker container
streampipes down # want to also clean docker data volumes when stopping the environment? # streampipes down -v
Start individual services only? We got you! You chose a template that suits your needs and now you only want to start individual services from it, e.g. only Kafka and CouchDB.
NOTE: the service names need to be present and match your current
.spenv
environment.
streampipes up -d kafka couchdb
Get current environment (if previously set using streampipes env --set <environment>
).
streampipes env
List containers of environment.
streampipes ps # include also stopped container streampieps ps --all
Get logs of specific service and use optional --follow
flag to stay attached to the logs.
streampipes logs --follow backend
Update all services of current environment
streampipes pull
Stop existing StreamPipes containers
streampipes stop extensions-all-jvm
Start existing StreamPipes containers
streampipes start extensions-all-jvm
Restart existing services
# restart backend couchdb container streampipes restart backend couchdb # restart existing services by removing and recreating container instance streampipes restart --force-create extensions-all-jvm
Clean your system and remove created StreamPipes Docker volumes, StreamPipes docker network and dangling StreamPipes images of old image layers.
streampipes clean # remove volumes, network and dangling images # streampipes clean --volumes
As you develop new pipeline elements, e.g., adapters, processors or sinks, you might want to add them to your existing environment. Adding these components requires minimal effort. We can simply use streampipes add
command to populate relevant Docker Compose files under the StreamPipes working directory (deploy/standalone/
).
NOTE: This requires your own pipeline elements to be already containerized and accessible to pull, e.g., from a public Docker repository such as DockerHub. If you use a private repository make sure your local Docker Daemon has access to it.
# check out streampipes add --help for more options # add new pipeline element to catalog, here my-processor streampipes add my-processor # add new pipeline element with custom image and ports streampipes add my-processor --image myrepo/myprocessor:0.68.0 --ports 8090:8090 --ports 8091:8091 # [Optional] add new pipeline element and persistently store service in environment (here: lite) and activate service in current environment (.spenv) streampipes add my-processor --store lite --activate
In general, this creates necessary Docker Compose files in the local service catalog:
cli/deploy/standalone/my-processor ├── docker-compose.dev.yml └── docker-compose.yml
If you activated the pipeline element while adding via the flag --activate
, you can directly start you new pipeline element using the following command. Otherwise, you will have to manually add it at a later point.
NOTE: The service name must match the directory name under
deploy/standalone/<service_name>
streampipes up -d my-processor
As of now, this step has to be done manually. All environments are located in environments/
.
├── adapter # developing a new connect adapter ├── backend # developing core backend features ├── basic # wanna run core, UI, connect etc from the IDE? ├── full # full version containing more pipeline elements ├── lite # few pipeline elements, less memory ├── pipeline-element # developing new pipeline-elements └── ui # developing UI features
Modifying an existing environment template. To modify an existing template, you can simply add a <YOUR_NEW_SERVICE>
to the template.
NOTE: You need to make sure, that the service your are adding exists in
deploy/standalone/service/<YOUR_NEW_SERVICE>
. If your're adding a completely new service take a look at existing ones, create a new service directory and include adocker-compose.yml
anddocker-compose.dev.yml
file.
[environment:backend] activemq kafka ... <YOUR_NEW_SERVICE>
Creating a new environment template. To create a new environment template, place a new file environments/<YOUR_NEW_ENVIRONMENT>
in the template directory. Open the file and use the following schema.
IMPORTANT: Please make sure to have
[environment:<YOUR_NEW_ENVIRONMENT>]
header in the first line of your new template matching the name of the file. Make sure to use small caps letters (lowercase) only.
[environment:<YOUR_NEW_ENVIRONMENT>] <SERVICE_1> <SERVICE_2> ...
streampipes
from anywhere? No problemSimply add the path to this cli directory to your $PATH
(on macOS, Linux) variable, e.g. in your .bashrc
or .zshrc
, or %PATH%
(on Windows).
For macOS, or Linux:
export PATH="/path/to/streampipes-installer/cli:$PATH"
For Windows 10 add installer\cli
to environment variables, e.g. check this documentation.
To upgrade to a new version, simply edit the version tag SP_VERSION
in the .env
file.
If you‘ve found a bug or have a feature that you’d love to see in StreamPipes, feel free to create an issue Find an issue on GitHub.
If you have any problems during the installation or questions around StreamPipes, you'll get help through one of our community channels:
And don't forget to follow us on Twitter!
We welcome contributions to StreamPipes. If you are interested in contributing to StreamPipes, let us know! You'll get to know an open-minded and motivated team working together to build the next IIoT analytics toolbox.
Here are some first steps in case you want to contribute:
Have fun!
We'd love to hear your feedback! Subscribe to users@streampipes.apache.org