StreamPipes CLI - The Developer's Favorite

The StreamPipes command-line interface (CLI) is focused on developers in order to provide an easy entrypoint to set up a suitable dev environment, either planning on developing

• new extensions such as connect adapters, processors, sinks or,
• new core features for backend and ui.

Current version: 0.67.0-SNAPSHOT

TL;DR

$ streampipes template -l
[INFO] Currently available StreamPipes environment templates
pipeline-element
...
$ streampipes template -s pipeline-element
$ streampipes up -d

Prerequisite

The CLI is basically a wrapper around multiple docker and docker-compose commands plus some additional sugar.

• Docker >= 17.06.0
• Docker-Compose >= 1.17.0 (Compose file format: 3.4)
• For Windows Developer: GitBash only

Tested on: macOS, Linux, Windows*)

NOTE: *) If you‘re using Windows the CLI only works in combination with GitBash - CMD, PowerShell won’t work.

CLI commands overview

StreamPipes CLI - Manage your StreamPipes environment with ease

Usage: streampipes COMMAND [OPTIONS]

Options:
  --help, -h      show help
  --version, -v   show version

Commands:
  clean       Remove StreamPipes data volumes, dangling images and network
  down        Stop and remove StreamPipes containers
  info        Get information
  logs        Get container logs for specific container
  ps          List all StreamPipes container for running environment
  pull        Download latest images from Dockerhub
  restart     Restart StreamPipes environment
  template    Select StreamPipes environment template
  up          Create and start StreamPipes container environment

Run 'streampipes COMMAND --help' for more info on a command.

Usage: Along dev life-cycle

List available environment templates

streampipes template --list

Inspect services in a given template to know what kind of services will be started as part this environment

streampipes template --inspect pipeline-element

Set environment template, e.g. pipeline-element if you want to write a new pipeline element

streampipes template --set pipeline-element

Start environment ( default: dev mode).

NOTE: dev mode is enabled by default since we rely on open ports to core service such as consul, couchdb, kafka etc. to reach from the IDE when developing. If you don't want to map ports (except the UI port), then use the --no-ports flag.

streampipes up -d
# start in production mode with unmapped ports
# streampipes up -d --no-ports

Now you're good to go to write your new pipeline element :tada: :tada: :tada:

HINT for extensions: Use our Maven archetypes to setup a project skeleton and use your IDE of choice for development. However, we do recommend using IntelliJ.

HINT for core: To work on backend or ui features you need to set the template to backend and clone the core repository incubator-streampipes - check the prerequisites there for more information.

Stop environment and remove docker container

streampipes down
# want to also clean docker data volumes when stopping the environment?
# streampipes down -v

Additionally, useful commands

Start individual services only? We got you! You chose a template that suits your needs and now you only want to start individual services from it, e.g. only Kafka and Consul.

NOTE: the service names need to be present and match your current .environment.

streampipes up -d kafka consul

Get logs of specific service

streampipes logs --follow backend

Update all services of current environment

streampipes pull

Restart all services of current environment or specific services

streampipes restart
# restart backend & consul
# streampipes restart backend consul

Clean your system and remove created StreamPipes Docker volumes, StreamPipes docker network and dangling StreamPipes images of old image layers.

streampipes clean
# remove volumes, network and dangling images
# streampipes clean --volumes

Modify/Create an environment template

As of now, this step has to be done manually. All environment templates are located in bin/templates/environments.

├── backend
├── full
├── lite
├── pipeline-element
└── ui

Modifying an existing template. To modify an existing template, you can simply add a <YOUR_NEW_SERVICE> to the template.

NOTE: You need to make sure, that the service your are adding exists in deploy/standalone/service/<YOUR_NEW_SERVICE>. If your're adding a completely new service take a look at existing ones, create a new service directory and include a docker-compose.yml and docker-compose.dev.yml file.

[environment:backend]
activemq
kafka
...
<YOUR_NEW_SERVICE>

Creating a new template. To create a new template, place a new file bin/templates/environments/<YOUR_NEW_TEMPLATE> in the template directory. Open the file and use the following schema.

IMPORTANT: Please make sure to have [environment:<YOUR_NEW_TEMPLATE>] header in the first line of your new template matching the name of the file. Make sure to use small caps letters (lowercase) only.

[environment:<YOUR_NEW_TEMPLATE>]
<SERVICE_1>
<SERVICE_2>
...

Run streampipes from anywhere? No problem

Simply add the path to this cli directory to your $PATH (on macOS, Linux) variable, e.g. in your .bashrc or .zshrc, or %PATH% (on Windows).

For macOS, or Linux:

export PATH="/path/to/incubator-streampipes-installer/cli:$PATH"

For Windows 10, e.g. check this documentation.

Upgrade to new version

To upgrade to a new version, simply edit the version tag in VERSION.

Get help

If you have any problems during the installation or questions around StreamPipes, you'll get help through one of our community channels:

• Slack
• Mailing Lists

And don't forget to follow us on Twitter!

License

Apache License 2.0