Google Git
Sign in
apache / streampipes / refs/tags/0.62.0
tage1d18807d1446d0fd4e16226a9e89db90774c303
taggerzehnder <zehnder@fzi.de>Wed May 22 18:28:05 2019 +0000
object6042055742a48cb3b91db7f0f9908e56694fa93c 
[RELEASE] [skip-ci]tagging release 0.62.0
commit6042055742a48cb3b91db7f0f9908e56694fa93c[log] [tgz]
authorzehnder <zehnder@fzi.de>Wed May 22 18:28:03 2019 +0000
committerzehnder <zehnder@fzi.de>Wed May 22 18:28:03 2019 +0000
treec18e2bfef93dee64f310ceacda6488bc506c1ca1
parentd01531a1801272e552a7faa1efe3fc609dbf16b9 [diff]
parenta20daa075721e2b8a19ffc5edffda9c5225305ee [diff]
[RELEASE] [skip-ci]merging 'release-0.62.0' into 'master'
tree: c18e2bfef93dee64f310ceacda6488bc506c1ca1
  1. .github/
  2. .gitlab/
  3. archetypes/
  4. streampipes-app-file-export/
  5. streampipes-backend/
  6. streampipes-code-generation/
  7. streampipes-commons/
  8. streampipes-config/
  9. streampipes-connect/
  10. streampipes-connect-container/
  11. streampipes-container/
  12. streampipes-container-embedded/
  13. streampipes-container-standalone/
  14. streampipes-dataformat/
  15. streampipes-dataformat-json/
  16. streampipes-logging/
  17. streampipes-measurement-units/
  18. streampipes-messaging/
  19. streampipes-messaging-jms/
  20. streampipes-messaging-kafka/
  21. streampipes-model/
  22. streampipes-model-client/
  23. streampipes-performance-tests/
  24. streampipes-pipeline-management/
  25. streampipes-rest/
  26. streampipes-rest-shared/
  27. streampipes-sdk/
  28. streampipes-serializers/
  29. streampipes-sources/
  30. streampipes-storage-api/
  31. streampipes-storage-couchdb/
  32. streampipes-storage-management/
  33. streampipes-storage-rdf4j/
  34. streampipes-test-utils/
  35. streampipes-user-management/
  36. streampipes-vocabulary/
  37. streampipes-wrapper/
  38. streampipes-wrapper-distributed/
  39. streampipes-wrapper-esper/
  40. streampipes-wrapper-flink/
  41. streampipes-wrapper-kafka-streams/
  42. streampipes-wrapper-siddhi/
  43. streampipes-wrapper-spark/
  44. streampipes-wrapper-standalone/
  45. streampipes-wrapper-storm/
  46. tools/
  47. .gitignore
  48. .gitlab-ci.yml
  49. .travis.yml
  50. CHANGELOG.md
  51. install.sh
  52. LICENSE
  53. package-lock.json
  54. pom.xml
  55. README.md
  56. sonar-project.properties
README.md

Travis Badge Codacy Badge Docker pulls Maven central License License

Table of contents

About StreamPipes

StreamPipes enables flexible modeling of stream processing pipelines by providing a graphical modeling editor on top of existing stream processing frameworks.

It leverages non-technical users to quickly define and execute processing pipelines based on an easily extensible toolbox of data sources, data processors and data sinks. StreamPipes has an exchangeable runtime execution layer and executes pipelines using one of the provided wrappers, e.g., for Apache Flink or Apache Kafka Streams.

Pipeline elements in StreamPipes can be installed at runtime - the built-in SDK allows to easily implement new pipeline elements according to your needs. Pipeline elements are standalone microservices that can run anywhere - centrally on your server, in a large-scale cluster or close at the edge.

Learn more about StreamPipes at https://www.streampipes.org/

Read the full documentation at https://docs.streampipes.org

Use Cases

StreamPipes allows you to connect IoT data sources using the SDK or the built-in graphical tool StreamPipes Connect.

The extensible toolbox of data processors and sinks supports use cases such as

  • Continuously store IoT data streams to third party systems (e.g., databases)
  • Filter measurements on streams (e.g., based on thresholds or value ranges)
  • Harmonize data by using data processors for transformations (e.g., by converting measurement units and data types or by aggregating measurements)
  • Detect situations that should be avoided (e.g., patterns based on time windows)
  • Wrap Machine Learning models into data processors to perform classifications or predictions on sensor and image data
  • Visualize real-time data from sensors and machines using the built-in Live Dashboard

Installation

The quickest way to run StreamPipes is the Docker-based installer script available for Unix, Mac and Windows (10).

It's easy to get started:

  1. Make sure you have Docker and Docker Compose installed.
  2. Clone or download the installer script from https://www.github.com/streampipes/streampipes-installer
  3. Execute ./streampipes start to run a lightweight StreamPipes version with few pipelines elements (not including Big Data frameworks) or start the full version (16GB RAM recommended) by executing ./streampipes start bigdata
  4. Open your browser, navigate to http://YOUR_HOSTNAME_HERE and follow the installation instructions.
  5. Once finished, switch to the pipeline editor and start the interactive tour or check the online tour to learn how to create your first pipeline!

For a more in-depth manual, read the installation guide at https://docs.streampipes.org/docs/user-guide-installation!

Pipeline Elements

StreamPipes includes a repository of ready-to-use pipeline elements. A description of the standard elements can be found in the Github repository streampipes-pipeline-elements.

Extending StreamPipes

You can easily add your own data streams, processors or sinks. A Java-based SDK and several run-time wrappers for popular streaming frameworks such as Apache Flink, Apache Spark and Apache Kafka Streams (and also plain Java programs) can be used to integrate your existing processing logic into StreamPipes. Pipeline elements are packaged as Docker images and can be installed at runtime, whenever your requirements change.

Check our developer guide at https://docs.streampipes.org/docs/dev-guide-introduction.

Get help

If you have any problems during the installation or questions around StreamPipes, you'll get help through one of our community channels:

And don't forget to follow us on Twitter!

Contribute

We welcome contributions to StreamPipes. If you are interested in contributing to StreamPipes, let us know!

Feedback

We'd love to hear your feedback! Contact us at feedback@streampipes.org

License

Apache License 2.0

StreamPipes is actively being developed by a dedicated group of people at FZI Research Center for Information Technology.