Google Git
Sign in
apache / streampipes-website / 0f28229a011a3a6b2bef7ae976656ea5e855390d / . / website-v2 / .docusaurus / docusaurus-plugin-content-blog / default / blog-archive-80c.json
blob: bbf1d2678f3379d336a3f043f63e63363bb8e1b4 [file] [log] [blame]
{
"blogPosts": [
{
"id": "/2023/06/16/release-092",
"metadata": {
"permalink": "/blog/2023/06/16/release-092",
"source": "@site/blog/2023-06-16-release-092.md",
"title": "Apache StreamPipes release 0.92.0",
"description": "8 minutes to read",
"date": "2023-06-16T00:00:00.000Z",
"formattedDate": "June 16, 2023",
"tags": [],
"readingTime": 2.255,
"hasTruncateMarker": true,
"authors": [
{
"name": "Philipp Zehnder",
"url": "https://github.com/tenthe",
"imageURL": "/docs/img/zehnder.png"
}
],
"frontMatter": {
"title": "Apache StreamPipes release 0.92.0",
"author": "Philipp Zehnder",
"authorURL": "https://github.com/tenthe",
"authorImageURL": "/docs/img/zehnder.png"
},
"nextItem": {
"title": "Apache StreamPipes release 0.91.0",
"permalink": "/blog/2023/04/12/release-091"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">8 minutes to read</div>**\r\n\r\nWe are pleased to announce the release of StreamPipes 0.92.0, which includes several bug fixes, performance improvements, and new features. \r\nThis version focuses on enhancing the technical aspects of StreamPipes. \r\nLet's explore the key updates in this release:\r\n\r\n<!--truncate-->\r\n\r\nThe current release can be downloaded <a href=\"https://streampipes.apache.org/download.html\">here</a>.\r\n\r\n## Improved Python Support:\r\nIn response to valuable user feedback, we have improved Python support in StreamPipes.\r\nUsers can now seamlessly integrate Python-based scripts, libraries, and modules into their functions. \r\nThis enhancement enables leveraging the power of machine learning and advanced analytics algorithms for IIoT data sources. \r\nPlease refer to the [Python documentation](https://streampipes.apache.org/docs/docs/python/latest/) for further details.\r\n\r\n\r\n## New Processing Elements: Introducing the Swinging Door Trending (SDT) Filter Processor:\r\nTo expand the functionality of StreamPipes, we have introduced several new processing elements.\r\nAmong them is the **Swinging Door Trending (SDT) Filter Processor**.\r\nThis processor provides a linear trend compression algorithm. \r\nEssentially, it replaces a series of continuous points (timestamp, value) with a straight line determined by the start and end points.\r\n\r\n<img src=\"/docs/blog/assets/2023-06-16/swinging_door_trending.gif\" alt=\"demonstration how to use the swinging door trending processor\"/>\r\n\r\n## Bug Fixes and Improvements:\r\nWe have addressed various bugs and issues reported by our users, resulting in a more stable and reliable StreamPipes experience.\r\nWe have also made several performance and usability improvements to enhance overall system efficiency.\r\n\r\n## Deprecation of Set Adapters\r\n\r\nIn order to streamline the StreamPipes ecosystem, we have deprecated the functionality of Set Adapters. \r\nGoing forward, our focus will solely be on streaming event sources. \r\nHowever, we understand the importance of working with files of sensor data, especially during development. \r\nTo accommodate this need, we have extended the **File Stream Adapter**, which now offers the following functionalities:\r\n1. Create a live simulation of sensor data based on the file.\r\n2. Replay the data in the file once, for example, to store it in the **Data Lake**.\r\n\r\n**IMPORTANT: The set adapters will be removed in the next release.**\r\n\r\nFrom this release onwards, we encourage you to utilize the **File Stream Adapter** instead of the **File Set Adapter**.\r\nTo simulate a file set using the **File Stream Adapter**, you can configure it with `Replay Once: Yes` and `Replay Speed: Fastest (Ignore original time)`.\r\nSee the example below:\r\n\r\n<img src=\"/docs/blog/assets/2023-06-16/file_stream_set.gif\" alt=\"demonstration how to use file sets with the file stream adapter\"/>\r\n\r\n## Final Remarks\r\nWe highly recommend updating to StreamPipes 0.92.0 to take advantage of these new features, bug fixes, and improvements. \r\nFor more detailed information, please refer to the [release notes](https://github.com/apache/incubator-streampipes/blob/release/0.92.0/RELEASE_NOTES.md#0920).\r\n\r\nWe appreciate your continued support and valuable feedback. \r\nStreamPipes is continually evolving, and we are dedicated to providing you with a powerful and reliable platform for your IIoT streaming data needs."
},
{
"id": "/2023/04/12/release-091",
"metadata": {
"permalink": "/blog/2023/04/12/release-091",
"source": "@site/blog/2023-04-12-release-091.md",
"title": "Apache StreamPipes release 0.91.0",
"description": "14 minutes to read",
"date": "2023-04-12T00:00:00.000Z",
"formattedDate": "April 12, 2023",
"tags": [],
"readingTime": 4.72,
"hasTruncateMarker": true,
"authors": [
{
"name": "Tim Bossenmaier",
"url": "https://github.com/bossenti",
"imageURL": "/docs/img/bossenmaier.png"
}
],
"frontMatter": {
"title": "Apache StreamPipes release 0.91.0",
"author": "Tim Bossenmaier",
"authorURL": "https://github.com/bossenti",
"authorImageURL": "/docs/img/bossenmaier.png"
},
"prevItem": {
"title": "Apache StreamPipes release 0.92.0",
"permalink": "/blog/2023/06/16/release-092"
},
"nextItem": {
"title": "Apache StreamPipes release 0.90.0",
"permalink": "/blog/2023/01/17/release-090"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">14 minutes to read</div>**\r\n<br/>\r\n\r\nThe Apache StreamPipes community is delighted to announce the release of Apache StreamPipes version 0.91.0!\r\nThe biggest highlight of this release is the birth of our official StreamPipes Python library.\r\nApart from that, we have made a lot of improvements under the hood, especially in terms of improving the development experience (e.g., by introducing and enforcing code style rules).\r\nNevertheless, our new version includes more than 90 improvements and bug fixes in total.\r\n\r\n<!--truncate-->\r\n\r\nThe current release can be downloaded <a href=\"https://streampipes.apache.org/download.html\">here</a>.\r\n\r\n## StreamPipes Python\r\n[Apache StreamPipes](https://streampipes.apache.org/) aims to enable non-technical users to connect and analyze IoT data streams.\r\nTo this end, it provides an easy-to-use and convenient user interface that allows one to connect to an IoT data source and create some visual\r\ngraphs within a few minutes. <br/>\r\nAlthough this is the main use case of Apache StreamPipes, it can also provide great value for people who are eager to work on data analysis or data science with IoT data, but don't want to get in touch with all the hassle associated with extracting data from devices in a suitable format.\r\nIn this scenario, StreamPipes helps you connect to your data source and extract the data for you.\r\nYou then can make the data available outside StreamPipes by writing it into an external source, such as a database, Kafka, etc.\r\nWhile this requires another component, you can also extract your data directly from StreamPipes programmatically using the StreamPipes API.\r\nFor convenience, we also provide you with a StreamPipes client both available for Java and Python.\r\nSpecifically with StreamPipes Python, we want to address the amazing data analytics and data science community in Python and benefit from the great universe of Python libraries out there.\r\n\r\n### Getting started\r\nBesides our official version, which you can access [here](https://streampipes.apache.org/download.html),\r\nwe provide Apache StreamPipes as a convenient release via PyPI.\r\nTherefore, you can easily install it via the following command\r\n```bash\r\npip install streampipes\r\n```\r\nAll documentation related to StreamPipes Python can be found [here](https://streampipes.apache.org/docs/docs/python/latest/).\r\n\r\nBut now enough words are written, let's just start with a few examples.\r\nAs a first step, we want to establish a connection to a running StreamPipes instance.\r\n```python\r\nfrom streampipes.client import StreamPipesClient\r\nfrom streampipes.client.config import StreamPipesClientConfig\r\nfrom streampipes.client.credential_provider import StreamPipesApiKeyCredentials\r\n\r\nconfig = StreamPipesClientConfig(\r\n credential_provider = StreamPipesApiKeyCredentials(\r\n username = \"test@streampipes.apache.org\",\r\n api_key = \"DEMO-KEY\",\r\n ),\r\n host_address = \"localhost\",\r\n https_disabled = True,\r\n port = 80\r\n)\r\n\r\nclient = StreamPipesClient(client_config=config)\r\n```\r\n\r\n`client.describe()` gives you an overview about what can be discovered via the Python client:\r\n```bash\r\nHi there!\r\nYou are connected to a StreamPipes instance running at http://localhost:80.\r\nThe following StreamPipes resources are available with this client:\r\n1x DataLakeMeasures\r\n1x DataStreams\r\n```\r\n\r\nThe required API key can be retrieved from StreamPipes via the UI:\r\n\r\n<img src=\"/docs/blog/assets/2023-04-12/how-to-get-api-key.gif\" alt=\"demonstration how to get an API key in form of a gif\"/>\r\n\r\n### Query Data from StreamPipes data lake\r\nAs a first step, we can see which data sets are available in the data lake:\r\n```python\r\nclient.dataLakeMeasureApi.all().to_pandas()\r\n```\r\n\r\nTo query data for a specific data lake measure, the following command takes you there:\r\n```python\r\nclient.dataLakeMeasureApi.get(identifier=\"flow-rate\").to_pandas()\r\n```\r\nThere are some parameters that allow you to query data more specifically, visit the [docs](https://streampipes.apache.org/docs/docs/python/latest/reference/endpoint/api/data_lake_measure/#streampipes.endpoint.api.data_lake_measure.MeasurementGetQueryConfig) for more detailed information.\r\n\r\nIf you are curious which features are available for StreamPipes Python, have a look at our [tutorial section](https://streampipes.apache.org/docs/docs/python/latest/tutorials/1-introduction-to-streampipes-python-client/).\r\n\r\nWe plan to further extend the Python library in the next releases - we welcome your ideas on any features you'd like to see!\r\n\r\n## Create Adapter without starting\r\n\r\nUntil now, when creating an adapter in StreamPipes, the adapter was started immediately after completion.\r\nFrom now on you can choose via the adapter overview in the last dialog whether the adapter should be started immediately or later (this is then possible in the adapter overview).\r\n\r\n<img src=\"/docs/blog/assets/2023-04-12/adapter-running.gif\" alt=\"demonstration how to disable automatic start of an adapter in form of a gif\"/>\r\n\r\n## Redirect after Login\r\n\r\nThis is another small user interface improvement: when a user opens a StreamPipes link even though they are not logged in, e.g.,\r\nhttp://streampipes.demo/#/connect, StreamPipes displays the login dialog and now redirects the user directly to the target view\r\n(in this case the connection overview).\r\n\r\n## Support Output Streams for Standalone Functions\r\nWithin the last release, [0.90.0](/docs/blog/2023/01/17/release-090.html), we introduced StreamPipes Functions as a lightweight alternative to processing elements. \r\nStreamPipes Functions do now support having output streams, which means that they allow not only to consume but to send data to StreamPipes as well.\r\n\r\nBelow you can see a simple example function that demonstrates how events can be pushed back to StreamPipes again.\r\n```java\r\npublic class FunctionPublishExample extends StreamPipesFunction {\r\n\r\n private static final Logger LOG = LoggerFactory.getLogger(FunctionPublishExample.class);\r\n\r\n private static final String FUNCTION_ID = \"org.apache.streampipes.example.function.publish\";\r\n private static final String STREAM_APP_ID = \"example-output-stream-1\";\r\n\r\n private SpOutputCollector outputCollector;\r\n\r\n @Override\r\n public List<String> requiredStreamIds() {\r\n return List.of(\"urn:streampipes.apache.org:eventstream:EtMUkN\");\r\n }\r\n\r\n @Override\r\n public void onServiceStarted(FunctionContext context) {\r\n LOG.info(\"Service started\");\r\n this.outputCollector = context.getOutputCollectors().get(STREAM_APP_ID);\r\n }\r\n\r\n @Override\r\n public void onEvent(Event event, String streamId) {\r\n LOG.info(\"on event\");\r\n var ev = new Event();\r\n ev.addField(\"timestamp\", System.currentTimeMillis());\r\n ev.addField(\"example-property\", \"abc\");\r\n this.outputCollector.collect(ev);\r\n }\r\n\r\n @Override\r\n public void onServiceStopped() {\r\n LOG.info(\"service stopped\");\r\n }\r\n\r\n @Override\r\n public FunctionConfig getFunctionConfig() {\r\n\r\n return FunctionConfigBuilder\r\n .create(FunctionId.from(FUNCTION_ID, 1))\r\n .withOutputStream(DataStreamBuilder.create(STREAM_APP_ID, \"My Function Stream\", \"\")\r\n .property(EpProperties.timestampProperty(\"timestamp\"))\r\n .property(EpProperties.stringEp(\r\n Labels.from(\"my-example-property\", \"test\", \"test\"),\r\n \"example-property\",\r\n SO.TEXT,\r\n PropertyScope.MEASUREMENT_PROPERTY))\r\n .format(Formats.jsonFormat())\r\n .protocol(Protocols.kafka(\"localhost\", 9094, STREAM_APP_ID))\r\n .build())\r\n .build();\r\n }\r\n}\r\n```\r\n\r\nThis functionality is also already supported in StreamPipes Python! 🚀\r\n\r\nThis means that you can write a StreamPipes function in Python that allows you to consume streams of data from StreamPipes,\r\nperform any data manipulation in Python, and send the data back to StreamPipes, where it can be further processed.\r\n\r\n\r\n<br/>\r\nTo get a comprehensive overview of all changes released within Apache StreamPipes 0.91.0\r\nplease visit our [release notes](https://github.com/apache/incubator-streampipes/blob/release/0.91.0/RELEASE_NOTES.md#0910)."
},
{
"id": "/2023/01/17/release-090",
"metadata": {
"permalink": "/blog/2023/01/17/release-090",
"source": "@site/blog/2023-01-17-release-090.md",
"title": "Apache StreamPipes release 0.90.0",
"description": "5 minutes to read",
"date": "2023-01-17T00:00:00.000Z",
"formattedDate": "January 17, 2023",
"tags": [],
"readingTime": 3.66,
"hasTruncateMarker": true,
"authors": [
{
"name": "Dominik Riemer",
"url": "https://github.com/dominikriemer",
"imageURL": "/docs/img/riemer.png"
}
],
"frontMatter": {
"title": "Apache StreamPipes release 0.90.0",
"author": "Dominik Riemer",
"authorURL": "https://github.com/dominikriemer",
"authorImageURL": "/docs/img/riemer.png"
},
"prevItem": {
"title": "Apache StreamPipes release 0.91.0",
"permalink": "/blog/2023/04/12/release-091"
},
"nextItem": {
"title": "Apache StreamPipes release 0.70.0",
"permalink": "/blog/2022/10/13/release-070"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">5 minutes to read</div>**\r\n<br/>\r\n\r\nThe Apache StreamPipes community is pleased to announce the immediate availability of Apache StreamPipes version 0.90.0!\r\nIn total, this release was focused on stabilization and closes 57 issues with improvements and bug fixes as well as some new features such as adapters and data sinks. \r\n\r\n<!--truncate-->\r\n\r\nThe current release can be downloaded <a href=\"https://streampipes.apache.org/download.html\">here</a>.\r\n\r\n\r\n## Improvements\r\nThe focus of this release cycle was on general usability improvements and bug fixes. We continued to harmonize the \r\nStreamPipes web interface, so that all views follow a similar navigation. Many improvements were added to the data explorer:\r\n\r\n* Incomplete events with missing values are now displayed in the data explorer\r\n* Improvements to the time selector and edit mode\r\n* More configuration options for data download\r\n* Improved overall data explorer performance\r\n\r\nIn addition, we improved features to monitor pipelines and extensions services. The pipeline monitoring component has been restructured and now works independent from the underlying message broker. \r\nIn addition, we extended the SDK so that runtime errors can be forwarded to the StreamPipes UI. \r\nFurther improvements to the monitoring stack will be added in the next versions.\r\n\r\n## New Adapters and Data Sinks\r\n\r\nThis version improves support for existing message brokers in form of adapters and sinks:\r\n\r\n* Improvements of the `Apache Pulsar` (which is also available as a data sink)\r\n* A new adapter and data sink for `Apache RocketMQ`\r\n* A new adapter for `NATS` (also available as a data sink)\r\n\r\n## GitHub Community Features\r\n\r\nApache StreamPipes is a community effort - everyone is welcome not only to contribute code, but also to provide feedback and other input which helps improving the product.\r\nTo improve community interaction, we switched our issue management from Jira to GitHub issues - making it much easier to create issues and to discuss future features.\r\nIn addition, we've enabled Github Discussions - feel free to participate in the existing discussions and ask questions if you have problems!\r\n\r\n* [GitHub Issues](https://github.com/apache/streampipes/issues)\r\n* [GitHub Discussions](https://github.com/apache/streampipes/discussions)\r\n\r\nAnd if you prefer the mailing list, subscribe to [dev@streampipes.apache.org](mailto:dev-subscribe@streampipes.apache.org) or [users@streampipes.apache.org](mailto:users-subscribe@streampipes.apache.org).\r\nTech-related and general project discussions as well as release votes will continue to happen on the dev list.\r\n\r\n<img src=\"/docs/blog/assets/2023-01-17/github-discussions.png\" alt=\"Github Discussions overview\"/>\r\n\r\n## Preview of Standalone Functions\r\nAn exciting new feature is available in preview mode: StreamPipes Functions are a new way to interact with connected data streams in StreamPipes.\r\nAs an open system, there are already several ways to extend the functionality of StreamPipes: New adapter can be added to connect to non-supported systems. \r\nIn addition, the SDK supports building custom data processors and sinks, so that StreamPipes can be easily extended at runtime in case new analytics operators are needed.\r\n\r\nBut providing extensions in form of pipeline elements are not always the best choice: E.g., in cases where non-generic algorithms should be executed on a specific data stream, \r\nproviding a reusable component for pipeline development is unsuitable.\r\n\r\nThis is where Functions come into play! StreamPipes Functions allow to define standalone components, which consume one or more connected data streams from StreamPipes and can be used to add custom processing logic.\r\nE.g., it becomes easy to apply an ML model on continuous data without the need to build a specific data processing pipeline.\r\n\r\nFunctions can be registered in an extensions service similar to data processors and sinks by extending a single class:\r\n\r\n```java\r\npublic class MyFunction extends StreamPipesFunction {\r\n @Override\r\n public FunctionId getFunctionId() {\r\n return FunctionId.from(\"my-app-id\", 1);\r\n }\r\n\r\n @Override\r\n public List<String> requiredStreamIds() {\r\n return List.of(\"my-stream-id\");\r\n }\r\n\r\n @Override\r\n public void onServiceStarted(FunctionContext context) {\r\n // called on service startup\r\n }\r\n\r\n @Override\r\n public void onEvent(Event event, String streamId) {\r\n // called for each incoming event\r\n }\r\n\r\n @Override\r\n public void onServiceStopped() {\r\n // called on service shutdown\r\n }\r\n}\r\n```\r\n\r\nThis concept, which will also come for more programming languages besides Java offer new ways for developers to interact with available data in StreamPipes and build custom extensions for IIoT analytics.\r\n\r\nFunctions are available in preview mode and their API should therefore be considered unstable.\r\n\r\n\r\n## Future Roadmap\r\n\r\nAfter this stabilization release, the next version will most probably bring a major new feature - Python support! \r\nThe community is currently working hard to bring the StreamPipes client and functions to the Python world. Stay tuned!\r\n\r\n<br/>\r\nTo get a comprehensive overview of all changes released within Apache StreamPipes 0.90.0\r\nplease visit our [release notes](https://github.com/apache/incubator-streampipes/blob/release/0.90.0/RELEASE_NOTES.md#0900)."
},
{
"id": "/2022/10/13/release-070",
"metadata": {
"permalink": "/blog/2022/10/13/release-070",
"source": "@site/blog/2022-10-13-release-070.md",
"title": "Apache StreamPipes release 0.70.0",
"description": "6 minutes to read",
"date": "2022-10-13T00:00:00.000Z",
"formattedDate": "October 13, 2022",
"tags": [],
"readingTime": 4.375,
"hasTruncateMarker": true,
"authors": [
{
"name": "Tim Bossenmaier",
"url": "https://github.com/bossenti",
"imageURL": "/docs/img/bossenmaier.png"
}
],
"frontMatter": {
"title": "Apache StreamPipes release 0.70.0",
"author": "Tim Bossenmaier",
"authorURL": "https://github.com/bossenti",
"authorImageURL": "/docs/img/bossenmaier.png"
},
"prevItem": {
"title": "Apache StreamPipes release 0.90.0",
"permalink": "/blog/2023/01/17/release-090"
},
"nextItem": {
"title": "Using Factory I/O with StreamPipes",
"permalink": "/blog/2022/09/05/using-factory.io-with-streampipes"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">6 minutes to read</div>**\r\n<br/>\r\n\r\nThe Apache StreamPipes (incubating) community is delighted to announce the release of Apache StreamPipes version 0.70.0!\r\nKey highlights of this release include many enhancements to the data explorer for fast exploration of IoT data and\r\na brand new asset management that allows importing and exporting of StreamPipes resources.\r\nIn addition, our new version includes more than 50 improvements and bug fixes.\r\n\r\n<!--truncate-->\r\n\r\nThe current release can be downloaded <a href=\"https://streampipes.apache.org/download.html\">here</a>.\r\n\r\n## Preview of Event Schema\r\nA very important and inevitable step when connecting to a data source with Apache StreamPipes \r\nis the definition of the event schema. By collecting some sample events, Apache StreamPipes is able to suggest \r\nan event schema that describes the data structure of the incoming events.\r\nAlthough this works quite well in the vast majority of cases, there is usually a need to modify the proposed\r\nevent schema to some degree or add additional fields. So far, a user has gotten little idea of what \r\nthe revised data source event schema looks like until the adapter has been created and some data was consumed.\r\nWith this release, we have included a before/after comparison of the events while defining the event schema.\r\nThis allows to compare the proposed events of Apache StreamPipes (`before`) with the events after \r\nall the user's changes have been applied (`after`).\r\nThe after view adapts to each interaction of the user.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-10-13/event-schema-preview.png\"/>\r\n\r\n## Export and Import StreamPipes Resource\r\nWith this release we introduce a completely new module called `Asset Management`.\r\nThis can be selected in the menu bar of the modules on the left side as the third last icon and \r\nis marked by a construction icon. This module allows to combine several StreamPipes resources to a so called `Asset`,\r\nwhich can be exported and re-imported later. To do this, a new asset must first be created.\r\nAfterwards the asset can be edited and the following detail view allows to link all resources \r\navailable in StreamPipes to the asset. After the asset has been finally saved, the asset (inlcuding all resources) can be downloaded \r\nwithin the configuration module by switching to the `Export/Import` tab. All existing assets are listed there and \r\ncan be selected for download.\r\nThe downloaded file can then be re-imported later in the same or another StreamPipes instance at the same place.\r\n\r\n<img src=\"/docs/blog/assets/2022-10-13/asset-manager.gif\" alt=\"demonstration how to create an asset and download it afterwards in form of a gif\"/>\r\n\r\n## Download Data of Configured Query from Data Lake\r\nAnother new feature is the ability to download or export data from the StreamPipes data lake\r\ndefined by a user-defined query. Previously, it was only possible to download all events belonging to a data lake index\r\nfrom the `Data Lake` tab of the configuration menu. In addition to this, each widget in the `Data Explorer`\r\noffers the possibility to download the corresponding data. The underlying query is created from the configuration \r\nof the widget and is also used to create the widget's figure.\r\nThe download supports both `CSV` and `JSON` format.\r\n\r\n<img src=\"/docs/blog/assets/2022-10-13/custom-query-download.gif\" alt=\"demonstration how to download the data of a custom data lake query in form of a gif\"/>\r\n\r\n## Slide View in Data Explorer\r\nTraditionally, the graphs created in the Data Explorer are displayed in a grid view, where the user can adjust\r\nthe arrangement of the figures as well as their height and width.\r\nWith this release, another option for displaying the dashboard widgets is available: the slide view.\r\nThis view allows to display a single graph in full-screen mode and switch between views using a horizontal menu bar.\r\nWhen a new `Data View` is created in the Data Explorer, a choice can be made between `grid view` and \r\n`slide view` as the default display mode.\r\nOnce a `Data View` is created, the display mode can be changed by clicking the icon in the upper left corner \r\nright next to the `Back` button.\r\n\r\n<img src=\"/docs/blog/assets/2022-10-13/slide-view.gif\" alt=\"demonstration how to use the slide view in the data explorer in form of a gif\"/>\r\n\r\n## Templates for Adapter Configuration\r\nSimilar to the processing elements, it is now possible to create templates for adapter configurations.\r\nThis allows you to start from a previously defined set of configuration settings when creating similar adapters.\r\nTo create a template for an adapter configuration, start by creating a new adapter and\r\nconfiguring all the settings that will be part of the template. Before proceeding to the field\r\nconfiguration (the second step of adapter creation), save the current configuration as a template.\r\nWhenever a new instance of the same adapter is created, the template can be selected from\r\nthe drop-down menu that appears. Templates are only available for the same type of adapter (obviously).\r\n\r\n<img src=\"/docs/blog/assets/2022-10-13/adapter-template.gif\" alt=\"demonstration how to create an adapter template in form of a gif\"/>\r\n\r\n## Upload Multiple Files at Once\r\nA rather small but very handy improvement is the possibility to upload multiple files at once to StreamPipes.\r\nWhen you click the `UPLOAD NEW FILE` button in the `File Management` module, it is now possible to select\r\nas many files as you want in the file upload dialog.\r\n\r\n<br/>\r\n\r\nTo get a comprehensive overview of all changes released within Apache StreamPipes 0.70.0\r\nplease visit our [release notes](https://github.com/apache/incubator-streampipes/blob/release/0.70.0/RELEASE_NOTES.md#0700)."
},
{
"id": "/2022/09/05/using-factory.io-with-streampipes",
"metadata": {
"permalink": "/blog/2022/09/05/using-factory.io-with-streampipes",
"source": "@site/blog/2022-09-05-using-factory.io-with-streampipes.md",
"title": "Using Factory I/O with StreamPipes",
"description": "10 minutes to read",
"date": "2022-09-05T00:00:00.000Z",
"formattedDate": "September 5, 2022",
"tags": [],
"readingTime": 5.43,
"hasTruncateMarker": true,
"authors": [
{
"name": "Sven Oehler"
}
],
"frontMatter": {
"title": "Using Factory I/O with StreamPipes",
"author": "Sven Oehler"
},
"prevItem": {
"title": "Apache StreamPipes release 0.70.0",
"permalink": "/blog/2022/10/13/release-070"
},
"nextItem": {
"title": "Apache StreamPipes release 0.69.0",
"permalink": "/blog/2022/03/21/_release-0690"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">10 minutes to read</div>**<br/>\r\n\r\nThis tutorial shows how you can stream sensor data into StreamPipes from a simulation environment (Factory I/O). The simulation environment is controlled by a \"Siemens S7-1200\" PLC and shows a conveyor belt which sorts packages by their height. The tutorial is based on the upcoming version 0.70.0 and teaches you how to load the sensor data, build a pipeline, preprocess the data and create visualizations.\r\n\r\n<img class=\"blog-image\" style=\"max-width:75%;\" src=\"/docs/blog/assets/2022-09-05/plc.jpg\" alt=\"plc\"><br/>\r\n\r\n<!--truncate-->\r\n\r\n## 1. Factory I/O\r\n<a href=\"https://docs.factoryio.com/\">*Factory* I/O</a> is a 3D simulation for virtual factories and is commonly used as a PLC training platform. We will need it to simulate the live data.\r\n- Set up Factory I/O and make sure that the <a href=\"https://docs.factoryio.com/tutorials/siemens/sample-s7-1200-1500/\">*Sorting by Height (Basic)*</a> simulation works.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-09-05/factory.io.png\" alt=\"dataView\"><br/>\r\n\r\n## 2. Adapter\r\nThe next step is to connect the PLC to StreamPipes. To achieve this we need to create an adapter.\r\n- Start <a href=\"https://streampipes.apache.org/docs/docs/user-guide-introduction.html\">*StreamPipes*</a>, go to the **CONNECT** menu and create a **NEW ADAPTER**. \r\n- Select the **PLC4X S7** adapter, insert the **IP-address** of your PLC and **import the file** with the PLC tags (see below). Then click **NEXT** on the bottom right.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-09-05/adapter1.png\" alt=\"adapter1\"/>\r\n\r\n- If the warning \"MISSING TIMESTAMP\" appears, press **ADD TIMESTAMP** and then **NEXT**. This warning occurs because some data sources don't provide a timestamp.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-09-05/adapter2.png\" alt=\"adapter2\"/>\r\n\r\n- You can name the adapter \"S7_sorting\" and select the option **PERSIST EVENTS** to make sure the data is persisted. The adapter is now ready to **START**. (You can check if the adapter is working by pressing the **`?`** symbol in the connect menu and clicking on **VALUES** to see a live preview).\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-09-05/adapter3.png\" alt=\"adapter3\"><br/>\r\n\r\n## 3. Dashboard for raw data\r\nA dashboard is used to monitor live data from a data stream. We will use it to make sure that the sensor values are updated when the simulation runs.\r\n- Go to the **DASHBOARD** menu and create a **NEW DASHBOARD** \"Dashboard_sorting\". Press the pen symbol to edit the dashboard. \r\n- Click **ADD VISUALIZATION** and select the automatically created pipeline \"Persist S7_sorting\". Choose the **TABLE** widget, configure it by pressing **SELECT ALL** and **CREATE** the widget.\r\n- Change the size of the table as you like and **SAVE** the dashboard.\r\n- The values should now update every 5 seconds. Test it by starting the simulation in Factory I/O.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-09-05/dashboard1.png\" alt=\"dashboard1\"><br/>\r\n\r\n## 4. Creating a Data View\r\nThe Data View is used to create visualization for persisted data in a Data Lake. You can choose various visualization to explore your data. We will create a table to see the stored data and a bar chart to take a closer look at a single sensor.\r\n- Create a **NEW DATA VIEW** \"Sorting\" in the **DATA EXPLORER** menu and edit the view.\r\n- Select the \"Persist S7_sorting\" pipeline at the **NEW WIDGET** menu, **SELECT ALL** fields and press **NEXT**.\r\n- Select the **TABLE** visualization. Click **SELECT ALL** and **CREATE**.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-09-05/dataView1.png\" alt=\"dataView1\"/>\r\n\r\n- Now we want to look closer at a single sensor value. Therefore click on the **`+`** symbol on the top left to add a new widget.\r\n- Choose the \"Persist S7_sorting\" pipeline again, set the **LIMIT** to 1000 and only select the **I_At_right_entry** field. Now select **DISTRIBUTION** as visualization and press **CREATE**. This chart shows how often the sensor returned true or false (package in front of it or not).\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-09-05/dataView2.png\" alt=\"dataView2\"/>\r\n\r\n## 5. Modifying the Pipeline\r\nThe next step is to change the pipeline to add some preproccessing steps. We want to count the number of large packages which get transported to the right side and we want to measure the time between two arriving packages.\r\n- Therefore **STOP** the running \"Persist S7_sorting\" pipeline in the **PIPELINE** menu and **MODIFY** the pipeline.\r\n- Add the **BOOLEAN COUNTER** as new pipeline element. Connect the data source to the counter and configure the element. Select **I_At_right_entry** as boolean field and change the flank parameter to **FALSE -> TRUE**. Then add a **DATA LAKE** and connect the **BOOLEAN COUNTER** to it. Use \"Boolean_Counter\" as identifier.\r\n- Add the **BOOLEAN TIMER** to the pipeline and connect the data source to it. Select **I_At_right_entry** as boolean field too and select **TRUE** and **SECONDS** in the configuration window. Add another **DATA LAKE** and connect the **BOOLEAN TIMER** to it. Use \"Boolean_Timer\" as identifier.\r\n- **SAVE** the pipeline and click the checkbox **START PIPELINE IMMEDIATELY**.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-09-05/pipeline.png\" alt=\"pipeline\"/>\r\n\r\n## 6. Dashboard for preprocessed data\r\nAfter we created two new data field in the last step, we will monitor them in the dashboard.\r\n- Go to the **DASHBOARD** menu and **EDIT** the dashboard.\r\n- Click **ADD VISUALIZATION** and select \"Persist S7_sorting: Boolean_Counter\". Create a **TABLE** widget as before and **SELECT ALL** fields.\r\n- Now add one last table. Choose \"Persist S7_sorting: Boolean_Timer\" and only select the **measured_time** field\r\n- Test your pipeline by starting the simulation again. The counter should now count up when a package passes the sensor on the right side and the \"measured_time\" should show the time since the last package arrived.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-09-05/dashboard2.png\" alt=\"dashboard2\"><br/>\r\n\r\n## 7. Final Data View\r\nFinally we want to improve the Data Explorer and create visualizations with the preprocessed data. We will create a time series to verify when a new package arrived and a heat map to check the time between two packages.\r\n- Go to the **DATA EXPLORER** menu and edit the data view.\r\n- Click **ADD WIDGET** and choose the pipeline \"Persist S7_sorting: Boolean_Counter\". Select the **counter** field and choose the **TIME SERIES** visualization.\r\n- Create another widget. Choose the \"Persist S7_sorting: BooleanCounter\" pipeline and select the **measured_time** field. Then select the **HEATMAP** and use \"measured_time\" as **HEAT VALUE**\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-09-05/dataView3.png\" alt=\"dataView3\"><br/>\r\n\r\n## Summary\r\nIn this tutorial we learned how to connect a PLC to StreamPipes, monitor the data stream with a dashboard, create a pipeline with different preprocessing steps and visualize the data in the Data View. But we just focused on the basics, so feel free to try out your own ideas and use different preprocessing steps, create new visualizations or try out another simulation environment.\r\n\r\n## CSV file for PLC tags\r\n```\r\nRuntime Name,Path,Data Type,Node Name,Comment,Hmi Visible,Hmi Accessible,Hmi Writeable,Typeobject ID,Version ID\r\nI_High_sensor,Tag table_1,Bool,%I0.0,,True,True,True,,\r\nI_Low_sensor,Tag table_1,Bool,%I0.1,,True,True,True,,\r\nI_Pallet_sensor,Tag table_1,Bool,%I0.2,,True,True,True,,\r\nI_Loaded,Tag table_1,Bool,%I0.3,,True,True,True,,\r\nI_At_left_entry,Tag table_1,Bool,%I0.4,,True,True,True,,\r\nI_At_left_exit,Tag table_1,Bool,%I0.5,,True,True,True,,\r\nI_At_right_entry,Tag table_1,Bool,%I0.6,,True,True,True,,\r\nI_At_right_exit,Tag table_1,Bool,%I0.7,,True,True,True,,\r\nO_Conveyor_entry,Tag table_1,Bool,%Q0.0,,True,True,True,,\r\nO_Load,Tag table_1,Bool,%Q0.1,,True,True,True,,\r\nO_Unload,Tag table_1,Bool,%Q0.2,,True,True,True,,\r\nO_Transf_left,Tag table_1,Bool,%Q0.3,,True,True,True,,\r\nO_Tranf_right,Tag table_1,Bool,%Q0.4,,True,True,True,,\r\nO_Conveyor_left,Tag table_1,Bool,%Q0.5,,True,True,True,,\r\n```"
},
{
"id": "/2022/03/21/_release-0690",
"metadata": {
"permalink": "/blog/2022/03/21/_release-0690",
"source": "@site/blog/2022-03-21_release-0690.md",
"title": "Apache StreamPipes release 0.69.0",
"description": "5 minutes to read",
"date": "2022-03-21T00:00:00.000Z",
"formattedDate": "March 21, 2022",
"tags": [],
"readingTime": 3.755,
"hasTruncateMarker": true,
"authors": [
{
"name": "Dominik Riemer",
"url": "https://twitter.com/dominikjriemer",
"imageURL": "/docs/img/riemer.png"
}
],
"frontMatter": {
"title": "Apache StreamPipes release 0.69.0",
"author": "Dominik Riemer",
"authorURL": "https://twitter.com/dominikjriemer",
"authorImageURL": "/docs/img/riemer.png"
},
"prevItem": {
"title": "Using Factory I/O with StreamPipes",
"permalink": "/blog/2022/09/05/using-factory.io-with-streampipes"
},
"nextItem": {
"title": "Apache StreamPipes release 0.68.0",
"permalink": "/blog/2021/07/22/_release-0680"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">5 minutes to read</div>**\r\n<br/>\r\n\r\nThe Apache StreamPipes (incubating) community is pleased to announce Apache StreamPipes version 0.69.0!\r\nThe most notable highlights of this release include a completely reworked data explorer for quick exploration of IoT data and advanced user and access rights management.\r\nIn addition, our new release includes more than 80 improvements and bug fixes.\r\n\r\n<!--truncate-->\r\n\r\nThe current release can be downloaded <a href=\"https://streampipes.apache.org/download.html\">here</a>.\r\n\r\n## Data Explorer\r\n\r\nApache StreamPipes 0.69.0 includes a heavily improved data explorer. The new user interface allows to visually explore data streams within minutes.\r\nAny pipeline which uses the ``Data Lake`` sink can be visualized in the data explorer. The completely new query interface allows to easily select date and time ranges and users can choose from either displaying raw data or aggregated time series.\r\nThe widget library includes charts to visualize time-series data (e.g., line chart, scatter plots or heatmaps), value distributions (e.g., density or histograms) or single values.\r\nThe widget interface is completely customized and users can build rich dashboards.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-03-21/sp-data-explorer-2.png\" alt=\"Data Explorer\"/>\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-03-21/sp-data-explorer-3.png\" alt=\"Data Explorer\"/>\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-03-21/sp-data-explorer-4.png\" alt=\"Data Explorer\"/>\r\n\r\n## User and access rights management\r\nStreamPipes now supports multiple users and roles. The new ``User management`` section in the settings page allows to create individual users and groups.\r\nVarious roles can be assigned to users and groups to manage permissions and visible views. For instance, you can now create users which only see dashboards, but cannot create pipelines or connect new data streams.\r\nIn addition, individual permissions can be assigned to various resource objects such as pipelines, data streams and dashboards. \r\n\r\nCommunication between extensions services and the core are now authenticated by default. Service accounts can be created to allow extensions services to access StreamPipes resources.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-03-21/sp-user-management-2.png\" alt=\"User Management\"/>\r\n\r\n\r\n## Improved pipeline element modeling\r\nWe've removed one of the major drawbacks of previous StreamPipes versions: Pipelines can now be flexibly reconfigured. In previous versions, due to the validation of pipelines and their configuration, only root elements could be modified. \r\nThe new version now gives users full flexibility when modeling pipelines (e.g., by removing or modifying elements from the middle of a pipeline), while pipelines are still continuously validated and modeling errors are shown.\r\nUpdating pipelines should now be much less distracting and faster.\r\n\r\n## Other new features\r\n\r\n### Mail configuration\r\n\r\nStreamPipes now includes a configuration interface for mail server settings. These settings are used by the email data sink used to trigger email notifications.\r\nIn addition, several system-wide mail services exist, e.g., to restore forgotten passwords.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-03-21/sp-config-mail.png\" alt=\"Mail configuration\"/>\r\n\r\n\r\n### Login configuration\r\n\r\nThe login mechanism can now be configured directly in the user interface. A self-registration can be activated to allow users to create new accounts.\r\nIn addition, passwords can be restored by using a two-stage email confirmation process. The profile view allows users to change their passwords individually.\r\n\r\n### Redesign of pipeline editor\r\nThe pipeline editor has a new look: The pipeline element selection bar has been moved from the top to the left side. The icons are smaller and for each pipeline element, the name and description are shown by default.\r\nThis makes it easier to find the elements you're looking for.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-03-21/sp-pipeline-editor.png\" alt=\"Dark mode\"/>\r\n\r\n\r\n### Improved service discovery\r\nUnder the hood, we've improved the discovery of extensions services. This is part of a larger effort which will be further improved in the next versions.\r\nExtensions services now automatically register within the core depending on the IP address of the service. Pipeline element descriptions are now free of the hostname and can be easily updated.\r\nSeveral environment variables exist to manually assign the hostname to an extensions service.\r\n\r\n### Performance\r\nWe've improved the overall system performance by replacing the triple store with a standard NoSQL database. This leads to huge performance improvements in terms of system start time, pipeline element installation and overall loading time.\r\n\r\n### Dark mode ;-)\r\nWe know that you wanted this: An experimental dark mode has been added to StreamPipes. The color scheme can be flexibly changed in the profile view and is stored for each user.\r\nThe dark mode should be considered experimental, so please expect some layout flaws.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2022-03-21/sp-dark-mode.png\" alt=\"Dark mode\"/>\r\n\r\n\r\n## Migration\r\nWhile we are not yet ready for automatic migration, a [migration guide](dev-guide-migration.md) explains several new concepts introduced in this StreamPipes version.\r\nWe aim at providing a backwards compatible version with release 1.0, planned for later this year."
},
{
"id": "/2021/07/22/_release-0680",
"metadata": {
"permalink": "/blog/2021/07/22/_release-0680",
"source": "@site/blog/2021-07-22_release-0680.md",
"title": "Apache StreamPipes release 0.68.0",
"description": "8 minutes to read",
"date": "2021-07-22T00:00:00.000Z",
"formattedDate": "July 22, 2021",
"tags": [],
"readingTime": 4.475,
"hasTruncateMarker": true,
"authors": [
{
"name": "Dominik Riemer",
"url": "https://twitter.com/dominikjriemer",
"imageURL": "/docs/img/riemer.png"
}
],
"frontMatter": {
"title": "Apache StreamPipes release 0.68.0",
"author": "Dominik Riemer",
"authorURL": "https://twitter.com/dominikjriemer",
"authorImageURL": "/docs/img/riemer.png"
},
"prevItem": {
"title": "Apache StreamPipes release 0.69.0",
"permalink": "/blog/2022/03/21/_release-0690"
},
"nextItem": {
"title": "Running StreamPipes the Kubernetes-Way using k3s",
"permalink": "/blog/2020/05/27/streampipes_on_k3s"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">8 minutes to read</div>**\r\n<br/>\r\n\r\nThe Apache StreamPipes (incubating) community is pleased to announce Apache StreamPipes version 0.68.0!\r\nThe latest release brings many improvements related to health management of pipelines and pipeline monitoring, improved user guidance such as a live preview in the pipeline editor and better handling of pipeline updates as well as several new data processors and sinks such as support for NATS. \r\nIn addition, the release contains several bug fixes related to K8s deployment, PLC adapters and pipeline management.\r\n\r\n<!--truncate-->\r\n\r\nThe current release can be downloaded <a href=\"https://streampipes.apache.org/download.html\">here</a>.\r\n\r\n## User interface\r\nThe user interface brings several improvements for users when working with pipelines. \r\nFor this release, we focused on providing a better user experience when pipelines and associated dashboard widgets are updated.\r\n\r\n### Pipeline preview\r\n\r\nA new pipeline preview allows to quickly inspect the output of data streams and processors during pipeline development.\r\nThe preview can be enabled in the pipeline canvas and displays the latest value of each pipeline element every second.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2021-07-22/pipeline-preview.png\" alt=\"Pipeline preview\"/>\r\n\r\n### Editable dashboard widgets\r\n\r\nApache StreamPipes 0.68.0 improves the modification of existing dashboard widgets. In the live dashboard, the visualization type and the underlying pipeline can now easily be changed without recreating a widget. \r\nIn addition, the dashboard shows additional error messages and hints in case pipelines are stopped or have been deleted. In the first case, pipelines can be conveniently started directly from the dashboard.\r\n\r\nBesides that, the dashboard now provides more visualization types (e.g., bar race chart, status light and word cloud) based on [Apache ECharts](https://echarts.apache.org).\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2021-07-22/edit-visualizations.png\" alt=\"Modify existing dashboard visualizations\"/>\r\n\r\n## Monitoring & Resilience\r\n\r\nThe latest version comes with several improvements related to better monitoring of running pipelines and internal health checking. \r\nTo allow users to inspect currently running pipelines, a new pipeline monitoring module has been added to the user interface which shows the current throughput of each pipeline element.\r\nIndividual pipeline elements can be selected in the preview panel. For each pipeline element, the number of processed input events, produced output events and the lag is displayed.\r\nThe displayed numbers may be slightly inaccurate for higher throughputs, but give an estimate whether the pipeline is running smoothly or lagging behind.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2021-07-22/pipeline-monitoring.png\" alt=\"Pipeline Monitoring\"/>\r\n\r\nBesides monitoring, StreamPipes 0.68.0 brings many improvements in terms of system resilience. Pipelines are now gracefully stopped once the StreamPipes installation is stopped. \r\nRunning pipelines are flagged and restarted at startup time. In addition, StreamPipes checks at every startup for any pipelines that were not properly stopped and restarts these pipelines accordingly.\r\nFor all running pipelines, StreamPipes now regularly checks the state of all involved pipeline elements and detects any failures. Problems are displayed in the UI and pipeline elements are automatically re-invoked once they become available after a failure. \r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2021-07-22/pipeline-failed.png\" alt=\"Awareness of pipeline failures in the UI\"/>\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2021-07-22/pipeline-restored.png\" alt=\"Recovery of failed pipeline elements\"/>\r\n\r\nAltogether, Apache StreamPipes is now able to recover and resume its state even when the power plug is pulled!\r\nResiliency is part of an ongoing larger community effort and will be further improved with a new service discovery approach that will be part of the upcoming version 0.69.0.\r\n\r\nThe Docker Compose-based installation now restarts StreamPipes automatically after a system restart by default.\r\n\r\n## StreamPipes Client\r\n\r\nTo allow third-party applications to access concepts managed by Apache StreamPipes, version 0.68.0 brings a preview of the StreamPipes client and REST API.\r\nThe public REST API and the client expose several features that were previously only internally available to developers, such as reading pipelines, performing operations on pipelines and subscribing to data sources managed by StreamPipes.\r\nAn API token can be easily generated in the profile section of the user interface.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2021-07-22/api-key-generation.png\" alt=\"Generating an API key for the StreamPipes client & REST API\"/>\r\n\r\nCurrently provided REST interfaces can be seen in the Swagger documentation, which is linked on the login page.\r\nThe client is provided as a Java library, an example of its usage is provided in our [Examples Github repository](https://github.com/apache/incubator-streampipes-examples/blob/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/client/example/StreamPipesClientExample.java)\r\n\r\n## StreamPipes Extensions\r\n\r\nWe also added several new adapters to connect data sources and pipeline elements:\r\n\r\n* A new NATS sink allows to forward events to a NATS broker\r\n* The PLC adapter now makes use of the PooledPLCDriverManager\r\n* Improvements to the MQTT sink\r\n* The IoTDB adapter now uses the latest IoTDB version\r\n* All SQL-based sinks now share a common and more stable codebase\r\n\r\n\r\n## Other features and improvements\r\n\r\n* The connect-master module has been removed and integrated into the core\r\n* Library upgrades to Angular (11), Spring Boot, and others\r\n* Improvements to the helm chart for K8s deployment\r\n\r\nSee the [release notes](https://issues.apache.org/jira/projects/STREAMPIPES/versions/12348698) for a complete list of new features and improvements.\r\n\r\n# Upgrading\r\nStreamPipes 0.68.0 is an intermediate release towards a stable version 1.0. Under the hood, we are performing several code migrations to further improve the overall stability and performance of StreamPipes. \r\nTherefore, we currently recommend to start with a clean installation when upgrading. We are working towards an auto-update feature which will be available later this year.\r\n\r\n## Feedback\r\n\r\nWe are absolutely open to your suggestions for further improvements! Let us know (by [mail](mailto:users@streampipes.apache.org) or [twitter](https://www.twitter.com/streampipes)) and we'll consider your feature request in the next release!\r\n\r\nAnd do not forget to subscribe to our new mailing lists: https://streampipes.apache.org/mailinglists.html"
},
{
"id": "/2020/05/27/streampipes_on_k3s",
"metadata": {
"permalink": "/blog/2020/05/27/streampipes_on_k3s",
"source": "@site/blog/2020-05-27-streampipes_on_k3s.md",
"title": "Running StreamPipes the Kubernetes-Way using k3s",
"description": "15 minutes to read",
"date": "2020-05-27T00:00:00.000Z",
"formattedDate": "May 27, 2020",
"tags": [],
"readingTime": 11.865,
"hasTruncateMarker": true,
"authors": [
{
"name": "Patrick Wiener",
"url": "https://twitter.com/patrickwiener",
"imageURL": "/docs/img/wiener.png"
}
],
"frontMatter": {
"title": "Running StreamPipes the Kubernetes-Way using k3s",
"author": "Patrick Wiener",
"authorURL": "https://twitter.com/patrickwiener",
"authorImageURL": "/docs/img/wiener.png"
},
"prevItem": {
"title": "Apache StreamPipes release 0.68.0",
"permalink": "/blog/2021/07/22/_release-0680"
},
"nextItem": {
"title": "Apache StreamPipes release 0.66.0",
"permalink": "/blog/2020/05/19/release-0660"
}
},
"content": "<img class=\"blog-image\" style=\"max-width:100%;\" src=\"/docs/blog/assets/2020-05-27/streampipes_k3s.png\" alt=\"StreamPipes on k3s\"/>\r\n**<div style=\"float: left; padding-right: 40px;\">15 minutes to read</div>**\r\n<br/>\r\n\r\n\r\nThis blog post is the second part of a series on how we want to bring StreamPipes closer to the edge. The [first part](/docs/blog/2020/01/23/streampipes_on_rpi) describes how we managed to build and deploy StreamPipes on one Raspberry Pi (Model 4), while the [second part](/docs/blog/2020/05/27/streampipes_on_k3s) further covers how to deploy StreamPipes over a Kubernetes cluster of two Raspberry Pi's via k3s using our StreamPipes helm chart.\r\n<!--truncate-->\r\n\r\n**<TL;DR>**\r\n\r\n**With new advancements that came in the last StreamPipes release, we managed to reduce the required resources (mainly memory) drastically. To this extent, we cross-compiled StreamPipes Docker images for ARM support and deployed our lite version on a Raspberry Pi (Model 4). We showcase, that it is feasible to run analytical pipelines with ease and operate StreamPipes at moderate load. Additionally, we demonstrate how to deploy StreamPipes over a two-node k3s cluster using our StreamPipes helm chart.**\r\n\r\n<img class=\"blog-image\" style=\"max-width:80%;\" src=\"/docs/blog/assets/2020-05-27/k3s.png\" alt=\"K3s\"/>\r\n\r\n## Introduction\r\n\r\nWhile it is easy to get started with StreamPipes in no time using our single-node deployment [instructions](/docs/docs/user-guide-installation) using Docker and Docker-Compose, StreamPipes naturally is designed to run in a distributed cluster setup through its microservice architecture.\r\n\r\nThis architectural design paired with the advancements of multi-architecture StreamPipes images as described in the previous blog post allows us to run certain analytical pipeline element containers with ease on various target nodes as well as target hardware architecture wether it is X86, ARM32v7, or ARM64v8. This becomes more valuable, as we shift the focus away from centralized cloud or data center only deployments where we generally are provided with a homogenous compute landscape in terms of resources towards edge deployments that oftentimes envolve resource-limited, constrained devices that not only differ in terms of compute but also in terms of their chip architecture.\r\n\r\nThereby, having multi-architecture images in your repertoire is key to deal with the challenge of heterogeneity in compute resources along the edge-cloud continuum. However, there are other challenges that occur when dealing with distributed container deployments such as automating deployments itself, scaling them, managing the containerized applications, taking care of networking between services, or realizing fault-tolerance by comparing the desired state against the current one. If there wasn't some solution we could leverage that takes care of all of this.\r\n\r\n## Well, which K to use? Kubernetes, K3s\r\n\r\nFortunately, there exists a few solutions in the container ecosystem with the open source container orchestrator [Kubernetes](https://kubernetes.io) (K8s) being the defacto standard, thereby incorporating the best ideas of Google's internal large-scale container cluster management system [Borg](https://kubernetes.io/blog/2015/04/borg-predecessor-to-kubernetes/) (see academic paper about the details published at [EuroSys 2015](https://research.google/pubs/pub43438/)). Since we are not going to write \"yet another K8s basics blog post\", we refer to their docs or other awesome blog posts such as the [Kubernetes 101](https://medium.com/google-cloud/kubernetes-101-pods-nodes-containers-and-clusters-c1509e409e16) to get familiarized with the concepts and terminology. With all the power, concepts and terminology K8s provides, it is oftentimes hard for newcomers to get started. Additionally, not all use cases necessarily need these extra features that K8s adds.\r\n\r\nOn the other hand, there exists [K3s](https://k3s.io/), a lightweight Kubernetes distro by Rancher that was introduced early 2019 and has since then gained huge popularity. As it is stated on their website:\r\n\r\n* **Perfect for Edge**: _K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances_\r\n* **Simplified & Secure**: _K3s is packaged as a single <40MB binary that reduces the dependencies and steps needed to install, run and auto-update a production Kubernetes cluster._\r\n* **Optimized for ARM**: _Both ARM64 and ARMv7 are supported with binaries and multiarch images available for both. K3s works great from something as small as a Raspberry Pi to an AWS a1.4xlarge 32GiB server._\r\n\r\nThus, this lead us to the point to say, hey why not use K3s to manage StreamPipes in all its glory. The containerized microservice design in addition to the newly available multiarch images are a great fit.\r\n\r\n\r\n## StreamPipes Helm Chart\r\nWhile you can easily write an deploy K8s YAML-based manifests for your deployments and services and so on the standard way, it is way more convenient to leverage [Helm](https://helm.sh/), the package manager for K8s. Helm describes complex, distributed deployments using **charts** in order to provide repeatable application installation, easy updates, rollbacks etc.\r\n\r\nTo this extent, we wrote a Helm chart for StreamPipes that can be found on our Github [incubator-streampipes-installer](https://github.com/apache/incubator-streampipes-installer/tree/dev/helm-chart) repository. This allows to deploy StreamPipes as one liner as shown later. Helm charts follow a certain directory structure as depicted in the following:\r\n\r\n```\r\nhelm-chart/\r\n Chart.yaml\r\n values.yaml\r\n templates/\r\n ...\r\n```\r\nThe `Chart.yaml` file contains a description of the chart, e.g., metadata such as chart name, version, maintainers etc.\r\n\r\nThe `values.yaml` file contains default values for all charts and in our case serves the purpose to provide a central place to store all variable/values, e.g., `streampipes.registry: \"apachestreampipes\"`, that replace chart templates such as the registry and version of the StreamPipes backend image:\r\n\r\n```yaml\r\n...\r\nimage: {{ .Values.streampipes.registry }}/backend:{{ .Values.streampipes.version }}\r\n...\r\n```\r\n\r\nLastly, the `template/` directory is for template files. When Helm evaluates a chart, it sends all of the files in this directory through the template rendering engine before collecting the results and finally forwarding them to K8s. If interested, you can read more on this in the [official Helm documentation](https://helm.sh/docs/).\r\n\r\n\r\n## The almighty StreamPi K3s Cluster\r\nAs pointed out in the first part of the series, we have gained valueable insights on how to run and deploy StreamPipes on a Raspberry Pi. This worked out so greatly, that we thought, why not step up the game, use another Pi and built the **almighty StreamPi cluster** by leveraging K3s for orchestration and management.\r\n\r\n**Note**: We are aware that this is far off from a production-grade setup, especially running StreamPipes core services such as the central pipeline management backend, or the UI in a two-node cluster. However, it provides a solid starting point for more elaborate deployments along the edge-cloud continuum.\r\n\r\nAs shown in the following figure, starting from the multiarch images hosted on [Dockerhub](https://hub.docker.com/search?q=apachestreampipes&type=image), we first provisioned the two Pi's with Docker, install and setup K3s and finally deploy our StreamPipes helm chart in order for users to connect, analyze and exploit their IIoT data with StreamPipes.\r\n\r\n\r\n<img class=\"blog-image\" style=\"max-width:100%;\" src=\"/docs/blog/assets/2020-05-27/deployment.png\" alt=\"Deployment\"/>\r\n\r\nIn the remainder, we cover how we installed Helm and K3s to form our almighty StreamPi K3s cluster. At the time of writing, we tested our deployment using the latest version of the Docker, Helm as well as K3s in the following setup\r\n\r\n* 2x Raspberry Pi Model 4 (4GiB RAM): `streampi-01`, `streampi-02`\r\n* Raspbian Buster 10\r\n* Docker 19.03.8\r\n* Helm 3.1.2\r\n* K3s v1.17.4+k3s1\r\n\r\n\r\n### 1. Install Docker\r\nSSH into your Raspberry Pi's and run the following command:\r\n```bash\r\ncurl -sSL https://get.docker.com | sh\r\n```\r\n\r\n### 2. Install K3s\r\nNow that we got Helm installed, we can proceed and install K3s. Thus, we must check wether we fulfill all the requirements. These are mainly network related, such that specific ports (server on `6443`) are accessible. You should carefully read about this in their [official documentation](https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/).\r\n\r\nOther than that, lean back and enjoy - the installation happens so fast, you might miss it if you get a coffee :)\r\n\r\n#### SSH into K3s server (`streampi-01`)\r\nOn the server node, run the following command:\r\n\r\n```bash\r\nsudo curl -sfL https://get.k3s.io | sh -\r\n```\r\nA kubeconfig file is written to `/etc/rancher/k3s/k3s.yaml` and the service is automatically started or restarted. The install script will install k3s and additional utilities, such as `kubectl`, `crictl`, `k3s-killall.sh`, and `k3s-uninstall.sh`.\r\n\r\nThe K3s installer creates a systemd service which can be used to `stop`, `start`, `restart` and check the `status` of the K3s server.\r\n\r\n```bash\r\nsudo systemctl status k3s.service\r\n\r\n● k3s.service - Lightweight Kubernetes\r\n Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)\r\n Active: active (running) since Sat 2020-04-11 15:54:07 BST; 3 days ago\r\n Docs: https://k3s.io\r\n Process: 10193 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)\r\n Process: 10194 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)\r\n Main PID: 10195 (k3s-server)\r\n (...)\r\n```\r\n\r\nNext, copy the server node token in order to connect additional agent nodes to the cluster.\r\n```bash\r\nsudo cat /var/lib/rancher/k3s/server/node-token\r\n```\r\n\r\n#### SSH into K3s agent (`streampi-02`)\r\nNow, open a new terminal session and SSH into your agent and run the following command, thereby adding the copied `K3S_TOKEN` to connect to the server, that is specified using `K3S_URL`.\r\n\r\n```bash\r\nsudo curl -sfL https://get.k3s.io | K3S_URL=https://streampi-01:6443 \\\r\nK3S_TOKEN=<NODE_TOKEN> sh -\r\n```\r\n\r\nThe K3s installer creates a systemd service which can be used to `stop`, `start`, `restart` and check the `status` of the K3s agent.\r\n\r\n```bash\r\nsudo systemctl status k3s-agent.service\r\n\r\n● k3s-agent.service - Lightweight Kubernetes\r\n Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)\r\n Active: active (running) since Sat 2020-04-11 16:05:37 BST; 3 days ago\r\n Docs: https://k3s.io\r\n Process: 24469 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)\r\n Process: 24470 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)\r\n Main PID: 24471 (k3s-agent)\r\n (...)\r\n```\r\n\r\nThat's it ... that's all!\r\n\r\nWhen you go back to the terminal session on the K3s server you can list all nodes in the cluster using standard `kubectl` commands.\r\n\r\n```bash\r\nkubectl get nodes\r\nNAME STATUS ROLES AGE VERSION\r\nstreampi-02 Ready <none> 3d v1.17.4+k3s1\r\nstreampi-01 Ready master 3d v1.17.4+k3s1\r\n```\r\n\r\nHmmm - but wait. We did only add one agent node to the cluster right? Well, that's true, however when installing K3s using the script the server automatically installs and runs an agent on the same node.\r\n\r\n### 3. Install Helm\r\nSSH into the RPi that should act as the K3s server later on - in our case this is `streampi-01` and install Helm. Since we are using Helm 3 there is no need to install tiller. Thus, installing helm is as simple as running the following command:\r\n\r\n```bash\r\ncurl -sSL https://get.helm.sh/helm-v3.1.2-linux-arm.tar.gz | sudo tar -xzf - -C /usr/local/bin/ linux-arm/helm --strip-components=1\r\n```\r\nThis command downloads the Helm binary release from Github, extracts only the binary and puts it in `/usr/local/bin`. Try go check `helm version` to see if it worked.\r\n\r\n```bash\r\nversion.BuildInfo{Version:\"v3.1.2\", GitCommit:\"d878d4d45863e42fd5cff6743294a11d28a9abce\", GitTreeState:\"clean\", GoVersion:\"go1.13.8\"}\r\n```\r\n\r\nTo use Helm, we can set and export `KUBECONFIG` environment variable to point to the kubeconfig under `/etc/rancher/k3s/k3s.yaml`.\r\n```bash\r\nexport KUBECONFIG=/etc/rancher/k3s/k3s.yaml\r\n```\r\nAlternatively, if you want this permanently you can either add it to your `.bashrc`/`.zshrc`, or create a `.kube` directory in your home directory and symlink the `k3s.yaml`.\r\n\r\n```bash\r\nmkdir $HOME/.kube\r\nln -s /etc/rancher/k3s/k3s.yaml $HOME/.kube/config\r\n```\r\n\r\n\r\n## Deploy StreamPipes on K3s using Helm\r\nNow, that we have all set up, we can go ahead and deploy StreamPipes using our Helm chart. Therefore, we go to the server node (`streampi-01`), where we installed Helm. Next, we clone our StreamPipes installer repository (`dev` branch) and navigate to the helm-chart directory.\r\n\r\n**Note**: At the point of writing, the helm chart is **only** available in the `dev` branch. This will change in the future as we update the repository.\r\n\r\n```bash\r\ngit clone -b dev --single-branch https://github.com/apache/incubator-streampipes-installer.git\r\ncd incubator-streampipes-installer\r\n```\r\n\r\nAt this point, we can leverage Helm's `install` command in order to install the StreamPipes Helm chart. Since Helm reads the K3s kubeconfig, it forwards the deployment request to K3s. As per default, we deploy StreamPipes `lite` version, where we only start standalone Java-based pipeline elements and disregard the ones building on top of our Flink wrapper.\r\n\r\n```bash\r\nhelm install streampipes helm-chart\r\n\r\nNAME: streampipes\r\nLAST DEPLOYED: Tue Apr 14 18:13:19 2020\r\nNAMESPACE: default\r\nSTATUS: deployed\r\nREVISION: 1\r\nTEST SUITE: None\r\n```\r\n\r\nNext, K3s receives this requests and goes ahead and downloads the corresponding Docker images and starts the deployments in the right order according to the specified dependencies, creates services as well as persistent volume claims when needed. This step takes some time depending on your internet connection and the hardware when starting the containers. You can check the status of the StreamPipes Helm installation using `kubectl get pods -o wide` (when adding `-w`, you can automatically watch for changes). You can also use the custom command below to only display the pod name, status as well as the node where it was scheduled and deployed to.\r\n\r\n```bash\r\nkubectl get pods -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName\r\n\r\nNAME STATUS NODE\r\nactivemq-66d58f47cf-b88nb Running streampi-01\r\nzookeeper-5d9947686f-7vpjb Running streampi-02\r\ncouchdb-77db98cf7b-6nn99 Running streampi-02\r\nconsul-55965f966b-tjcfb Running streampi-02\r\ninfluxdb-b95b6479-mvgbg Running streampi-02\r\nbackend-6cdfb7bbf9-b4vgh Running streampi-02\r\npipeline-elements-all-jvm-64795974-g2xhv Running streampi-02\r\nkafka-657b5fb77-xtw8p Running streampi-01\r\nconnect-master-56f5f947d-n2t5b Running streampi-01\r\nsources-watertank-simulator-6958f7844c-2jv78 Running streampi-01\r\nconnect-worker-fdbd6f464-m6vm8 Running streampi-01\r\nnginx-5d4498d4f5-fcccq Running streampi-02\r\n```\r\n\r\nAs can be seen, a fair amount of StreamPipes containers aka pods were deployed on `streampi-01` and `streampi-02`. When everything goes well, you should see all containers in the status `RUNNING` after a couple of minutes. To access K8s cluster UI pod a `NodePort` was created. The `NodePort` publishes the UI (nginx) service on every node using using its public IP and a port. Then you can access the service using any of the cluster IPs and the assigned port.\r\n\r\n```yaml\r\napiVersion: v1\r\nkind: Service\r\nmetadata:\r\n name: nginx\r\nspec:\r\n selector:\r\n app: nginx\r\n type: NodePort\r\n ports:\r\n - name: main\r\n protocol: TCP\r\n port: 80 # container port\r\n targetPort: 80 # node port\r\n```\r\n\r\nThat means, you can go and check the StreamPipes UI either on `http://streampi-01/` or `http://streampi-02/` - K8s service handles the routing to the nginx pod containing the UI - just to name one nice feature.\r\n\r\n<img class=\"blog-image\" style=\"max-width:100%;\" src=\"/docs/blog/assets/2020-05-27/streampi_installation.png\" alt=\"Installation\"/>\r\n\r\n## Sum up & Where to go from here?\r\nStarting out as a simple hack session on a late afternoon, we build multi-archicture StreamPipes Docker images and deployed them on a Raspberry Pi in the first part of the series. Though, this is was feasible StreamPipes architectural microservice design provides a natural fit for a distributed cluster deployment. To this extent, we provide a StreamPipes Helm chart that lets you easily deploy StreamPipes in a one-liner in a Kubernetes cluster as showcased using a testbed of two Raspberry Pi's and lightweight K3s.\r\n\r\nAs of now, we let K8s handle the scheduling of containers for us in a random fashion. From here, we can further explore K8s feature such as `nodeSelector` or `nodeAffinity` that helps you use labels (`key=value` pairs) for pod to node assignment.\r\n\r\n<img class=\"blog-image\" style=\"max-width:60%;\" src=\"/docs/blog/assets/2020-05-27/helm_k3s.png\" alt=\"HelmK3s\"/>\r\n\r\n- [Part 1: StreamPi(pes) - Running StreamPipes on Raspberry Pi](/docs/blog/2020/01/23/streampipes_on_rpi)\r\n- **[Part 2: Running StreamPipes the Kubernetes-Way using K3s](/docs/blog/2020/05/27/streampipes_on_k3s)**\r\n\r\n\r\n### Useful links\r\n\r\n- [1] [K3s - lightweight Kubernetes distro](https://k3s.io/)\r\n- [2] [Helm - the package manager for Kubernetes](https://helm.sh/)\r\n- [3] [Github: StreamPipes Helm chart](https://github.com/apache/incubator-streampipes-installer/tree/dev/helm-chart)"
},
{
"id": "/2020/05/19/release-0660",
"metadata": {
"permalink": "/blog/2020/05/19/release-0660",
"source": "@site/blog/2020-05-19-release-0660.md",
"title": "Apache StreamPipes release 0.66.0",
"description": "6 minutes to read",
"date": "2020-05-19T00:00:00.000Z",
"formattedDate": "May 19, 2020",
"tags": [],
"readingTime": 3.415,
"hasTruncateMarker": true,
"authors": [
{
"name": "Dominik Riemer",
"url": "https://twitter.com/dominikjriemer",
"imageURL": "/docs/img/riemer.png"
}
],
"frontMatter": {
"title": "Apache StreamPipes release 0.66.0",
"author": "Dominik Riemer",
"authorURL": "https://twitter.com/dominikjriemer",
"authorImageURL": "/docs/img/riemer.png"
},
"prevItem": {
"title": "Running StreamPipes the Kubernetes-Way using k3s",
"permalink": "/blog/2020/05/27/streampipes_on_k3s"
},
"nextItem": {
"title": "StreamPi(pes) - Running StreamPipes on Raspberry Pi",
"permalink": "/blog/2020/01/23/streampipes_on_rpi"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">6 minutes to read</div>**\r\n<br/>\r\n\r\nThe Apache StreamPipes (incubating) community is proud to announce Apache StreamPipes version 0.66.0! This release is our first official Apache release and brings some very cool features and improvements the community has worked on over the past months.\r\n<!--truncate-->\r\n\r\nThe latest release brings major improvements to the live dashboard, the notification system, multiple new pipeline elements and StreamPipes Connect adapters, a simplified installation, more deployment options and a variety of bug fixes and minor improvements.\r\nThe current release can be downloaded <a href=\"https://streampipes.apache.org/download.html\">here</a>.\r\n\r\n\r\n## User interface\r\n\r\nThis release introduces a completely redesigned live dashboard. The old dashboard had some issues, most importantly, widgets could not be freely moved on the dashboard panel and the layout of dashboard widget was not persisted. \r\nWith the new dashboard, users are able to customize and layout widgets in a much easier and more powerful way. The dashboard brings the following improvements:\r\n* Multiple dashboards can be created and viewed\r\n* Widgets can be flexibly resized on a grid\r\n* More configuration options for widgets, e.g., change the color and appearance of widgets.\r\n* A presentation mode allows to display a live dashboard without the navigation bars.\r\n\r\nThe new dashboard has full feature parity with the old dashboard in terms of available widgets. The new architecture allows us to integrate even more visualizations with less effort in the upcoming versions.\r\n\r\n<div class=\"my-carousel\">\r\n<img class=\"blog-image\" style=\"max-width:100%;\" src=\"/docs/blog/assets/2020-05-19/dashboard-02.png\" alt=\"New dashboard\"/>\r\n<img class=\"blog-image\" style=\"max-width:100%;\" src=\"/docs/blog/assets/2020-05-19/dashboard-01.png\" alt=\"New dashboard\"/>\r\n</div>\r\n\r\nIn addition, the notification system has been significantly improved. The new notification view now offers a chat-like interface which allows you to select notifications per pipeline. The history of notifications is fully available and can be opened by scrolling to the top - similar to the behaviour of popular messaging services:\r\n\r\n<div class=\"my-carousel\">\r\n<img class=\"blog-image\" style=\"max-width:100%;\" src=\"/docs/blog/assets/2020-05-19/notification-01.png\" alt=\"New dashboard\"/>\r\n<img class=\"blog-image\" style=\"max-width:100%;\" src=\"/docs/blog/assets/2020-05-19/notification-02.png\" alt=\"New dashboard\"/>\r\n</div> \r\n\r\nA technical note:\r\nBoth features are part of our current effort to gradually migrate AngularJS components to Angular 9 (2+), now leaving the pipeline editor the only component that is still implemented in AngularJS.\r\nOnce the editor has been migrated as well (which is currently targeted for the upcoming release 0.67.0), we'll be able to release global UI features (e.g., better user management) in a much faster way.\r\n\r\n\r\n## Apache StreamPipes Extensions\r\n\r\nWe also added several new adapters to connect data sources and pipeline elements:\r\n\r\n\r\n* We added several new processors for handling boolean values, which are targeted towards better processing of PLC data.\r\n* processors-geo-jvm includes more processors for geo operations, e.g., distance calculation, speed calculation and more.\r\n* processors-text-mining includes several processors relevant for text mining (e.g., chunking, language detection)\r\n* New adapters for NETIO power sockets, and several new sinks (e.g., MQTT and Eclipse Ditto) \r\n\r\n\r\n## Installation & Deployment\r\n\r\nThe community also improved the installation process. Now, the installer offered for download includes all three supported deployment options and tools:\r\n* The installer, targeted at users, with installation scripts for Linux, MacOS and Windows\r\n* The CLI, targeted at developers, for configurable deployment of individual services\r\n* Helm charts for Kubernetes\r\n\r\nA very cool new feature are Docker images for ARM, which allow to run StreamPipes on a Raspberry Pi. This will be part of an upcoming blog post on running Apache StreamPipes at the edge.\r\n\r\nFrom now on, users are no longer required to enter the server IP upon startup, which should further reduce errors at installation time.\r\n\r\n## Other features and improvements\r\n\r\n* Connect adapters expose their own icons and documentation.\r\n* The UI has been upgraded to Angular 9 and the build has more optimizations enabled, leading to significant performance gains.\r\n* Pipelines are now auto-saved, so that you can resume building your pipeline after refreshing the browser.\r\n\r\nSee the [release notes](https://issues.apache.org/jira/projects/STREAMPIPES/versions/12347025) for a complete list of new features and improvements.\r\n\r\n\r\n## Feedback\r\n\r\nWe are absolutely open to your suggestions for further improvements! Let us know (by [mail](mailto:users@streampipes.apache.org) or [twitter](https://www.twitter.com/streampipes)) and we'll consider your feature request in the next release!\r\n\r\nAnd do not forget to subscribe to our new mailing lists: https://streampipes.apache.org/mailinglists.html"
},
{
"id": "/2020/01/23/streampipes_on_rpi",
"metadata": {
"permalink": "/blog/2020/01/23/streampipes_on_rpi",
"source": "@site/blog/2020-01-23-streampipes_on_rpi.md",
"title": "StreamPi(pes) - Running StreamPipes on Raspberry Pi",
"description": "5 minutes to read",
"date": "2020-01-23T00:00:00.000Z",
"formattedDate": "January 23, 2020",
"tags": [],
"readingTime": 5.285,
"hasTruncateMarker": true,
"authors": [
{
"name": "Patrick Wiener",
"url": "https://twitter.com/patrickwiener",
"imageURL": "/docs/img/wiener.png"
}
],
"frontMatter": {
"title": "StreamPi(pes) - Running StreamPipes on Raspberry Pi",
"author": "Patrick Wiener",
"authorURL": "https://twitter.com/patrickwiener",
"authorImageURL": "/docs/img/wiener.png"
},
"prevItem": {
"title": "Apache StreamPipes release 0.66.0",
"permalink": "/blog/2020/05/19/release-0660"
},
"nextItem": {
"title": "Moving to the Apache Incubator",
"permalink": "/blog/2019/11/25/moving-to-the-apache-incubator"
}
},
"content": "<img class=\"blog-image\" style=\"max-width:100%;\" src=\"/docs/blog/assets/2020-01-23/streampipes_raspberrypi.png\" alt=\"StreamPipes on Raspberry Pi\"/>\r\n**<div style=\"float: left; padding-right: 40px;\">5 minutes to read</div>**\r\n<br/>\r\n\r\n\r\nThis is the start of a two part series on how we want to bring StreamPipes closer to the edge. The [first part](/docs/blog/2020/01/23/streampipes_on_rpi) describes how we managed to build and deploy StreamPipes on one Raspberry Pi (Model 4), while the [second part](/docs/blog/2020/05/27/streampipes_on_k3s) further covers how to deploy StreamPipes over a Kubernetes cluster of two Raspberry Pi's via K3s using our StreamPipes helm chart.\r\n<!--truncate-->\r\n\r\n<TL;DR>\r\n\r\n_With new advancements that came in the last StreamPipes release, we managed to reduce the required resources (mainly memory) drastically. To this extent, we cross-compiled StreamPipes Docker images for ARM support and deployed our lite version on a Raspberry Pi (Model 4). We showcase, that it is feasible to run analytical pipelines with ease and operate StreamPipes at moderate load._\r\n\r\nApache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to enable non-technical users to easily connect, analyze and explore IoT data streams. Users can leverage algorithms from an extensible toolbox to model their analytical pipelines. By building on a microservice architecture, StreamPipes components can be distributed in a cluster (e.g. Kubernetes) as well as run on a single node for prototypical and development purposes using Docker. With the latest release of StreamPipes (0.65.0), we managed to reduce the memory consumption drastically to use less than 2Gig‘s of RAM for the lite version. Since we had a Raspberry Pi Model 4 laying around, we thought: hey why not just deploy StreamPipes on a Pi - and that’s what we did. Along the way, we learned quite some interesting things, that we want to share in this blog post.\r\n\r\nWhen users model their analytical pipelines using the graphical pipeline editor, the corresponding pipeline element containers, hosting these algorithms, are invoked by a rest call from the backend to start a new thread with the corresponding algorithm. In typical edge use cases such as in the industrial internet of things (IIoT), it is oftentimes neither feasible nor necessary to simply send all raw data to the cloud to perform analytics. On the contrary, requirements often induce the challenges to either derive decisions in a time-sensitive manner (near realtime), preprocess data to reduce and harmonize it, or anonymize it by pixeling personal information such as faces in video footages. In these scenarios, StreamPipes can aid by running dedicated pipeline element containers directly on the edge using edge nodes such as Raspberry Pi‘s close to where data origins to address the aforementioned challenges.\r\n\r\n<img class=\"blog-image\" align=\"center\" style=\"max-width:80%;\" src=\"/docs/blog/assets/2020-01-23/00_idea.png\" alt=\"StreamPipes lite on Raspberry Pi\"/>\r\n\r\nStreamPipes Docker images were only built for x86 based architectures. However, oftentimes edge devices such as Raspberry Pi's or Nvidia Jetson Nano's run ARM based 32 or 64bit architectures. Thus, we could not simply use our existing Docker images to deploy StreamPipes as is. Luckily, Docker offers solutions to cross-compile images. Together with Docker‘s experimental manifest feature, the deployment works like a charms [[1](https://docs.docker.com/engine/reference/commandline/manifest/)]. There are awesome blog posts online [[2](https://ownyourbits.com/2018/06/27/running-and-building-arm-docker-containers-in-x86/)] that guide you through the process of cross-compilation, so we are not going to cover it in this blog post. With that, we were able to build StreamPipes images for AMD, ARM32v7 and ARM64v8 by just adding a few lines to our Dockerfiles and CI/CD pipelines. After having built and pushed our newly created Docker images, we were able to use our existing installation workflow, e.g. StreamPipes CLI, to start StreamPipes on the Raspberry Pi.\r\n\r\n<img class=\"blog-image\" style=\"max-width:100%;\" src=\"/docs/blog/assets/2020-01-23/01_start.png\" alt=\"StreamPipes CLI starting StreamPipes lite version on Raspberry Pi 4\"/>\r\n\r\nThe figure shows StreamPipes CLI starting our internal `0.65.1-SNAPSHOT` version, where we tested the multiarch build of our images. As depicted, all StreamPipes lite services are started. Due to the fact, that we created a manifest for all our Docker images, we did not have to change a single image description specified in the various `docker-compose.yml` files in the service repository. This is pretty cool! The Docker daemon automatically pulls the correct image for the corresponding architecture - in our case ARM32v7. Taking a quick look at `htop` indicates that we operate on a constant level of consumed memory at around ~1.7GB.\r\n\r\n<img class=\"blog-image\" style=\"max-width:100%;\" src=\"/docs/blog/assets/2020-01-23/02_htop.png\" alt=\"htop showing around 1.7GB memory consumption\"/>\r\n\r\n### Demo: Temperature Monitoring Pipeline\r\nTo test our setup, we deployed a simple temperature monitoring pipeline, that throws a notification whenever the temperature value exceeds a certain threshold (in this case: 65°C) and visualizes the results in a basic dashboard (see figure). Therefore, we use existing pipeline elements of the algorithm toolbox provided within the lite version. Thereby, StreamPipes lite already hooks you up with sample data sources - in this case a flow rate sensor - producing temperature, as well as mass flow measurements. So our temperature monitoring pipeline consists of the following pipeline elements:\r\n\r\n- **Data source:** Flow rate sensor (mock data generating `<timestamp;sensorid;massflow;temperature>` each second)\r\n- **Data processor:** Numerical filter (specified temperature threshold: > 65°C)\r\n- **Data sinks:** Dashboard visualization, Notification (internal)\r\n\r\n<img class=\"blog-image\" style=\"max-width:100%;\" src=\"/docs/blog/assets/2020-01-23/03_pipeline.png\" alt=\"Simple pipeline visualizing and notifying about temperature values above threshold (here 65°C)\"/>\r\n\r\nAfter saving and starting the pipeline, we built our dashboard based on the results of the pipeline, only showing values above 65°C. Additionally, we receive a notification for every event, that exceeds this temperature threshold.\r\n\r\n<img class=\"blog-image\" style=\"max-width:100%;\" src=\"/docs/blog/assets/2020-01-23/04_dashboard.png\" alt=\"Dashboard\"/>\r\n\r\n\r\n### So, what's next?\r\nStarting out as a simple hack session on a late afternoon, trying to test where we could potentially run StreamPipes on and push it to it's limits, we gained valueable insights on how to build and run it on various architectures. Even though we do not recommend deploying StreamPipes in it's full glory on single low-resourced devices such as Raspberry Pi's, we aim to bring certain algorithms closer to where data is generated - on the edge. Consequently - apart from x86 images - we plan on providing our StreamPipes Docker images for ARM based architectures in the future. As a sneak peak - we also plan on deploying StreamPipes over a K3s [[3](https://k3s.io/)] cluster, a lightweight Kubernetes distro especially suited for IoT and edge scenarios.\r\n\r\n- **[Part 1: StreamPi(pes) - Running StreamPipes on Raspberry Pi](/docs/blog/2020/01/23/streampipes_on_rpi)**\r\n- [Part 2: Running StreamPipes the Kubernetes-Way using K3s](/docs/blog/2020/05/27/streampipes_on_k3s)\r\n\r\n\r\n### Useful links\r\n- [1] [Docker manifest docs](https://docs.docker.com/engine/reference/commandline/manifest/)\r\n- [2] [Running and Building ARM Docker containers in x86](https://ownyourbits.com/2018/06/27/running-and-building-arm-docker-containers-in-x86/)\r\n- [3] [K3s - lightweight Kubernetes distro](https://k3s.io/)"
},
{
"id": "/2019/11/25/moving-to-the-apache-incubator",
"metadata": {
"permalink": "/blog/2019/11/25/moving-to-the-apache-incubator",
"source": "@site/blog/2019-11-25-moving-to-the-apache-incubator.md",
"title": "Moving to the Apache Incubator",
"description": "8 minutes to read",
"date": "2019-11-25T00:00:00.000Z",
"formattedDate": "November 25, 2019",
"tags": [],
"readingTime": 3.675,
"hasTruncateMarker": true,
"authors": [
{
"name": "Dominik Riemer",
"url": "https://twitter.com/dominikjriemer",
"imageURL": "/docs/img/riemer.png"
}
],
"frontMatter": {
"title": "Moving to the Apache Incubator",
"author": "Dominik Riemer",
"authorURL": "https://twitter.com/dominikjriemer",
"authorImageURL": "/docs/img/riemer.png"
},
"prevItem": {
"title": "StreamPi(pes) - Running StreamPipes on Raspberry Pi",
"permalink": "/blog/2020/01/23/streampipes_on_rpi"
},
"nextItem": {
"title": "StreamPipes release 0.65.0",
"permalink": "/blog/2019/11/23/release-0650"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">8 minutes to read</div>**\r\n<br/>\r\nGreat news: StreamPipes is moving to the Apache Incubator!\r\n<!--truncate-->\r\n\r\n## The StreamPipes journey so far\r\nWe started to build the early foundations of StreamPipes back in 2014 as part of the EU research project _ProaSense_. Our goal was to make event processing accessible to end users with a focus on the Industrial IoT, without the technical expertise required by stream processing tooling.\r\nAfter some very early prototypes (see an early screenshot below :-)) demonstrating the usefulness of the pipeline editor and the underlying semantics-based data model in two manufacturing-related use cases, we open-sourced StreamPipes in 2017 and continued development in various research projects.\r\nSince then, we presented and showcased StreamPipes at various events: fairs, meetups, conference talks and so on. For instance, in 2019 we attended over 20 events where we gave demos or talked about StreamPipes.\r\nIt is great to see that the solution is now steadily getting increased traction in the Industrial IoT space.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;margin-bottom:20px;\" src=\"/docs/blog/assets/2019-11-25/2019-11_StreamPipes-v0.png\" alt=\"1st prototype of the StreamPipes UI\"/>\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2019-11-25/2019-11_StreamPipes-now.png\" alt=\"1st prototype of the StreamPipes UI\"/>\r\n\r\n## Moving to the Apache Incubator\r\nEarlier this year, we discussed how to continue the development of StreamPipes. For the core team at FZI, our main goal was to widen the developer base, attract external developers and to make StreamPipes a real community-driven open source project.\r\nWe had a look at various open source foundations (namely Apache, CNCF and Eclipse), but from the beginning, there was no doubt that we wanted to see StreamPipes in the Apache Software Foundation. Using a lot of Apache Big Data projects under the hood ([Apache Kafka](https://kafka.apache.org), [Apache Flink](https://flink.apache.org) and others), an emerging Apache IoT ecosystem, and the community-driven, open development process of ASF projects, this was an easy decision.\r\nFinally, a few weeks before we went to ApacheCon North America in Las Vegas in September this year to talk about StreamPipes, we discovered the awesome [Apache PLC4X](https://plc4x.apache.org) project, which perfectly aligns with StreamPipes tackling the hard task to connect proprietary industrial protocols with modern software tooling.\r\nWe met Chris Dutz together with other people from PLC4X and other Apache projects in Las Vegas and were overwhelmed by the _Apache Way_ and the great collaboration between the various projects within the ASF.\r\n\r\nSo after coming back home, we immediately started to write the proposal to enter the Apache Incubator, which got accepted on 2019-11-11.\r\n\r\n## New project setup\r\nWith the great help from our mentors, we already performed some initial steps required to move our infrastructure over to the ASF, which will continue during the next week:\r\n\r\n* The new website will be soon available under [https://streampipes.apache.org](https://streampipes.apache.org)\r\n* We will move our code repositories over to the ASF on Github. Before that, we'll merge some repos to make future releases easier: The **backend and user interface** will be merged to the repository [https://www.github.com/apache/incubator-streampipes]([https://www.github.com/apache/incubator-streampipes). Pipeline elements and StreamPipes Connect adapters will also be merged to a new repo called **streampipes-extensions**, soon available at [https://www.github.com/apache/incubator-streampipes-extensions](https://www.github.com/apache/incubator-streampipes-extensions). Both **website and documentation** will be available at [https://www.github.com/apache/incubator-streampipes-website](https://www.github.com/apache/incubator-streampipes-website). Finally, the project **streampipes-installer** at [https://www.github.com/apache/incubator-streampipes-installer](https://www.github.com/apache/incubator-streampipes-installer) will contain the installer for Windows and OSX/Linux, the developer-oriented CLI tool and helm charts to install StreamPipes in Kubernetes.\r\n* Apache project communication mainly relies on mailing lists. Subscribe to the [user list](mailto:users-subscribe@streampipes.apache.org) and [dev list](mailto:dev-subscribe@streampipes.apache.org) and help us to improve StreamPipes!\r\n* Issues can be created in the new [Jira](https://issues.apache.org/jira/projects/STREAMPIPES).\r\n* All previous releases will be marked as non-ASF-releases to clearly indicate that previous releases do not follow the Apache release policy. Yesterday, we released the final pre-Apache version [0.65.0](/docs/blog/2019/11/23/release-0650), the next release will be our first Apache release. \r\n\r\n## Thanks\r\nWe would like to thank Chris Dutz for his great support in doing the first steps in the Apache Incubator.\r\n\r\nWe would also like to thank our employee, [FZI Research Center for Information Technology](https://www.fzi.de/en), for the continued great support - starting from giving us the opportunity to present StreamPipes at large fairs such as Hannover Messe, over media support such as creating the great StreamPipes logo, up to the fast legal support we received when we got accepted by the Apache Incubator.\r\nIf you want to build cool stuff in a great applied research institute while pursuing a PhD at KIT, FZI is surely the best place to be ;-)\r\n\r\nLet's continue development and transform Apache StreamPipes (incubating) into the next Apache community!"
},
{
"id": "/2019/11/23/release-0650",
"metadata": {
"permalink": "/blog/2019/11/23/release-0650",
"source": "@site/blog/2019-11-23-release-0650.md",
"title": "StreamPipes release 0.65.0",
"description": "5 minutes to read",
"date": "2019-11-23T00:00:00.000Z",
"formattedDate": "November 23, 2019",
"tags": [],
"readingTime": 2.145,
"hasTruncateMarker": true,
"authors": [
{
"name": "Dominik Riemer",
"url": "https://twitter.com/dominikjriemer",
"imageURL": "/docs/img/riemer.png"
}
],
"frontMatter": {
"title": "StreamPipes release 0.65.0",
"author": "Dominik Riemer",
"authorURL": "https://twitter.com/dominikjriemer",
"authorImageURL": "/docs/img/riemer.png"
},
"prevItem": {
"title": "Moving to the Apache Incubator",
"permalink": "/blog/2019/11/25/moving-to-the-apache-incubator"
},
"nextItem": {
"title": "StreamPipes release 0.64.0",
"permalink": "/blog/2019/09/19/release-0640"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">5 minutes to read</div>**\r\n<br/>\r\n\r\nWe are happy to announce StreamPipes version 0.65.0! This release brings performance improvements related to the installation process and further reduces the minimum hardware requirements to run StreamPipes.\r\n<!--truncate-->\r\n\r\n## Installation\r\n\r\nIf you're installing StreamPipes using the provided installer, you might have noticed that the initial setup takes some time: Docker images are downloaded, all dependent microservices are started and the RDF-based database that we use internally to store pipeline elements is initialized.\r\nOn less-performant hardware, this process could take a few minutes. For this release, we optimized the installation process:\r\n\r\n* You don't need to provide your IP address during the installation process anymore - StreamPipes now runs without manually providing an IP address.\r\n* A new initialization screen is shown in the StreamPipes user interface that automatically disappears and guides you to the setup process once all mandatory services are running.\r\n* The setup process in the user interface should now be much faster due to some changes related to the caching of pipeline elements.\r\n\r\n## Lite version\r\n\r\nTo further reduce the minimum hardware requirements to run StreamPipes, we added a new Docker container to the backend that includes all data processors and sinks that depend on the lightweight JVM wrapper. Flink and other modules are excluded.\r\nThis approach allows to test StreamPipes with less memory (e.g., on a laptop) while still having a larger set of pipeline elements installed. \r\nWe now recommend a minimum of 4GB RAM for the StreamPipes Lite version (make sure that the RAM is also assigned to Docker Desktop), but our tests have shown that the memory consumption of the StreamPipes lite version is around 2GB.\r\n\r\n## Pipeline Elements\r\n\r\n* 0.65.0 includes some new data processors: A CSV enrichment processors that allows to enrich events with metadata from CSV files, an event counter and some processors that perform operations on boolean values, e.g., coming from PLCs.\r\n\r\n## Other features and improvements\r\n\r\n* Use the latest Flink version 1.9.1\r\n* We improved the OPC-UA adapter, e.g., you can now directly set the URL of the OPC-UA server.\r\n* Proper error messages are now shown in the pipeline view in case something goes wrong during the start of a pipeline.\r\n\r\nSee the [release notes](https://github.com/apache/incubator-streampipes/releases/tag/0.65.0) for a complete list of new features and improvements.\r\n\r\n\r\n## Feedback\r\n\r\nWe are absolutely open to your suggestions for further improvements! Let us know (by [mail](mailto:feedback@streampipes.org), [slack](https://slack.streampipes.org) or [twitter](https://www.twitter.com/streampipes)) and we'll consider your feature request in the next release!\r\n\r\nAnd do not forget to subscribe to our new mailing lists: https://streampipes.apache.org/mailinglists.html"
},
{
"id": "/2019/09/19/release-0640",
"metadata": {
"permalink": "/blog/2019/09/19/release-0640",
"source": "@site/blog/2019-09-19-release-0640.md",
"title": "StreamPipes release 0.64.0",
"description": "5 minutes to read",
"date": "2019-09-19T00:00:00.000Z",
"formattedDate": "September 19, 2019",
"tags": [],
"readingTime": 2.105,
"hasTruncateMarker": true,
"authors": [
{
"name": "Dominik Riemer",
"url": "https://twitter.com/dominikjriemer",
"imageURL": "/docs/img/riemer.png"
}
],
"frontMatter": {
"title": "StreamPipes release 0.64.0",
"author": "Dominik Riemer",
"authorURL": "https://twitter.com/dominikjriemer",
"authorImageURL": "/docs/img/riemer.png"
},
"prevItem": {
"title": "StreamPipes release 0.65.0",
"permalink": "/blog/2019/11/23/release-0650"
},
"nextItem": {
"title": "StreamPipes release 0.63.0",
"permalink": "/blog/2019/09/05/release-0630"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">5 minutes to read</div>**\r\n<br/>\r\n\r\nUsually, we try to publish a new StreamPipes release every three months. But after attending a very exciting [ApacheCon](https://www.apachecon.com/acna19/) last week, where we worked with some Apache communities on a few really cool new features, we decided to release these features as soon as possible.\r\nSo here's StreamPipes 0.64.0!\r\n<!--truncate-->\r\n\r\n## StreamPipes Connect\r\n\r\n* StreamPipes Connect now has a new (experimental) connector for [ApachePLC4X](https://plc4x.apache.org/). PLC4X is a (really cool) universal protocol adapter for the Industrial IoT with many adapters for systems like Modbus, Beckhoff, S7 and OPC-UA. The new StreamPipes Connect adapter allows you to easily connect to an S7 device, we will add more integrations in the upcoming releases.\r\n* We added another adapter to retrieve data from the [Apache Pulsar](https://pulsar.apache.org) messaging system.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2019-09-19/spconnect.png\" alt=\"PLC4X adapter for StreamPipes Connect\"/>\r\n\r\n## Pipeline Elements\r\n\r\n* 0.64.0 also includes a new data sink to send data to Apache Pulsar.\r\n\r\n## Deployment\r\n\r\nStreamPipes relies on a microservice-based architecture and therefore requires quite a few services (> 15 for the full version) to be up and running. \r\nThis has impact on the memory consumption of the server where StreamPipes is running. On the other hand, we want to make it as easy as possible to try StreamPipes even on laptops with less powerful hardware. \r\n\r\nHowever, the lite version still required > 8G memory and the full version even more. Additionally, after the last release, we received feedback from the community indicating that the memory consumption has significantly increased.\r\nSo we looked deeper into the issue and discovered that the Docker base images we were using to deliver the StreamPipes services caused high memory consumption.\r\n\r\nBefore StreamPipes 0.63.0, we used the Alpine Oracle JDK image for most services. In 0.63.0, we switched to an OpenJDK/Alpine distribution. This had an enormous effect on memory, having individual services that reserved more than 1.5GB of memory.\r\n\r\nSo in this version, we switched to AdoptOpenJDK along with [OpenJ9](https://www.eclipse.org/openj9/). The results are fantastic: The full version including all pipeline elements now needs only 6GB memory (compared to > 16 in the last version).\r\n\r\nThe screenshot below shows that StreamPipes now is much less resource hungry:\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2019-09-19/memory.png\" alt=\"PLC4X adapter for StreamPipes Connect\"/>\r\n\r\nIn future versions, we will continue our efforts to decrease the memory consumption of StreamPipes.\r\n\r\n## Feedback\r\n\r\nWe are absolutely open to your suggestions for further improvements! Let us know (by [mail](mailto:feedback@streampipes.org), [slack](https://slack.streampipes.org) or [twitter](https://www.twitter.com/streampipes)) and we'll consider your feature request in the next release!"
},
{
"id": "/2019/09/05/release-0630",
"metadata": {
"permalink": "/blog/2019/09/05/release-0630",
"source": "@site/blog/2019-09-05-release-0630.md",
"title": "StreamPipes release 0.63.0",
"description": "5 minutes to read",
"date": "2019-09-05T00:00:00.000Z",
"formattedDate": "September 5, 2019",
"tags": [],
"readingTime": 3.835,
"hasTruncateMarker": true,
"authors": [
{
"name": "Dominik Riemer",
"url": "https://twitter.com/dominikjriemer",
"imageURL": "/docs/img/riemer.png"
}
],
"frontMatter": {
"title": "StreamPipes release 0.63.0",
"author": "Dominik Riemer",
"authorURL": "https://twitter.com/dominikjriemer",
"authorImageURL": "/docs/img/riemer.png"
},
"prevItem": {
"title": "StreamPipes release 0.64.0",
"permalink": "/blog/2019/09/19/release-0640"
},
"nextItem": {
"title": "StreamPipes release 0.62.0",
"permalink": "/blog/2019/05/23/release-0620"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">5 minutes to read</div>**\r\n<br/>\r\n\r\nStreamPipes version 0.63.0 has been released! Following our release plan to alternate between user-faced and backend-oriented releases, this release contains many (more than 70) features and improvements, mainly on the backend side.\r\nThe new version is available on Docker Hub and can be downloaded using our [installer](https://github.com/apache/incubator-streampipes-installer).\r\n<!--truncate-->\r\n\r\nFor this release, we focused on improving the stability and resilience of StreamPipes, adding many new features \"under the hood\".\r\n\r\n## Backend & Data Model\r\n\r\n* First, v0.63.0 brings **support for binary messaging formats**. Prior to 0.63.0, the only messaging format used at runtime was JSON. Now, several other binary formats are supported: Smile and CBOR as binary JSON representations and FST, a fast Java-based serializer.\r\n* Second, we improved **resilience of pipelines**. Now, when StreamPipes is stopped, all running pipelines are stopped beforehand and can be restarted later on, so that you don't end up with orphaned pipelines anymore. \r\n\r\nWe also extended the data model and SDK with new features:\r\n\r\n* Pipeline elements can now provide their own assets such as documentation. Simply add an icon and a markdown file containing the documentation of the pipeline element and both will be visible directly in the pipeline editor after the installation of the pipeline element.\r\n* We added several new static properties to define required user input: **SecretStaticProperties** can be used to define secret values (e.g., passwords). In addition, **StaticPropertyAlternatives** were introduced to let users select between multiple combinations of static properties (e.g., a time window and a count window along with window settings).\r\n\r\nIn addition, we migrated all pipeline element container and the backend to Spring Boot applications. The triple store used to store pipeline element descriptions has been extracted to an own microservice. Btw, have you seen the new [helm chart](https://www.github.com/streampipes/streampipes-helm-chart) that simplifies running StreamPipes on Kubernetes?\r\n \r\n## UI and Pipeline Editor\r\nWe improved the **interactive user guide**, so that it now depends on pipeline elements that are also available in the lite version.\r\nThree different interactive guides exist: One that teaches you how to create pipelines, another one that illustrates how to connect data with StreamPipes connect and a third one that shows how to create live visualizations using the built-in dashboard.\r\n\r\nAs **user feedback** is absolutely important to help us improving StreamPipes, we added a feature that allows users to directly give feedback from the StreamPipes UI. Simply click the feedback icon in the top navigation bar and submit your anonymous feedback to us! \r\n\r\nFinally, there are also some functional improvements in the pipeline editor: **Pipeline validation** has improved and gives direct feedback. And, finally, you can now also add multiple dashboards sink to a single pipeline :-)\r\n\r\nAnd a nice feature you can't see but which you'll probably feel: All UI files are now **Gzip compressed** by default, so that StreamPipes will now load much faster. \r\n\r\n## Pipeline Elements\r\n\r\nStreamPipes 0.63.0 includes several new pipeline elements:\r\n* A sink for Apache IoTDB\r\n* A sink for PostgreSQL\r\n* A processor to convert boolean values\r\n* A processor to extract timestamps\r\n* A processor to compute trigonometric functions\r\n\r\n## StreamPipes Connect\r\nWe added more adapters and improved some adapters to StreamPipes Connect:\r\n\r\n* An improved MQTT adapter that can handle authentication\r\n* A new MySQL adapter\r\n* An improved OPC-UA adapter\r\n* A new random data generator that eases testing of pipeline elements\r\n\r\nIn addition, we completely reworked the Connect UI. The schema view now lets you edit events more conveniently and timestamp fields can be directly marked in the UI.\r\n\r\n\r\n## Documentation\r\nWe updated and restructured the user guide, which now consists of four parts: [Introduction](/docs/user-guide-introduction), [Tour](/docs/user-guide-tour), [Installation](/docs/user-guide-installation) and [First Steps](/docs/user-guide-first-steps).\r\nWe also updated all screenshots to reflect the current look and feel of StreamPipes.\r\n\r\nIn addition, the developer guide was further extended (e.g., there is now a new tutorial on creating data sinks). Maven archetypes are now the recommended way to create new pipeline elements.\r\n\r\n\r\n## Other features and improvements\r\n\r\n* An improved installer that lets you choose between two StreamPipes versions for different hardware setups (full or lite, previously named big data and desktop)\r\n* We updated some core libraries like Apache Flink and Apache Kafka to newer versions.\r\n* Improved support for Firefox and Edge: Besides Chrome, StreamPipes should also behave and look similar in Firefox and Edge browsers.\r\n\r\nSee the [release notes](https://github.com/apache/incubator-streampipes/releases/tag/0.63.0) for a complete list of new features and improvements.\r\n\r\n## Feedback\r\n\r\nWe are absolutely open to your suggestions for further improvements! Let us know (by [mail](mailto:feedback@streampipes.org), [slack](https://slack.streampipes.org) or [twitter](https://www.twitter.com/streampipes)) and we'll consider your feature request in the next release!"
},
{
"id": "/2019/05/23/release-0620",
"metadata": {
"permalink": "/blog/2019/05/23/release-0620",
"source": "@site/blog/2019-05-23-release-0620.md",
"title": "StreamPipes release 0.62.0",
"description": "6 minutes to read",
"date": "2019-05-23T00:00:00.000Z",
"formattedDate": "May 23, 2019",
"tags": [],
"readingTime": 2.875,
"hasTruncateMarker": true,
"authors": [
{
"name": "Dominik Riemer",
"url": "http://twitter.com/dominikjriemer",
"imageURL": "/docs/img/riemer.png"
}
],
"frontMatter": {
"title": "StreamPipes release 0.62.0",
"author": "Dominik Riemer",
"authorURL": "http://twitter.com/dominikjriemer",
"authorImageURL": "/docs/img/riemer.png"
},
"prevItem": {
"title": "StreamPipes release 0.63.0",
"permalink": "/blog/2019/09/05/release-0630"
},
"nextItem": {
"title": "StreamPipes release 0.61.0",
"permalink": "/blog/2019/04/04/release-0610"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">6 minutes to read</div>**\r\n<br/>\r\n\r\nStreamPipes version 0.62.0 has been released! While the last version brought many features under the hood, this new version totally focuses on features and improvements visible to end users.\r\nThe new version is available on Docker Hub and can be downloaded using our [installer](https://github.com/streampipes/streampipes-installer).\r\n<!--truncate-->\r\n\r\nFor this release, we focused on improving error handling in the pipeline editor, added more pipeline elements, added more adapters to StreamPipes Connect and, added a documentation on available pipeline elements.\r\n\r\n## Pipeline Elements \r\nPipeline elements are the most important part of the StreamPipes toolbox. Although pipeline elements can be easily developed using our SDK, we aim to provide as many ready-to-use elements as possible to provide you with meaningful data harmonization and analytics features right from the beginning.\r\nStreamPipes v0.62.0 brings many new data processors and data sinks, an overview can be found [here](/docs/pipeline-elements). \r\n\r\n## Pipeline Editor\r\nTo better support users while developing pipelines, we improved the verification of user input in the pipeline editor.\r\nUsers now get hints (directly shown in the pipeline assembly) on missing pipeline elements and error messages are shown if invalid values are entered while configuring pipeline elements (see the screenshot below).\r\nIn addition, the editor now disallows adding new pipeline elements until the active pipeline element is fully configured, otherwise, the element is marked with a red alert symbol, which makes it much easier to discover modeling errors.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2019-05-23/pipeline-editor.png\" alt=\"Pipeline Element Verification\"/>\r\n\r\nOne feature request which was coming from one of our users was to better understand what pipeline elements are doing. For that reason, we now added a feature that allows to see a full documentation of any pipeline element directly in the pipeline editor.\r\nOnce users start to configure a new pipeline element, the documentation can be opened:\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2019-05-23/pipeline-element-documentation.png\" alt=\"Pipeline Element Documentation\"/>\r\n\r\n\r\n## StreamPipes Connect\r\nWe added more adapters to StreamPipes Connect. \r\nBesides generic adapters such as frequently used protocols like OPC UA, MQTT and Kafka, Connect now also includes some new open data adapters and adapters for web APIs to make starting with StreamPipes more funny and easier:\r\n\r\n* A Slack source to connect Slack channels with StreamPipes\r\n* A Coindesk sources that allows you to get the latest Bitcoin rates\r\n* A Wikipedia sources to get latest edits and created articles from Wikipedia\r\n* IEXCloud adapters to provide access to stock quotes and news from IEXCloud (an API token is required).\r\n* A ROS source to access data from the Robot Operating System (ROS)\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2019-05-23/connect.png\" alt=\"StreamPipes Connect\"/>\r\n\r\n\r\n## Documentation\r\nThe documentation for pipeline elements that is shown in the pipeline editor is now also available in the online documentation.\r\nNavigate to [pipeline elements](/docs/pipeline-elements) to see an overview of all pipeline elements that are currently provided by StreamPipes.\r\n\r\n## Other features and improvements\r\n\r\n* Adapters can be sorted and filtered in StreamPipes Connect.\r\n* Labels and descriptions of pipeline elements can be provided by external resources files as a first step towards internationalization.\r\n* We upgraded the UI to the latest Angular version 7 and drastically reduced the size of the UI source bundle, which reduces loading time.\r\n* Stream requirements are now correctly ordered\r\n\r\nSee the [release notes](https://github.com/streampipes/streampipes/releases/tag/0.62.0) for more details.\r\n\r\n## Feedback\r\n\r\nWe are absolutely open to your suggestions for further improvements! Let us know (by [mail](mailto:feedback@streampipes.org), [slack](https://slack.streampipes.org) or [twitter](https://www.twitter.com/streampipes)) and we'll consider your feature request in the next release!"
},
{
"id": "/2019/04/04/release-0610",
"metadata": {
"permalink": "/blog/2019/04/04/release-0610",
"source": "@site/blog/2019-04-04-release-0610.md",
"title": "StreamPipes release 0.61.0",
"description": "5 minutes to read",
"date": "2019-04-04T00:00:00.000Z",
"formattedDate": "April 4, 2019",
"tags": [],
"readingTime": 3.545,
"hasTruncateMarker": true,
"authors": [
{
"name": "Dominik Riemer",
"url": "http://twitter.com/dominikjriemer",
"imageURL": "/docs/img/riemer.png"
}
],
"frontMatter": {
"title": "StreamPipes release 0.61.0",
"author": "Dominik Riemer",
"authorURL": "http://twitter.com/dominikjriemer",
"authorImageURL": "/docs/img/riemer.png"
},
"prevItem": {
"title": "StreamPipes release 0.62.0",
"permalink": "/blog/2019/05/23/release-0620"
},
"nextItem": {
"title": "ContainerCLI: A CLI tool for your Docker Application",
"permalink": "/blog/2018/11/24/container-cli"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">5 minutes to read</div>**\r\n<br/>\r\n\r\nWe are proud to release StreamPipes version 0.61.0, which brings many improvements and some exciting new features that simplify pipeline development!\r\nThe new version, which is already available on Docker Hub and can be downloaded using our [installer](https://github.com/streampipes/streampipes-installer) brings over 20 improvements. Under the hood, a reworked event model simplifies the development of new pipeline elements using the built-in SDK.\r\n<!--truncate-->\r\n\r\nOur main mission is to make IoT data management & analytics accessible to everyone.\r\nThis release focuses on better integrating StreamPipes with other systems and data sources and contains UX improvements to ease the development of pipelines.\r\n\r\n## StreamPipes Connect\r\nStreamPipes Connect, which aims to easily connect arbitrary data sources with StreamPipes, has been first introduced as a public beta in version 0.60.0.\r\nVersion 0.61.0 brings many new features to Connect and also implements many bug fixes that now make Connect the recommended way to connect data sources.\r\n\r\nStreamPipes Connect relies on an extensible architecture to connect either generic protocols or specific adapters.\r\n\r\nIn more detail, StreamPipes Connect supports the following protocols, with a completely code-free integration process:\r\n\r\n* Data Streams\r\n * HTTP/REST\r\n * MQTT\r\n * Apache Kafka\r\n * File\r\n * ROS (preview)\r\n * GDELT\r\n\r\n* Data Sets\r\n * HTTP/REST\r\n * HDFS\r\n * File\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2019-04-04/connect.png\" alt=\"StreamPipes Connect\"/>\r\n\r\nThe next version will also include an adapter for OPC UA, which is currently being tested in a real setting.\r\n\r\nFor protocols that are not yet supported by StreamPipes Connect, a new Maven archetype for easily connecting data sources using the [SDK](/docs/dev-guide-tutorial-sources) is provided.\r\n\r\n## Improved Pipeline Modeling\r\n\r\nOn the UI side, we try to make the graphical modeling of pipelines as simple as possible. Besides some minor improvements (e.g., case-insensitive search for pipeline elements), pipelines are now constantly validated and users get feedback on steps that are still missing (e.g., missing pipeline elements).\r\nAs many other features we are currently implementing, this feature directly comes from a user request - so if you have any suggestions for improvement, let us know!\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2019-04-04/validation.png\" alt=\"Pipeline Validation\"/>\r\n\r\n\r\n## Reworked Event Model\r\n\r\nOne of the most important features of v0.61.0 is not directly visible to end users, but drastically eases the development of new pipeline elements with the SDK.\r\nFrom this release on, events at runtime are represented in form of a new ``Event Object`` instead of the former ``Map`` representation.\r\nThis feature allows us for much better support of complex data processors which have more than one input data stream and comes with many convenience methods to better extrat fields from an event for processing.\r\n\r\nFor instance, prior to v0.61.0 you had to extract fields from an incoming events as follows:\r\n\r\n```java\r\n@Override\r\n public void onEvent(Map<String, Object> in, String s, SpOutputCollector out) {\r\n Integer numberValue = Integer.parseInt(String.valueOf(in.get(MAPPING_PROPERTY_ID));\r\n }\r\n```\r\n\r\nThe new event model is much simpler and less error-prone:\r\n\r\n```java\r\n public void onEvent(Event in, SpOutputCollector out) throws SpRuntimeException {\r\n Integer numberValue = in\r\n .getFieldBySelector(MAPPING_PROPERTY_SELECTOR)\r\n .getAsPrimitive()\r\n .getAsInteger();\r\n }\r\n```\r\n\r\nThe new event model is even more useful in cases where your data streams have a complex nested structure or contain list fields.\r\nIf you connect two data streams containing the same fields to a processor, field names are automatically renamed in the background to avoid naming conflicts.\r\nA complete guide can be found in the [developer guide](/docs/dev-guide-event-model).\r\n\r\n## Documentation\r\n\r\nThe developer guide has been extended:\r\nBesides reworked tutorials on developing new data processors and data sources, the new documentation includes instructions on adding [stream requirements](/docs/dev-guide-stream-requirements), the new [event model](/docs/dev-guide-event-model), and in-depth guides on defining [static properties](/docs/dev-guide-static-properties) and [output strategies](/docs/dev-guide-output-strategies).\r\n\r\n## Other features and improvements\r\n\r\n* SSL support: The UI can now be configured to support SSL, see the [documentation](/docs/dev-guide-ssl) on required configurations in your docker-compose file.\r\n* A new map visualization to visualize spatial data streams\r\n* The customize dialog is now hidden in case no static properties are present\r\n* Adapter descriptions can be exported and imported\r\n* The UI does no longer load any external sources\r\n* Updated Maven archetypes\r\n\r\nSee the [release notes](https://github.com/streampipes/streampipes/releases/tag/0.61.0) for more details.\r\n\r\n## Feedback\r\n\r\nWe are absolutely open to your suggestions for further improvements! Let us know (by [mail](mailto:feedback@streampipes.org), [slack](https://slack.streampipes.org) or [twitter](https://www.twitter.com/streampipes)) and we'll consider your feature request in the next release!"
},
{
"id": "/2018/11/24/container-cli",
"metadata": {
"permalink": "/blog/2018/11/24/container-cli",
"source": "@site/blog/2018-11-24-container-cli.md",
"title": "ContainerCLI: A CLI tool for your Docker Application",
"description": "6 minutes to read",
"date": "2018-11-24T00:00:00.000Z",
"formattedDate": "November 24, 2018",
"tags": [],
"readingTime": 3.895,
"hasTruncateMarker": true,
"authors": [
{
"name": "Philipp Zehnder",
"url": "http://twitter.com/philipp10der",
"imageURL": "/docs/img/zehnder.png"
}
],
"frontMatter": {
"title": "ContainerCLI: A CLI tool for your Docker Application",
"author": "Philipp Zehnder",
"authorURL": "http://twitter.com/philipp10der",
"authorImageURL": "/docs/img/zehnder.png"
},
"prevItem": {
"title": "StreamPipes release 0.61.0",
"permalink": "/blog/2019/04/04/release-0610"
},
"nextItem": {
"title": "Preview: StreamPipes release 0.60.0",
"permalink": "/blog/2018/09/17/preview-060"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">6 minutes to read</div>**\r\n<br/>\r\n\r\nIn this blog post, we show how we developed a CLI tool to install and manage StreamPipes.\r\nIn addition, we present a template project that enables you to easily create a CLI tool for your own docker application.\r\nAll you need is a working docker-compose.yml file.\r\n\r\n<!--truncate-->\r\n\r\nStreamPipes has a Microservice architecture and we use Docker and Docker Compose for all of our development, testing, and deployment processes.\r\nIn the beginning, we developed the CLI only to ease the installation process (e.g. managing multiple versions), but over time we realized that the tool is much more powerful.\r\nFor example, StreamPipes has multiple developer roles: Some developers are working on the frontend, others on the backend, and some are developing new algorithms and data sinks for StreamPipes.\r\nIn the figure below, you can see different configurations that are used by different developers.\r\nAll services are running in docker, except for the ones the developer is currently working on.\r\nAs a result, we had many different docker-compose.yml files and when we made changes, we had to synchronize them all.\r\n\r\n<img class=\"blog-image\" style=\"max-width:60%;\" src=\"/docs/blog/assets/2018-11-24/roles.png\" alt=\"Three different developer roles\"/>\r\n\r\nTherefore, our goal was to speed up the setup of a development environment, so that developers can focus on their specific programming task.\r\nThat’s why we developed a command line tool that automates the setup of different environments.\r\nAll what is needed is a docker-compose.yml file that describes all services of the application and the template CLI project from [GitHub](https://github.com/streampipes/container-cli).\r\nAfterwards, you need to split up your compose file and create an individual file for each service. Put those files in individual folders in the services folder.\r\nEach folder should have the same name as the service name in the docker-compose file. That’s it!\r\nNow you can use the CLI.\r\nWith the default commands you are able to start, stop, and configure the application. But how is this better than just using docker itself?\r\nThe advantage is the flexibility and extensibility of the tool.\r\nYou can extend the script with tasks you have to perform regularly, like cleaning up volumes, or setting up testing environments.\r\n\r\n\r\nIn the following we will describe the tool in more detail.\r\nYou can see all files and folders of the CLI Tool in the next listing.\r\nThe first file “container-cli.m4” contains the script.\r\nWe use [Argbash](https://argbash.io/), which eases the use of bash script parameters and flags.\r\nIn this file you can easily add new custom commands, flags or other functionalities.\r\nThe next file is the actual bash script. It is generated by argbash.\r\nSince this file is generated automatically, you should not change anything.\r\nThe docker-compose.yml file is the main file.\r\nThis should contain all services, networks, or volume information that is relevant for all the other services of your application.\r\nAll your other services are located in the services folder.\r\nFor each service, create a folder with the name of the service and place a docker-compose.yml in the folder configuring the service.\r\nTherefore, you can copy and paste the configuration from your current docker-compose.yml.\r\nThe script checks the services folder dynamically, this means you can add services at a later point in time without changing anything else.\r\nIn our example, we have four services: Backend, UI, database, and a test database containing data for the end2end tests.\r\nWhich services should be started when running the command **\"container-cli start\"** is defined in the active-service file.\r\nThis file contains the names of all services, each in one line, that should be started.\r\nOften it is required to have multiple configurations of your app. You can save them in the templates folder.\r\nBy running **“container-cli set-template ui_developer”**, you can select the template ui_developer.\r\nWith this command, all services defined in the template are written to the active-services file. (e.g. the ui_developer template contains just the database and backend and the e2e-test template contains the backend, ui, and test-database).\r\n\r\n\r\n<img class=\"blog-image\" style=\"max-width:30%;\" src=\"/docs/blog/assets/2018-11-24/files.png\" alt=\"Structure of the containter cli project\"/>\r\n\r\n\r\nThe last file is the tmpl_env file.\r\nIt contains configuration variables for the individual services.\r\nThose variables can also be changed by the bash script dynamically, because the tmpl_env file is moved to the .env file in the same folder.\r\nWe use the configuration for example to dynamically set the IP Address of the host system or set the application version.\r\nTo use a variable in a docker-compose file just write it in braces, as shown below.\r\n\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2018-11-24/variables.png\" alt=\"How to use Environment Variables\"/>\r\n\r\nWe hope this tool is helpful for your use case.\r\nWhen you have any problems, feedback, or ideas for improvements, contact us or open an issue on [GitHub](https://github.com/streampipes/container-cli)."
},
{
"id": "/2018/09/17/preview-060",
"metadata": {
"permalink": "/blog/2018/09/17/preview-060",
"source": "@site/blog/2018-09-17-preview-060.md",
"title": "Preview: StreamPipes release 0.60.0",
"description": "5 minutes to read",
"date": "2018-09-17T00:00:00.000Z",
"formattedDate": "September 17, 2018",
"tags": [],
"readingTime": 3.37,
"hasTruncateMarker": true,
"authors": [
{
"name": "Dominik Riemer",
"url": "http://twitter.com/dominikjriemer",
"imageURL": "/docs/img/riemer.png"
}
],
"frontMatter": {
"title": "Preview: StreamPipes release 0.60.0",
"author": "Dominik Riemer",
"authorURL": "http://twitter.com/dominikjriemer",
"authorImageURL": "/docs/img/riemer.png"
},
"prevItem": {
"title": "ContainerCLI: A CLI tool for your Docker Application",
"permalink": "/blog/2018/11/24/container-cli"
},
"nextItem": {
"title": "How to understand your machine data?",
"permalink": "/blog/2018/06/18/how-to-understand-your-machine-data"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">5 minutes to read</div>**\r\n<br/>\r\n\r\nIt's been a while since our last blog post. However, in the background we are working hard to release the next version of StreamPipes which is currently in the final phase.\r\nIn this post, we'd like to give you a preview of the upcoming StreamPipes version 0.60.0, which brings many new features and improvements.\r\n\r\n<!--truncate-->\r\n\r\nOur first public release is now about 6 months ago. In the meantime, we gathered lots of feedback we got from users concerned with the installation, available pipeline elements and usability.\r\nWe learned what users expect from the tool and really tried to implement feature suggestions from the community.\r\n\r\nHere are the major things you'll find in the next version which will be released by the end of September:\r\n\r\n## Installation and first-time usage\r\nWe aim to make it as easy as possible for first-time users to try and test StreamPipes. Although we have provided Docker images and a docker-compose file from the beginning, we've learned that our target group looks for an easier way to install the tool, without the need to manually configuring things like ports or hostnames.\r\nFor that reason, we have implemented an installer that is already on [Github](https://www.github.com/streampipes/streampipes-installer). With the next release, you'll find an even easier installer along with a dev-installer targeted at more advanced users that allows to configure the services and pipeline element containers you'd like to use in your StreamPipes instance.\r\n\r\nIn addition, we wanted to provide better support for users that are trying to build a pipeline for the first time. Although we provide a tour and a user guide, we are aware that nobody likes reading documentations ;-)\r\nTherefore, the next release includes an interactive product tour that guides you through the first steps with StreamPipes and explains how to build pipelines - right in the pipeline editor!\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2018-09-17/streampipes-060-product-tour.png\" alt=\"Product Tour\"/>\r\n\r\n## StreamPipes Connect\r\n\r\nAlthough the SDK provides a developer-oriented way to connect new data sources, many users have asked us to provide an easier way to connect sources and data streams to StreamPipes.\r\nThe next release includes a beta release of StreamPipes Connect. Connect is a major effort that integrates work we've done over the past year and provides an easy way to integrate your data streams right from the UI.\r\nA data marketplace contains a list of pre-defined adapters (which can be rather generic by selecting a protocol and a data format or quite specific as, e.g., an openSenseMap source).\r\nThese adapters can be configured according to your needs and directly started on a set of worker nodes.\r\n\r\nStreamPipes Connect also lays the foundation for some cool new features that make it even easier to connect sources and will be added over the next months.\r\nIn addition, pre-processing features will be integrated that will support edge computing use cases.\r\n\r\nBe aware that StreamPipes Connect will be released in a rather early version, so don't use it for production yet.\r\n\r\n<img class=\"blog-image\" style=\"max-width:90%;\" src=\"/docs/blog/assets/2018-09-17/streampipes-060.png\" alt=\"StreamPipes Connect\"/>\r\n\r\n## Pipeline Elements\r\n\r\nPipeline elements (data processors and sinks) are the part of StreamPipes that generate added-value.\r\nWhile the first release contained a set of around 10 pipeline elements, the next release will provide many new elements that are already integrated.\r\nNew elements will include pattern detection algorithms, geospatial operators, image processing algorithms (e.g., a generic classifier) and even a Slack sink!\r\n\r\nIn total, the next version includes more than 40 pipeline elements and we will add many more during the next months.\r\nThe new pipeline elements are already available on [Github](https://www.github.com/streampipes/streampipes-pipeline-elements) and will be released as Docker images by the end of September.\r\n\r\n## Other features and improvements\r\n\r\n* A new wrapper for the [Siddhi](https://github.com/wso2/siddhi) CEP engine\r\n* various bug fixes and stability improvements\r\n* many UX improvements\r\n* many more, stay tuned for the next blog post!\r\n\r\n## Feedback\r\n\r\nWe are absolutely open to your suggestions for further improvements! Let us know (by [mail](mailto:feedback@streampipes.org), [slack](https://slack.streampipes.org) or [twitter](https://www.twitter.com/streampipes)) and we'll consider your feature request in the next release!"
},
{
"id": "/2018/06/18/how-to-understand-your-machine-data",
"metadata": {
"permalink": "/blog/2018/06/18/how-to-understand-your-machine-data",
"source": "@site/blog/2018-06-18-how-to-understand-your-machine-data.md",
"title": "How to understand your machine data?",
"description": "8 minutes to read",
"date": "2018-06-18T00:00:00.000Z",
"formattedDate": "June 18, 2018",
"tags": [],
"readingTime": 6.69,
"hasTruncateMarker": true,
"authors": [
{
"name": "Philipp Zehnder",
"url": "http://twitter.com/philipp10der",
"imageURL": "/docs/img/zehnder.png"
}
],
"frontMatter": {
"title": "How to understand your machine data?",
"author": "Philipp Zehnder",
"authorURL": "http://twitter.com/philipp10der",
"authorImageURL": "/docs/img/zehnder.png"
},
"prevItem": {
"title": "Preview: StreamPipes release 0.60.0",
"permalink": "/blog/2018/09/17/preview-060"
},
"nextItem": {
"title": "Welcome to StreamPipes!",
"permalink": "/blog/2018/06/14/welcome"
}
},
"content": "**<div style=\"float: left; padding-right: 40px;\">8 minutes to read</div>**\r\n<br/>\r\n\r\n\r\nData is the new oil. Probably everybody of you has heard or read this sentence before.\r\nIt is often stated how important data and data understanding is for companies today.\r\nBut what does this actually mean? How does the data look like that is produced by your machines and systems?\r\nAnd how can we solve the big challenge to turn this raw data into insights that can be understood by humans?\r\n\r\n<!--truncate-->\r\n\r\nWhen it comes to data analytics people often talk about the big players like Google, Facebook, or Netflix which collect a lot of data about their users and their usage behavior.\r\nThe core of their business model is to use data and turn it into profit. Nowadays all companies have the requirement to analyze their data.\r\nBut since this was not originally part of their business model it is very challenging for them to catch up.\r\nTo gain insights from data, it is often referred to advanced analytics and machine learning. Before such techniques can be used some work must be done.\r\nA first, basic part of that is to understand your data. We do that in four steps, as can be seen in figure below.\r\nFirst data sources must be accessed, then they can be integrated and for example be stored in a data lake.\r\nAfter that, we employ rule-based analytics to find patterns and situations.\r\nIn the end, machine learning and advanced analytics algorithms can be used to get more out of the data.\r\nThe idea behind those steps is to generate value as early as possible by collecting the low-hanging fruits first.\r\nWe develop methodologies and tools for the individual steps to also allow domain experts to perform them to bridge the gap between the big tech companies and more traditional industries.\r\n\r\n<img class=\"blog-image\" style=\"max-width:50%;\" src=\"/docs/blog/assets/2018-06-18/01_motivation.png\" alt=\"Motivation Graphic\"/>\r\n\r\nThe questions we try to answer in these blog posts are:\r\n\r\n* **How does data produced by machines look like?**\r\n* **How can machines and other data sources be accessed?**\r\n* **How is it possible to process the continuously produced flow of data?**\r\n\r\nWe illustrate our ideas with an example which we also use during the whole blog series.\r\nThe example is a water circle of a cooling system. You can see such a system in the next image.\r\nIt contains multiple water tanks and pipes that connect those tanks.\r\nWithin the tanks and the pipes are multiple sensors, which measure for example the water level in the tank, the flow rate in the water pipe, or the temperature of the water.\r\nAll those sensors are very simple, they make iterative observations and send the sensed measurements to a computer system.\r\n\r\n<img class=\"blog-image\" style=\"max-width:70%;\" src=\"/docs/blog/assets/2018-06-18/02_anlage.png\" alt=\"Water Circle Image\"/>\r\n\r\nBefore we go into detail, we want to explain the term streaming data, because we focus on machine and sensor data which is continually produced in a streaming fashion.\r\nThe opposite of streaming data are fixed sized data sets, called batch data.\r\nSince always new data is produced by sensors, it is not possible to wait till all data is completely produced and then process it.\r\nDifferent to that is batch data.\r\nThe advantage of batch data is, when the processing starts no new data is added anymore.\r\nStreaming data can also be processed with batch processing systems.\r\nTherefore, a data stream must be separated into fixed chunks which are then processed in a batch fashion.\r\nThis is possible but the more natural fit is to use streaming systems since they were designed to process streaming data.\r\n\r\nIn an environment where we process machine data we have different components.\r\nThe individual components exchange information in the form of **events**.\r\nAll assets continually producing data are called **data sources** (i.e. a sensor, machine or a software system).\r\nTo access these data sources, **adapters** are needed.\r\nAdapters are software components (which can run anywhere, for example directly on the machine or separately on a edge device near the machine) that are able to connect to the data source, extract the sensor information and emit events to the **middleware**.\r\nThe middleware is a central component which is an intermediate between the individual components.\r\nEven if the middleware is a central component, it is not a bottleneck and also not a single point of failure since it is often realized in form of a distributed system.\r\n**Processing systems** can access data from the middleware, transform it and apply algorithms.\r\nThis is the component where the data is analyzed and higher-level information is generated.\r\nIn the end, results can be stored in **data bases**.\r\nFrom there, other applications like for example dashboards can access the calculated results.\r\nThis is the high-level architecture of processing data streams.\r\n\r\n<img class=\"blog-image\" src=\"/docs/blog/assets/2018-06-18/03_architecture.png\" alt=\"Overall architecture\"/>\r\n\r\nOnce the events are integrated into a middleware, it must be processed to detect situations and generate higher-level events.\r\nThis can be done in an **Event Processing Network (EPN)**.\r\nAn EPN consists of one or multiple **data sources, event processors** transforming and detecting patterns in the data and finally **data sinks**, which can be data bases, alarms or other systems.\r\nBelow such a pipeline is shown where we have two sensors as data sources: A flow rate sensor measuring the flow rate in a water pipe and a water level sensor in a water tank.\r\nIt further contains three processing elements, each with a different functionality.\r\nThe first one detects increases in the water tank, the second one filters out values under a defined threshold.\r\nThe last of the processing elements detects when both situations occur within a defined time window.\r\nIn the end, we employ a data sink, which triggers a notification for a service technician once the modeled situation occurs.\r\nWe call such instances of EPNs processing **pipelines**.\r\n\r\n<img class=\"blog-image\" src=\"/docs/blog/assets/2018-06-18/04_pipeline.png\" alt=\"Example Pipeline\"/>\r\n\r\nThe pipeline in the image before contains three different processing elements.\r\nThose elements contain the actual algorithms.\r\nSuch algorithms can be as easy as simple filters on a sensor value or can also be more complex, such as patterns that occur over a period of time.\r\nBut they can also contain machine learning and advanced analytics algorithms (e.g. neural networks) that perform predictions on data streams.\r\nEvent processors are just an abstraction that take a data stream as an input and expose a “new” data stream.\r\nFurthermore, they contain configuration parameters which can be defined by a user.\r\nWith this concept, we can create reusable components that can be used in many different pipelines.\r\n\r\nThe goal of StreamPipes is to enable domain experts to do their data analysis themselves by providing tooling, where such processing pipelines can be modeled in a graphical user interface and then executed automatically without the need of an IT expert.\r\nProcessing engines which are used as the basis of the processing elements in our solution often require a higher technological understanding.\r\nIf this understanding is available, they can also be used on their own.\r\nWe use such systems, like Apache Flink, Apache Spark or the brand-new KSQL by Confluent, for the processing but provide an abstraction over them to enable domain experts to model pipelines themselves.\r\nThis has also the advantages that individual runtime solutions can be exchanged over time.\r\nFurthermore, we provide a semantic layer for all elements, including data sources, processing elements, and data sinks.\r\nWith that layer, our system is capable of understanding the meaning and further support the user to ensure that just semantically correct pipelines are created.\r\n\r\nIn this first blog post of our series, we gave a first introduction how data produced by machines and systems can be analyzed and value can be generated out of it.\r\nIn the following blog posts we explain the individual parts in more detail to provide you with enough knowledge to start analyzing your own data.\r\n\r\n* **[Part 1: Overview](/docs/blog/2018/06/18/how-to-understand-your-machine-data)**\r\n* Part 2: Data Sources\r\n* Part 3: Data Processing Elements\r\n* Part 4: Data Sinks\r\n* Part 5: Putting it all together (Example Pipeline)\r\n* Part 6: Practical Tips"
},
{
"id": "/2018/06/14/welcome",
"metadata": {
"permalink": "/blog/2018/06/14/welcome",
"source": "@site/blog/2018-06-14-welcome.md",
"title": "Welcome to StreamPipes!",
"description": "Welcome to our new blog. StreamPipes is the new, easy-to-use self-service data analytics toolbox for the Industrial IoT. In the coming weeks, we'll publish a series of blog posts that explain some ideas of StreamPipes. You'll read about use cases, best practices as well as technical details.",
"date": "2018-06-14T00:00:00.000Z",
"formattedDate": "June 14, 2018",
"tags": [],
"readingTime": 0.245,
"hasTruncateMarker": false,
"authors": [
{
"name": "Dominik Riemer",
"url": "http://twitter.com/dominikjriemer",
"imageURL": "/docs/img/riemer.png"
}
],
"frontMatter": {
"title": "Welcome to StreamPipes!",
"author": "Dominik Riemer",
"authorURL": "http://twitter.com/dominikjriemer",
"authorImageURL": "/docs/img/riemer.png"
},
"prevItem": {
"title": "How to understand your machine data?",
"permalink": "/blog/2018/06/18/how-to-understand-your-machine-data"
}
},
"content": "Welcome to our new blog. StreamPipes is the new, easy-to-use self-service data analytics toolbox for the Industrial IoT. In the coming weeks, we'll publish a series of blog posts that explain some ideas of StreamPipes. You'll read about use cases, best practices as well as technical details.\r\n\r\nStay tuned!"
}
]
}