| <!-- |
| ▄▄▄ ██▓███ ▄▄▄ ▄████▄ ██░ ██ ▓█████ ██▓ ▄████ ███▄ █ ██▓▄▄▄█████▓▓█████ |
| ▒████▄ ▓██░ ██▒▒████▄ ▒██▀ ▀█ ▓██░ ██▒▓█ ▀ ▓██▒ ██▒ ▀█▒ ██ ▀█ █ ▓██▒▓ ██▒ ▓▒▓█ ▀ |
| ▒██ ▀█▄ ▓██░ ██▓▒▒██ ▀█▄ ▒▓█ ▄ ▒██▀▀██░▒███ ▒██▒▒██░▄▄▄░▓██ ▀█ ██▒▒██▒▒ ▓██░ ▒░▒███ |
| ░██▄▄▄▄██ ▒██▄█▓▒ ▒░██▄▄▄▄██ ▒▓▓▄ ▄██▒░▓█ ░██ ▒▓█ ▄ ░██░░▓█ ██▓▓██▒ ▐▌██▒░██░░ ▓██▓ ░ ▒▓█ ▄ |
| ▓█ ▓██▒▒██▒ ░ ░ ▓█ ▓██▒▒ ▓███▀ ░░▓█▒░██▓░▒████▒ ░██░░▒▓███▀▒▒██░ ▓██░░██░ ▒██▒ ░ ░▒████▒ |
| ▒▒ ▓▒█░▒▓▒░ ░ ░ ▒▒ ▓▒█░░ ░▒ ▒ ░ ▒ ░░▒░▒░░ ▒░ ░ ░▓ ░▒ ▒ ░ ▒░ ▒ ▒ ░▓ ▒ ░░ ░░ ▒░ ░ |
| ▒ ▒▒ ░░▒ ░ ▒ ▒▒ ░ ░ ▒ ▒ ░▒░ ░ ░ ░ ░ ▒ ░ ░ ░ ░ ░░ ░ ▒░ ▒ ░ ░ ░ ░ ░ |
| ░ ▒ ░░ ░ ▒ ░ ░ ░░ ░ ░ ▒ ░░ ░ ░ ░ ░ ░ ▒ ░ ░ ░ |
| ░ ░ ░ ░░ ░ ░ ░ ░ ░ ░ ░ ░ ░ ░ ░ ░ |
| --> |
| |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| |
| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| |
| <link rel="canonical" href="https://ignite.apache.org/features/streaming.html" /> |
| <!--#include virtual="/includes/scriptshead.html" --> |
| |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" /> |
| <meta http-equiv="Pragma" content="no-cache" /> |
| <meta http-equiv="Expires" content="0" /> |
| <title>Data Loading and Streaming - Apache Ignite</title> |
| <!--#include virtual="/includes/styles.html" --> |
| |
| |
| </head> |
| <body> |
| <!--#include virtual="/includes/header.html" --> |
| <article> |
| <header> |
| <div class="container"> |
| <h1 >Data Loading and <strong>Streaming</strong></h1> |
| </div> |
| </header> |
| <div class="container"> |
| |
| |
| <p> |
| Apache Ignite® data loading and streaming capabilities allow ingesting large finite as well as |
| never-ending volumes of data in a scalable and fault-tolerant way into the cluster. |
| The rate at which data can be injected into Ignite is very high and easily exceeds millions |
| of events per second on a moderately sized cluster. |
| </p> |
| <img class="img-responsive diagram-right" src="/images/ignite-stream-query.png" alt="Data Loading and Streaming" /> |
| <p> |
| Apache Ignite integrates with major streaming technologies and frameworks such as Kafka, Camel, |
| Storm or JMS to bring even more advanced streaming capabilities to Ignite-based architectures. |
| </p> |
| |
| |
| <h2>Data Loading</h2> |
| <p> |
| Ignite provides several <a href="/docs/latest/data-streaming" target="_blank" rel="noopener"> |
| techniques</a> for initial data loading. For instance, Ignite streaming APIs are a good |
| choice for clusters with Ignite native persistence enabled, while the clusters that persist |
| data in a 3rd party store can connect to it directly with <code>CacheStore</code> API. |
| </p> |
| |
| <h2>How Ignite Streaming Works:</h2> |
| <ol class="page-list"> |
| <li>Clients inject streams of data into Ignite.</li> |
| <li>Data is automatically partitioned between Ignite data nodes.</li> |
| <li>Data is concurrently processed across all cluster nodes.</li> |
| <li>Clients perform concurrent <code>SQL queries</code> on the streamed data.</li> |
| <li>Clients subscribe to <code>continuous queries</code> as data changes.</li> |
| </ol> |
| |
| |
| <h2>Code Examples:</h2> |
| <!-- Nav tabs --> |
| <ul id="datagrid-examples" class="nav nav-tabs"> |
| <li ><a class="active" href="#streaming-data" role="tab" data-toggle="tab">Stream Data</a></li> |
| <li><a href="#streaming-transformer" role="tab" data-toggle="tab">Transform Data</a></li> |
| |
| </ul> |
| |
| <!-- Tab panes --> |
| <div class="tab-content"> |
| <div role="tabpanel" class="tab-pane active" id="streaming-data"> |
| <pre class="line-numbers"><code class="language-java"> |
| // Get the data streamer reference and stream data. |
| try (IgniteDataStreamer<Integer, String> stmr = ignite.dataStreamer("myStreamCache")) { |
| // Stream entries. |
| for (int i = 0; i < 100000; i++) |
| stmr.addData(i, Integer.toString(i)); |
| } |
| </code></pre> |
| </div> |
| <div role="tabpanel" class="tab-pane" id="streaming-transformer"> |
| <pre class="line-numbers"><code class="language-java"> |
| CacheConfiguration cfg = new CacheConfiguration("wordCountCache"); |
| |
| IgniteCache<Integer, Long> stmCache = ignite.getOrCreateCache(cfg); |
| |
| try (IgniteDataStreamer<String, Long> stmr = ignite.dataStreamer(stmCache.getName())) { |
| // Allow data updates. |
| stmr.allowOverwrite(true); |
| |
| // Configure data transformation to count instances of the same word. |
| stmr.receiver(StreamTransformer.from((e, arg) -> { |
| // Get current count. |
| Long val = e.getValue(); |
| |
| // Increment count by 1. |
| e.setValue(val == null ? 1L : val + 1); |
| |
| return null; |
| })); |
| |
| // Stream words into the streamer cache. |
| for (String word : text) |
| stmr.addData(word, 1L); |
| } |
| </code></pre> |
| </div> |
| </div> |
| |
| <h2>GitHub Examples:</h2> |
| <p>Also see |
| <a href="https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheContinuousQueryExample.java" target="github">continuous queries</a>, |
| <a href="https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/streaming/wordcount" target="github">word count</a>, |
| and <a href="https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/streaming" target="github">other streaming examples</a> available on GitHub. |
| </p> |
| |
| <h2>Streaming Features</h2> |
| |
| <table class="table table-bordered table-striped" name="Streaming Features"> |
| <thead> |
| <tr> |
| <th width="35%" class="left">Feature</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td class="left">Data Streamers</td> |
| <td> |
| <p> |
| Data streamers are defined by <code>IgniteDataStreamer</code> API and are built to inject large |
| amounts of continuous streams of data into Ignite stream caches. Data streamers are |
| built in a scalable and fault-tolerant fashion and provide <b>at-least-once-guarantee</b> |
| semantics for all the data streamed into Ignite. |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/data-streaming">Docs for this Feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left">Data Loading</td> |
| <td> |
| <p> |
| Data streamers can be used to load large amounts of data into Ignite cahes. |
| They can be used for initial data loading from a 3rd party database or another source. |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/data-streaming">Docs for this Feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left">Collocated Processing</td> |
| <td> |
| <p> |
| For cases when you need to execute some custom logic instead of just adding new data, |
| you can take advantage of <code>StreamReceiver</code> API. |
| </p> |
| <p> |
| Stream receivers allow you to react to the streamed data in collocated fashion, |
| directly on the nodes where it will be cached. You can change the data or add any |
| custom pre-processing logic to it, before putting the data into cache. |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/data-streaming">Docs for this Feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left">Continuous Queries</td> |
| <td> |
| <p> |
| Continuous queries are useful for cases when you want to execute a query and then |
| continue to get notified about the data changes that fall into your query filter. |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/key-value-api/continuous-queries">Docs for this Feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left">JMS Data Streamer</td> |
| <td> |
| <p> |
| Apache Ignite JMS 1.1 module provides a streamer to consume JMS queue and topic messages into |
| Apache Ignite caches. |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/extensions-and-integrations/streaming/jms-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left"> |
| <div class="page-links"> |
| <img class="img-responsive diagram-right" src="/images/apache_flume.png" alt="Apache Flume" width="120" height="60"/> |
| </div> |
| Apache Flume Sink |
| </td> |
| <td> |
| <p> |
| IgniteSink is a Flume sink that extracts Events from an associated Flume channel and injects them into an Ignite cache. |
| Flume 1.7.0 is supported. |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/extensions-and-integrations/streaming/flume-sink">Docs for this feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left">MQTT Streamer</td> |
| <td> |
| <p> |
| Apache Ignite MQTT module provides a streamer to consume MQTT topic messages into |
| Apache Ignite caches. |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/extensions-and-integrations/streaming/mqtt-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left">Twitter Streamer</td> |
| <td> |
| <p> |
| Ignite Twitter Streamer consumes messages from a Twitter Streaming API and inserts them into an Ignite cache. |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/extensions-and-integrations/streaming/twitter-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left"> |
| <div class="page-links"> |
| <img class="img-responsive diagram-right" src="/images/apache_kafka.png" alt="Apache Kafka" width="120" height="60"/> |
| </div> |
| Apache Kafka Streamer |
| </td> |
| <td> |
| <p> |
| Apache Ignite Kafka Streamer module provides streaming from Kafka to Ignite cache. |
| |
| There are two ways this can be achieved: |
| <ul> |
| <li> importing Kafka Streamer module in your Maven project and instantiate KafkaStreamer for |
| data streaming; |
| </li> |
| <li> using Kafka Connect functionality.</li> |
| </ul> |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/extensions-and-integrations/streaming/kafka-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left"> |
| <div class="page-links"> |
| <img class="img-responsive diagram-right" src="/images/apache_camel.png" alt="Apache Camel" width="120" height="60"/> |
| </div> |
| Apache Camel streamer |
| </td> |
| <td> |
| <p> |
| Apache Ignite Camel provides a streamer to consume cache tuples from a Camel endpoint such as |
| HTTP, TCP, File, FTP, AMQP, SNMP, databases, etc. For more information on available components, |
| refer to <a href="http://camel.apache.org/components.html">Camel Components</a> |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/extensions-and-integrations/streaming/camel-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left"> |
| <div class="page-links"> |
| <img class="img-responsive diagram-right" src="/images/apache_storm.png" alt="Apache Storm" width="120" height="60"/> |
| </div> |
| Apache Storm Streamer |
| </td> |
| <td> |
| <p> |
| Ignite Storm Streamer consumes messages from an Apache Storm consumer endpoint and feeds them into an Ignite cache. |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/extensions-and-integrations/streaming/storm-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left"> |
| <div class="page-links"> |
| <img class="img-responsive diagram-right" src="/images/apache_flink.png" alt="Apache Flink" width="120" height="60"/> |
| </div> |
| Apache Flink Streamer |
| </td> |
| <td> |
| <p> |
| Ignite Flink Streamer consumes messages from an Apache Flink consumer endpoint and feeds them into an Ignite cache. |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/extensions-and-integrations/streaming/flink-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left"> |
| <div class="page-links"> |
| <img class="img-responsive diagram-right" src="/images/rocket_mq.png" alt="Apache RocketMQ" width="120" height="60"/> |
| </div> |
| Apache RocketMQ Streamer |
| </td> |
| <td> |
| <p> |
| Ignite RocketMQ Streamer consumes messages from an Apache RocketMQ consumer endpoint and feeds them into an Ignite cache. |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/extensions-and-integrations/streaming/rocketmq-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left">ZeroMQ Streamer</td> |
| <td> |
| <p> |
| Apache Ignite ZeroMQ Streamer module provides streaming from ZeroMQ to Ignite cache. |
| </p> |
| <div class="page-links"> |
| <a href="/docs/latest/extensions-and-integrations/streaming/zeromq-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a> |
| </div> |
| </td> |
| </tr> |
| <tr> |
| <td class="left">Pub/Sub Streamer</td> |
| <td> |
| <p> |
| Pub/Sub module is a streaming connector to inject Pub/Sub data into the Ignite cache. |
| </p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| </article> |
| <!--#include virtual="/includes/footer.html" --> |
| |
| <!--#include virtual="/includes/scripts.html" --> |
| </body> |
| </html> |
