 <!--
 ▄▄▄       ██▓███   ▄▄▄       ▄████▄   ██░ ██ ▓█████     ██▓  ▄████  ███▄    █  ██▓▄▄▄█████▓▓█████
▒████▄    ▓██░  ██▒▒████▄    ▒██▀ ▀█  ▓██░ ██▒▓█   ▀    ▓██▒ ██▒ ▀█▒ ██ ▀█   █ ▓██▒▓  ██▒ ▓▒▓█   ▀
▒██  ▀█▄  ▓██░ ██▓▒▒██  ▀█▄  ▒▓█    ▄ ▒██▀▀██░▒███      ▒██▒▒██░▄▄▄░▓██  ▀█ ██▒▒██▒▒ ▓██░ ▒░▒███
░██▄▄▄▄██ ▒██▄█▓▒ ▒░██▄▄▄▄██ ▒▓▓▄ ▄██▒░▓█ ░██ ▒▓█  ▄    ░██░░▓█  ██▓▓██▒  ▐▌██▒░██░░ ▓██▓ ░ ▒▓█  ▄
 ▓█   ▓██▒▒██▒ ░  ░ ▓█   ▓██▒▒ ▓███▀ ░░▓█▒░██▓░▒████▒   ░██░░▒▓███▀▒▒██░   ▓██░░██░  ▒██▒ ░ ░▒████▒
 ▒▒   ▓▒█░▒▓▒░ ░  ░ ▒▒   ▓▒█░░ ░▒ ▒  ░ ▒ ░░▒░▒░░ ▒░ ░   ░▓   ░▒   ▒ ░ ▒░   ▒ ▒ ░▓    ▒ ░░   ░░ ▒░ ░
  ▒   ▒▒ ░░▒ ░       ▒   ▒▒ ░  ░  ▒    ▒ ░▒░ ░ ░ ░  ░    ▒ ░  ░   ░ ░ ░░   ░ ▒░ ▒ ░    ░     ░ ░  ░
  ░   ▒   ░░         ░   ▒   ░         ░  ░░ ░   ░       ▒ ░░ ░   ░    ░   ░ ░  ▒ ░  ░         ░
      ░  ░               ░  ░░ ░       ░  ░  ░   ░  ░    ░        ░          ░  ░              ░  ░
-->

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">

<link rel="canonical" href="https://ignite.apache.org/features/streaming.html" />
    <!--#include virtual="/includes/scriptshead.html" -->

    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" />
    <meta http-equiv="Pragma" content="no-cache" />
    <meta http-equiv="Expires" content="0" />
    <title>Data Loading and Streaming - Apache Ignite</title>
    <!--#include virtual="/includes/styles.html" -->

    
</head>
<body>
<!--#include virtual="/includes/header.html" -->
<article>
    <header>
        <div class="container">
            <h1 >Data Loading and <strong>Streaming</strong></h1>
        </div>
    </header>
    <div class="container">
                
        
        <p>
            Apache Ignite® data loading and streaming capabilities allow ingesting large finite as well as
            never-ending volumes of data in a scalable and fault-tolerant way into the cluster.
            The rate at which data can be injected into Ignite is very high and easily exceeds millions
            of events per second on a moderately sized cluster.
        </p>
        <img class="img-responsive diagram-right" src="/images/ignite-stream-query.png" alt="Data Loading and Streaming" />
        <p>
            Apache Ignite integrates with major streaming technologies and frameworks such as Kafka, Camel,
            Storm or JMS to bring even more advanced streaming capabilities to Ignite-based architectures.
        </p>
            
                    
        <h2>Data Loading</h2>
        <p>
            Ignite provides several <a href="/docs/latest/data-streaming" target="_blank" rel="noopener">
            techniques</a> for initial data loading. For instance, Ignite streaming APIs are a good
            choice for clusters with Ignite native persistence enabled, while the clusters that persist
            data in a 3rd party store can connect to it directly with <code>CacheStore</code> API.
        </p>

        <h2>How Ignite Streaming Works:</h2>
        <ol class="page-list">
            <li>Clients inject streams of data into Ignite.</li>
            <li>Data is automatically partitioned between Ignite data nodes.</li>
            <li>Data is concurrently processed across all cluster nodes.</li>
            <li>Clients perform concurrent <code>SQL queries</code> on the streamed data.</li>
            <li>Clients subscribe to <code>continuous queries</code> as data changes.</li>
        </ol>
    
            
        <h2>Code Examples:</h2>
        <!-- Nav tabs -->
        <ul id="datagrid-examples" class="nav nav-tabs">
            <li ><a class="active" href="#streaming-data" role="tab" data-toggle="tab">Stream Data</a></li>
            <li><a href="#streaming-transformer" role="tab" data-toggle="tab">Transform Data</a></li>
            
        </ul>

        <!-- Tab panes -->
        <div class="tab-content">
            <div role="tabpanel" class="tab-pane active" id="streaming-data">
                <pre class="line-numbers"><code class="language-java">
                    // Get the data streamer reference and stream data.
                    try (IgniteDataStreamer&lt;Integer, String&gt; stmr = ignite.dataStreamer("myStreamCache")) {
                        // Stream entries.
                        for (int i = 0; i &lt; 100000; i++)
                            stmr.addData(i, Integer.toString(i));
                    }
                </code></pre>
            </div>
            <div role="tabpanel" class="tab-pane" id="streaming-transformer">
                <pre class="line-numbers"><code class="language-java">
                    CacheConfiguration cfg = new CacheConfiguration("wordCountCache");

                    IgniteCache&lt;Integer, Long&gt; stmCache = ignite.getOrCreateCache(cfg);

                    try (IgniteDataStreamer&lt;String, Long&gt; stmr = ignite.dataStreamer(stmCache.getName())) {
                        // Allow data updates.
                        stmr.allowOverwrite(true);

                        // Configure data transformation to count instances of the same word.
                        stmr.receiver(StreamTransformer.from((e, arg) -> {
                        // Get current count.
                        Long val = e.getValue();

                        // Increment count by 1.
                        e.setValue(val == null ? 1L : val + 1);

                        return null;
                        }));

                        // Stream words into the streamer cache.
                        for (String word : text)
                        stmr.addData(word, 1L);
                    }
                </code></pre>
            </div>
        </div>
            
        <h2>GitHub Examples:</h2>
            <p>Also see
                <a href="https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheContinuousQueryExample.java" target="github">continuous queries</a>,
    <a href="https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/streaming/wordcount" target="github">word count</a>,
    and <a href="https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/streaming" target="github">other streaming examples</a> available on GitHub.
            </p>
		
        <h2>Streaming Features</h2>

        <table class="table table-bordered table-striped" name="Streaming Features">
        <thead>
            <tr>
                <th width="35%" class="left">Feature</th>
                <th>Description</th>
            </tr>
        </thead>
        <tbody>
            <tr>
                <td class="left">Data Streamers</td>
                <td>
                    <p>
                        Data streamers are defined by <code>IgniteDataStreamer</code> API and are built to inject large
                        amounts of continuous streams of data into Ignite stream caches. Data streamers are
                        built in a scalable and fault-tolerant fashion and provide <b>at-least-once-guarantee</b>
                        semantics for all the data streamed into Ignite.
                    </p>
                    <div class="page-links">
                        <a href="/docs/latest/data-streaming">Docs for this Feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">Data Loading</td>
                <td>
                    <p>
                        Data streamers can be used to load large amounts of data into Ignite cahes.
                        They can be used for initial data loading from a 3rd party database or another source.
                    </p>
                    <div class="page-links">
                        <a href="/docs/latest/data-streaming">Docs for this Feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">Collocated Processing</td>
                <td>
                    <p>
                        For cases when you need to execute some custom logic instead of just adding new data,
                        you can take advantage of <code>StreamReceiver</code> API.
                    </p>
                    <p>
                        Stream receivers allow you to react to the streamed data in collocated fashion,
                        directly on the nodes where it will be cached. You can change the data or add any
                        custom pre-processing logic to it, before putting the data into cache.
                    </p>
                    <div class="page-links">
                        <a href="/docs/latest/data-streaming">Docs for this Feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">Continuous Queries</td>
                <td>
                    <p>
                        Continuous queries are useful for cases when you want to execute a query and then
                        continue to get notified about the data changes that fall into your query filter.
                    </p>
                    <div class="page-links">
                        <a href="/docs/latest/key-value-api/continuous-queries">Docs for this Feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">JMS Data Streamer</td>
                <td>
                    <p>
                        Apache Ignite JMS 1.1 module provides a streamer to consume JMS queue and topic messages into
                        Apache Ignite caches.
                    </p>
                    <div  class="page-links">
                        <a href="/docs/latest/extensions-and-integrations/streaming/jms-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">
                    <div  class="page-links">
                        <img class="img-responsive diagram-right" src="/images/apache_flume.png" alt="Apache Flume" width="120" height="60"/>
                    </div>
                    Apache Flume Sink
                </td>
                <td>
                    <p>
                        IgniteSink is a Flume sink that extracts Events from an associated Flume channel and injects them into an Ignite cache.
                        Flume 1.7.0 is supported.
                    </p>
                    <div  class="page-links">
                        <a href="/docs/latest/extensions-and-integrations/streaming/flume-sink">Docs for this feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">MQTT Streamer</td>
                <td>
                    <p>
                        Apache Ignite MQTT module provides a streamer to consume MQTT topic messages into
                        Apache Ignite caches.
                    </p>
                    <div  class="page-links">
                        <a href="/docs/latest/extensions-and-integrations/streaming/mqtt-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">Twitter Streamer</td>
                <td>
                    <p>
                        Ignite Twitter Streamer consumes messages from a Twitter Streaming API and inserts them into an Ignite cache.
                    </p>
                    <div  class="page-links">
                        <a href="/docs/latest/extensions-and-integrations/streaming/twitter-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">
                    <div  class="page-links">
                        <img class="img-responsive diagram-right" src="/images/apache_kafka.png" alt="Apache Kafka" width="120" height="60"/>
                    </div>
                    Apache Kafka Streamer
                </td>
                <td>
                    <p>
                        Apache Ignite Kafka Streamer module provides streaming from Kafka to Ignite cache.

                        There are two ways this can be achieved:
                    <ul>
                        <li> importing Kafka Streamer module in your Maven project and instantiate KafkaStreamer for
                            data streaming;
                        </li>
                        <li> using Kafka Connect functionality.</li>
                    </ul>
                    </p>
                    <div  class="page-links">
                        <a href="/docs/latest/extensions-and-integrations/streaming/kafka-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">
                    <div  class="page-links">
                        <img class="img-responsive diagram-right" src="/images/apache_camel.png" alt="Apache Camel" width="120" height="60"/>
                    </div>
                    Apache Camel streamer
                </td>
                <td>
                    <p>
                        Apache Ignite Camel provides a streamer to consume cache tuples from a Camel endpoint such as
                        HTTP, TCP, File, FTP, AMQP, SNMP, databases, etc. For more information on available components,
                        refer to <a href="http://camel.apache.org/components.html">Camel Components</a>
                    </p>
                    <div  class="page-links">
                        <a href="/docs/latest/extensions-and-integrations/streaming/camel-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">
                    <div  class="page-links">
                        <img class="img-responsive diagram-right" src="/images/apache_storm.png" alt="Apache Storm" width="120" height="60"/>
                    </div>
                    Apache Storm Streamer
                </td>
                <td>
                    <p>
                        Ignite Storm Streamer consumes messages from an Apache Storm consumer endpoint and feeds them into an Ignite cache.
                    </p>
                    <div  class="page-links">
                        <a href="/docs/latest/extensions-and-integrations/streaming/storm-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">
                    <div  class="page-links">
                        <img class="img-responsive diagram-right" src="/images/apache_flink.png" alt="Apache Flink" width="120" height="60"/>
                    </div>
                    Apache Flink Streamer
                </td>
                <td>
                    <p>
                        Ignite Flink Streamer consumes messages from an Apache Flink consumer endpoint and feeds them into an Ignite cache.
                    </p>
                    <div  class="page-links">
                        <a href="/docs/latest/extensions-and-integrations/streaming/flink-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">
                    <div  class="page-links">
                        <img class="img-responsive diagram-right" src="/images/rocket_mq.png" alt="Apache RocketMQ" width="120" height="60"/>
                    </div>
                    Apache RocketMQ Streamer
                </td>
                <td>
                    <p>
                        Ignite RocketMQ Streamer consumes messages from an Apache RocketMQ consumer endpoint and feeds them into an Ignite cache.
                    </p>
                    <div  class="page-links">
                        <a href="/docs/latest/extensions-and-integrations/streaming/rocketmq-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">ZeroMQ Streamer</td>
                <td>
                    <p>
                        Apache Ignite ZeroMQ Streamer module provides streaming from ZeroMQ to Ignite cache.
                    </p>
                    <div  class="page-links">
                        <a href="/docs/latest/extensions-and-integrations/streaming/zeromq-streamer">Docs for this feature <i class="fas fa-angle-double-right"></i></a>
                    </div>
                </td>
            </tr>
            <tr>
                <td class="left">Pub/Sub Streamer</td>
                <td>
                    <p>
                        Pub/Sub module is a streaming connector to inject Pub/Sub data into the Ignite cache.
                    </p>
                </td>
            </tr>
        </tbody>
    </table>
</div>
</article>
    <!--#include virtual="/includes/footer.html" -->

<!--#include virtual="/includes/scripts.html" -->
</body>
</html>