<!--
 ▄▄▄       ██▓███   ▄▄▄       ▄████▄   ██░ ██ ▓█████     ██▓  ▄████  ███▄    █  ██▓▄▄▄█████▓▓█████
▒████▄    ▓██░  ██▒▒████▄    ▒██▀ ▀█  ▓██░ ██▒▓█   ▀    ▓██▒ ██▒ ▀█▒ ██ ▀█   █ ▓██▒▓  ██▒ ▓▒▓█   ▀
▒██  ▀█▄  ▓██░ ██▓▒▒██  ▀█▄  ▒▓█    ▄ ▒██▀▀██░▒███      ▒██▒▒██░▄▄▄░▓██  ▀█ ██▒▒██▒▒ ▓██░ ▒░▒███
░██▄▄▄▄██ ▒██▄█▓▒ ▒░██▄▄▄▄██ ▒▓▓▄ ▄██▒░▓█ ░██ ▒▓█  ▄    ░██░░▓█  ██▓▓██▒  ▐▌██▒░██░░ ▓██▓ ░ ▒▓█  ▄
 ▓█   ▓██▒▒██▒ ░  ░ ▓█   ▓██▒▒ ▓███▀ ░░▓█▒░██▓░▒████▒   ░██░░▒▓███▀▒▒██░   ▓██░░██░  ▒██▒ ░ ░▒████▒
 ▒▒   ▓▒█░▒▓▒░ ░  ░ ▒▒   ▓▒█░░ ░▒ ▒  ░ ▒ ░░▒░▒░░ ▒░ ░   ░▓   ░▒   ▒ ░ ▒░   ▒ ▒ ░▓    ▒ ░░   ░░ ▒░ ░
  ▒   ▒▒ ░░▒ ░       ▒   ▒▒ ░  ░  ▒    ▒ ░▒░ ░ ░ ░  ░    ▒ ░  ░   ░ ░ ░░   ░ ▒░ ▒ ░    ░     ░ ░  ░
  ░   ▒   ░░         ░   ▒   ░         ░  ░░ ░   ░       ▒ ░░ ░   ░    ░   ░ ░  ▒ ░  ░         ░
      ░  ░               ░  ░░ ░       ░  ░  ░   ░  ░    ░        ░          ░  ░              ░  ░
-->

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

<!DOCTYPE html>
<html lang="en">
<head>
<link rel="canonical" href="https://ignite.apache.org/use-cases/spark-acceleration.html"/>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">

    <meta name="description"
          content="Apache Ignite integrates with Apache Spark to accelerate the performance of Spark applications
          and APIs by keeping data in a shared in-memory cluster."/>

    <title>Apache Spark Performance Acceleration With Apache Ignite</title>

    <!--#include virtual="/includes/styles.html" -->

    <!--#include virtual="/includes/sh.html" -->
</head>
<body>
<!--#include virtual="/includes/header.html" -->	
<article>
  <header><div class="container">
					 <h1>Apache Spark Performance <br />Acceleration <strong>With Apache Ignite</strong></h1>
</div>
</header>
<div class="container">
            
          
           <p>
                Apache Ignite integrates with Apache Spark to accelerate the performance of Spark applications
                and APIs by keeping data in a shared in-memory cluster. Spark users can use Ignite as a data
                source in a way similar to Hadoop or a relational database. Just start an Ignite cluster, set
                it as a data source for Spark workers, and keep using Spark RDDs or DataFrames APIs or gain
                even more speed by running Ignite SQL or compute APIs directly.
            </p>
            <img class="img-fluid diagram-right" src="/images/svg-diagrams/spark_acceleration.svg" />

            <p>
                In addition to the performance acceleration of Spark applications, Ignite is used as a shared
                in-memory layer by those Spark workers that need to share both data and state.
            </p>

                
            <p>
                The performance increase is achievable for several reasons. First, Ignite is designed to store data sets
                in memory across a cluster of nodes reducing latency of Spark operations that usually need to pull date
                from disk-based systems. Second, Ignite tries to minimize data shuffling over the network between its
                store and Spark applications by running certain Spark tasks, produced by RDDs or DataFrames APIs,
                in-place on Ignite nodes. This optimization helps to reduce the effect of network latency on the
                performance of Spark calls. Finally, the network impact can be further reduced if the native
                Ignite APIs, such as SQL, are called from Spark applications directly. By doing so, you can eliminate
                data shuffling between Spark and Ignite as long as Ignite SQL queries are always executed on
                Ignite nodes returning a much smaller final result set to the application layer.
            </p>				
			
            <h2>Ignite Shared RDDs</h2>
            <p>
                Apache Ignite provides an implementation of the Spark RDD, which allows any data and state to be shared
                in memory as RDDs across Spark jobs. The Ignite RDD provides a shared, mutable view of the data stored
                in Ignite caches across different Spark jobs, workers, or applications.
            </p>

            <p>
                The Ignite RDD is implemented as a view over a distributed Ignite table (aka. cache). It can be deployed
                with an Ignite node either within the Spark job executing process, on a Spark worker, or in a separate
                Ignite cluster. This means that depending on the chosen deployment mode, the shared state may either
                exist only during the lifespan of a Spark application (embedded mode), or it may out-survive the Spark
                application (standalone mode).
            </p>				
      
            <h2>Ignite DataFrames</h2>
            <p>
                The Apache Spark DataFrame API introduced the concept of a schema to describe the data,
                allowing Spark to manage the schema and organize the data into a tabular format. To put it simply,
                a DataFrame is a distributed collection of data organized into named columns. It is conceptually
                equivalent to a table in a relational database and allows Spark to leverage the Catalyst query
                optimizer to produce much more efficient query execution plans in comparison to RDDs, which are
                collections of elements partitioned across the nodes of the cluster.
            </p>
            <p>
                Ignite supports DataFrame APIs allowing Spark to write to and read from Ignite through that interface.
                Furthermore, Ignite analyses execution plans produced by Spark's Catalyst engine and can execute
                parts of the plan on Ignite nodes directly, which will reduce data shuffling and consequently make your
                SparkSQL perform better.
            </p>				

            
      <div class="jumbotron jumbotron-fluid">
        <div class="container">
          <div class="title display-6">Learn More</div>
          <hr class="my-4">
          <div class="row">
            <div class="col-sm-6">
              <ul>
                <li>
                  <a href="https://apacheignite-fs.readme.io/docs/installation-deployment" target="docs">
                    Ignite and Spark Installation and Deployment <i class="fa fa-angle-double-right"></i>
                </a>
                </li>
                <li>
                  <a href="https://apacheignite-fs.readme.io/docs/ignitecontext-igniterdd" target="docs">
                    Ignite RDDs in Details <i class="fa fa-angle-double-right"></i>
                </a>
                </li>
              </ul>
            </div>
            <div class="col-sm-6">
              <ul>
                <li>
                  <a href="https://apacheignite-fs.readme.io/docs/ignite-data-frame" target="docs">
                    Ignite DataFrames in Details <i class="fa fa-angle-double-right"></i>
                </a>
                </li>
                <li>
                  
					<a href="/use-cases/dih.html">
                    Ignite as a Digital Integration Hub <i class="fa fa-angle-double-right"></i>
                </a>
					
                </li>
              </ul>
            </div>
          </div>
        </div>
      </div>
    
  </div>			

</article>
<!--#include virtual="/includes/footer.html" -->
<!--#include virtual="/includes/scripts.html" -->
</body>
</html>