| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1" /> |
| <title>Accelerate Apache Spark Applications| Apache Ignite</title> |
| <link rel="stylesheet" href="/js/vendor/hystmodal/hystmodal.min.css" /> |
| <link rel="stylesheet" href="/js/vendor/swiper/swiper-bundle.min.css" /> |
| <link rel="stylesheet" href="/css/utils.css" /> |
| <link rel="stylesheet" href="/css/site.css" /> |
| <link rel="stylesheet" href="/css/media.css" media="only screen and (max-width:1199px)" /> |
| <link rel="icon" type="image/png" href="/img/favicon.png" /> |
| <link rel="stylesheet" href="../css/native-persistence.css" /> |
| <link rel="stylesheet" href="../css/compute-apis.css" /> |
| <link rel="stylesheet" href="../css/digital-hub.css" /> |
| <link rel="stylesheet" href="../css/spark.css" /> |
| </head> |
| <body> |
| <!-- MOBILE MENU START --> |
| <div class="hystmodal" id="jsMenuModal" aria-hidden="true"> |
| <div class="hystmodal__wrap"> |
| <div class="hystmodal__window mobmenu" role="dialog" aria-modal="true"> |
| <button data-hystclose class="hystmodal__close">Close</button> |
| <div class="mobmenu__wrap"> |
| <a href="/" class="mobmenu__logo"><img src="/img/logo.svg" alt="Логотип" /></a> |
| <div class="mobmenu__menu"> |
| <div class="mobmenu__h1 berlin">Navigation</div> |
| <ul> |
| <li> |
| <a href="/">Get started</a> |
| </li> |
| <li> |
| <a href="/features/">Features</a> |
| </li> |
| <li> |
| <a href="/community.html">Community</a> |
| </li> |
| <li> |
| <a href="/use-cases/provenusecases.html">Powered by</a> |
| </li> |
| <li> |
| <a href="https://ignite.apache.org/docs/latest/">Docs</a> |
| </li> |
| </ul> |
| </div> |
| <!-- //mobmenu__menu --> |
| </div> |
| <!-- //mobmenu__wrap --> |
| </div> |
| <!-- //mobmenu --> |
| </div> |
| </div> |
| <!-- MOBILE MENU END --> |
| <header class="hdr hdr__blue"> |
| <div class="cmtyhero__wrap flexi"> |
| <button class="hdr__burger" data-menumodal="#jsMenuModal"><img src="/img/menu.svg" alt="" /></button><a class="hdr__logo" href="/"><img class="hdr__logoimg" src="/img/logo-white.svg" alt="" /></a |
| ><a class="button hdr__button" href="/download.html">Download Ignite 2.11 </a> |
| <nav class="hdrmenu"> |
| <ul class="flexi"> |
| <li><a href="/">Get started</a></li> |
| <li><a href="/features">Features</a></li> |
| <li><a href="/community.html">Community</a></li> |
| <li><a href="/use-cases/provenusecases.html">Powered by</a></li> |
| <li><a href="///ignite.apache.org/docs/latest/">Docs</a></li> |
| </ul> |
| </nav> |
| </div> |
| </header> |
| <header class="hdrfloat"> |
| <div class="cmtyhero__wrap flexi"> |
| <button class="hdr__burger" data-menumodal="#jsMenuModal"><img src="/img/menu.svg" alt="" /></button><a class="hdr__logo" href="/"><img class="hdr__logoimg" src="/img/logo.svg" alt="" /></a |
| ><a class="button hdr__button" href="/download.html">Download Ignite 2.11 </a> |
| <nav class="hdrmenu"> |
| <ul class="flexi"> |
| <li><a href="/">Get started</a></li> |
| <li><a href="/features">Features</a></li> |
| <li><a href="/community.html">Community</a></li> |
| <li><a href="/use-cases/provenusecases.html">Powered by</a></li> |
| <li><a href="/docs.html">Docs</a></li> |
| </ul> |
| </nav> |
| </div> |
| </header> |
| <section class="innerhero"> |
| <div class="container innerhero__cont"> |
| <div class="innerhero__main"> |
| <h1 class="h1 innerhero__h1">Accelerate Apache Spark Applications</h1> |
| <div class="h3 pt-2 with-apache">With Apache Ignite</div> |
| <div class="innerhero__descr pt-2 h5"> |
| Minimize data shuffling over the network with the Apache<br /> |
| Ignite implementation of RDD and Dataframe APIs |
| </div> |
| <div class="innerhero__action"><a class="button innerhero__button" href="https://ignite.apache.org/docs/latest/index">Start Coding</a></div> |
| </div> |
| <img class="innerhero__pic innerhero__pic--spark" src="/img/usecases/spark/hero-image.svg" alt="hero-image" /> |
| </div> |
| </section> |
| <!-- /.innerhero--> |
| <section class="spark1"> |
| <div class="container"> |
| <h2 class="h5 spark1__h2">How Does Ignite Accelerate Spark Applications?</h2> |
| <div class="spark1__block"> |
| <div class="spark1__item"> |
| <h3 class="spark1__h3 fz20">Horizontally scalable and shared in-memory layer</h3> |
| <p class="spark1__text">Ignite is designed to store data sets in memory across a cluster of nodes reducing latency of Spark operations that usually need to pull date from disk-based systems.</p> |
| </div> |
| <div class="spark1__item"> |
| <h3 class="spark1__h3 fz20">Minimal data shuffling over the network</h3> |
| <p class="spark1__text">Ignite tries to minimize data shuffling over the network between its store and Spark applications by running certain Spark tasks, produced by RDDs or DataFrames APIs, in-place on Ignite nodes.</p> |
| </div> |
| <div class="spark1__item"> |
| <h3 class="spark1__h3 fz20">Extra performance boost with native Ignite APIs</h3> |
| <p class="spark1__text">Use native Ignite APIs, such as SQL from Spark applications directly and eliminate data shuffling completely between Spark and Ignite.</p> |
| </div> |
| </div> |
| <img class="spark1__image" src="/img/usecases/spark/image.svg" alt="image" /> |
| </div> |
| </section> |
| <!-- /.spark1--> |
| <section class="spark2"> |
| <div class="container"> |
| <h2 class="h5 spark2__h2">Ignite Shared RDDs</h2> |
| <div class="spark2__block"> |
| <div class="spark2__item"> |
| <p class="spark2__text">Apache Ignite provides an implementation of the Spark RDD, which allows any data and state to be shared in memory as RDDs across Spark jobs.</p> |
| <p class="spark2__text pt-2">The Ignite RDD provides a shared, mutable view of the data stored in Ignite caches across different Spark jobs, workers, or applications.</p> |
| </div> |
| <div class="spark2__item"> |
| <p class="spark2__text"> |
| The Ignite RDD is implemented as a view over a distributed Ignite table (aka. cache). It can be deployed with an Ignite node either within the Spark job executing process, on a Spark worker, or in a separate Ignite cluster. |
| </p> |
| <p class="spark2__text pt-2"> |
| This means that depending on the chosen deployment mode, the shared state may either exist only during the lifespan of a Spark application (embedded mode), or it may out-survive the Spark application (standalone mode). |
| </p> |
| </div> |
| </div> |
| <h2 class="h5 spark2__h2">Ignite DataFrames</h2> |
| <div class="spark2__block"> |
| <div class="spark2__item"> |
| <p class="spark2__text">The Apache Spark DataFrame API introduced the concept of a schema to describe the data, allowing Spark to manage the schema and organize the data into a tabular format.</p> |
| <p class="spark2__text pt-2"> |
| To put it simply, a DataFrame is a distributed collection of data organized into named columns. It is conceptually equivalent to a table in a relational database and allows Spark to leverage the Catalyst query optimizer to |
| produce much more efficient query execution plans in comparison to RDDs, which are collections of elements partitioned across the nodes of the cluster. |
| </p> |
| </div> |
| <div class="spark2__item"> |
| <p class="spark2__text">Ignite supports DataFrame APIs allowing Spark to write to and read from Ignite through that interface.</p> |
| <p class="spark2__text pt-1"> |
| Furthermore, Ignite analyses execution plans produced by Spark's Catalyst engine and can execute parts of the plan on Ignite nodes directly, which will reduce data shuffling and consequently make your SparkSQL perform better. |
| </p> |
| </div> |
| </div> |
| </div> |
| </section> |
| <section class="native-bottom container"> |
| <div class="native-bottom__grid"> |
| <article class="nativebotblock"> |
| <h3 class="h4 nativebotblock__title"><img class="nativebotblock__icon" src="/img/features/native-rocket.svg" alt="" /><span>Ready to Start?</span></h3> |
| <p class="nativebotblock__text">Discover our quick start guide and build your first application in 5-10 minutes</p> |
| <a class="nativebotblock__link arrowlink" href="https://ignite.apache.org/docs/latest/" target="_blank">Quick Start Guide</a> |
| </article> |
| <article class="nativebotblock nativebotblock--learn"> |
| <h3 class="h4 nativebotblock__title"><img class="nativebotblock__icon" src="/img/features/native-docs.svg" alt="" /><span>Want to Learn More?</span></h3> |
| <p class="nativebotblock__text">Using Hadoop with Spark? See how Ignite accelerates Hadoop-based deployments</p> |
| <a class="nativebotblock__link arrowlink" href="https://ignite.apache.org/use-cases/hadoop-acceleration.html" target="_blank">Apache Hadoop Acceleration</a> |
| </article> |
| </div> |
| </section> |
| <a class="scrollTop" href="#" |
| ><svg class="feather feather-chevron-up" xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewbox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> |
| <polyline points="18 15 12 9 6 15"></polyline></svg |
| ></a> |
| <script src="/js/vendor/hystmodal/hystmodal.min.js"></script> |
| <script src="/js/vendor/swiper/swiper-bundle.min.js"></script> |
| <script src="/js/vendor/waypoints.min.js"></script> |
| <script src="/js/main.js"></script> |
| </body> |
| </html> |
