| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <title> |
| Spark Release 0.3 | Apache Spark |
| |
| </title> |
| |
| |
| |
| |
| |
| <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css" rel="stylesheet" |
| integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC" crossorigin="anonymous"> |
| <link rel="preconnect" href="https://fonts.googleapis.com"> |
| <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> |
| <link href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,wght@0,400;0,500;0,700;1,400;1,500;1,700&Courier+Prime:wght@400;700&display=swap" rel="stylesheet"> |
| <link href="/css/custom.css" rel="stylesheet"> |
| <!-- Code highlighter CSS --> |
| <link href="/css/pygments-default.css" rel="stylesheet"> |
| <link rel="icon" href="/favicon.ico" type="image/x-icon"> |
| |
| <!-- Matomo --> |
| <script> |
| var _paq = window._paq = window._paq || []; |
| /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ |
| _paq.push(["disableCookies"]); |
| _paq.push(['trackPageView']); |
| _paq.push(['enableLinkTracking']); |
| (function() { |
| var u="https://analytics.apache.org/"; |
| _paq.push(['setTrackerUrl', u+'matomo.php']); |
| _paq.push(['setSiteId', '40']); |
| var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; |
| g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); |
| })(); |
| </script> |
| <!-- End Matomo Code --> |
| </head> |
| <body class="global"> |
| <nav class="navbar navbar-expand-lg navbar-dark p-0 px-4" style="background: #1D6890;"> |
| <a class="navbar-brand" href="/"> |
| <img src="/images/spark-logo-rev.svg" alt="" width="141" height="72"> |
| </a> |
| <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarContent" |
| aria-controls="navbarContent" aria-expanded="false" aria-label="Toggle navigation"> |
| <span class="navbar-toggler-icon"></span> |
| </button> |
| <div class="collapse navbar-collapse col-md-12 col-lg-auto pt-4" id="navbarContent"> |
| |
| <ul class="navbar-nav me-auto"> |
| <li class="nav-item"> |
| <a class="nav-link active" aria-current="page" href="/downloads.html">Download</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="libraries" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Libraries |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="libraries"> |
| <li><a class="dropdown-item" href="/sql/">SQL and DataFrames</a></li> |
| <li><a class="dropdown-item" href="/spark-connect/">Spark Connect</a></li> |
| <li><a class="dropdown-item" href="/streaming/">Spark Streaming</a></li> |
| <li><a class="dropdown-item" href="/pandas-on-spark/">pandas on Spark</a></li> |
| <li><a class="dropdown-item" href="/mllib/">MLlib (machine learning)</a></li> |
| <li><a class="dropdown-item" href="/graphx/">GraphX (graph)</a></li> |
| <li> |
| <hr class="dropdown-divider"> |
| </li> |
| <li><a class="dropdown-item" href="/third-party-projects.html">Third-Party Projects</a></li> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="documentation" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Documentation |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="documentation"> |
| <li><a class="dropdown-item" href="/docs/latest/">Latest Release</a></li> |
| <li><a class="dropdown-item" href="/documentation.html">Older Versions and Other Resources</a></li> |
| <li><a class="dropdown-item" href="/faq.html">Frequently Asked Questions</a></li> |
| </ul> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link active" aria-current="page" href="/examples.html">Examples</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="community" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Community |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="community"> |
| <li><a class="dropdown-item" href="/community.html">Mailing Lists & Resources</a></li> |
| <li><a class="dropdown-item" href="/contributing.html">Contributing to Spark</a></li> |
| <li><a class="dropdown-item" href="/improvement-proposals.html">Improvement Proposals (SPIP)</a> |
| </li> |
| <li><a class="dropdown-item" href="https://issues.apache.org/jira/browse/SPARK">Issue Tracker</a> |
| </li> |
| <li><a class="dropdown-item" href="/powered-by.html">Powered By</a></li> |
| <li><a class="dropdown-item" href="/committers.html">Project Committers</a></li> |
| <li><a class="dropdown-item" href="/history.html">Project History</a></li> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="developers" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Developers |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="developers"> |
| <li><a class="dropdown-item" href="/developer-tools.html">Useful Developer Tools</a></li> |
| <li><a class="dropdown-item" href="/versioning-policy.html">Versioning Policy</a></li> |
| <li><a class="dropdown-item" href="/release-process.html">Release Process</a></li> |
| <li><a class="dropdown-item" href="/security.html">Security</a></li> |
| </ul> |
| </li> |
| </ul> |
| <ul class="navbar-nav ml-auto"> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="apacheFoundation" role="button" |
| data-bs-toggle="dropdown" aria-expanded="false"> |
| Apache Software Foundation |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="apacheFoundation"> |
| <li><a class="dropdown-item" href="https://www.apache.org/">Apache Homepage</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/licenses/">License</a></li> |
| <li><a class="dropdown-item" |
| href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/security/">Security</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/events/current-event">Event</a></li> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| </nav> |
| |
| <div class="container"> |
| <div class="row mt-4"> |
| <div class="col-12 col-md-9"> |
| <h2>Spark Release 0.3</h2> |
| |
| |
| <p>Spark 0.3 brings a variety of new features. You can download it for either <a href="https://github.com/mesos/spark/tarball/0.3-scala-2.9">Scala 2.9</a> or <a href="https://github.com/mesos/spark/tarball/0.3-scala-2.8">Scala 2.8</a>.</p> |
| |
| <h3>Scala 2.9 Support</h3> |
| |
| <p>This is the first release to support Scala 2.9 in addition to 2.8. Future releases are likely to be 2.9-only unless there is high demand for 2.8.</p> |
| |
| <h3>Save Operations</h3> |
| |
| <p>You can now save distributed datasets to the Hadoop filesystem (HDFS), Amazon S3, Hypertable, and any other storage system supported by Hadoop. There are convenience methods for several common formats, like text files and SequenceFiles. For example, to save a dataset as text:</p> |
| |
| <div class="code"> |
| <span class="keyword">val</span> numbers = spark.parallelize(1 to 100)<br /> numbers.<span class="sparkop">saveAsTextFile</span>(<span class="string">"hdfs://..."</span>) |
| </div> |
| |
| <h3>Native Types for SequenceFiles</h3> |
| |
| <p>In working with SequenceFiles, which store objects that implement Hadoop’s Writable interface, Spark will now let you use native types for certain common Writable types, like IntWritable and Text. For example:</p> |
| |
| <div class="code"> |
| <span class="comment">// Will read a SequenceFile of (IntWritable, Text)</span><br /> |
| <span class="keyword">val</span> data = spark.sequenceFile[Int, String](<span class="string">"hdfs://..."</span>) |
| </div> |
| |
| <p>Similarly, you can save datasets of basic types directly as SequenceFiles:</p> |
| |
| <div class="code"> |
| <span class="comment">// Will write a SequenceFile of (IntWritable, IntWritable)</span><br /> |
| <span class="keyword">val</span> squares = spark.parallelize(1 to 100).<span class="sparkop">map</span>(<span class="closure">n => (n, n*n)</span>)<br /> |
| squares.saveAsSequenceFile(<span class="string">"hdfs://..."</span>) |
| </div> |
| |
| <h3>Maven Integration</h3> |
| |
| <p>Spark now fetches dependencies via Maven and can publish Maven artifacts for easier dependency management.</p> |
| |
| <h3>Faster Broadcast & Shuffle</h3> |
| |
| <p>This release includes broadcast and shuffle algorithms from <a href="http://www.mosharaf.com/wp-content/uploads/orchestra-sigcomm11.pdf">this paper</a> to better support applications that communicate large amounts of data.</p> |
| |
| <h3>Support for Non-Filesystem Hadoop Input Formats</h3> |
| |
| <p>The new <tt>SparkContext.hadoopRDD</tt> method allows reading data from Hadoop-compatible storage systems other than file systems, such as HBase, Hypertable, etc.</p> |
| |
| <h3>Other Features</h3> |
| |
| <ul> |
| <li>Outer join operators (<tt>leftOuterJoin</tt>, <tt>rightOuterJoin</tt>, etc).</li> |
| <li>Support for Scala 2.9 interpreter features (history search, Ctrl-C current line, etc) in the 2.9 version.</li> |
| <li>Better default levels of parallelism for various operations.</li> |
| <li>Ability to control number of splits in a file.</li> |
| <li>Various bug fixes.</li> |
| </ul> |
| |
| |
| <p> |
| <br/> |
| <a href="/news/">Spark News Archive</a> |
| </p> |
| |
| </div> |
| <div class="col-12 col-md-3"> |
| <div class="news" style="margin-bottom: 20px;"> |
| <h5>Latest News</h5> |
| <ul class="list-unstyled"> |
| |
| <li><a href="/news/spark-3-4-3-released.html">Spark 3.4.3 released</a> |
| <span class="small">(Apr 18, 2024)</span></li> |
| |
| <li><a href="/news/spark-3-5-1-released.html">Spark 3.5.1 released</a> |
| <span class="small">(Feb 23, 2024)</span></li> |
| |
| <li><a href="/news/spark-3-3-4-released.html">Spark 3.3.4 released</a> |
| <span class="small">(Dec 16, 2023)</span></li> |
| |
| <li><a href="/news/spark-3-4-2-released.html">Spark 3.4.2 released</a> |
| <span class="small">(Nov 30, 2023)</span></li> |
| |
| </ul> |
| <p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p> |
| </div> |
| <div style="text-align:center; margin-bottom: 20px;"> |
| <a href="https://www.apache.org/events/current-event.html"> |
| <img src="https://www.apache.org/events/current-event-234x60.png" style="max-width: 100%;"/> |
| </a> |
| </div> |
| <div class="hidden-xs hidden-sm"> |
| <a href="/downloads.html" class="btn btn-cta btn-lg d-grid" style="margin-bottom: 30px;"> |
| Download Spark |
| </a> |
| <p style="font-size: 16px; font-weight: 500; color: #555;"> |
| Built-in Libraries: |
| </p> |
| <ul class="list-none"> |
| <li><a href="/sql/">SQL and DataFrames</a></li> |
| <li><a href="/streaming/">Spark Streaming</a></li> |
| <li><a href="/mllib/">MLlib (machine learning)</a></li> |
| <li><a href="/graphx/">GraphX (graph)</a></li> |
| </ul> |
| <a href="/third-party-projects.html">Third-Party Projects</a> |
| </div> |
| </div> |
| </div> |
| |
| |
| |
| <footer class="small"> |
| <hr> |
| Apache Spark, Spark, Apache, the Apache feather logo, and the Apache Spark project logo are either registered |
| trademarks or trademarks of The Apache Software Foundation in the United States and other countries. |
| See guidance on use of Apache Spark <a href="/trademarks.html">trademarks</a>. |
| All other marks mentioned may be trademarks or registered trademarks of their respective owners. |
| Copyright © 2018 The Apache Software Foundation, Licensed under the |
| <a href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>. |
| </footer> |
| </div> |
| |
| <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/js/bootstrap.bundle.min.js" |
| integrity="sha384-MrcW6ZMFYlzcLA8Nl+NtUVF0sA7MsXsP1UyJoMp4YLEuNSfAP+JcXn/tWtIaxVXM" |
| crossorigin="anonymous"></script> |
| <script src="https://code.jquery.com/jquery.js"></script> |
| <script src="/js/lang-tabs.js"></script> |
| <script src="/js/downloads.js"></script> |
| </body> |
| </html> |
