| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <title> |
| Spark Release 1.0.1 | Apache Spark |
| |
| </title> |
| |
| |
| |
| |
| |
| <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css" rel="stylesheet" |
| integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC" crossorigin="anonymous"> |
| <link rel="preconnect" href="https://fonts.googleapis.com"> |
| <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> |
| <link href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,wght@0,400;0,500;0,700;1,400;1,500;1,700&Courier+Prime:wght@400;700&display=swap" rel="stylesheet"> |
| <link href="/css/custom.css" rel="stylesheet"> |
| <!-- Code highlighter CSS --> |
| <link href="/css/pygments-default.css" rel="stylesheet"> |
| <link rel="icon" href="/favicon.ico" type="image/x-icon"> |
| |
| <!-- Matomo --> |
| <script> |
| var _paq = window._paq = window._paq || []; |
| /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ |
| _paq.push(["disableCookies"]); |
| _paq.push(['trackPageView']); |
| _paq.push(['enableLinkTracking']); |
| (function() { |
| var u="https://analytics.apache.org/"; |
| _paq.push(['setTrackerUrl', u+'matomo.php']); |
| _paq.push(['setSiteId', '40']); |
| var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; |
| g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); |
| })(); |
| </script> |
| <!-- End Matomo Code --> |
| </head> |
| <body class="global"> |
| <nav class="navbar navbar-expand-lg navbar-dark p-0 px-4" style="background: #1D6890;"> |
| <a class="navbar-brand" href="/"> |
| <img src="/images/spark-logo-rev.svg" alt="" width="141" height="72"> |
| </a> |
| <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarContent" |
| aria-controls="navbarContent" aria-expanded="false" aria-label="Toggle navigation"> |
| <span class="navbar-toggler-icon"></span> |
| </button> |
| <div class="collapse navbar-collapse col-md-12 col-lg-auto pt-4" id="navbarContent"> |
| |
| <ul class="navbar-nav me-auto"> |
| <li class="nav-item"> |
| <a class="nav-link active" aria-current="page" href="/downloads.html">Download</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="libraries" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Libraries |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="libraries"> |
| <li><a class="dropdown-item" href="/sql/">SQL and DataFrames</a></li> |
| <li><a class="dropdown-item" href="/spark-connect/">Spark Connect</a></li> |
| <li><a class="dropdown-item" href="/streaming/">Spark Streaming</a></li> |
| <li><a class="dropdown-item" href="/mllib/">MLlib (machine learning)</a></li> |
| <li><a class="dropdown-item" href="/graphx/">GraphX (graph)</a></li> |
| <li> |
| <hr class="dropdown-divider"> |
| </li> |
| <li><a class="dropdown-item" href="/third-party-projects.html">Third-Party Projects</a></li> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="documentation" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Documentation |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="documentation"> |
| <li><a class="dropdown-item" href="/docs/latest/">Latest Release</a></li> |
| <li><a class="dropdown-item" href="/documentation.html">Older Versions and Other Resources</a></li> |
| <li><a class="dropdown-item" href="/faq.html">Frequently Asked Questions</a></li> |
| </ul> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link active" aria-current="page" href="/examples.html">Examples</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="community" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Community |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="community"> |
| <li><a class="dropdown-item" href="/community.html">Mailing Lists & Resources</a></li> |
| <li><a class="dropdown-item" href="/contributing.html">Contributing to Spark</a></li> |
| <li><a class="dropdown-item" href="/improvement-proposals.html">Improvement Proposals (SPIP)</a> |
| </li> |
| <li><a class="dropdown-item" href="https://issues.apache.org/jira/browse/SPARK">Issue Tracker</a> |
| </li> |
| <li><a class="dropdown-item" href="/powered-by.html">Powered By</a></li> |
| <li><a class="dropdown-item" href="/committers.html">Project Committers</a></li> |
| <li><a class="dropdown-item" href="/history.html">Project History</a></li> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="developers" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Developers |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="developers"> |
| <li><a class="dropdown-item" href="/developer-tools.html">Useful Developer Tools</a></li> |
| <li><a class="dropdown-item" href="/versioning-policy.html">Versioning Policy</a></li> |
| <li><a class="dropdown-item" href="/release-process.html">Release Process</a></li> |
| <li><a class="dropdown-item" href="/security.html">Security</a></li> |
| </ul> |
| </li> |
| </ul> |
| <ul class="navbar-nav ml-auto"> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="apacheFoundation" role="button" |
| data-bs-toggle="dropdown" aria-expanded="false"> |
| Apache Software Foundation |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="apacheFoundation"> |
| <li><a class="dropdown-item" href="https://www.apache.org/">Apache Homepage</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/licenses/">License</a></li> |
| <li><a class="dropdown-item" |
| href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/security/">Security</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/events/current-event">Event</a></li> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| </nav> |
| |
| <div class="container"> |
| <div class="row mt-4"> |
| <div class="col-12 col-md-9"> |
| <h2>Spark Release 1.0.1</h2> |
| |
| |
| <p>Spark 1.0.1 is a maintenance release with several stability fixes and a few new features in Spark’s SQL (alpha) library. This release is based on the <a href="https://github.com/apache/spark/tree/branch-1.0">branch-1.0</a> maintenance branch of Spark. We recommend users follow the head of this branch to get the most recent stable version of Spark.</p> |
| |
| <p>You can download Spark 1.0.1 as either a |
| <a href="http://d3kbcqa49mib13.cloudfront.net/spark-1.0.1.tgz" onclick="trackOutboundLink(this, 'Release Download Links', 'cloudfront_spark-1.0.1.tgz'); return false;">source package</a> |
| (5 MB tgz) or a prebuilt package for |
| <a href="http://d3kbcqa49mib13.cloudfront.net/spark-1.0.1-bin-hadoop1.tgz" onclick="trackOutboundLink(this, 'Release Download Links', 'cloudfront_spark-1.0.1-bin-hadoop1.tgz'); return false;">Hadoop 1 / CDH3</a>, |
| <a href="http://d3kbcqa49mib13.cloudfront.net/spark-1.0.1-bin-cdh4.tgz" onclick="trackOutboundLink(this, 'Release Download Links', 'cloudfront_spark-1.0.1-bin-cdh4.tgz'); return false;">CDH4</a>, or |
| <a href="http://d3kbcqa49mib13.cloudfront.net/spark-1.0.1-bin-hadoop2.tgz" onclick="trackOutboundLink(this, 'Release Download Links', 'cloudfront_spark-1.0.1-bin-hadoop2.tgz'); return false;">Hadoop 2 / CDH5 / HDP2</a> |
| (160 MB tgz). Release signatures and checksums are available at the official <a href="http://www.apache.org/dist/spark/spark-1.0.1/">Apache download site</a>.</p> |
| |
| <h3 id="fixes">Fixes</h3> |
| <p>Spark 1.0.1 contains stability fixes in several components. Some of the more important fixes are highlighted below. You can visit the <a href="http://s.apache.org/5zh">Spark issue tracker</a> for an exhaustive list of fixes.</p> |
| |
| <h4 id="spark-core">Spark Core</h4> |
| <ul> |
| <li>Issue with missing keys during external aggregations (<a href="https://issues.apache.org/jira/browse/SPARK-2043">SPARK-2043</a>)</li> |
| <li>Issue during job failures in Mesos mode (<a href="https://issues.apache.org/jira/browse/SPARK-1749">SPARK-1749</a>)</li> |
| <li>Error when defining case classes in Scala shell (<a href="https://issues.apache.org/jira/browse/SPARK-1199">SPARK-1199</a>)</li> |
| <li>Proper support for r3.xlarge instances on AWS (<a href="https://issues.apache.org/jira/browse/SPARK-1790">SPARK-1790</a>)</li> |
| </ul> |
| |
| <h4 id="pyspark">PySpark</h4> |
| <ul> |
| <li>Issue causing crashes when large numbers of tasks finish quickly (<a href="https://issues.apache.org/jira/browse/SPARK-2282">SPARK-2282</a>)</li> |
| <li>Issue importing MLlib in YARN-client mode (<a href="https://issues.apache.org/jira/browse/SPARK-2172">SPARK-2172</a>)</li> |
| <li>Incorrect behavior when hashing None (<a href="https://issues.apache.org/jira/browse/SPARK-1468">SPARK-1468</a>)</li> |
| </ul> |
| |
| <h4 id="mllib">MLlib</h4> |
| <ul> |
| <li>Added compatibility for numpy 1.4 (<a href="https://issues.apache.org/jira/browse/SPARK-2091">SPARK-2091</a>)</li> |
| <li>Concurrency issue in random sampler (<a href="https://issues.apache.org/jira/browse/SPARK-2251">SPARK-2251</a>)</li> |
| <li>NotSerializable exception in ALS (<a href="https://issues.apache.org/jira/browse/SPARK-1977">SPARK-1977</a>)</li> |
| </ul> |
| |
| <h4 id="streaming">Streaming</h4> |
| <ul> |
| <li>Key not found when slow receiver starts (<a href="https://issues.apache.org/jira/browse/SPARK-2009">SPARK-2009</a>)</li> |
| <li>Resource clean-up with KafkaInputDStream (<a href="https://issues.apache.org/jira/browse/SPARK-2034">SPARK-2034</a>)</li> |
| <li>Issue with Flume events larger than 1020 bytes (<a href="https://issues.apache.org/jira/browse/SPARK-1916">SPARK-1916</a>)</li> |
| </ul> |
| |
| <h3 id="sparksql-features">SparkSQL Features</h3> |
| <ul> |
| <li>Support for querying JSON datasets (<a href="https://issues.apache.org/jira/browse/SPARK-2060">SPARK-2060</a>).</li> |
| <li>Improved reading and writing Parquet data, including support for nested records and arrays (<a href="https://issues.apache.org/jira/browse/SPARK-1293">SPARK-1293</a>, <a href="https://issues.apache.org/jira/browse/SPARK-2195">SPARK-2195</a>, <a href="https://issues.apache.org/jira/browse/SPARK-1913">SPARK-1913</a>, and <a href="https://issues.apache.org/jira/browse/SPARK-1487">SPARK-1487</a>).</li> |
| <li>Improved support for SQL commands (<code class="language-plaintext highlighter-rouge">CACHE TABLE</code>, <code class="language-plaintext highlighter-rouge">DESCRIBE</code>, SHOW TABLES) (<a href="https://issues.apache.org/jira/browse/SPARK-1968">SPARK-1968</a>, <a href="https://issues.apache.org/jira/browse/SPARK-2128">SPARK-2128</a>, and <a href="https://issues.apache.org/jira/browse/SPARK-1704">SPARK-1704</a>).</li> |
| <li>Support for SQL specific configuration (initially used for setting number of partitions) (<a href="https://issues.apache.org/jira/browse/SPARK-1508">SPARK-1508</a>).</li> |
| <li>Idempotence for DDL operations (<a href="https://issues.apache.org/jira/browse/SPARK-2191">SPARK-2191</a>).</li> |
| </ul> |
| |
| <h3 id="known-issues">Known Issues</h3> |
| <p>This release contains one known issue: multi-statement lines the REPL with internal references (<code class="language-plaintext highlighter-rouge">> val x = 10; val y = x + 10</code>) produce exceptions (<a href="https://issues.apache.org/jira/browse/SPARK-2452">SPARK-2452</a>). This will be fixed shortly on the 1.0 branch; the fix will be included in the 1.0.2 release.</p> |
| |
| <h3 id="contributors">Contributors</h3> |
| <p>The following developers contributed to this release:</p> |
| |
| <ul> |
| <li>Aaron Davidson – bug fixes in PySpark and Spark core</li> |
| <li>Ali Ghodsi – documentation update</li> |
| <li>Anant – compatibility fix for spark-ec2 script</li> |
| <li>Anatoli Fomenko – MLlib doc fix</li> |
| <li>Andre Schumacher – nested Parquet data</li> |
| <li>Andrew Ash – documentation</li> |
| <li>Andrew Or – bug fixes and documentation</li> |
| <li>Ankur Dave – bug fixes</li> |
| <li>Arkadiusz Komarzewski – doc fix</li> |
| <li>Baishuo – sql fix</li> |
| <li>Chen Chao – comment fix and bug fix</li> |
| <li>Cheng Hao – SQL features</li> |
| <li>Cheng Lian – SQL features</li> |
| <li>Christian Tzolov – build improvement</li> |
| <li>ClĂ©ment MATHIEU – doc updates</li> |
| <li>CodingCat – doc updates and bug fix</li> |
| <li>Colin McCabe – bug fix</li> |
| <li>Daoyuan – SQL joins</li> |
| <li>David Lemieux – bug fix</li> |
| <li>Derek Ma – bug fix</li> |
| <li>Doris Xin – bug fix</li> |
| <li>Erik Selin – PySpark fix</li> |
| <li>Gang Bai – bug fix</li> |
| <li>Guoqiang Li – bug fixes</li> |
| <li>Henry Saputra – documentation</li> |
| <li>Jiang – doc fix</li> |
| <li>Joy Yoj – bug fix</li> |
| <li>Jyotiska NK – test improvement</li> |
| <li>Kan Zhang – PySpark SQL features</li> |
| <li>Kay Ousterhout – documentation fix</li> |
| <li>LY Lai – bug fix</li> |
| <li>Lars Albertsson – bug fix</li> |
| <li>Lei Zhang – SQL fix and feature</li> |
| <li>Mark Hamstra – bug fix</li> |
| <li>Matei Zaharia – doc updates and bug fix</li> |
| <li>Matthew Farrellee – bug fixes</li> |
| <li>Michael Armbrust – sql features and fixes</li> |
| <li>Neville Li – buf fix</li> |
| <li>Nick Chammas – doc fix</li> |
| <li>Ori Kremer – bug fix</li> |
| <li>Patrick Wendell – documentation and release manager</li> |
| <li>Prashant Sharma – bug and doc fixes</li> |
| <li>Qiuzhuang.Lian – bug fix</li> |
| <li>Raymond Liu – bug fix</li> |
| <li>Ravikanth Nawada – bug fixes</li> |
| <li>Reynold Xin – bug and doc fixes</li> |
| <li>Sameer Agarwal – optimization</li> |
| <li>Sandy Ryza – doc fix</li> |
| <li>Sean Owen – bug fix</li> |
| <li>Sebastien Rainville – bug fix</li> |
| <li>Shixiong Zhu – code clean-up</li> |
| <li>Szul, Piotr – bug fix</li> |
| <li>Takuya UESHIN – bug fixes and SQL features</li> |
| <li>Thomas Graves – bug fix</li> |
| <li>Uri Laserson – bug fix</li> |
| <li>Vadim Chekan – bug fix</li> |
| <li>Varakhedi Sujeet – ec2 r3 support</li> |
| <li>Vlad – doc fix</li> |
| <li>Wang Lianhui – bug fix</li> |
| <li>Wenchen Fan – optimization</li> |
| <li>William Benton – SQL feature</li> |
| <li>Xi Liu – SQL feature</li> |
| <li>Xiangrui Meng – bug fix</li> |
| <li>Ximo Guanter Gonzalbez – SQL feature</li> |
| <li>Yadid Ayzenberg – doc fix</li> |
| <li>Yijie Shen – buf fix</li> |
| <li>Yin Huai – JSON support and bug fixes</li> |
| <li>Zhen Peng – bug fix</li> |
| <li>Zichuan Ye – ec2 fixes</li> |
| <li>Zongheng Yang – sql fixes</li> |
| </ul> |
| |
| <p><em>Thanks to everyone who contributed!</em></p> |
| |
| |
| <p> |
| <br/> |
| <a href="/news/">Spark News Archive</a> |
| </p> |
| |
| </div> |
| <div class="col-12 col-md-3"> |
| <div class="news" style="margin-bottom: 20px;"> |
| <h5>Latest News</h5> |
| <ul class="list-unstyled"> |
| |
| <li><a href="/news/spark-3-5-1-released.html">Spark 3.5.1 released</a> |
| <span class="small">(Feb 23, 2024)</span></li> |
| |
| <li><a href="/news/spark-3-3-4-released.html">Spark 3.3.4 released</a> |
| <span class="small">(Dec 16, 2023)</span></li> |
| |
| <li><a href="/news/spark-3-4-2-released.html">Spark 3.4.2 released</a> |
| <span class="small">(Nov 30, 2023)</span></li> |
| |
| <li><a href="/news/spark-3-5-0-released.html">Spark 3.5.0 released</a> |
| <span class="small">(Sep 13, 2023)</span></li> |
| |
| </ul> |
| <p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p> |
| </div> |
| <div style="text-align:center; margin-bottom: 20px;"> |
| <a href="https://www.apache.org/events/current-event.html"> |
| <img src="https://www.apache.org/events/current-event-234x60.png" style="max-width: 100%;"/> |
| </a> |
| </div> |
| <div class="hidden-xs hidden-sm"> |
| <a href="/downloads.html" class="btn btn-cta btn-lg d-grid" style="margin-bottom: 30px;"> |
| Download Spark |
| </a> |
| <p style="font-size: 16px; font-weight: 500; color: #555;"> |
| Built-in Libraries: |
| </p> |
| <ul class="list-none"> |
| <li><a href="/sql/">SQL and DataFrames</a></li> |
| <li><a href="/streaming/">Spark Streaming</a></li> |
| <li><a href="/mllib/">MLlib (machine learning)</a></li> |
| <li><a href="/graphx/">GraphX (graph)</a></li> |
| </ul> |
| <a href="/third-party-projects.html">Third-Party Projects</a> |
| </div> |
| </div> |
| </div> |
| |
| |
| |
| <footer class="small"> |
| <hr> |
| Apache Spark, Spark, Apache, the Apache feather logo, and the Apache Spark project logo are either registered |
| trademarks or trademarks of The Apache Software Foundation in the United States and other countries. |
| See guidance on use of Apache Spark <a href="/trademarks.html">trademarks</a>. |
| All other marks mentioned may be trademarks or registered trademarks of their respective owners. |
| Copyright © 2018 The Apache Software Foundation, Licensed under the |
| <a href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>. |
| </footer> |
| </div> |
| |
| <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/js/bootstrap.bundle.min.js" |
| integrity="sha384-MrcW6ZMFYlzcLA8Nl+NtUVF0sA7MsXsP1UyJoMp4YLEuNSfAP+JcXn/tWtIaxVXM" |
| crossorigin="anonymous"></script> |
| <script src="https://code.jquery.com/jquery.js"></script> |
| <script src="/js/lang-tabs.js"></script> |
| <script src="/js/downloads.js"></script> |
| </body> |
| </html> |
