| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <title> |
| Spark Release 3.0.0 | Apache Spark |
| |
| </title> |
| |
| |
| |
| |
| |
| <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css" rel="stylesheet" |
| integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC" crossorigin="anonymous"> |
| <link rel="preconnect" href="https://fonts.googleapis.com"> |
| <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> |
| <link href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,wght@0,400;0,500;0,700;1,400;1,500;1,700&Courier+Prime:wght@400;700&display=swap" rel="stylesheet"> |
| <link href="/css/custom.css" rel="stylesheet"> |
| <!-- Code highlighter CSS --> |
| <link href="/css/pygments-default.css" rel="stylesheet"> |
| <link rel="icon" href="/favicon.ico" type="image/x-icon"> |
| |
| <!-- Matomo --> |
| <script> |
| var _paq = window._paq = window._paq || []; |
| /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ |
| _paq.push(["disableCookies"]); |
| _paq.push(['trackPageView']); |
| _paq.push(['enableLinkTracking']); |
| (function() { |
| var u="https://analytics.apache.org/"; |
| _paq.push(['setTrackerUrl', u+'matomo.php']); |
| _paq.push(['setSiteId', '40']); |
| var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; |
| g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); |
| })(); |
| </script> |
| <!-- End Matomo Code --> |
| </head> |
| <body class="global"> |
| <nav class="navbar navbar-expand-lg navbar-dark p-0 px-4" style="background: #1D6890;"> |
| <a class="navbar-brand" href="/"> |
| <img src="/images/spark-logo-rev.svg" alt="" width="141" height="72"> |
| </a> |
| <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarContent" |
| aria-controls="navbarContent" aria-expanded="false" aria-label="Toggle navigation"> |
| <span class="navbar-toggler-icon"></span> |
| </button> |
| <div class="collapse navbar-collapse col-md-12 col-lg-auto pt-4" id="navbarContent"> |
| |
| <ul class="navbar-nav me-auto"> |
| <li class="nav-item"> |
| <a class="nav-link active" aria-current="page" href="/downloads.html">Download</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="libraries" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Libraries |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="libraries"> |
| <li><a class="dropdown-item" href="/sql/">SQL and DataFrames</a></li> |
| <li><a class="dropdown-item" href="/spark-connect/">Spark Connect</a></li> |
| <li><a class="dropdown-item" href="/streaming/">Spark Streaming</a></li> |
| <li><a class="dropdown-item" href="/pandas-on-spark/">pandas on Spark</a></li> |
| <li><a class="dropdown-item" href="/mllib/">MLlib (machine learning)</a></li> |
| <li><a class="dropdown-item" href="/graphx/">GraphX (graph)</a></li> |
| <li> |
| <hr class="dropdown-divider"> |
| </li> |
| <li><a class="dropdown-item" href="/third-party-projects.html">Third-Party Projects</a></li> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="documentation" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Documentation |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="documentation"> |
| <li><a class="dropdown-item" href="/docs/latest/">Latest Release</a></li> |
| <li><a class="dropdown-item" href="/documentation.html">Older Versions and Other Resources</a></li> |
| <li><a class="dropdown-item" href="/faq.html">Frequently Asked Questions</a></li> |
| </ul> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link active" aria-current="page" href="/examples.html">Examples</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="community" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Community |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="community"> |
| <li><a class="dropdown-item" href="/community.html">Mailing Lists & Resources</a></li> |
| <li><a class="dropdown-item" href="/contributing.html">Contributing to Spark</a></li> |
| <li><a class="dropdown-item" href="/improvement-proposals.html">Improvement Proposals (SPIP)</a> |
| </li> |
| <li><a class="dropdown-item" href="https://issues.apache.org/jira/browse/SPARK">Issue Tracker</a> |
| </li> |
| <li><a class="dropdown-item" href="/powered-by.html">Powered By</a></li> |
| <li><a class="dropdown-item" href="/committers.html">Project Committers</a></li> |
| <li><a class="dropdown-item" href="/history.html">Project History</a></li> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="developers" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Developers |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="developers"> |
| <li><a class="dropdown-item" href="/developer-tools.html">Useful Developer Tools</a></li> |
| <li><a class="dropdown-item" href="/versioning-policy.html">Versioning Policy</a></li> |
| <li><a class="dropdown-item" href="/release-process.html">Release Process</a></li> |
| <li><a class="dropdown-item" href="/security.html">Security</a></li> |
| </ul> |
| </li> |
| </ul> |
| <ul class="navbar-nav ml-auto"> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="apacheFoundation" role="button" |
| data-bs-toggle="dropdown" aria-expanded="false"> |
| Apache Software Foundation |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="apacheFoundation"> |
| <li><a class="dropdown-item" href="https://www.apache.org/">Apache Homepage</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/licenses/">License</a></li> |
| <li><a class="dropdown-item" |
| href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/security/">Security</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/events/current-event">Event</a></li> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| </nav> |
| |
| <div class="container"> |
| <div class="row mt-4"> |
| <div class="col-12 col-md-9"> |
| <h2>Spark Release 3.0.0</h2> |
| |
| |
| <p>Apache Spark 3.0.0 is the first release of the 3.x line. The vote passed on the 10th of June, 2020. This release is based on git tag <code class="language-plaintext highlighter-rouge">v3.0.0</code> which includes all commits up to June 10. Apache Spark 3.0 builds on many of the innovations from Spark 2.x, bringing new ideas as well as continuing long-term projects that have been in development. With the help of tremendous contributions from the open-source community, this release resolved more than 3400 tickets as the result of contributions from over 440 contributors.</p> |
| |
| <p>This year is Spark’s 10-year anniversary as an open source project. Since its initial release in 2010, Spark has grown to be one of the most active open source projects. Nowadays, Spark is the de facto unified engine for big data processing, data science, machine learning and data analytics workloads.</p> |
| |
| <p>Spark SQL is the top active component in this release. 46% of the resolved tickets are for Spark SQL. These enhancements benefit all the higher-level libraries, including structured streaming and MLlib, and higher level APIs, including SQL and DataFrames. Various related optimizations are added in this release. In TPC-DS 30TB benchmark, Spark 3.0 is roughly two times faster than Spark 2.4.</p> |
| |
| <p>Python is now the most widely used language on Spark. PySpark has more than 5 million monthly downloads on PyPI, the Python Package Index. This release improves its functionalities and usability, including the pandas UDF API redesign with Python type hints, new pandas UDF types, and more Pythonic error handling.</p> |
| |
| <p>Here are the feature highlights in Spark 3.0: adaptive query execution; dynamic partition pruning; ANSI SQL compliance; significant improvements in pandas APIs; new UI for structured streaming; up to 40x speedups for calling R user-defined functions; accelerator-aware scheduler; and SQL reference documentation.</p> |
| |
| <p>To download Apache Spark 3.0.0, visit the <a href="https://spark.apache.org/downloads.html">downloads</a> page. You can consult JIRA for the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12339177">detailed changes</a>. We have curated a list of high level changes here, grouped by major modules.</p> |
| |
| <ul id="markdown-toc"> |
| <li><a href="#core-spark-sql-structured-streaming" id="markdown-toc-core-spark-sql-structured-streaming">Core, Spark SQL, Structured Streaming</a></li> |
| <li><a href="#mllib" id="markdown-toc-mllib">MLlib</a></li> |
| <li><a href="#sparkr" id="markdown-toc-sparkr">SparkR</a></li> |
| <li><a href="#graphx" id="markdown-toc-graphx">GraphX</a></li> |
| <li><a href="#deprecations" id="markdown-toc-deprecations">Deprecations</a></li> |
| <li><a href="#known-issues" id="markdown-toc-known-issues">Known Issues</a></li> |
| <li><a href="#credits" id="markdown-toc-credits">Credits</a></li> |
| </ul> |
| |
| <h3 id="core-spark-sql-structured-streaming">Core, Spark SQL, Structured Streaming</h3> |
| |
| <p><strong>Highlight</strong></p> |
| |
| <ul> |
| <li>[Project Hydrogen] Accelerator-aware Scheduler (<a href="https://issues.apache.org/jira/browse/SPARK-24615">SPARK-24615</a>)</li> |
| <li>Adaptive Query Execution (<a href="https://issues.apache.org/jira/browse/SPARK-31412">SPARK-31412</a>)</li> |
| <li>Dynamic Partition Pruning (<a href="https://issues.apache.org/jira/browse/SPARK-11150">SPARK-11150</a>)</li> |
| <li>Redesigned pandas UDF API with type hints (<a href="https://issues.apache.org/jira/browse/SPARK-28264">SPARK-28264</a>)</li> |
| <li>Structured Streaming UI (<a href="https://issues.apache.org/jira/browse/SPARK-29543">SPARK-29543</a>)</li> |
| <li>Catalog plugin API (<a href="https://issues.apache.org/jira/browse/SPARK-31121">SPARK-31121</a>)</li> |
| <li>Java 11 support (<a href="https://issues.apache.org/jira/browse/SPARK-24417">SPARK-24417</a>)</li> |
| <li>Hadoop 3 support (<a href="https://issues.apache.org/jira/browse/SPARK-23534">SPARK-23534</a>)</li> |
| <li>Better ANSI SQL compatibility</li> |
| </ul> |
| |
| <p><strong>Performance Enhancements</strong></p> |
| |
| <ul> |
| <li>Adaptive Query Execution (<a href="https://issues.apache.org/jira/browse/SPARK-31412">SPARK-31412</a>) |
| <ul> |
| <li>Basic framework (<a href="https://issues.apache.org/jira/browse/SPARK-23128">SPARK-23128</a>)</li> |
| <li>Post shuffle partition number adjustment (<a href="https://issues.apache.org/jira/browse/SPARK-28177">SPARK-28177</a>)</li> |
| <li>Dynamic subquery reuse (<a href="https://issues.apache.org/jira/browse/SPARK-28753">SPARK-28753</a>)</li> |
| <li>Local shuffle reader (<a href="https://issues.apache.org/jira/browse/SPARK-28560">SPARK-28560</a>)</li> |
| <li>Skew join optimization (<a href="https://issues.apache.org/jira/browse/SPARK-29544">SPARK-29544</a>)</li> |
| <li>Optimize reading contiguous shuffle blocks (<a href="https://issues.apache.org/jira/browse/SPARK-9853">SPARK-9853</a>)</li> |
| </ul> |
| </li> |
| <li>Dynamic Partition Pruning (<a href="https://issues.apache.org/jira/browse/SPARK-11150">SPARK-11150</a>)</li> |
| <li>Other optimizer rules |
| <ul> |
| <li>Rule TransposeWindow (<a href="https://issues.apache.org/jira/browse/SPARK-20636">SPARK-20636</a>)</li> |
| <li>Rule ReuseSubquery (<a href="https://issues.apache.org/jira/browse/SPARK-27279">SPARK-27279</a>)</li> |
| <li>Rule PushDownLeftSemiAntiJoin (<a href="https://issues.apache.org/jira/browse/SPARK-19712">SPARK-19712</a>)</li> |
| <li>Rule PushLeftSemiLeftAntiThroughJoin (<a href="https://issues.apache.org/jira/browse/SPARK-19712">SPARK-19712</a>)</li> |
| <li>Rule ReplaceNullWithFalse (<a href="https://issues.apache.org/jira/browse/SPARK-25860">SPARK-25860</a>)</li> |
| <li>Rule Eliminate sorts without limit in the subquery of Join/Aggregation (<a href="https://issues.apache.org/jira/browse/SPARK-29343">SPARK-29343</a>)</li> |
| <li>Rule PruneHiveTablePartitions (<a href="https://issues.apache.org/jira/browse/SPARK-15616">SPARK-15616</a>)</li> |
| <li>Pruning unnecessary nested fields from Generate (<a href="https://issues.apache.org/jira/browse/SPARK-27707">SPARK-27707</a>)</li> |
| <li>Rule RewriteNonCorrelatedExists (<a href="https://issues.apache.org/jira/browse/SPARK-29800">SPARK-29800</a>)</li> |
| </ul> |
| </li> |
| <li>Minimize table cache synchronization costs (<a href="https://issues.apache.org/jira/browse/SPARK-26917">SPARK-26917</a>, <a href="https://issues.apache.org/jira/browse/SPARK-26617">SPARK-26617</a>, <a href="https://issues.apache.org/jira/browse/SPARK-26548">SPARK-26548</a>)</li> |
| <li>Split aggregation code into small functions (<a href="https://issues.apache.org/jira/browse/SPARK-21870">SPARK-21870</a>)</li> |
| <li>Add batching in INSERT and ALTER TABLE ADD PARTITION command (<a href="https://issues.apache.org/jira/browse/SPARK-29938">SPARK-29938</a>)</li> |
| <li>Allows Aggregator to be registered as a UDAF (<a href="https://issues.apache.org/jira/browse/SPARK-27296">SPARK-27296</a>)</li> |
| </ul> |
| |
| <p><strong>SQL Compatibility Enhancements</strong></p> |
| |
| <ul> |
| <li>Switch to Proleptic Gregorian calendar (<a href="https://issues.apache.org/jira/browse/SPARK-26651">SPARK-26651</a>)</li> |
| <li>Build Spark’s own datetime pattern definition (<a href="https://issues.apache.org/jira/browse/SPARK-31408">SPARK-31408</a>)</li> |
| <li>Introduce ANSI store assignment policy for table insertion (<a href="https://issues.apache.org/jira/browse/SPARK-28495">SPARK-28495</a>)</li> |
| <li>Follow ANSI store assignment rule in table insertion by default (<a href="https://issues.apache.org/jira/browse/SPARK-28885">SPARK-28885</a>)</li> |
| <li>Add a SQLConf <code class="language-plaintext highlighter-rouge">spark.sql.ansi.enabled</code> (<a href="https://issues.apache.org/jira/browse/SPARK-28989">SPARK-28989</a>)</li> |
| <li>Support ANSI SQL filter clause for aggregate expression (<a href="https://issues.apache.org/jira/browse/SPARK-27986">SPARK-27986</a>)</li> |
| <li>Support ANSI SQL OVERLAY function (<a href="https://issues.apache.org/jira/browse/SPARK-28077">SPARK-28077</a>)</li> |
| <li>Support ANSI nested bracketed comments (<a href="https://issues.apache.org/jira/browse/SPARK-28880">SPARK-28880</a>)</li> |
| <li>Throw exception on overflow for integers (<a href="https://issues.apache.org/jira/browse/SPARK-26218">SPARK-26218</a>)</li> |
| <li>Overflow check for interval arithmetic operations (<a href="https://issues.apache.org/jira/browse/SPARK-30341">SPARK-30341</a>)</li> |
| <li>Throw Exception when invalid string is cast to numeric type (<a href="https://issues.apache.org/jira/browse/SPARK-30292">SPARK-30292</a>)</li> |
| <li>Make interval multiply and divide’s overflow behavior consistent with other operations (<a href="https://issues.apache.org/jira/browse/SPARK-30919">SPARK-30919</a>)</li> |
| <li>Add ANSI type aliases for char and decimal (<a href="https://issues.apache.org/jira/browse/SPARK-29941">SPARK-29941</a>)</li> |
| <li>SQL Parser defines ANSI compliant reserved keywords (<a href="https://issues.apache.org/jira/browse/SPARK-26215">SPARK-26215</a>)</li> |
| <li>Forbid reserved keywords as identifiers when ANSI mode is on (<a href="https://issues.apache.org/jira/browse/SPARK-26976">SPARK-26976</a>)</li> |
| <li>Support ANSI SQL: LIKE … ESCAPE syntax (<a href="https://issues.apache.org/jira/browse/SPARK-28083">SPARK-28083</a>)</li> |
| <li>Support ANSI SQL Boolean-Predicate syntax (<a href="https://issues.apache.org/jira/browse/SPARK-27924">SPARK-27924</a>)</li> |
| <li>Better support for correlated subquery processing (<a href="https://issues.apache.org/jira/browse/SPARK-18455">SPARK-18455</a>)</li> |
| </ul> |
| |
| <p><strong>PySpark Enhancements</strong></p> |
| |
| <ul> |
| <li>Redesigned pandas UDFs with type hints (<a href="https://issues.apache.org/jira/browse/SPARK-28264">SPARK-28264</a>)</li> |
| <li>Allow Pandas UDF to take an iterator of pd.DataFrames (<a href="https://issues.apache.org/jira/browse/SPARK-26412">SPARK-26412</a>)</li> |
| <li>Support StructType as arguments and return types for Scalar Pandas UDF (<a href="https://issues.apache.org/jira/browse/SPARK-27240">SPARK-27240</a> )</li> |
| <li>Support Dataframe Cogroup via Pandas UDFs (<a href="https://issues.apache.org/jira/browse/SPARK-27463">SPARK-27463</a>)</li> |
| <li>Add mapInPandas to allow an iterator of DataFrames (<a href="https://issues.apache.org/jira/browse/SPARK-28198">SPARK-28198</a>)</li> |
| <li>Certain SQL functions should take column names as well (<a href="https://issues.apache.org/jira/browse/SPARK-26979">SPARK-26979</a>)</li> |
| <li>Make PySpark SQL exceptions more Pythonic (<a href="https://issues.apache.org/jira/browse/SPARK-31849">SPARK-31849</a>)</li> |
| </ul> |
| |
| <p><strong>Extensibility Enhancements</strong></p> |
| |
| <ul> |
| <li>Catalog plugin API (<a href="https://issues.apache.org/jira/browse/SPARK-31121">SPARK-31121</a>)</li> |
| <li>Data source V2 API refactoring (<a href="https://issues.apache.org/jira/browse/SPARK-25390">SPARK-25390</a>)</li> |
| <li>Hive 3.0 and 3.1 metastore support (<a href="https://issues.apache.org/jira/browse/SPARK-27970">SPARK-27970</a>, <a href="https://issues.apache.org/jira/browse/SPARK-24360">SPARK-24360</a>)</li> |
| <li>Extend Spark plugin interface to driver (<a href="https://issues.apache.org/jira/browse/SPARK-29396">SPARK-29396</a>)</li> |
| <li>Extend Spark metrics system with user-defined metrics using executor plugins (<a href="https://issues.apache.org/jira/browse/SPARK-28091">SPARK-28091</a>)</li> |
| <li>Developer APIs for extended Columnar Processing Support (<a href="https://issues.apache.org/jira/browse/SPARK-27396">SPARK-27396</a>)</li> |
| <li>Built-in source migration using DSV2: parquet, ORC, CSV, JSON, Kafka, Text, Avro (<a href="https://issues.apache.org/jira/browse/SPARK-27589">SPARK-27589</a>)</li> |
| <li>Allow FunctionInjection in SparkExtensions (<a href="https://issues.apache.org/jira/browse/SPARK-25560">SPARK-25560</a>)</li> |
| </ul> |
| |
| <p><strong>Connector Enhancements</strong></p> |
| |
| <ul> |
| <li>Support High Performance S3A committers (<a href="https://issues.apache.org/jira/browse/SPARK-23977">SPARK-23977</a>)</li> |
| <li>Column pruning through nondeterministic expressions (<a href="https://issues.apache.org/jira/browse/SPARK-29768">SPARK-29768</a>)</li> |
| <li>Support <code class="language-plaintext highlighter-rouge">spark.sql.statistics.fallBackToHdfs</code> in data source tables (<a href="https://issues.apache.org/jira/browse/SPARK-25474">SPARK-25474</a>)</li> |
| <li>Allow partition pruning with subquery filters on file source (<a href="https://issues.apache.org/jira/browse/SPARK-26893">SPARK-26893</a>)</li> |
| <li>Avoid pushdown of subqueries in data source filters (<a href="https://issues.apache.org/jira/browse/SPARK-25482">SPARK-25482</a>)</li> |
| <li>Recursive data loading from file sources (<a href="https://issues.apache.org/jira/browse/SPARK-27990">SPARK-27990</a>)</li> |
| <li>Parquet/ORC |
| <ul> |
| <li>Pushdown of disjunctive predicates (<a href="https://issues.apache.org/jira/browse/SPARK-27699">SPARK-27699</a>)</li> |
| <li>Generalize Nested Column Pruning (<a href="https://issues.apache.org/jira/browse/SPARK-25603">SPARK-25603</a>) and turned on by default (<a href="https://issues.apache.org/jira/browse/SPARK-29805">SPARK-29805</a>)</li> |
| <li>Parquet Only |
| <ul> |
| <li>Parquet predicate pushdown for nested fields (<a href="https://issues.apache.org/jira/browse/SPARK-17636">SPARK-17636</a>)</li> |
| </ul> |
| </li> |
| <li>ORC Only |
| <ul> |
| <li>Support merge schema for ORC (<a href="https://issues.apache.org/jira/browse/SPARK-11412">SPARK-11412</a>)</li> |
| <li>Nested schema pruning for ORC (<a href="https://issues.apache.org/jira/browse/SPARK-27034">SPARK-27034</a>)</li> |
| <li>Predicate conversion complexity reduction for ORC (<a href="https://issues.apache.org/jira/browse/SPARK-27105">SPARK-27105</a>, <a href="https://issues.apache.org/jira/browse/SPARK-28108">SPARK-28108</a>)</li> |
| <li>Upgrade Apache ORC to 1.5.9 (<a href="https://issues.apache.org/jira/browse/SPARK-30695">SPARK-30695</a>)</li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li>CSV |
| <ul> |
| <li>Support filters pushdown in CSV datasource (<a href="https://issues.apache.org/jira/browse/SPARK-30323">SPARK-30323</a>)</li> |
| </ul> |
| </li> |
| <li>Hive Serde |
| <ul> |
| <li>No schema inference when reading Hive serde table with native data source (<a href="https://issues.apache.org/jira/browse/SPARK-27119">SPARK-27119</a>)</li> |
| <li>Hive CTAS commands should use data source if it is convertible (<a href="https://issues.apache.org/jira/browse/SPARK-25271">SPARK-25271</a>)</li> |
| <li>Use native data source to optimize inserting partitioned Hive table (<a href="https://issues.apache.org/jira/browse/SPARK-28573">SPARK-28573</a>)</li> |
| </ul> |
| </li> |
| <li>Kafka |
| <ul> |
| <li>Add support for Kafka headers (<a href="https://issues.apache.org/jira/browse/SPARK-23539">SPARK-23539</a>)</li> |
| <li>Add Kafka delegation token support (<a href="https://issues.apache.org/jira/browse/SPARK-25501">SPARK-25501</a>)</li> |
| <li>Introduce new option to Kafka source: offset by timestamp (starting/ending) (<a href="https://issues.apache.org/jira/browse/SPARK-26848">SPARK-26848</a>)</li> |
| <li>Support the “minPartitions” option in Kafka batch source and streaming source v1 (<a href="https://issues.apache.org/jira/browse/SPARK-30656">SPARK-30656</a>)</li> |
| <li>Upgrade Kafka to 2.4.1 (<a href="https://issues.apache.org/jira/browse/SPARK-31126">SPARK-31126</a>)</li> |
| </ul> |
| </li> |
| <li>New built-in data sources |
| <ul> |
| <li>New build-in binary file data sources (<a href="https://issues.apache.org/jira/browse/SPARK-25348">SPARK-25348</a>)</li> |
| <li>New no-op batch data sources (<a href="https://issues.apache.org/jira/browse/SPARK-26550">SPARK-26550</a>) and no-op streaming sink (<a href="https://issues.apache.org/jira/browse/SPARK-26649">SPARK-26649</a>)</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <p><strong>Feature Enhancements</strong></p> |
| |
| <ul> |
| <li>[Hydrogen] Accelerator-aware Scheduler (<a href="https://issues.apache.org/jira/browse/SPARK-24615">SPARK-24615</a>)</li> |
| <li>Introduce a complete set of Join Hints (<a href="https://issues.apache.org/jira/browse/SPARK-27225">SPARK-27225</a>)</li> |
| <li>Add PARTITION BY hint for SQL queries (<a href="https://issues.apache.org/jira/browse/SPARK-28746">SPARK-28746</a>)</li> |
| <li>Metadata Handling in Thrift Server (<a href="https://issues.apache.org/jira/browse/SPARK-28426">SPARK-28426</a>)</li> |
| <li>Add higher order functions to scala API (<a href="https://issues.apache.org/jira/browse/SPARK-27297">SPARK-27297</a>)</li> |
| <li>Support simple all gather in barrier task context (<a href="https://issues.apache.org/jira/browse/SPARK-30667">SPARK-30667</a>)</li> |
| <li>Hive UDFs supports the UDT type (<a href="https://issues.apache.org/jira/browse/SPARK-28158">SPARK-28158</a>)</li> |
| <li>Support DELETE/UPDATE/MERGE Operators in Catalyst (<a href="https://issues.apache.org/jira/browse/SPARK-28351">SPARK-28351</a>, <a href="https://issues.apache.org/jira/browse/SPARK-28892">SPARK-28892</a>, <a href="https://issues.apache.org/jira/browse/SPARK-28893">SPARK-28893</a>)</li> |
| <li>Implement DataFrame.tail (<a href="https://issues.apache.org/jira/browse/SPARK-30185">SPARK-30185</a>)</li> |
| <li>New built-in functions |
| <ul> |
| <li>sinh, cosh, tanh, asinh, acosh, atanh (<a href="https://issues.apache.org/jira/browse/SPARK-28133">SPARK-28133</a>)</li> |
| <li>any, every, some (<a href="https://issues.apache.org/jira/browse/SPARK-19851">SPARK-19851</a>)</li> |
| <li>bit_and, bit_or (<a href="https://issues.apache.org/jira/browse/SPARK-27879">SPARK-27879</a>)</li> |
| <li>bit_count (<a href="https://issues.apache.org/jira/browse/SPARK-29491">SPARK-29491</a>)</li> |
| <li>bit_xor (<a href="https://issues.apache.org/jira/browse/SPARK-29545">SPARK-29545</a>)</li> |
| <li>bool_and, bool_or (<a href="https://issues.apache.org/jira/browse/SPARK-30184">SPARK-30184</a>)</li> |
| <li>count_if (<a href="https://issues.apache.org/jira/browse/SPARK-27425">SPARK-27425</a>)</li> |
| <li>date_part (<a href="https://issues.apache.org/jira/browse/SPARK-28690">SPARK-28690</a>)</li> |
| <li>extract (<a href="https://issues.apache.org/jira/browse/SPARK-23903">SPARK-23903</a>)</li> |
| <li>forall (<a href="https://issues.apache.org/jira/browse/SPARK-27905">SPARK-27905</a>)</li> |
| <li>from_csv (<a href="https://issues.apache.org/jira/browse/SPARK-25393">SPARK-25393</a>)</li> |
| <li>make_date (<a href="https://issues.apache.org/jira/browse/SPARK-28432">SPARK-28432</a>)</li> |
| <li>make_interval (<a href="https://issues.apache.org/jira/browse/SPARK-29393">SPARK-29393</a>)</li> |
| <li>make_timestamp (<a href="https://issues.apache.org/jira/browse/SPARK-28459">SPARK-28459</a>)</li> |
| <li>map_entries (<a href="https://issues.apache.org/jira/browse/SPARK-23935">SPARK-23935</a>)</li> |
| <li>map_filter (<a href="https://issues.apache.org/jira/browse/SPARK-23937">SPARK-23937</a>)</li> |
| <li>map_zip_with (<a href="https://issues.apache.org/jira/browse/SPARK-23938">SPARK-23938</a>)</li> |
| <li>max_by, min_by (<a href="https://issues.apache.org/jira/browse/SPARK-27653">SPARK-27653</a>)</li> |
| <li>schema_of_csv (<a href="https://issues.apache.org/jira/browse/SPARK-25672">SPARK-25672</a>)</li> |
| <li>to_csv (<a href="https://issues.apache.org/jira/browse/SPARK-25638">SPARK-25638</a>)</li> |
| <li>transform_keys (<a href="https://issues.apache.org/jira/browse/SPARK-23939">SPARK-23939</a>)</li> |
| <li>transform_values (<a href="https://issues.apache.org/jira/browse/SPARK-23940">SPARK-23940</a>)</li> |
| <li>typeof (<a href="https://issues.apache.org/jira/browse/SPARK-29961">SPARK-29961</a>)</li> |
| <li>version (<a href="https://issues.apache.org/jira/browse/SPARK-29554">SPARK-29554</a>)</li> |
| <li>xxhash64 (<a href="https://issues.apache.org/jira/browse/SPARK-27099">SPARK-27099</a>)</li> |
| </ul> |
| </li> |
| <li>Improvements on the existing built-in functions |
| <ul> |
| <li>built-in date-time functions/operations improvement (<a href="https://issues.apache.org/jira/browse/SPARK-31415">SPARK-31415</a>)</li> |
| <li>Support FAILFAST mode for from_json (<a href="https://issues.apache.org/jira/browse/SPARK-25243">SPARK-25243</a>)</li> |
| <li>array_sort adds a new comparator parameter (<a href="https://issues.apache.org/jira/browse/SPARK-29020">SPARK-29020</a>)</li> |
| <li>filter can now take the index as input as well as the element (<a href="https://issues.apache.org/jira/browse/SPARK-28962">SPARK-28962</a>)</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <p><strong>Monitoring and Debuggability Enhancements</strong></p> |
| |
| <ul> |
| <li>New Structured Streaming UI (<a href="https://issues.apache.org/jira/browse/SPARK-29543">SPARK-29543</a>)</li> |
| <li>SHS: Allow event logs for running streaming apps to be rolled over (<a href="https://issues.apache.org/jira/browse/SPARK-28594">SPARK-28594</a>)</li> |
| <li>JDBC tab in SHS (<a href="https://issues.apache.org/jira/browse/SPARK-29724">SPARK-29724</a>, <a href="https://issues.apache.org/jira/browse/SPARK-29726">SPARK-29726</a>)</li> |
| <li>Add an API that allows a user to define and observe arbitrary metrics on batch and streaming queries (<a href="https://issues.apache.org/jira/browse/SPARK-29345">SPARK-29345</a>)</li> |
| <li>Instrumentation for tracking per-query planning time (<a href="https://issues.apache.org/jira/browse/SPARK-26129">SPARK-26129</a>)</li> |
| <li>Put the basic shuffle metrics in the SQL exchange operator (<a href="https://issues.apache.org/jira/browse/SPARK-26139">SPARK-26139</a>)</li> |
| <li>SQL statement is shown in SQL Tab instead of callsite (<a href="https://issues.apache.org/jira/browse/SPARK-27045">SPARK-27045</a>)</li> |
| <li>Add tooltip to SparkUI (<a href="https://issues.apache.org/jira/browse/SPARK-29449">SPARK-29449</a>)</li> |
| <li>Improve the concurrent performance of History Server (<a href="https://issues.apache.org/jira/browse/SPARK-29043">SPARK-29043</a>)</li> |
| <li>EXPLAIN FORMATTED command (<a href="https://issues.apache.org/jira/browse/SPARK-27395">SPARK-27395</a>)</li> |
| <li>Support Dumping truncated plans and generated code to a file (<a href="https://issues.apache.org/jira/browse/SPARK-26023">SPARK-26023</a>)</li> |
| <li>Enhance describe framework to describe the output of a query (<a href="https://issues.apache.org/jira/browse/SPARK-26982">SPARK-26982</a>)</li> |
| <li>Add SHOW VIEWS command (<a href="https://issues.apache.org/jira/browse/SPARK-31113">SPARK-31113</a>)</li> |
| <li>Improve the error messages of SQL parser (<a href="https://issues.apache.org/jira/browse/SPARK-27901">SPARK-27901</a>)</li> |
| <li>Support Prometheus monitoring natively (<a href="https://issues.apache.org/jira/browse/SPARK-29429">SPARK-29429</a>)</li> |
| <li>Add executor memory metrics to heartbeat and expose in executors REST API (<a href="https://issues.apache.org/jira/browse/SPARK-23429">SPARK-23429</a>)</li> |
| <li>Add Executor metrics and memory usage instrumentation to the metrics system (<a href="https://issues.apache.org/jira/browse/SPARK-27189">SPARK-27189</a>)</li> |
| </ul> |
| |
| <p><strong>Documentation and Test Coverage Enhancements</strong></p> |
| |
| <ul> |
| <li>Build a SQL Reference (<a href="https://issues.apache.org/jira/browse/SPARK-28588">SPARK-28588</a>)</li> |
| <li>Build a user guide for WebUI (<a href="https://issues.apache.org/jira/browse/SPARK-28372">SPARK-28372</a>)</li> |
| <li>Build a page for SQL configuration documentation (<a href="https://issues.apache.org/jira/browse/SPARK-30510">SPARK-30510</a>)</li> |
| <li>Add version information for Spark configuration (<a href="https://issues.apache.org/jira/browse/SPARK-30839">SPARK-30839</a>)</li> |
| <li>Port regression tests from PostgreSQL (<a href="https://issues.apache.org/jira/browse/SPARK-27763">SPARK-27763</a>)</li> |
| <li>Thrift-server test coverage (<a href="https://issues.apache.org/jira/browse/SPARK-28608">SPARK-28608</a>)</li> |
| <li>Test coverage of UDFs (python UDF, pandas UDF, scala UDF) (<a href="https://issues.apache.org/jira/browse/SPARK-27921">SPARK-27921</a>)</li> |
| </ul> |
| |
| <p><strong>Native Spark App in Kubernetes</strong></p> |
| |
| <ul> |
| <li>Support user-specified driver and executor pod templates (<a href="https://issues.apache.org/jira/browse/SPARK-24434">SPARK-24434</a>)</li> |
| <li>Allow dynamic allocation without an external shuffle service (<a href="https://issues.apache.org/jira/browse/SPARK-27963">SPARK-27963</a>)</li> |
| <li>More responsive dynamic allocation with K8S (<a href="https://issues.apache.org/jira/browse/SPARK-28487">SPARK-28487</a>)</li> |
| <li>Kerberos Support for Spark on K8S (<a href="https://issues.apache.org/jira/browse/SPARK-23257">SPARK-23257</a>)</li> |
| <li>Kerberos Support in Kubernetes resource manager (Client Mode) (<a href="https://issues.apache.org/jira/browse/SPARK-25815">SPARK-25815</a>)</li> |
| <li>Support client dependencies with a Hadoop Compatible File System (<a href="https://issues.apache.org/jira/browse/SPARK-23153">SPARK-23153</a>)</li> |
| <li>Add configurable auth secret source in k8s backend (<a href="https://issues.apache.org/jira/browse/SPARK-26239">SPARK-26239</a>)</li> |
| <li>Support subpath mounting with Kubernetes (<a href="https://issues.apache.org/jira/browse/SPARK-25960">SPARK-25960</a>)</li> |
| <li>Make Python 3 the default in PySpark Bindings for K8S (<a href="https://issues.apache.org/jira/browse/SPARK-24516">SPARK-24516</a>)</li> |
| </ul> |
| |
| <p><strong>Other Notable Changes</strong></p> |
| |
| <ul> |
| <li>Java 11 support (<a href="https://issues.apache.org/jira/browse/SPARK-24417">SPARK-24417</a>)</li> |
| <li>Hadoop 3 support (<a href="https://issues.apache.org/jira/browse/SPARK-23534">SPARK-23534</a>)</li> |
| <li>Built-in Hive execution upgrade from 1.2.1 to 2.3.7 (<a href="https://issues.apache.org/jira/browse/SPARK-23710">SPARK-23710</a>, <a href="https://issues.apache.org/jira/browse/SPARK-28723">SPARK-28723</a>, <a href="https://issues.apache.org/jira/browse/SPARK-31381">SPARK-31381</a>)</li> |
| <li>Use Apache Hive 2.3 dependency by default (<a href="https://issues.apache.org/jira/browse/SPARK-30034">SPARK-30034</a>)</li> |
| <li>GA Scala 2.12 and remove 2.11 (<a href="https://issues.apache.org/jira/browse/SPARK-26132">SPARK-26132</a>)</li> |
| <li>Improve logic for timing out executors in dynamic allocation (<a href="https://issues.apache.org/jira/browse/SPARK-20286">SPARK-20286</a>)</li> |
| <li>Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation (<a href="https://issues.apache.org/jira/browse/SPARK-27677">SPARK-27677</a>)</li> |
| <li>Acquire new executors to avoid hang because of blacklisting (<a href="https://issues.apache.org/jira/browse/SPARK-22148">SPARK-22148</a>)</li> |
| <li>Allow sharing Netty’s memory pool allocators (<a href="https://issues.apache.org/jira/browse/SPARK-24920">SPARK-24920</a>)</li> |
| <li>Fix deadlock between TaskMemoryManager and UnsafeExternalSorter$SpillableIterator (<a href="https://issues.apache.org/jira/browse/SPARK-27338">SPARK-27338</a>)</li> |
| <li>Introduce AdmissionControl APIs for StructuredStreaming (<a href="https://issues.apache.org/jira/browse/SPARK-30669">SPARK-30669</a>)</li> |
| <li>Spark History Main page performance improvement (<a href="https://issues.apache.org/jira/browse/SPARK-25973">SPARK-25973</a>)</li> |
| <li>Speed up and slim down metric aggregation in SQL listener (<a href="https://issues.apache.org/jira/browse/SPARK-29562">SPARK-29562</a>)</li> |
| <li>Avoid the network when shuffle blocks are fetched from the same host (<a href="https://issues.apache.org/jira/browse/SPARK-27651">SPARK-27651</a>)</li> |
| <li>Improve file listing for DistributedFileSystem (<a href="https://issues.apache.org/jira/browse/SPARK-27801">SPARK-27801</a>)</li> |
| <li>Remove Support for hadoop 2.6 (<a href="https://issues.apache.org/jira/browse/SPARK-25016">SPARK-25016</a>)</li> |
| </ul> |
| |
| <p><strong>Changes of behavior</strong></p> |
| |
| <p>Please read the migration guides for each component: <a href="https://spark.apache.org/docs/3.0.0/core-migration-guide.html">Spark Core</a>, <a href="https://spark.apache.org/docs/3.0.0/sql-migration-guide.html">Spark SQL</a>, <a href="https://spark.apache.org/docs/3.0.0/ss-migration-guide.html">Structured Streaming</a> and <a href="https://spark.apache.org/docs/3.0.0/pyspark-migration-guide.html">PySpark</a>.</p> |
| |
| <p>A few other behavior changes that are missed in the migration guide:</p> |
| |
| <ul> |
| <li>In Spark 3.0, the deprecated class <code class="language-plaintext highlighter-rouge">org.apache.spark.sql.streaming.ProcessingTime</code> has been removed. Use <code class="language-plaintext highlighter-rouge">org.apache.spark.sql.streaming.Trigger.ProcessingTime</code> instead. Likewise, <code class="language-plaintext highlighter-rouge">org.apache.spark.sql.execution.streaming.continuous.ContinuousTrigger</code> has been removed in favor of <code class="language-plaintext highlighter-rouge">Trigger.Continuous</code>, and <code class="language-plaintext highlighter-rouge">org.apache.spark.sql.execution.streaming.OneTimeTrigger</code> has been hidden in favor of <code class="language-plaintext highlighter-rouge">Trigger.Once</code>. (<a href="https://issues.apache.org/jira/browse/SPARK-28199">SPARK-28199</a>)</li> |
| <li>Due to the upgrade of Scala 2.12, <code class="language-plaintext highlighter-rouge">DataStreamWriter.foreachBatch</code> is not source compatible for Scala program. You need to update your Scala source code to disambiguate between Scala function and Java lambda. (<a href="https://issues.apache.org/jira/browse/SPARK-26132">SPARK-26132</a>)</li> |
| </ul> |
| |
| <p><em>Programming guides: <a href="https://spark.apache.org/docs/3.0.0/rdd-programming-guide.html">Spark RDD Programming Guide</a> and <a href="https://spark.apache.org/docs/3.0.0/sql-programming-guide.html">Spark SQL, DataFrames and Datasets Guide</a> and <a href="https://spark.apache.org/docs/3.0.0/structured-streaming-programming-guide.html">Structured Streaming Programming Guide</a>.</em></p> |
| |
| <h3 id="mllib">MLlib</h3> |
| |
| <p><strong>Highlight</strong></p> |
| |
| <ul> |
| <li>Multiple columns support was added to Binarizer (<a href="https://issues.apache.org/jira/browse/SPARK-23578">SPARK-23578</a>), StringIndexer (<a href="https://issues.apache.org/jira/browse/SPARK-11215">SPARK-11215</a>), StopWordsRemover (<a href="https://issues.apache.org/jira/browse/SPARK-29808">SPARK-29808</a>) and PySpark QuantileDiscretizer (<a href="https://issues.apache.org/jira/browse/SPARK-22796">SPARK-22796</a>)</li> |
| <li>Support Tree-Based Feature Transformation(<a href="https://issues.apache.org/jira/browse/SPARK-13677">SPARK-13677</a>)</li> |
| <li>Two new evaluators MultilabelClassificationEvaluator (<a href="https://issues.apache.org/jira/browse/SPARK-16692">SPARK-16692</a>) and RankingEvaluator (<a href="https://issues.apache.org/jira/browse/SPARK-28045">SPARK-28045</a>) were added</li> |
| <li>Sample weights support was added in DecisionTreeClassifier/Regressor (<a href="https://issues.apache.org/jira/browse/SPARK-19591">SPARK-19591</a>), RandomForestClassifier/Regressor (<a href="https://issues.apache.org/jira/browse/SPARK-9478">SPARK-9478</a>), GBTClassifier/Regressor (<a href="https://issues.apache.org/jira/browse/SPARK-9612">SPARK-9612</a>), RegressionEvaluator (<a href="https://issues.apache.org/jira/browse/SPARK-24102">SPARK-24102</a>), BinaryClassificationEvaluator (<a href="https://issues.apache.org/jira/browse/SPARK-24103">SPARK-24103</a>), BisectingKMeans (<a href="https://issues.apache.org/jira/browse/SPARK-30351">SPARK-30351</a>), KMeans (<a href="https://issues.apache.org/jira/browse/SPARK-29967">SPARK-29967</a>) and GaussianMixture (<a href="https://issues.apache.org/jira/browse/SPARK-30102">SPARK-30102</a>)</li> |
| <li>R API for PowerIterationClustering was added (<a href="https://issues.apache.org/jira/browse/SPARK-19827">SPARK-19827</a>)</li> |
| <li>Added Spark ML listener for tracking ML pipeline status (<a href="https://issues.apache.org/jira/browse/SPARK-23674">SPARK-23674</a>)</li> |
| <li>Fit with validation set was added to Gradient Boosted Trees in Python (<a href="https://issues.apache.org/jira/browse/SPARK-24333">SPARK-24333</a>)</li> |
| <li><a href="https://github.com/huaxingao/spark/blob/e00a5a751bcd769ed92ff26a84c702b176cf1671/docs/ml-features.html#robustscaler">RobustScaler</a> transformer was added (<a href="https://issues.apache.org/jira/browse/SPARK-28399">SPARK-28399</a>)</li> |
| <li><a href="https://github.com/huaxingao/spark/blob/e00a5a751bcd769ed92ff26a84c702b176cf1671/docs/ml-classification-regression.html#factorization-machines">Factorization Machines</a> classifier and regressor were added (<a href="https://issues.apache.org/jira/browse/SPARK-29224">SPARK-29224</a>)</li> |
| <li>Gaussian Naive Bayes (<a href="https://issues.apache.org/jira/browse/SPARK-16872">SPARK-16872</a>) and Complement Naive Bayes (<a href="https://issues.apache.org/jira/browse/SPARK-29942">SPARK-29942</a>) were added</li> |
| <li>ML function parity between Scala and Python (<a href="https://issues.apache.org/jira/browse/SPARK-28958">SPARK-28958</a>)</li> |
| <li>predictRaw is made public in all the Classification models. predictProbability is made public in all the Classification models except LinearSVCModel (<a href="https://issues.apache.org/jira/browse/SPARK-30358">SPARK-30358</a>)</li> |
| </ul> |
| |
| <p><strong>Changes of behavior</strong></p> |
| |
| <p>Please read the <a href="https://spark.apache.org/docs/3.0.0/ml-migration-guide.html">migration guide</a> for details.</p> |
| |
| <p>A few other behavior changes that are missed in the migration guide:</p> |
| |
| <ul> |
| <li>In Spark 3.0, a multiclass logistic regression in Pyspark will now (correctly) return LogisticRegressionSummary, not the subclass BinaryLogisticRegressionSummary. The additional methods exposed by BinaryLogisticRegressionSummary would not work in this case anyway. (<a href="https://issues.apache.org/jira/browse/SPARK-31681">SPARK-31681</a>)</li> |
| <li>In Spark 3.0, pyspark.ml.param.shared.Has* mixins do not provide any set<em>(self, value) setter methods anymore, use the respective self.set(self.</em>, value) instead. See SPARK-29093 for details. (<a href="https://issues.apache.org/jira/browse/SPARK-29093">SPARK-29093</a>)</li> |
| </ul> |
| |
| <p><em>Programming guide: <a href="https://spark.apache.org/docs/3.0.0/ml-guide.html">Machine Learning Library (MLlib) Guide</a>.</em></p> |
| |
| <h3 id="sparkr">SparkR</h3> |
| |
| <ul> |
| <li>Arrow optimization in SparkR’s interoperability (<a href="https://issues.apache.org/jira/browse/SPARK-26759">SPARK-26759</a>) |
| <ul> |
| <li>Performance enhancement via vectorized R gapply(), dapply(), createDataFrame, collect()</li> |
| </ul> |
| </li> |
| <li>“eager execution” for R shell, IDE (<a href="https://issues.apache.org/jira/browse/SPARK-24572">SPARK-24572</a>)</li> |
| <li>R API for Power Iteration Clustering (<a href="https://issues.apache.org/jira/browse/SPARK-19827">SPARK-19827</a>)</li> |
| </ul> |
| |
| <p><strong>Changes of behavior</strong></p> |
| |
| <p>Please read the <a href="https://spark.apache.org/docs/3.0.0/sparkr-migration-guide.html">migration guide</a> for details.</p> |
| |
| <p><em>Programming guide: <a href="https://spark.apache.org/docs/3.0.0/sparkr.html">SparkR (R on Spark)</a>.</em></p> |
| |
| <h3 id="graphx">GraphX</h3> |
| |
| <p><em>Programming guide: <a href="https://spark.apache.org/docs/3.0.0/graphx-programming-guide.html">GraphX Programming Guide</a>.</em></p> |
| |
| <h3 id="deprecations">Deprecations</h3> |
| |
| <ul> |
| <li>Deprecate Python 2 support (<a href="https://issues.apache.org/jira/browse/SPARK-27884">SPARK-27884</a>)</li> |
| <li>Deprecate R < 3.4 support (<a href="https://issues.apache.org/jira/browse/SPARK-26014">SPARK-26014</a>)</li> |
| <li>Deprecate UserDefinedAggregateFunction (<a href="https://issues.apache.org/jira/browse/SPARK-30423">Spark-30423</a>)</li> |
| </ul> |
| |
| <h3 id="known-issues">Known Issues</h3> |
| |
| <ul> |
| <li>Streaming queries with <code class="language-plaintext highlighter-rouge">dropDuplicates</code> operator may not be able to restart with the checkpoint written by Spark 2.x. This will be fixed in Spark 3.0.1. (<a href="https://issues.apache.org/jira/browse/SPARK-31990">SPARK-31990</a>)</li> |
| <li>In Web UI, the job list page may hang for more than 40 seconds. This will be fixed in Spark 3.0.1. (<a href="https://issues.apache.org/jira/browse/SPARK-31967">SPARK-31967</a>)</li> |
| <li>Set <code class="language-plaintext highlighter-rouge">io.netty.tryReflectionSetAccessible</code> for Arrow on JDK9+ (<a href="https://issues.apache.org/jira/browse/SPARK-29923">SPARK-29923</a>)</li> |
| <li>With AWS SDK upgrade to 1.11.655, we strongly encourage the users that use S3N file system (open-source NativeS3FileSystem that is based on jets3t library) on Hadoop 2.7.3 to upgrade to use AWS Signature V4 and set the bucket endpoint or migrate to S3A (“s3a://” prefix) - jets3t library uses AWS v2 by default and s3.amazonaws.com as an endpoint. Otherwise, the 403 Forbidden error may be thrown in the following cases: |
| <ul> |
| <li>If a user accesses an S3 path that contains “+” characters and uses the legacy S3N file system, e.g. s3n://bucket/path/+file.</li> |
| <li>If a user has configured AWS V2 signature to sign requests to S3 with S3N file system.</li> |
| </ul> |
| |
| <p>Note that if you use S3AFileSystem, e.g. (“s3a://bucket/path”) to access S3 in S3Select or SQS connectors, then everything will work as expected. (<a href="https://issues.apache.org/jira/browse/SPARK-30968">SPARK-30968</a>)</p> |
| </li> |
| <li>Parsing day of year using pattern letter ‘D’ returns the wrong result if the year field is missing. This can happen in SQL functions like <code class="language-plaintext highlighter-rouge">to_timestamp</code> which parses datetime string to datetime values using a pattern string. This will be fixed in Spark 3.0.1. (<a href="https://issues.apache.org/jira/browse/SPARK-31939">SPARK-31939</a>)</li> |
| <li>Join/Window/Aggregate inside subqueries may lead to wrong results if the keys have values -0.0 and 0.0. This will be fixed in Spark 3.0.1. (<a href="https://issues.apache.org/jira/browse/SPARK-31958">SPARK-31958</a>)</li> |
| <li>A window query may fail with ambiguous self-join error unexpectedly. This will be fixed in Spark 3.0.1. (<a href="https://issues.apache.org/jira/browse/SPARK-31956">SPARK-31956</a>)</li> |
| </ul> |
| |
| <h3 id="credits">Credits</h3> |
| |
| <p>Last but not least, this release would not have been possible without the following contributors: Aaruna Godthi, Adam Binford, Adi Muraru, Adrian Tanase, Ajith S, Akshat Bordia, Ala Luszczak, Aleksandr Kashkirov, Alessandro Bellina, Alex Hagerman, Ali Afroozeh, Ali Smesseim, Alon Doron, Aman Omer, Anastasios Zouzias, Anca Sarb, Andre Sa De Mello, Andrew Crosby, Andy Grove, Andy Zhang, Ankit Raj Boudh, Ankur Gupta, Anton Kirillov, Anton Okolnychyi, Anton Yanchenko, Artem Kalchenko, Artem Kupchinskiy, Artsiom Yudovin, Arun Mahadevan, Arun Pandian, Asaf Levy, Attila Zsolt Piros, Bago Amirbekian, Baohe Zhang, Bartosz Konieczny, Behroz Sikander, Ben Ryves, Bo Hai, Bogdan Ghit, Boris Boutkov, Boris Shminke, Branden Smith, Brandon Krieger, Brian Scannell, Brooke Wenig, Bruce Robbins, Bryan Cutler, Burak Yavuz, Carson Wang, Chaerim Yeo, Chakravarthi, Chandni Singh, Chandu Kavar, Chaoqun Li, Chen Hao, Cheng Lian, Chenxiao Mao, Chitral Verma, Chris Martin, Chris Zhao, Christian Clauss, Christian Stuart, Cody Koeninger, Colin Ma, Cong Du, DB Tsai, Dang Minh Dung, Daoyuan Wang, Darcy Shen, Darren Tirto, Dave DeCaprio, David Lewis, David Lindelof, David Navas, David Toneian, David Vogelbacher, David Vrba, David Yang, Deepyaman Datta, Devaraj K, Dhruve Ashar, Dianjun Ma, Dilip Biswal, Dima Kamalov, Dongdong Hong, Dongjoon Hyun, Dooyoung Hwang, Douglas R Colkitt, Drew Robb, Dylan Guedes, Edgar Rodriguez, Edwina Lu, Emil Sandsto, Enrico Minack, Eren Avsarogullari, Eric Chang, Eric Liang, Eric Meisel, Eric Wu, Erik Christiansen, Erik Erlandson, Eyal Zituny, Fei Wang, Felix Cheung, Fokko Driesprong, Fuwang Hu, Gabbi Merz, Gabor Somogyi, Gengliang Wang, German Schiavon Matteo, Giovanni Lanzani, Greg Senia, Guangxin Wang, Guilherme Souza, Guy Khazma, Haiyang Yu, Helen Yu, Hemanth Meka, Henrique Goulart, Henry D, Herman Van Hovell, Hirobe Keiichi, Holden Karau, Hossein Falaki, Huaxin Gao, Huon Wilson, Hyukjin Kwon, Icysandwich, Ievgen Prokhorenko, Igor Calabria, Ilan Filonenko, Ilya Matiach, Imran Rashid, Ivan Gozali, Ivan Vergiliev, Izek Greenfield, Jacek Laskowski, Jackey Lee, Jagadesh Kiran, Jalpan Randeri, James Lamb, Jamison Bennett, Jash Gala, Jatin Puri, Javier Fuentes, Jeff Evans, Jenny, Jesse Cai, Jiaan Geng, Jiafu Zhang, Jiajia Li, Jian Tang, Jiaqi Li, Jiaxin Shan, Jing Chen He, Joan Fontanals, Jobit Mathew, Joel Genter, John Ayad, John Bauer, John Zhuge, Jorge Machado, Jose Luis Pedrosa, Jose Torres, Joseph K. Bradley, Josh Rosen, Jules Damji, Julien Peloton, Juliusz Sompolski, Jungtaek Lim, Junjie Chen, Justin Uang, Kang Zhou, Karthikeyan Singaravelan, Karuppayya Rajendran, Kazuaki Ishizaki, Ke Jia, Keiji Yoshida, Keith Sun, Kengo Seki, Kent Yao, Ketan Kunde, Kevin Yu, Koert Kuipers, Kousuke Saruta, Kris Mok, Lantao Jin, Lee Dongjin, Lee Moon Soo, Li Hao, Li Jin, Liang Chen, Liang Li, Liang Zhang, Liang-Chi Hsieh, Lijia Liu, Lingang Deng, Lipeng Zhu, Liu Xiao, Liu, Linhong, Liwen Sun, Luca Canali, MJ Tang, Maciej Szymkiewicz, Manu Zhang, Marcelo Vanzin, Marco Gaido, Marek Simunek, Mark Pavey, Martin Junghanns, Martin Loncaric, Maryann Xue, Masahiro Kazama, Matt Hawes, Matt Molek, Matt Stillwell, Matthew Cheah, Maxim Gekk, Maxim Kolesnikov, Mellacheruvu Sandeep, Michael Allman, Michael Chirico, Michael Styles, Michal Senkyr, Mick Jermsurawong, Mike Kaplinskiy, Mingcong Han, Mukul Murthy, Nagaram Prasad Addepally, Nandor Kollar, Neal Song, Neo Chien, Nicholas Chammas, Nicholas Marion, Nick Karpov, Nicola Bova, Nicolas Fraison, Nihar Sheth, Nik Vanderhoof, Nikita Gorbachevsky, Nikita Konda, Ninad Ingole, Niranjan Artal, Nishchal Venkataramana, Norman Maurer, Ohad Raviv, Oleg Kuznetsov, Oleksii Kachaiev, Oleksii Shkarupin, Oliver Urs Lenz, Onur Satici, Owen O’Malley, Ozan Cicekci, Pablo Langa Blanco, Parker Hegstrom, Parth Chandra, Parth Gandhi, Patrick Brown, Patrick Cording, Patrick Pisciuneri, Pavithra Ramachandran, Peng Bo, Pengcheng Liu, Petar Petrov, Peter G. Horvath, Peter Parente, Peter Toth, Philipse Guo, Prakhar Jain, Pralabh Kumar, Praneet Sharma, Prashant Sharma, Qi Shao, Qianyang Yu, Rafael Renaudin, Rahij Ramsharan, Rahul Mahadev, Rakesh Raushan, Rekha Joshi, Reynold Xin, Reza Safi, Rob Russo, Rob Vesse, Robert (Bobby) Evans, Rong Ma, Ross Lodge, Ruben Fiszel, Ruifeng Zheng, Ruilei Ma, Russell Spitzer, Ryan Blue, Ryne Yang, Sahil Takiar, Saisai Shao, Sam Tran, Samuel L. Setegne, Sandeep Katta, Sangram Gaikwad, Sanket Chintapalli, Sanket Reddy, Sarth Frey, Saurabh Chawla, Sean Owen, Sergey Zhemzhitsky, Seth Fitzsimmons, Shahid, Shahin Shakeri, Shane Knapp, Shanyu Zhao, Shaochen Shi, Sharanabasappa G Keriwaddi, Sharif Ahmad, Shiv Prashant Sood, Shivakumar Sondur, Shixiong Zhu, Shuheng Dai, Shuming Li, Simeon Simeonov, Song Jun, Stan Zhai, Stavros Kontopoulos, Stefaan Lippens, Steve Loughran, Steven Aerts, Steven Rand, Sujith Chacko, Sun Ke, Sunitha Kambhampati, Szilard Nemeth, Tae-kyeom, Kim, Takanobu Asanuma, Takeshi Yamamuro, Takuya UESHIN, Tarush Grover, Tathagata Das, Terry Kim, Thomas D’Silva, Thomas Graves, Tianshi Zhu, Tiantian Han, Tibor Csogor, Tin Hang To, Ting Yang, Tingbing Zuo, Tom Van Bussel, Tomoko Komiyama, Tony Zhang, TopGunViper, Udbhav Agrawal, Uncle Gen, Vaclav Kosar, Venkata Krishnan Sowrirajan, Viktor Tarasenko, Vinod KC, Vinoo Ganesh, Vladimir Kuriatkov, Wang Shuo, Wayne Zhang, Wei Zhang, Weichen Xu, Weiqiang Zhuang, Weiyi Huang, Wenchen Fan, Wenjie Wu, Wesley Hoffman, William Hyun, William Montaz, William Wong, Wing Yew Poon, Woudy Gao, Wu, Xiaochang, XU Duo, Xian Liu, Xiangrui Meng, Xianjin YE, Xianyang Liu, Xianyin Xin, Xiao Li, Xiaoyuan Ding, Ximo Guanter, Xingbo Jiang, Xingcan Cui, Xinglong Wang, Xinrong Meng, XiuLi Wei, Xuedong Luan, Xuesen Liang, Xuewen Cao, Yadong Song, Yan Ma, Yanbo Liang, Yang Jie, Yanlin Wang, Yesheng Ma, Yi Wu, Yi Zhu, Yifei Huang, Yiheng Wang, Yijie Fan, Yin Huai, Yishuang Lu, Yizhong Zhang, Yogesh Garg, Yongjin Zhou, Yongqiang Chai, Younggyu Chun, Yuanjian Li, Yucai Yu, Yuchen Huo, Yuexin Zhang, Yuhao Yang, Yuli Fiterman, Yuming Wang, Yun Zou, Zebing Lin, Zhenhua Wang, Zhou Jiang, Zhu, Lipeng, codeborui, cxzl25, dengziming, deshanxiao, eatoncys, hehuiyuan, highmoutain, huangtianhua, liucht-inspur, mob-ai, nooberfsh, roland1982, teeyog, tools4origins, triplesheep, ulysses-you, wackxu, wangjiaochun, wangshisan, wenfang6, wenxuanguan</p> |
| |
| |
| <p> |
| <br/> |
| <a href="/news/">Spark News Archive</a> |
| </p> |
| |
| </div> |
| <div class="col-12 col-md-3"> |
| <div class="news" style="margin-bottom: 20px;"> |
| <h5>Latest News</h5> |
| <ul class="list-unstyled"> |
| |
| <li><a href="/news/spark-3-4-3-released.html">Spark 3.4.3 released</a> |
| <span class="small">(Apr 18, 2024)</span></li> |
| |
| <li><a href="/news/spark-3-5-1-released.html">Spark 3.5.1 released</a> |
| <span class="small">(Feb 23, 2024)</span></li> |
| |
| <li><a href="/news/spark-3-3-4-released.html">Spark 3.3.4 released</a> |
| <span class="small">(Dec 16, 2023)</span></li> |
| |
| <li><a href="/news/spark-3-4-2-released.html">Spark 3.4.2 released</a> |
| <span class="small">(Nov 30, 2023)</span></li> |
| |
| </ul> |
| <p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p> |
| </div> |
| <div style="text-align:center; margin-bottom: 20px;"> |
| <a href="https://www.apache.org/events/current-event.html"> |
| <img src="https://www.apache.org/events/current-event-234x60.png" style="max-width: 100%;"/> |
| </a> |
| </div> |
| <div class="hidden-xs hidden-sm"> |
| <a href="/downloads.html" class="btn btn-cta btn-lg d-grid" style="margin-bottom: 30px;"> |
| Download Spark |
| </a> |
| <p style="font-size: 16px; font-weight: 500; color: #555;"> |
| Built-in Libraries: |
| </p> |
| <ul class="list-none"> |
| <li><a href="/sql/">SQL and DataFrames</a></li> |
| <li><a href="/streaming/">Spark Streaming</a></li> |
| <li><a href="/mllib/">MLlib (machine learning)</a></li> |
| <li><a href="/graphx/">GraphX (graph)</a></li> |
| </ul> |
| <a href="/third-party-projects.html">Third-Party Projects</a> |
| </div> |
| </div> |
| </div> |
| |
| |
| |
| <footer class="small"> |
| <hr> |
| Apache Spark, Spark, Apache, the Apache feather logo, and the Apache Spark project logo are either registered |
| trademarks or trademarks of The Apache Software Foundation in the United States and other countries. |
| See guidance on use of Apache Spark <a href="/trademarks.html">trademarks</a>. |
| All other marks mentioned may be trademarks or registered trademarks of their respective owners. |
| Copyright © 2018 The Apache Software Foundation, Licensed under the |
| <a href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>. |
| </footer> |
| </div> |
| |
| <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/js/bootstrap.bundle.min.js" |
| integrity="sha384-MrcW6ZMFYlzcLA8Nl+NtUVF0sA7MsXsP1UyJoMp4YLEuNSfAP+JcXn/tWtIaxVXM" |
| crossorigin="anonymous"></script> |
| <script src="https://code.jquery.com/jquery.js"></script> |
| <script src="/js/lang-tabs.js"></script> |
| <script src="/js/downloads.js"></script> |
| </body> |
| </html> |
