| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <title> |
| Versioning Policy | Apache Spark |
| |
| </title> |
| |
| |
| |
| |
| |
| <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css" rel="stylesheet" |
| integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC" crossorigin="anonymous"> |
| <link rel="preconnect" href="https://fonts.googleapis.com"> |
| <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> |
| <link href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,wght@0,400;0,500;0,700;1,400;1,500;1,700&Courier+Prime:wght@400;700&display=swap" rel="stylesheet"> |
| <link href="/css/custom.css" rel="stylesheet"> |
| <!-- Code highlighter CSS --> |
| <link href="/css/pygments-default.css" rel="stylesheet"> |
| <link rel="icon" href="/favicon.ico" type="image/x-icon"> |
| |
| <!-- Matomo --> |
| <script> |
| var _paq = window._paq = window._paq || []; |
| /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ |
| _paq.push(["disableCookies"]); |
| _paq.push(['trackPageView']); |
| _paq.push(['enableLinkTracking']); |
| (function() { |
| var u="https://analytics.apache.org/"; |
| _paq.push(['setTrackerUrl', u+'matomo.php']); |
| _paq.push(['setSiteId', '40']); |
| var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; |
| g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); |
| })(); |
| </script> |
| <!-- End Matomo Code --> |
| </head> |
| <body class="global"> |
| <nav class="navbar navbar-expand-lg navbar-dark p-0 px-4" style="background: #1D6890;"> |
| <a class="navbar-brand" href="/"> |
| <img src="/images/spark-logo-rev.svg" alt="" width="141" height="72"> |
| </a> |
| <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarContent" |
| aria-controls="navbarContent" aria-expanded="false" aria-label="Toggle navigation"> |
| <span class="navbar-toggler-icon"></span> |
| </button> |
| <div class="collapse navbar-collapse col-md-12 col-lg-auto pt-4" id="navbarContent"> |
| |
| <ul class="navbar-nav me-auto"> |
| <li class="nav-item"> |
| <a class="nav-link active" aria-current="page" href="/downloads.html">Download</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="libraries" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Libraries |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="libraries"> |
| <li><a class="dropdown-item" href="/sql/">SQL and DataFrames</a></li> |
| <li><a class="dropdown-item" href="/spark-connect/">Spark Connect</a></li> |
| <li><a class="dropdown-item" href="/streaming/">Spark Streaming</a></li> |
| <li><a class="dropdown-item" href="/pandas-on-spark/">pandas on Spark</a></li> |
| <li><a class="dropdown-item" href="/mllib/">MLlib (machine learning)</a></li> |
| <li><a class="dropdown-item" href="/graphx/">GraphX (graph)</a></li> |
| <li> |
| <hr class="dropdown-divider"> |
| </li> |
| <li><a class="dropdown-item" href="/third-party-projects.html">Third-Party Projects</a></li> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="documentation" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Documentation |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="documentation"> |
| <li><a class="dropdown-item" href="/docs/latest/">Latest Release</a></li> |
| <li><a class="dropdown-item" href="/documentation.html">Older Versions and Other Resources</a></li> |
| <li><a class="dropdown-item" href="/faq.html">Frequently Asked Questions</a></li> |
| </ul> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link active" aria-current="page" href="/examples.html">Examples</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="community" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Community |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="community"> |
| <li><a class="dropdown-item" href="/community.html">Mailing Lists & Resources</a></li> |
| <li><a class="dropdown-item" href="/contributing.html">Contributing to Spark</a></li> |
| <li><a class="dropdown-item" href="/improvement-proposals.html">Improvement Proposals (SPIP)</a> |
| </li> |
| <li><a class="dropdown-item" href="https://issues.apache.org/jira/browse/SPARK">Issue Tracker</a> |
| </li> |
| <li><a class="dropdown-item" href="/powered-by.html">Powered By</a></li> |
| <li><a class="dropdown-item" href="/committers.html">Project Committers</a></li> |
| <li><a class="dropdown-item" href="/history.html">Project History</a></li> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="developers" role="button" data-bs-toggle="dropdown" |
| aria-expanded="false"> |
| Developers |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="developers"> |
| <li><a class="dropdown-item" href="/developer-tools.html">Useful Developer Tools</a></li> |
| <li><a class="dropdown-item" href="/versioning-policy.html">Versioning Policy</a></li> |
| <li><a class="dropdown-item" href="/release-process.html">Release Process</a></li> |
| <li><a class="dropdown-item" href="/security.html">Security</a></li> |
| </ul> |
| </li> |
| </ul> |
| <ul class="navbar-nav ml-auto"> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" id="apacheFoundation" role="button" |
| data-bs-toggle="dropdown" aria-expanded="false"> |
| Apache Software Foundation |
| </a> |
| <ul class="dropdown-menu" aria-labelledby="apacheFoundation"> |
| <li><a class="dropdown-item" href="https://www.apache.org/">Apache Homepage</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/licenses/">License</a></li> |
| <li><a class="dropdown-item" |
| href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/security/">Security</a></li> |
| <li><a class="dropdown-item" href="https://www.apache.org/events/current-event">Event</a></li> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| </nav> |
| |
| <div class="container"> |
| <div class="row mt-4"> |
| <div class="col-12 col-md-9"> |
| <h2>Versioning policy</h2> |
| |
| <p>Starting with Spark 1.0.0, the Spark project will follow the |
| <a href="http://semver.org/">semantic versioning guidelines</a> with a few deviations. |
| These small differences account for Spark’s nature as a multi-module project.</p> |
| |
| <h3>Spark versions</h3> |
| |
| <p>Each Spark release will be versioned: <code class="language-plaintext highlighter-rouge">[MAJOR].[FEATURE].[MAINTENANCE]</code></p> |
| |
| <ul> |
| <li><strong>MAJOR</strong>: All releases with the same major version number will have API compatibility. |
| Major version numbers will remain stable over long periods of time. For instance, 1.X.Y may last |
| 1 year or more.</li> |
| <li><strong>FEATURE</strong>: Feature releases will typically contain new features, improvements, and bug fixes. |
| Each feature release will have a merge window where new patches can be merged, a QA window when |
| only fixes can be merged, then a final period where voting occurs on release candidates. These |
| windows will be announced immediately after the previous feature release to give people plenty |
| of time, and over time, we might make the whole release process more regular (similar to Ubuntu).</li> |
| <li><strong>MAINTENANCE</strong>: Maintenance releases will occur more frequently and depend on specific patches |
| introduced (e.g. bug fixes) and their urgency. In general these releases are designed to patch bugs. |
| However, higher level libraries may introduce small features, such as a new algorithm, provided |
| they are entirely additive and isolated from existing code paths. Spark core may not introduce |
| any features.</li> |
| </ul> |
| |
| <h3>Alpha components</h3> |
| |
| <p>When new components are added to Spark, they may initially be marked as “alpha”. Alpha components |
| do not have to abide by the above guidelines, however, to the maximum extent possible, they should |
| try to. Once they are marked “stable” they have to follow these guidelines.</p> |
| |
| <h3>API compatibility</h3> |
| |
| <p>An API is any public class or interface exposed in Spark that is not marked as “developer API” or |
| “experimental”. Release A is API compatible with release B if code compiled against release A |
| <em>compiles cleanly</em> against B. Currently, does not guarantee that a compiled application that is |
| linked against version A will link cleanly against version B without re-compiling. Link-level |
| compatibility is something we’ll try to guarantee in future releases.</p> |
| |
| <p>Note, however, that even for features “developer API” and “experimental”, we strive to maintain |
| maximum compatibility. Code should not be merged into the project as “experimental” if there is |
| a plan to change the API later, because users expect the maximum compatibility from all |
| available APIs.</p> |
| |
| <h3>Considerations when breaking APIs</h3> |
| |
| <p>The Spark project strives to avoid breaking APIs or silently changing behavior, even at major versions. While this is not always possible, the balance of the following factors should be considered before choosing to break an API.</p> |
| |
| <h4>Cost of breaking an API</h4> |
| |
| <p>Breaking an API almost always has a non-trivial cost to the users of Spark. A broken API means that Spark programs need to be rewritten before they can be upgraded. However, there are a few considerations when thinking about what the cost will be:</p> |
| |
| <ul> |
| <li><strong>Usage</strong> - an API that is actively used in many different places, is always very costly to break. While it is hard to know usage for sure, there are a bunch of ways that we can estimate: |
| <ul> |
| <li> |
| <p>How long has the API been in Spark?</p> |
| </li> |
| <li> |
| <p>Is the API common even for basic programs?</p> |
| </li> |
| <li> |
| <p>How often do we see recent questions in JIRA or mailing lists?</p> |
| </li> |
| <li> |
| <p>How often does it appear in StackOverflow or blogs?</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p><strong>Behavior after the break</strong> - How will a program that works today, work after the break? The following are listed roughly in order of increasing severity:</p> |
| |
| <ul> |
| <li> |
| <p>Will there be a compiler or linker error?</p> |
| </li> |
| <li> |
| <p>Will there be a runtime exception?</p> |
| </li> |
| <li> |
| <p>Will that exception happen after significant processing has been done?</p> |
| </li> |
| <li> |
| <p>Will we silently return different answers? (very hard to debug, might not even notice!)</p> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| |
| <h4>Cost of maintaining an API</h4> |
| |
| <p>Of course, the above does not mean that we will <strong>never</strong> break <strong>any</strong> APIs. We must also consider the cost both to the project and to our users of keeping the API in question.</p> |
| |
| <ul> |
| <li> |
| <p><strong>Project Costs</strong> - Every API we have needs to be tested and needs to keep working as other parts of the project changes. These costs are significantly exacerbated when external dependencies change (the JVM, Scala, etc). In some cases, while not completely technically infeasible, the cost of maintaining a particular API can become too high.</p> |
| </li> |
| <li> |
| <p><strong>User Costs</strong> - APIs also have a cognitive cost to users learning Spark or trying to understand Spark programs. This cost becomes even higher when the API in question has confusing or undefined semantics.</p> |
| </li> |
| </ul> |
| |
| <h4>Alternatives to breaking an API</h4> |
| |
| <p>In cases where there is a “Bad API”, but where the cost of removal is also high, there are alternatives that should be considered that do not hurt existing users but do address some of the maintenance costs.</p> |
| |
| <ul> |
| <li> |
| <p><strong>Avoid Bad APIs</strong> - While this is a bit obvious, it is an important point. Anytime we are adding a new interface to Spark we should consider that we might be stuck with this API forever. Think deeply about how new APIs relate to existing ones, as well as how you expect them to evolve over time.</p> |
| </li> |
| <li> |
| <p><strong>Deprecation Warnings</strong> - All deprecation warnings should point to a clear alternative and should never just say that an API is deprecated.</p> |
| </li> |
| <li> |
| <p><strong>Updated Docs</strong> - Documentation should point to the “best” recommended way of performing a given task. In the cases where we maintain legacy documentation, we should clearly point to newer APIs and suggest to users the “right” way.</p> |
| </li> |
| <li> |
| <p><strong>Community Work</strong> - Many people learn Spark by reading blogs and other sites such as StackOverflow. However, many of these resources are out of date. Update them, to reduce the cost of eventually removing deprecated APIs.</p> |
| </li> |
| </ul> |
| |
| <h2>Release cadence</h2> |
| |
| <p>The branch is cut every January and July, so feature (“minor”) releases occur about every 6 months in general. |
| Hence, Spark 2.3.0 would generally be released about 6 months after 2.2.0. Maintenance releases happen as needed |
| in between feature releases. Major releases do not happen according to a fixed schedule.</p> |
| |
| <h3>Spark 3.5 release window</h3> |
| |
| <table> |
| <thead> |
| <tr> |
| <th>Date</th> |
| <th>Event</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>July 16th 2023</td> |
| <td>Code freeze. Release branch cut.</td> |
| </tr> |
| <tr> |
| <td>Late July 2023</td> |
| <td>QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.</td> |
| </tr> |
| <tr> |
| <td>August 2023</td> |
| <td>Release candidates (RC), voting, etc. until final release passes</td> |
| </tr> |
| </tbody> |
| </table> |
| |
| <h2>Maintenance releases and EOL</h2> |
| |
| <p>Feature release branches will, generally, be maintained with bug fix releases for a period of 18 months. |
| For example, branch 2.3.x is no longer considered maintained as of September 2019, 18 months after the release |
| of 2.3.0 in February 2018. No more 2.3.x releases should be expected after that point, even for bug fixes.</p> |
| |
| <p>The last minor release within a major a release will typically be maintained for longer as an “LTS” release. |
| For example, 2.4.0 was released in November 2nd 2018 and had been maintained for 31 months until 2.4.8 was released on May 2021. 2.4.8 is the last release and no more 2.4.x releases should be expected even for bug fixes.</p> |
| |
| |
| </div> |
| <div class="col-12 col-md-3"> |
| <div class="news" style="margin-bottom: 20px;"> |
| <h5>Latest News</h5> |
| <ul class="list-unstyled"> |
| |
| <li><a href="/news/spark-3-4-3-released.html">Spark 3.4.3 released</a> |
| <span class="small">(Apr 18, 2024)</span></li> |
| |
| <li><a href="/news/spark-3-5-1-released.html">Spark 3.5.1 released</a> |
| <span class="small">(Feb 23, 2024)</span></li> |
| |
| <li><a href="/news/spark-3-3-4-released.html">Spark 3.3.4 released</a> |
| <span class="small">(Dec 16, 2023)</span></li> |
| |
| <li><a href="/news/spark-3-4-2-released.html">Spark 3.4.2 released</a> |
| <span class="small">(Nov 30, 2023)</span></li> |
| |
| </ul> |
| <p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p> |
| </div> |
| <div style="text-align:center; margin-bottom: 20px;"> |
| <a href="https://www.apache.org/events/current-event.html"> |
| <img src="https://www.apache.org/events/current-event-234x60.png" style="max-width: 100%;"/> |
| </a> |
| </div> |
| <div class="hidden-xs hidden-sm"> |
| <a href="/downloads.html" class="btn btn-cta btn-lg d-grid" style="margin-bottom: 30px;"> |
| Download Spark |
| </a> |
| <p style="font-size: 16px; font-weight: 500; color: #555;"> |
| Built-in Libraries: |
| </p> |
| <ul class="list-none"> |
| <li><a href="/sql/">SQL and DataFrames</a></li> |
| <li><a href="/streaming/">Spark Streaming</a></li> |
| <li><a href="/mllib/">MLlib (machine learning)</a></li> |
| <li><a href="/graphx/">GraphX (graph)</a></li> |
| </ul> |
| <a href="/third-party-projects.html">Third-Party Projects</a> |
| </div> |
| </div> |
| </div> |
| |
| |
| |
| <footer class="small"> |
| <hr> |
| Apache Spark, Spark, Apache, the Apache feather logo, and the Apache Spark project logo are either registered |
| trademarks or trademarks of The Apache Software Foundation in the United States and other countries. |
| See guidance on use of Apache Spark <a href="/trademarks.html">trademarks</a>. |
| All other marks mentioned may be trademarks or registered trademarks of their respective owners. |
| Copyright © 2018 The Apache Software Foundation, Licensed under the |
| <a href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>. |
| </footer> |
| </div> |
| |
| <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/js/bootstrap.bundle.min.js" |
| integrity="sha384-MrcW6ZMFYlzcLA8Nl+NtUVF0sA7MsXsP1UyJoMp4YLEuNSfAP+JcXn/tWtIaxVXM" |
| crossorigin="anonymous"></script> |
| <script src="https://code.jquery.com/jquery.js"></script> |
| <script src="/js/lang-tabs.js"></script> |
| <script src="/js/downloads.js"></script> |
| </body> |
| </html> |
