| |
| <!DOCTYPE html> |
| <!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--> |
| <!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--> |
| <!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]--> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> |
| <title>Migration Guide: PySpark (Python on Spark) - Spark 3.0.0-preview Documentation</title> |
| |
| |
| |
| |
| <link rel="stylesheet" href="css/bootstrap.min.css"> |
| <style> |
| body { |
| padding-top: 60px; |
| padding-bottom: 40px; |
| } |
| </style> |
| <meta name="viewport" content="width=device-width"> |
| <link rel="stylesheet" href="css/bootstrap-responsive.min.css"> |
| <link rel="stylesheet" href="css/main.css"> |
| |
| <script src="js/vendor/modernizr-2.6.1-respond-1.1.0.min.js"></script> |
| |
| <link rel="stylesheet" href="css/pygments-default.css"> |
| |
| |
| <!-- Google analytics script --> |
| <script type="text/javascript"> |
| var _gaq = _gaq || []; |
| _gaq.push(['_setAccount', 'UA-32518208-2']); |
| _gaq.push(['_trackPageview']); |
| |
| (function() { |
| var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; |
| ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; |
| var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); |
| })(); |
| </script> |
| |
| |
| </head> |
| <body> |
| <!--[if lt IE 7]> |
| <p class="chromeframe">You are using an outdated browser. <a href="https://browsehappy.com/">Upgrade your browser today</a> or <a href="http://www.google.com/chromeframe/?redirect=true">install Google Chrome Frame</a> to better experience this site.</p> |
| <![endif]--> |
| |
| <!-- This code is taken from http://twitter.github.com/bootstrap/examples/hero.html --> |
| |
| <div class="navbar navbar-fixed-top" id="topbar"> |
| <div class="navbar-inner"> |
| <div class="container"> |
| <div class="brand"><a href="index.html"> |
| <img src="img/spark-logo-hd.png" style="height:50px;"/></a><span class="version">3.0.0-preview</span> |
| </div> |
| <ul class="nav"> |
| <!--TODO(andyk): Add class="active" attribute to li some how.--> |
| <li><a href="index.html">Overview</a></li> |
| |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">Programming Guides<b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li><a href="quick-start.html">Quick Start</a></li> |
| <li><a href="rdd-programming-guide.html">RDDs, Accumulators, Broadcasts Vars</a></li> |
| <li><a href="sql-programming-guide.html">SQL, DataFrames, and Datasets</a></li> |
| <li><a href="structured-streaming-programming-guide.html">Structured Streaming</a></li> |
| <li><a href="streaming-programming-guide.html">Spark Streaming (DStreams)</a></li> |
| <li><a href="ml-guide.html">MLlib (Machine Learning)</a></li> |
| <li><a href="graphx-programming-guide.html">GraphX (Graph Processing)</a></li> |
| <li><a href="sparkr.html">SparkR (R on Spark)</a></li> |
| </ul> |
| </li> |
| |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">API Docs<b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li><a href="api/scala/index.html#org.apache.spark.package">Scala</a></li> |
| <li><a href="api/java/index.html">Java</a></li> |
| <li><a href="api/python/index.html">Python</a></li> |
| <li><a href="api/R/index.html">R</a></li> |
| <li><a href="api/sql/index.html">SQL, Built-in Functions</a></li> |
| </ul> |
| </li> |
| |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">Deploying<b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li><a href="cluster-overview.html">Overview</a></li> |
| <li><a href="submitting-applications.html">Submitting Applications</a></li> |
| <li class="divider"></li> |
| <li><a href="spark-standalone.html">Spark Standalone</a></li> |
| <li><a href="running-on-mesos.html">Mesos</a></li> |
| <li><a href="running-on-yarn.html">YARN</a></li> |
| <li><a href="running-on-kubernetes.html">Kubernetes</a></li> |
| </ul> |
| </li> |
| |
| <li class="dropdown"> |
| <a href="api.html" class="dropdown-toggle" data-toggle="dropdown">More<b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li><a href="configuration.html">Configuration</a></li> |
| <li><a href="monitoring.html">Monitoring</a></li> |
| <li><a href="tuning.html">Tuning Guide</a></li> |
| <li><a href="job-scheduling.html">Job Scheduling</a></li> |
| <li><a href="security.html">Security</a></li> |
| <li><a href="hardware-provisioning.html">Hardware Provisioning</a></li> |
| <li><a href="migration-guide.html">Migration Guide</a></li> |
| <li class="divider"></li> |
| <li><a href="building-spark.html">Building Spark</a></li> |
| <li><a href="https://spark.apache.org/contributing.html">Contributing to Spark</a></li> |
| <li><a href="https://spark.apache.org/third-party-projects.html">Third Party Projects</a></li> |
| </ul> |
| </li> |
| </ul> |
| <!--<p class="navbar-text pull-right"><span class="version-text">v3.0.0-preview</span></p>--> |
| </div> |
| </div> |
| </div> |
| |
| <div class="container-wrapper"> |
| |
| |
| |
| <div class="left-menu-wrapper"> |
| <div class="left-menu"> |
| <h3><a href="migration-guide.html">Migration Guide</a></h3> |
| |
| <ul> |
| |
| <li> |
| <a href="core-migration-guide.html"> |
| |
| Spark Core |
| |
| </a> |
| </li> |
| |
| |
| |
| <li> |
| <a href="sql-migration-guide.html"> |
| |
| SQL, Datasets and DataFrame |
| |
| </a> |
| </li> |
| |
| |
| |
| <li> |
| <a href="ss-migration-guide.html"> |
| |
| Structured Streaming |
| |
| </a> |
| </li> |
| |
| |
| |
| <li> |
| <a href="ml-migration-guide.html"> |
| |
| MLlib (Machine Learning) |
| |
| </a> |
| </li> |
| |
| |
| |
| <li> |
| <a href="pyspark-migration-guide.html"> |
| |
| <b>PySpark (Python on Spark)</b> |
| |
| </a> |
| </li> |
| |
| |
| |
| <li> |
| <a href="sparkr-migration-guide.html"> |
| |
| SparkR (R on Spark) |
| |
| </a> |
| </li> |
| |
| |
| |
| </ul> |
| |
| </div> |
| </div> |
| |
| |
| <input id="nav-trigger" class="nav-trigger" checked type="checkbox"> |
| <label for="nav-trigger"></label> |
| <div class="content-with-sidebar" id="content"> |
| |
| <h1 class="title">Migration Guide: PySpark (Python on Spark)</h1> |
| |
| |
| <ul id="markdown-toc"> |
| <li><a href="#upgrading-from-pyspark-24-to-30" id="markdown-toc-upgrading-from-pyspark-24-to-30">Upgrading from PySpark 2.4 to 3.0</a></li> |
| <li><a href="#upgrading-from-pyspark-23-to-24" id="markdown-toc-upgrading-from-pyspark-23-to-24">Upgrading from PySpark 2.3 to 2.4</a></li> |
| <li><a href="#upgrading-from-pyspark-230-to-231-and-above" id="markdown-toc-upgrading-from-pyspark-230-to-231-and-above">Upgrading from PySpark 2.3.0 to 2.3.1 and above</a></li> |
| <li><a href="#upgrading-from-pyspark-22-to-23" id="markdown-toc-upgrading-from-pyspark-22-to-23">Upgrading from PySpark 2.2 to 2.3</a></li> |
| <li><a href="#upgrading-from-pyspark-14-to-15" id="markdown-toc-upgrading-from-pyspark-14-to-15">Upgrading from PySpark 1.4 to 1.5</a></li> |
| <li><a href="#upgrading-from-pyspark-10-12-to-13" id="markdown-toc-upgrading-from-pyspark-10-12-to-13">Upgrading from PySpark 1.0-1.2 to 1.3</a></li> |
| </ul> |
| |
| <p>Note that this migration guide describes the items specific to PySpark. |
| Many items of SQL migration can be applied when migrating PySpark to higher versions. |
| Please refer <a href="sql-migration-guide.html">Migration Guide: SQL, Datasets and DataFrame</a>.</p> |
| |
| <h2 id="upgrading-from-pyspark-24-to-30">Upgrading from PySpark 2.4 to 3.0</h2> |
| |
| <ul> |
| <li> |
| <p>Since Spark 3.0, PySpark requires a Pandas version of 0.23.2 or higher to use Pandas related functionality, such as <code>toPandas</code>, <code>createDataFrame</code> from Pandas DataFrame, etc.</p> |
| </li> |
| <li> |
| <p>Since Spark 3.0, PySpark requires a PyArrow version of 0.12.1 or higher to use PyArrow related functionality, such as <code>pandas_udf</code>, <code>toPandas</code> and <code>createDataFrame</code> with “spark.sql.execution.arrow.enabled=true”, etc.</p> |
| </li> |
| <li> |
| <p>In PySpark, when creating a <code>SparkSession</code> with <code>SparkSession.builder.getOrCreate()</code>, if there is an existing <code>SparkContext</code>, the builder was trying to update the <code>SparkConf</code> of the existing <code>SparkContext</code> with configurations specified to the builder, but the <code>SparkContext</code> is shared by all <code>SparkSession</code>s, so we should not update them. Since 3.0, the builder comes to not update the configurations. This is the same behavior as Java/Scala API in 2.3 and above. If you want to update them, you need to update them prior to creating a <code>SparkSession</code>.</p> |
| </li> |
| <li>In PySpark, when Arrow optimization is enabled, if Arrow version is higher than 0.11.0, Arrow can perform safe type conversion when converting Pandas.Series to Arrow array during serialization. Arrow will raise errors when detecting unsafe type conversion like overflow. Setting <code>spark.sql.execution.pandas.arrowSafeTypeConversion</code> to true can enable it. The default setting is false. PySpark’s behavior for Arrow versions is illustrated in the table below: |
| <table class="table"> |
| <tr> |
| <th> |
| <b>PyArrow version</b> |
| </th> |
| <th> |
| <b>Integer Overflow</b> |
| </th> |
| <th> |
| <b>Floating Point Truncation</b> |
| </th> |
| </tr> |
| <tr> |
| <td> |
| version < 0.11.0 |
| </td> |
| <td> |
| Raise error |
| </td> |
| <td> |
| Silently allows |
| </td> |
| </tr> |
| <tr> |
| <td> |
| version > 0.11.0, arrowSafeTypeConversion=false |
| </td> |
| <td> |
| Silent overflow |
| </td> |
| <td> |
| Silently allows |
| </td> |
| </tr> |
| <tr> |
| <td> |
| version > 0.11.0, arrowSafeTypeConversion=true |
| </td> |
| <td> |
| Raise error |
| </td> |
| <td> |
| Raise error |
| </td> |
| </tr> |
| </table> |
| </li> |
| <li>Since Spark 3.0, <code>createDataFrame(..., verifySchema=True)</code> validates <code>LongType</code> as well in PySpark. Previously, <code>LongType</code> was not verified and resulted in <code>None</code> in case the value overflows. To restore this behavior, <code>verifySchema</code> can be set to <code>False</code> to disable the validation.</li> |
| </ul> |
| |
| <h2 id="upgrading-from-pyspark-23-to-24">Upgrading from PySpark 2.3 to 2.4</h2> |
| |
| <ul> |
| <li>In PySpark, when Arrow optimization is enabled, previously <code>toPandas</code> just failed when Arrow optimization is unable to be used whereas <code>createDataFrame</code> from Pandas DataFrame allowed the fallback to non-optimization. Now, both <code>toPandas</code> and <code>createDataFrame</code> from Pandas DataFrame allow the fallback by default, which can be switched off by <code>spark.sql.execution.arrow.fallback.enabled</code>.</li> |
| </ul> |
| |
| <h2 id="upgrading-from-pyspark-230-to-231-and-above">Upgrading from PySpark 2.3.0 to 2.3.1 and above</h2> |
| |
| <ul> |
| <li>As of version 2.3.1 Arrow functionality, including <code>pandas_udf</code> and <code>toPandas()</code>/<code>createDataFrame()</code> with <code>spark.sql.execution.arrow.enabled</code> set to <code>True</code>, has been marked as experimental. These are still evolving and not currently recommended for use in production.</li> |
| </ul> |
| |
| <h2 id="upgrading-from-pyspark-22-to-23">Upgrading from PySpark 2.2 to 2.3</h2> |
| |
| <ul> |
| <li> |
| <p>In PySpark, now we need Pandas 0.19.2 or upper if you want to use Pandas related functionalities, such as <code>toPandas</code>, <code>createDataFrame</code> from Pandas DataFrame, etc.</p> |
| </li> |
| <li> |
| <p>In PySpark, the behavior of timestamp values for Pandas related functionalities was changed to respect session timezone. If you want to use the old behavior, you need to set a configuration <code>spark.sql.execution.pandas.respectSessionTimeZone</code> to <code>False</code>. See <a href="https://issues.apache.org/jira/browse/SPARK-22395">SPARK-22395</a> for details.</p> |
| </li> |
| <li> |
| <p>In PySpark, <code>na.fill()</code> or <code>fillna</code> also accepts boolean and replaces nulls with booleans. In prior Spark versions, PySpark just ignores it and returns the original Dataset/DataFrame.</p> |
| </li> |
| <li> |
| <p>In PySpark, <code>df.replace</code> does not allow to omit <code>value</code> when <code>to_replace</code> is not a dictionary. Previously, <code>value</code> could be omitted in the other cases and had <code>None</code> by default, which is counterintuitive and error-prone.</p> |
| </li> |
| </ul> |
| |
| <h2 id="upgrading-from-pyspark-14-to-15">Upgrading from PySpark 1.4 to 1.5</h2> |
| |
| <ul> |
| <li> |
| <p>Resolution of strings to columns in Python now supports using dots (<code>.</code>) to qualify the column or |
| access nested values. For example <code>df['table.column.nestedField']</code>. However, this means that if |
| your column name contains any dots you must now escape them using backticks (e.g., <code>table.`column.with.dots`.nested</code>).</p> |
| </li> |
| <li> |
| <p>DataFrame.withColumn method in PySpark supports adding a new column or replacing existing columns of the same name.</p> |
| </li> |
| </ul> |
| |
| <h2 id="upgrading-from-pyspark-10-12-to-13">Upgrading from PySpark 1.0-1.2 to 1.3</h2> |
| |
| <h4 class="no_toc" id="python-datatypes-no-longer-singletons">Python DataTypes No Longer Singletons</h4> |
| |
| <p>When using DataTypes in Python you will need to construct them (i.e. <code>StringType()</code>) instead of |
| referencing a singleton.</p> |
| |
| |
| </div> |
| |
| <!-- /container --> |
| </div> |
| |
| <script src="js/vendor/jquery-3.4.1.min.js"></script> |
| <script src="js/vendor/bootstrap.min.js"></script> |
| <script src="js/vendor/anchor.min.js"></script> |
| <script src="js/main.js"></script> |
| |
| <!-- MathJax Section --> |
| <script type="text/x-mathjax-config"> |
| MathJax.Hub.Config({ |
| TeX: { equationNumbers: { autoNumber: "AMS" } } |
| }); |
| </script> |
| <script> |
| // Note that we load MathJax this way to work with local file (file://), HTTP and HTTPS. |
| // We could use "//cdn.mathjax...", but that won't support "file://". |
| (function(d, script) { |
| script = d.createElement('script'); |
| script.type = 'text/javascript'; |
| script.async = true; |
| script.onload = function(){ |
| MathJax.Hub.Config({ |
| tex2jax: { |
| inlineMath: [ ["$", "$"], ["\\\\(","\\\\)"] ], |
| displayMath: [ ["$$","$$"], ["\\[", "\\]"] ], |
| processEscapes: true, |
| skipTags: ['script', 'noscript', 'style', 'textarea', 'pre'] |
| } |
| }); |
| }; |
| script.src = ('https:' == document.location.protocol ? 'https://' : 'http://') + |
| 'cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js' + |
| '?config=TeX-AMS-MML_HTMLorMML'; |
| d.getElementsByTagName('head')[0].appendChild(script); |
| }(document)); |
| </script> |
| </body> |
| </html> |
