layout: post title: Spark Release 2.4.6 categories: [] tags: [] status: publish type: post published: true meta: _edit_last: ‘4’ _wpas_done_all: ‘1’
Spark 2.4.6 is a maintenance release containing stability, correctness, and security fixes. This release is based on the branch-2.4 maintenance branch of Spark. We strongly recommend all 2.4 users to upgrade to this stable release.
Notable changes
- [SPARK-29419]: Seq.toDS / spark.createDataset(Seq) is not thread-safe
- [SPARK-31519]: Cast in having aggregate expressions returns the wrong result
- [SPARK-26293]: Cast exception when having python udf in subquery
- [SPARK-30826]: LIKE returns wrong result from external table using parquet
- [SPARK-30857]: Wrong truncations of timestamps before the epoch to hours and days
- [SPARK-31256]: Dropna doesn't work for struct columns
- [SPARK-31312]: Transforming Hive simple UDF (using JAR) expression may incur CNFE in later evaluation
- [SPARK-31420]: Infinite timeline redraw in job details page
- [SPARK-31485]: Barrier stage can hang if only partial tasks launched
- [SPARK-31500]: collect_set() of BinaryType returns duplicate elements
- [SPARK-31503]: fix the SQL string of the TRIM functions
- [SPARK-31663]: Grouping sets with having clause returns the wrong result
- [SPARK-26908]: Fix toMilis
- [SPARK-31563]: Failure of Inset.sql for UTF8String collection
Dependency Changes
While being a maintence release we did still upgrade some dependencies in this release they are:
- netty-all to 4.1.47.Final ([CVE-2019-20445])
- Janino to 3.0.16 (SQL Generated code)
- aws-java-sdk-sts to 1.11.655 (required for kinesis client upgrade)
- snappy 1.1.7.5 (stability improvements & ppc64le performance)
Known issues
- [SPARK-31170]: Spark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
- [SPARK-26021]: -0.0 and 0.0 not treated consistently, doesn't match Hive
- [SPARK-26154]: Stream-stream joins - left outer join gives inconsistent outpu
- [SPARK-28344]: Fail the query if detect ambiguous self join
You can consult JIRA for the detailed changes.
We would like to acknowledge all community members for contributing patches to this release.