Apache Spark 2.4.0 is the fifth release in the 2.x line. This release adds Barrier Execution Mode for better integration with deep learning frameworks, introduces 30+ built-in and higher-order functions to deal with complex data type easier, improves the K8s integration, along with experimental Scala 2.12 support. Other major updates include the built-in Avro data source, Image data source, flexible streaming sinks, elimination of the 2GB block size limitation during transfer, Pandas UDF improvements. In addition, this release continues to focus on usability, stability, and polish while resolving around 1100 tickets.
To download Apache Spark 2.4.0, visit the downloads page. You can consult JIRA for the detailed changes. We have curated a list of high level changes here, grouped by major modules.
Major features
API
Performance and stability
Connectors
Kubernetes Scheduler Backend
PySpark
Other notable changes
Programming guides: Spark RDD Programming Guide and Spark SQL, DataFrames and Datasets Guide.
Major features
Other notable changes
Programming guide: Structured Streaming Programming Guide.
Major features
Other notable changes
Programming guide: Machine Learning Library (MLlib) Guide.
Programming guide: SparkR (R on Spark).
Programming guide: GraphX Programming Guide.
Spark Core
Spark SQL
Please read the Migration Guide for all the behavior changes
-i
option correctlyLast but not least, this release would not have been possible without the following contributors: Achuth17, Adam Bradbury, Adamyuanyuan, Adelbert Chang, Ala Luszczak, Aleksandr Koriagin, Alessandro Bellina, Alessandro Solimando, Andrew Korzhuev, Anton Okolnychyi, Antonio Murgia, Arseniy Tashoyan, Artem Rudoy, Arun Mahadevan, Asher Saban, Bago Amirbekian, Benjamin Peterson, Bo Meng, Bogdan Raducanu, Bounkong Khamphousone, Brandon Krieger, Brian Lindblom, Bruce Robbins, Bryan Cutler, Cheng Lian, Chongguang LIU, Chris Horn, Chris Martin, Cody Koeninger, DB Tsai, Daniel Sakuma, Daniel Van Der Ende, Darcy Shen, David Vogelbacher, Devaraj K, Dhruve Ashar, Dilip Biswal, Dongjoon Hyun, DylanGuedes, Efim Poberezkin, Eric Chang, Eric Liang, Erik Erlandson, Eyal Farago, Fangshi Li, Felix Cheung, Feng Liu, Fernando Pereira, Florent Pepin, Fokko Driesprong, Gabor Somogyi, Gengliang Wang, Ger Van Rossum, Gera Shegalov, Goun Na, Hao Ren, Henry Robinson, Herman Van Hovell, Hieu Huynh, Holden Karau, Huang Tengfei, Huaxin Gao, Hyukjin Kwon, Ilan Filonenko, Imran Rashid, Jacek Laskowski, Jake Charland, James Thompson, James Yu, Jaroslav Chladek, Jeff Zhang, JiahuiJiang, Jim Kleckner, Joey Krabacher, John Zhuge, Jongyoul Lee, Jooseong Kim, Jose Torres, Joseph Bradley, Joseph K. Bradley, Josh Rosen, Julien Cuquemelle, Juliusz Sompolski, Jungtaek Lim, KaiXinXIaoLei, Kallman, Steven, Karthik Palaniappan, Kaya Kupferschmidt, Kazuaki Ishizaki, Kelley Robinson, Kent Yao, Kevin Yu, KevinZwx, Koert Kuipers, Kousuke Saruta, Kris Mok, LantaoJin, Lee Dongjin, Lemonjing, Li Jin, Liang-Chi Hsieh, Lu WANG, LucaCanali, Marcelo Vanzin, Marco Gaido, Marek Novotny, Mario Molina, Mark Petruska, Maryann Xue, Mathieu St-Louis, Matthew Cheah, Matthew Tovbin, Mauro Palsgraaf, Maxim Gekk, Michael (Stu) Stewart, Michael Allman, Michael Chirico, Michael Mior, Michal Switakowski, Mihaly Toth, Miklos C, Miles Yucht, Misha Dmitriev, Mukul Murthy, Mykhailo Shtelma, Neal Song, Ngone51, Nihar Sheth, Nolan Emirot, Norman Maurer, Onur Satici, Patrick McGloin, Patrick Pisciuneri, Paul Mackles, Peter Toth, Prashant Sharma, Rao Fu, Ray Burgemeestre, Rekha Joshi, Reynold Xin, Reza Safi, Ricardo Martinelli De Oliveira, Rob Vesse, Robert Kruszewski, Rong Tang, Ryan Blue, Sahil Takiar, Saisai Shao, Sandeep Singh, Sandor Murakozi, Sanket Chintapalli, Santiago Saavedra, Sean Owen, Sean Suchter, Sebastian Arzt, Shane Knapp, Shixiong Zhu, Soham Aurangabadkar, Stacy Kerkela, Stan Zhai, Stavros Kontopoulos, Steve Loughran, Sunitha Kambhampati, Takeshi Yamamuro, Takuya UESHIN, Tathagata Das, Ted Yu, Teng Peng, Thiruvasakan Paramasivan, Thomas Graves, Tom Saleeba, Vayda, Oleksandr: IT (PRG), Vinod KC, Vladimir Kuriatkov, Wang Gengliang, Weichen Xu, Wenbo Zhao, Wenchen Fan, William Sheu, XD-DENG, Xiangrui Meng, Xianjin YE, Xianyang Liu, Xiao Li, Xiaogang Tu, Xiayun Sun, Xingbo Jiang, Yacine Mazari, Yash Sharma, Ye Zhou, Yinan Li, Yogesh Garg, Yuanbo Liu, Yuanjian Li, Yuchen Huo, Yuexin Zhang, Yuming Wang, Yuval Itzchakov, Zhan Zhang, Zhang Le, Zheng RuiFeng, Zoltan C. Toth