layout: post title: “Spark Shuffle: SparkRDMA vs Crail” author: Jonas Pfefferle, Patrick Stuedi, Animesh Trivedi, Bernard Metzler, Adrian Schuepbach category: blog comments: true
Hardware Configuration
The specific cluster configuration used for the experiments in this blog:
- Cluster
- 8 compute + 1 management node x86_64 cluster
- Node configuration
- CPU: 2 x Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
- DRAM: 96GB DDR3
- Network: 1x100Gbit/s Mellanox ConnectX-5
- Software
- Ubuntu 16.04.3 LTS (Xenial Xerus) with Linux kernel version 4.10.0-33-generic
- Crail 1.0, commit a45c8382050f471e9342e1c6cf25f9f2001af6b5
- Crail Shuffle plugin, commit 2273b5dd53405cab3389f5c1fc2ee4cd30f02ae6
- SparkRDMA, commit d95ce3e370a8e3b5146f4e0ab5e67a19c6f405a5 (latest master on 8th of November 2017)
Overview
Performance comparison
Summary