layout: post title: “Spark Shuffle: SparkRDMA vs Crail” author: Jonas Pfefferle, Patrick Stuedi, Animesh Trivedi, Bernard Metzler, Adrian Schuepbach category: blog comments: true

Hardware Configuration

The specific cluster configuration used for the experiments in this blog:

  • Cluster
    • 8 compute + 1 management node x86_64 cluster
  • Node configuration
    • CPU: 2 x Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
    • DRAM: 96GB DDR3
    • Network: 1x100Gbit/s Mellanox ConnectX-5
  • Software
    • Ubuntu 16.04.3 LTS (Xenial Xerus) with Linux kernel version 4.10.0-33-generic
    • Crail 1.0, commit a45c8382050f471e9342e1c6cf25f9f2001af6b5
    • Crail Shuffle plugin, commit 2273b5dd53405cab3389f5c1fc2ee4cd30f02ae6
    • SparkRDMA, commit d95ce3e370a8e3b5146f4e0ab5e67a19c6f405a5 (latest master on 8th of November 2017)

Overview

Performance comparison

Summary