SparkRDMA is a high performance ShuffleManager plugin for Apache Spark that uses RDMA (instead of TCP) when performing Shuffle data transfers in Spark jobs.

This open-source project is developed, maintained and supported by Mellanox Technologies.

Running 320GB TeraSort workload with SparkRDMA is x2.63 faster than standard Spark (runtime in seconds)

Test environment:

7 Spark standalone workers on Azure "h16mr" VM instance, Intel Haswell E5-2667 V3,

224GB RAM, 2000GB SSD for temporary storage, Mellanox InfiniBand FDR (56Gb/s)

Also featured at the Spark+AI Summit 2018, please see more info on our session: https://databricks.com/session/accelerated-spark-on-azure-seamless-and-scalable-hardware-offloads-in-the-cloud

Running 19GB Pagerank with SparkRDMA is x2.01 faster than standard Spark (runtime in seconds)

Test environment:

5 Spark standalone workers, 2x Intel Xeon E5-2697 v3 @ 2.60GHz, 25 cores per Worker, 150GB RAM, non-flash storage (HDD)

Mellanox ConnectX-5 network adapter with 100GbE RoCE fabric, connected with a Mellanox Spectrum switch

For more information on configuration, performance tuning and troubleshooting, please visit the SparkRDMA GitHub Wiki

• Apache Spark 2.0.0/2.1.0/2.2.0/2.3.0/2.4.0
• Java 8
• An RDMA-supported network, e.g. RoCE or Infiniband

Please use the "Releases" page to download pre-built binaries.
If you would like to build the project yourself, please refer to the "Build" section below.

The pre-built binaries are packed as an archive that contains the following files:

• spark-rdma-3.1-for-spark-2.0.0-jar-with-dependencies.jar
• spark-rdma-3.1-for-spark-2.1.0-jar-with-dependencies.jar
• spark-rdma-3.1-for-spark-2.2.0-jar-with-dependencies.jar
• spark-rdma-3.1-for-spark-2.3.0-jar-with-dependencies.jar
• spark-rdma-3.1-for-spark-2.4.0-jar-with-dependencies.jar
• libdisni.so

libdisni.so must be in java.library.path on every Spark Master and Worker (usually in /usr/lib)

Provide Spark the location of the SparkRDMA plugin jars by using the extraClassPath option. For standalone mode this can be added to either spark-defaults.conf or any runtime configuration file. For client mode this must be added to spark-defaults.conf. For Spark 2.0.0 (Replace with 2.1.0, 2.2.0, 2.3.0, 2.4.0 according to your Spark version):

spark.driver.extraClassPath   /path/to/SparkRDMA/target/spark-rdma-3.1-for-spark-2.0.0-jar-with-dependencies.jar
spark.executor.extraClassPath /path/to/SparkRDMA/target/spark-rdma-3.1-for-spark-2.0.0-jar-with-dependencies.jar

To enable the SparkRDMA Shuffle Manager plugin, add the following line to either spark-defaults.conf or any runtime configuration file:

spark.shuffle.manager   org.apache.spark.shuffle.rdma.RdmaShuffleManager

Building the SparkRDMA plugin requires Apache Maven and Java 8

1. Obtain a clone of SparkRDMA

2. Build the plugin for your Spark version (either 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0), e.g. for Spark 2.0.0:

mvn -DskipTests clean package -Pspark-2.0.0

3. Obtain a clone of DiSNI for building libdisni:

git clone https://github.com/zrlio/disni.git
cd disni
git checkout tags/v1.7 -b v1.7

4. Compile and install only libdisni (the jars are already included in the SparkRDMA plugin):

cd libdisni
autoprepare.sh
./configure --with-jdk=/path/to/java8/jdk
make
make install

For any questions, issues or suggestions, please use our Google group: https://groups.google.com/forum/#!forum/sparkrdma

Any PR submissions are welcome