Spark-2.4 support

In spark-2.4 mapOutputTracker.getMapSizesByExecutorId returns iterator, rather then seq.

Change-Id: I4a65a6e66af34792e29ed758fe81281df2cb908b

Author: petro-rudenko

README.md

@@ -34,7 +34,7 @@ Mellanox ConnectX-5 network adapter with 100GbE RoCE fabric, connected with a Me
 For more information on configuration, performance tuning and troubleshooting, please visit the [SparkRDMA GitHub Wiki](https://github.com/Mellanox/SparkRDMA/wiki)
 
 ## Runtime requirements
-* Apache Spark 2.0.0/2.1.0/2.2.0/2.3.0
+* Apache Spark 2.0.0/2.1.0/2.2.0/2.3.0/2.4.0
 * Java 8
 * An RDMA-supported network, e.g. RoCE or Infiniband
 
@@ -49,14 +49,15 @@ The pre-built binaries are packed as an archive that contains the following file
 * spark-rdma-3.1-for-spark-2.1.0-jar-with-dependencies.jar
 * spark-rdma-3.1-for-spark-2.2.0-jar-with-dependencies.jar
 * spark-rdma-3.1-for-spark-2.3.0-jar-with-dependencies.jar
+* spark-rdma-3.1-for-spark-2.4.0-jar-with-dependencies.jar
 * libdisni.so
 
 libdisni.so **must** be in `java.library.path` on every Spark Master and Worker (usually in /usr/lib)
 
 ### Configuration
 
 Provide Spark the location of the SparkRDMA plugin jars by using the extraClassPath option.  For standalone mode this can
-be added to either spark-defaults.conf or any runtime configuration file.  For client mode this **must** be added to spark-defaults.conf. For Spark 2.0.0 (Replace with 2.1.0, 2.2.0 or 2.3.0 according to your Spark version):
+be added to either spark-defaults.conf or any runtime configuration file.  For client mode this **must** be added to spark-defaults.conf. For Spark 2.0.0 (Replace with 2.1.0, 2.2.0, 2.3.0, 2.4.0 according to your Spark version):
 ```
 spark.driver.extraClassPath   /path/to/SparkRDMA/target/spark-rdma-3.1-for-spark-2.0.0-jar-with-dependencies.jar
 spark.executor.extraClassPath /path/to/SparkRDMA/target/spark-rdma-3.1-for-spark-2.0.0-jar-with-dependencies.jar
@@ -76,7 +77,7 @@ Building the SparkRDMA plugin requires [Apache Maven](http://maven.apache.org/)
 
 1. Obtain a clone of [SparkRDMA](https://github.com/Mellanox/SparkRDMA)
 
-2. Build the plugin for your Spark version (either 2.0.0, 2.1.0, 2.2.0 or 2.3.0), e.g. for Spark 2.0.0:
+2. Build the plugin for your Spark version (either 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0), e.g. for Spark 2.0.0:
 ```
 mvn -DskipTests clean package -Pspark-2.0.0
 ```

pom.xml

@@ -61,6 +61,12 @@
         <spark.version>2.3.0</spark.version>
       </properties>
     </profile>
+    <profile>
+      <id>spark-2.4.0</id>
+      <properties>
+        <spark.version>2.4.0</spark.version>
+      </properties>
+    </profile>
   </profiles>
 
   <dependencies>

src/main/scala/org/apache/spark/shuffle/rdma/RdmaShuffleReader.scala

@@ -46,7 +46,8 @@ private[spark] class RdmaShuffleReader[K, C](
       startPartition,
       endPartition,
       handle.shuffleId,
-      mapOutputTracker.getMapSizesByExecutorId(handle.shuffleId, startPartition, endPartition))
+      mapOutputTracker.getMapSizesByExecutorId(handle.shuffleId,
+        startPartition, endPartition).toSeq)
 
     val dummyShuffleBlockId = ShuffleBlockId(0, 0, 0)
     // Wrap the streams for compression based on configuration