Skip to content
This repository was archived by the owner on Dec 20, 2022. It is now read-only.

Commit 5519dd9

Browse files
Merge pull request #9 from Mellanox/release-2.0
Release 2.0
2 parents d95ce3e + 090f075 commit 5519dd9

32 files changed

+1889
-1761
lines changed

README.md

100644100755
Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -5,24 +5,24 @@ performing Shuffle data transfers in Spark jobs.
55
This open-source project is developed, maintained and supported by [Mellanox Technologies](http://www.mellanox.com).
66

77
## Performance results
8-
Example performance speedup for HiBench TeraSort:
9-
![Alt text](https://user-images.githubusercontent.com/20062725/28947340-30d45c6a-7864-11e7-96ea-ca3cf505ce7a.png)
8+
Example performance speedup for [HiBench](https://github.com/intel-hadoop/HiBench) TeraSort:
9+
![Alt text](https://user-images.githubusercontent.com/1121987/37930913-cdc3591c-314c-11e8-9794-d30db5900751.png)
1010

11-
Running TeraSort with SparkRDMA is x1.41 faster than standard Spark (runtime in seconds)
11+
Running TeraSort with SparkRDMA is x1.53 faster than standard Spark (runtime in seconds)
1212

1313
Testbed:
1414

1515
175GB Workload
1616

17-
15 Workers, 2x Intel Xeon E5-2697 v3 @ 2.60GHz, 28 cores per Worker, 256GB RAM, non-flash storage (HDD)
17+
16 Workers, 2x Intel Xeon E5-2697 v3 @ 2.60GHz, 30 cores per Worker, 256GB RAM, non-flash storage (HDD)
1818

1919
Mellanox ConnectX-4 network adapter with 100GbE RoCE fabric, connected with a Mellanox Spectrum switch
2020

2121
## Wiki pages
2222
For more information on configuration, performance tuning and troubleshooting, please visit the [SparkRDMA GitHub Wiki](https://github.com/Mellanox/SparkRDMA/wiki)
2323

2424
## Runtime requirements
25-
* Apache Spark 2.0.0/2.1.0/2.2.0
25+
* Apache Spark 2.0.0/2.1.0/2.2.0/2.3.0
2626
* Java 8
2727
* An RDMA-supported network, e.g. RoCE or Infiniband
2828

@@ -33,20 +33,21 @@ Please use the ["Releases"](https://github.com/Mellanox/SparkRDMA/releases) page
3333
<br>If you would like to build the project yourself, please refer to the ["Build"](https://github.com/Mellanox/SparkRDMA#build) section below.
3434

3535
The pre-built binaries are packed as an archive that contains the following files:
36-
* spark-rdma-1.0-for-spark-2.0.0-jar-with-dependencies.jar
37-
* spark-rdma-1.0-for-spark-2.1.0-jar-with-dependencies.jar
38-
* spark-rdma-1.0-for-spark-2.2.0-jar-with-dependencies.jar
36+
* spark-rdma-2.0-for-spark-2.0.0-jar-with-dependencies.jar
37+
* spark-rdma-2.0-for-spark-2.1.0-jar-with-dependencies.jar
38+
* spark-rdma-2.0-for-spark-2.2.0-jar-with-dependencies.jar
39+
* spark-rdma-2.0-for-spark-2.3.0-jar-with-dependencies.jar
3940
* libdisni.so
4041

41-
libdisni.so **must** be installed on every Spark Master and Worker (usually in /usr/lib)
42+
libdisni.so **must** be in `java.library.path` on every Spark Master and Worker (usually in /usr/lib)
4243

4344
### Configuration
4445

4546
Provide Spark the location of the SparkRDMA plugin jars by using the extraClassPath option. For standalone mode this can
46-
be added to either spark-defaults.conf or any runtime configuration file. For client mode this **must** be added to spark-defaults.conf. For Spark 2.0.0 (Replace with 2.1.0 or 2.2.0 according to your Spark version):
47+
be added to either spark-defaults.conf or any runtime configuration file. For client mode this **must** be added to spark-defaults.conf. For Spark 2.0.0 (Replace with 2.1.0, 2.2.0 or 2.3.0 according to your Spark version):
4748
```
48-
spark.driver.extraClassPath /path/to/SparkRDMA/target/spark-rdma-1.0-for-spark-2.0.0-jar-with-dependencies.jar
49-
spark.executor.extraClassPath /path/to/SparkRDMA/target/spark-rdma-1.0-for-spark-2.0.0-jar-with-dependencies.jar
49+
spark.driver.extraClassPath /path/to/SparkRDMA/target/spark-rdma-2.0-for-spark-2.0.0-jar-with-dependencies.jar
50+
spark.executor.extraClassPath /path/to/SparkRDMA/target/spark-rdma-2.0-for-spark-2.0.0-jar-with-dependencies.jar
5051
```
5152

5253
### Running
@@ -63,7 +64,7 @@ Building the SparkRDMA plugin requires [Apache Maven](http://maven.apache.org/)
6364

6465
1. Obtain a clone of [SparkRDMA](https://github.com/Mellanox/SparkRDMA)
6566

66-
2. Build the plugin for your Spark version (either 2.0.0, 2.1.0 or 2.2.0), e.g. for Spark 2.0.0:
67+
2. Build the plugin for your Spark version (either 2.0.0, 2.1.0, 2.2.0 or 2.3.0), e.g. for Spark 2.0.0:
6768
```
6869
mvn -DskipTests clean package -Pspark-2.0.0
6970
```
@@ -73,7 +74,7 @@ mvn -DskipTests clean package -Pspark-2.0.0
7374
```
7475
git clone https://github.com/zrlio/disni.git
7576
cd disni
76-
git checkout tags/v1.3 -b v1.3
77+
git checkout tags/v1.4 -b v1.4
7778
```
7879

7980
4. Compile and install only libdisni (the jars are already included in the SparkRDMA plugin):

checkstyle.xml

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
<!--
2+
~ Licensed to the Apache Software Foundation (ASF) under one or more
3+
~ contributor license agreements. See the NOTICE file distributed with
4+
~ this work for additional information regarding copyright ownership.
5+
~ The ASF licenses this file to You under the Apache License, Version 2.0
6+
~ (the "License"); you may not use this file except in compliance with
7+
~ the License. You may obtain a copy of the License at
8+
~
9+
~ http://www.apache.org/licenses/LICENSE-2.0
10+
~
11+
~ Unless required by applicable law or agreed to in writing, software
12+
~ distributed under the License is distributed on an "AS IS" BASIS,
13+
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
~ See the License for the specific language governing permissions and
15+
~ limitations under the License.
16+
-->
17+
18+
<!DOCTYPE module PUBLIC
19+
"-//Puppy Crawl//DTD Check Configuration 1.3//EN"
20+
"http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
21+
22+
<!--
23+
24+
Checkstyle configuration based on the Google coding conventions from:
25+
26+
- Google Java Style
27+
https://google-styleguide.googlecode.com/svn-history/r130/trunk/javaguide.html
28+
29+
with Spark-specific changes from:
30+
31+
http://spark.apache.org/contributing.html#code-style-guide
32+
33+
Checkstyle is very configurable. Be sure to read the documentation at
34+
http://checkstyle.sf.net (or in your downloaded distribution).
35+
36+
Most Checks are configurable, be sure to consult the documentation.
37+
38+
To completely disable a check, just comment it out or delete it from the file.
39+
40+
Authors: Max Vetrenko, Ruslan Diachenko, Roman Ivanov.
41+
42+
-->
43+
44+
<module name = "Checker">
45+
<property name="charset" value="UTF-8"/>
46+
47+
<property name="severity" value="error"/>
48+
49+
<property name="fileExtensions" value="java, properties, xml"/>
50+
51+
<!-- Checks for whitespace -->
52+
<!-- See http://checkstyle.sf.net/config_whitespace.html -->
53+
<module name="FileTabCharacter">
54+
<property name="eachLine" value="true"/>
55+
</module>
56+
57+
<module name="RegexpSingleline">
58+
<!-- \s matches whitespace character, $ matches end of line. -->
59+
<property name="format" value="\s+$"/>
60+
<property name="message" value="No trailing whitespace allowed."/>
61+
</module>
62+
63+
<module name="NewlineAtEndOfFile">
64+
<property name="lineSeparator" value="lf"/>
65+
</module>
66+
<module name="SuppressWarningsFilter" />
67+
68+
<module name="TreeWalker">
69+
<module name="SuppressWarningsHolder" />
70+
<module name="OuterTypeFilename"/>
71+
<module name="IllegalTokenText">
72+
<property name="tokens" value="STRING_LITERAL, CHAR_LITERAL"/>
73+
<property name="format" value="\\u00(08|09|0(a|A)|0(c|C)|0(d|D)|22|27|5(C|c))|\\(0(10|11|12|14|15|42|47)|134)"/>
74+
<property name="message" value="Avoid using corresponding octal or Unicode escape."/>
75+
</module>
76+
<module name="AvoidEscapedUnicodeCharacters">
77+
<property name="allowEscapesForControlCharacters" value="true"/>
78+
<property name="allowByTailComment" value="true"/>
79+
<property name="allowNonPrintableEscapes" value="true"/>
80+
</module>
81+
<module name="LineLength">
82+
<property name="max" value="100"/>
83+
<property name="ignorePattern" value="^package.*|^import.*|a href|href|http://|https://|ftp://"/>
84+
</module>
85+
<module name="NoLineWrap"/>
86+
<module name="EmptyBlock">
87+
<property name="option" value="TEXT"/>
88+
<property name="tokens" value="LITERAL_TRY, LITERAL_FINALLY, LITERAL_IF, LITERAL_ELSE, LITERAL_SWITCH"/>
89+
</module>
90+
<module name="NeedBraces">
91+
<property name="allowSingleLineStatement" value="true"/>
92+
</module>
93+
<module name="OneStatementPerLine"/>
94+
<module name="ArrayTypeStyle"/>
95+
<module name="FallThrough"/>
96+
<module name="UpperEll"/>
97+
<module name="ModifierOrder"/>
98+
<module name="SeparatorWrap">
99+
<property name="tokens" value="DOT"/>
100+
<property name="option" value="nl"/>
101+
</module>
102+
<module name="SeparatorWrap">
103+
<property name="tokens" value="COMMA"/>
104+
<property name="option" value="EOL"/>
105+
</module>
106+
<module name="PackageName">
107+
<property name="format" value="^[a-z]+(\.[a-z][a-z0-9]*)*$"/>
108+
<message key="name.invalidPattern"
109+
value="Package name ''{0}'' must match pattern ''{1}''."/>
110+
</module>
111+
<module name="ClassTypeParameterName">
112+
<property name="format" value="([A-Z][a-zA-Z0-9]*$)"/>
113+
<message key="name.invalidPattern"
114+
value="Class type name ''{0}'' must match pattern ''{1}''."/>
115+
</module>
116+
<module name="MethodTypeParameterName">
117+
<property name="format" value="([A-Z][a-zA-Z0-9]*)"/>
118+
<message key="name.invalidPattern"
119+
value="Method type name ''{0}'' must match pattern ''{1}''."/>
120+
</module>
121+
<module name="GenericWhitespace">
122+
<message key="ws.followed"
123+
value="GenericWhitespace ''{0}'' is followed by whitespace."/>
124+
<message key="ws.preceded"
125+
value="GenericWhitespace ''{0}'' is preceded with whitespace."/>
126+
<message key="ws.illegalFollow"
127+
value="GenericWhitespace ''{0}'' should followed by whitespace."/>
128+
<message key="ws.notPreceded"
129+
value="GenericWhitespace ''{0}'' is not preceded with whitespace."/>
130+
</module>
131+
132+
<module name="Indentation">
133+
<property name="basicOffset" value="2"/>
134+
<property name="braceAdjustment" value="0"/>
135+
<property name="caseIndent" value="2"/>
136+
<property name="throwsIndent" value="2"/>
137+
<property name="lineWrappingIndentation" value="2"/>
138+
<property name="arrayInitIndent" value="2"/>
139+
</module>
140+
141+
142+
<module name="MethodParamPad"/>
143+
<module name="AnnotationLocation">
144+
<property name="tokens" value="CLASS_DEF, INTERFACE_DEF, ENUM_DEF, METHOD_DEF, CTOR_DEF"/>
145+
</module>
146+
<module name="AnnotationLocation">
147+
<property name="tokens" value="VARIABLE_DEF"/>
148+
<property name="allowSamelineMultipleAnnotations" value="true"/>
149+
</module>
150+
<module name="MethodName">
151+
<property name="format" value="^[a-z][a-z0-9][a-zA-Z0-9_]*$"/>
152+
<message key="name.invalidPattern"
153+
value="Method name ''{0}'' must match pattern ''{1}''."/>
154+
</module>
155+
<module name="EmptyCatchBlock">
156+
<property name="exceptionVariableName" value="expected"/>
157+
</module>
158+
<module name="CommentsIndentation"/>
159+
<module name="UnusedImports"/>
160+
<module name="RedundantImport"/>
161+
<module name="RedundantModifier"/>
162+
</module>
163+
</module>

pom.xml

100644100755
Lines changed: 58 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77

88
<groupId>com.github.mellanox</groupId>
99
<artifactId>spark-rdma</artifactId>
10-
<version>1.0</version>
10+
<version>2.0</version>
1111
<name>${project.artifactId}</name>
1212
<description>SparkRDMA Shuffle Manager Plugin</description>
1313
<inceptionYear>2017</inceptionYear>
@@ -40,7 +40,7 @@
4040
<profile>
4141
<id>spark-2.0.0</id>
4242
<properties>
43-
<spark.version>2.0.0</spark.version>
43+
<spark.version>2.0.1</spark.version>
4444
</properties>
4545
</profile>
4646
<profile>
@@ -55,6 +55,12 @@
5555
<spark.version>2.2.0</spark.version>
5656
</properties>
5757
</profile>
58+
<profile>
59+
<id>spark-2.3.0</id>
60+
<properties>
61+
<spark.version>2.3.0</spark.version>
62+
</properties>
63+
</profile>
5864
</profiles>
5965

6066
<dependencies>
@@ -67,7 +73,7 @@
6773
<dependency>
6874
<groupId>com.ibm.disni</groupId>
6975
<artifactId>disni</artifactId>
70-
<version>1.3</version>
76+
<version>1.4</version>
7177
<exclusions>
7278
<exclusion>
7379
<groupId>org.slf4j</groupId>
@@ -84,6 +90,55 @@
8490
<build>
8591
<finalName>${project.artifactId}-${project.version}-for-spark-${spark.version}</finalName>
8692
<plugins>
93+
<plugin>
94+
<groupId>org.apache.maven.plugins</groupId>
95+
<artifactId>maven-checkstyle-plugin</artifactId>
96+
<version>3.0.0</version>
97+
<executions>
98+
<execution>
99+
<id>validate</id>
100+
<phase>validate</phase>
101+
<configuration>
102+
<configLocation>checkstyle.xml</configLocation>
103+
<encoding>UTF-8</encoding>
104+
<consoleOutput>true</consoleOutput>
105+
<failsOnError>true</failsOnError>
106+
<linkXRef>false</linkXRef>
107+
<outputFile>${basedir}/target/checkstyle-output.xml</outputFile>
108+
<inputEncoding>${project.build.sourceEncoding}</inputEncoding>
109+
<outputEncoding>${project.reporting.outputEncoding}</outputEncoding>
110+
</configuration>
111+
<goals>
112+
<goal>check</goal>
113+
</goals>
114+
</execution>
115+
</executions>
116+
</plugin>
117+
<plugin>
118+
<groupId>org.scalastyle</groupId>
119+
<artifactId>scalastyle-maven-plugin</artifactId>
120+
<version>1.0.0</version>
121+
<configuration>
122+
<verbose>false</verbose>
123+
<failOnViolation>true</failOnViolation>
124+
<includeTestSourceDirectory>true</includeTestSourceDirectory>
125+
<failOnWarning>false</failOnWarning>
126+
<sourceDirectory>${project.basedir}/src/main/scala</sourceDirectory>
127+
<testSourceDirectory>${project.basedir}/src/test/scala</testSourceDirectory>
128+
<configLocation>scalastyle_config.xml</configLocation>
129+
<outputFile>${basedir}/target/scalastyle-output.xml</outputFile>
130+
<outputEncoding>UTF-8</outputEncoding>
131+
</configuration>
132+
<executions>
133+
<execution>
134+
<id>validate</id>
135+
<phase>validate</phase>
136+
<goals>
137+
<goal>check</goal>
138+
</goals>
139+
</execution>
140+
</executions>
141+
</plugin>
87142
<plugin>
88143
<groupId>org.scala-tools</groupId>
89144
<artifactId>maven-scala-plugin</artifactId>

0 commit comments

Comments
 (0)