MAHOUT-2000 : Integrated spark 2.2.0 with Mahout 13.1-snapshot

[dustin vanstee]

Hi , I tried a quick smoke test build to see if I can build a Mahout binary
that supports Spark 2.2.0

After a quick test I had some errors .. I will continue to debug, but
posting here to see if anyone has some hints to help me get past this ...

pom.xml modification (added) -->
<profile>
<id>spark-2.2</id>
<activation>
<property><name>spark-2.2</name></property>
</activation>
<properties>
<spark.version>2.2.0</spark.version>
<spark.compat.version>2.2</spark.compat.version>
</properties>
</profile>


*Build command ...*

mvn clean install -Phadoop2 -Pspark-2.2

*result snippet *

a bunch of these type errors

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
in stage 17.0 failed 1 times, most recent failure: Lost task 0.0 in stage
17.0 (TID 55, localhost, executor driver): java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FileSystem$Statistics.getThreadStatistics()Lorg/
apache/hadoop/fs/FileSystem$Statistics$StatisticsData;
at org.apache.spark.deploy.SparkHadoopUtil$$anonfun$3.
apply(SparkHadoopUtil.scala:180)

*** 19 TESTS FAILED ***
[INFO] ------------------------------------------------------------
------------
[INFO] Reactor Summary:
[INFO]
[INFO] Mahout Build Tools ................................. SUCCESS [
1.763 s]
[INFO] Apache Mahout ...................................... SUCCESS [
0.036 s]
[INFO] Mahout Math ........................................ SUCCESS [
46.831 s]
[INFO] Mahout HDFS ........................................ SUCCESS [
2.890 s]
[INFO] Mahout Map-Reduce .................................. SUCCESS [09:59
min]
[INFO] Mahout Integration ................................. SUCCESS [
33.694 s]
[INFO] Mahout Examples .................................... SUCCESS [
14.022 s]
[INFO] Mahout Math Scala bindings ......................... SUCCESS [02:36
min]
[INFO] Mahout Spark bindings .............................. FAILURE [01:05
min]
[INFO] Mahout Release Package ............................. SKIPPED
[INFO] Mahout H2O backend ................................. SKIPPED
[INFO] ------------------------------------------------------------
------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------
------------
[INFO] Total time: 15:21 min
[INFO] Finished at: 2017-07-13T15:15:24-04:00
[INFO] Final Memory: 93M/1555M



*One other thing I tried was a spark 2.1 build which passed.*
--> mvn clean install -Phadoop2 -Pspark-2.1
[INFO] Mahout Build Tools ................................. SUCCESS [
1.444 s]
[INFO] Apache Mahout ...................................... SUCCESS [
0.029 s]
[INFO] Mahout Math ........................................ SUCCESS [
43.992 s]
[INFO] Mahout HDFS ........................................ SUCCESS [
2.485 s]
[INFO] Mahout Map-Reduce .................................. SUCCESS [09:39
min]
[INFO] Mahout Integration ................................. SUCCESS [
34.532 s]
[INFO] Mahout Examples .................................... SUCCESS [
15.654 s]
[INFO] Mahout Math Scala bindings ......................... SUCCESS [02:45
min]
[INFO] Mahout Spark bindings .............................. SUCCESS [01:18
min]
[INFO] Mahout Release Package ............................. SUCCESS [
1.230 s]
[INFO] Mahout H2O backend ................................. SUCCESS [03:13
min]
[INFO] ------------------------------------------------------------
------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------
------------
[INFO] Total time: 18:36 min
[INFO] Finished at: 2017-07-13T15:39:17-04:00
[INFO] Final Memory: 144M/1653M