[jira] [Created] (MAHOUT-1889) Mahout doesn't work with Spark 2.0

Discussion:

Sergey Svinarchuk (JIRA)

2016-10-17 11:01:03 UTC

Sergey Svinarchuk created MAHOUT-1889:
-----------------------------------------

Summary: Mahout doesn't work with Spark 2.0
Key: MAHOUT-1889
URL: https://issues.apache.org/jira/browse/MAHOUT-1889
Project: Mahout
Issue Type: Bug
Reporter: Sergey Svinarchuk

In Spark 2.0 was changes path to libraries and classpath. If change classpath to correct for Spark 2.0, all Spark job failed with java.lang.NoSuchMethodError: , because Spark API was changed.

Example for spark-shell:
{code}
./bin/mahout spark-shell

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.repl.SparkILoop.setPrompt(Ljava/lang/String;)V
at org.apache.mahout.sparkbindings.shell.MahoutSparkILoop.<init>(MahoutSparkILoop.scala:58)
at org.apache.mahout.sparkbindings.shell.Main$.main(Main.scala:32)
at org.apache.mahout.sparkbindings.shell.Main.main(Main.scala)

{code}

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Suneel Marthi (JIRA)

2016-10-17 12:16:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582071#comment-15582071 ]

Suneel Marthi commented on MAHOUT-1889:
---------------------------------------

Thanks for reporting this, it makes sense to work on this if any of the Hadoop vendors are already packaging Spark 2x in their distros. Feel free to submit a patch nevertheless.

Post by Sergey Svinarchuk (JIRA)
Mahout doesn't work with Spark 2.0
----------------------------------
Key: MAHOUT-1889
URL: https://issues.apache.org/jira/browse/MAHOUT-1889
Project: Mahout
Issue Type: Bug
Reporter: Sergey Svinarchuk
In Spark 2.0 was changes path to libraries and classpath. If change classpath to correct for Spark 2.0, all Spark job failed with java.lang.NoSuchMethodError: , because Spark API was changed.
{code}
./bin/mahout spark-shell
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.repl.SparkILoop.setPrompt(Ljava/lang/String;)V
at org.apache.mahout.sparkbindings.shell.MahoutSparkILoop.<init>(MahoutSparkILoop.scala:58)
at org.apache.mahout.sparkbindings.shell.Main$.main(Main.scala:32)
at org.apache.mahout.sparkbindings.shell.Main.main(Main.scala)
{code}

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Lucas Araujo (JIRA)

2016-10-25 19:38:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606259#comment-15606259 ]

Lucas Araujo commented on MAHOUT-1889:
--------------------------------------

I would argue that this is not a bug because there is no guarantee that Mahout should work with a version of Spark that it's not its current dependency. As is stated in the FAQ for using Mahout with Spark - https://mahout.apache.org/users/sparkbindings/faq.html:

"If you are getting method signature like errors, most probably you have mismatch between Mahout's Spark dependency and actual Spark installed."

The current mahout/pom.xml states that the dependency is Spark 1.6.2 (and not Spark 2.x), and spark-shell works fine with the 1.6.2 version. So, this bug should either be closed or be changed to a feature request or something.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Lucas Araujo (JIRA)

2016-10-26 13:13:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606259#comment-15606259 ]

Lucas Araujo edited comment on MAHOUT-1889 at 10/26/16 1:13 PM:
----------------------------------------------------------------

I would argue that this is not a bug because there is no guarantee that Mahout should work with a version of Spark that it's not its current dependency. As is stated in the FAQ for using Mahout with Spark - https://mahout.apache.org/users/sparkbindings/faq.html :

"If you are getting method signature like errors, most probably you have mismatch between Mahout's Spark dependency and actual Spark installed."

The current mahout/pom.xml states that the dependency is Spark 1.6.2 (and not Spark 2.x), and spark-shell works fine with the 1.6.2 version. So, this bug should either be closed or be changed to a feature request or something.

was (Author: laou.mh):
I would argue that this is not a bug because there is no guarantee that Mahout should work with a version of Spark that it's not its current dependency. As is stated in the FAQ for using Mahout with Spark - https://mahout.apache.org/users/sparkbindings/faq.html:

"If you are getting method signature like errors, most probably you have mismatch between Mahout's Spark dependency and actual Spark installed."

The current mahout/pom.xml states that the dependency is Spark 1.6.2 (and not Spark 2.x), and spark-shell works fine with the 1.6.2 version. So, this bug should either be closed or be changed to a feature request or something.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Sergey Svinarchuk (JIRA)

2016-10-26 14:11:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608562#comment-15608562 ]

Sergey Svinarchuk commented on MAHOUT-1889:
-------------------------------------------

Mahout also cannot rebuild if change Spark version in pom.xml to 2.0. It fails with

{code}
[ERROR] /home/vaifer/mapr_project/private-mahout/spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala:62: error: value out in class IMain cannot be accessed in scala.tools.nsc.interpreter.IMain
[ERROR] Access to protected value out not permitted because
[ERROR] enclosing class MahoutSparkILoop in package shell is not a subclass of
[ERROR] class IMain in package interpreter where target is defined
[ERROR] _interp.out.println(str)
[ERROR] ^
[ERROR] /home/vaifer/mapr_project/private-mahout/spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala:69: error: value master is not a member of org.apache.spark.repl.SparkILoop
[ERROR] val master = _interp.master match {
[ERROR] ^
[ERROR] /home/vaifer/mapr_project/private-mahout/spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala:77: error: value getAddedJars is not a member of object org.apache.spark.repl.SparkILoop
[ERROR] val jars = SparkILoop.getAddedJars.map(new java.io.File(_).getAbsolutePath)
................
................
{code}

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Suneel Marthi (JIRA)

2016-10-26 16:01:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608860#comment-15608860 ]

Suneel Marthi commented on MAHOUT-1889:
---------------------------------------

We don't support Spark 2.x yet and yes the Mahout spark shell had to be tweaked with each Spark upgrade, in large part thanks to the Spark community for incompatible and breaking changes that come with each Spark release.

Its not on the immediate roadmap to fix this, and its the least priority at the moment for the project to be supporting Spark 2.x. Most Mahout users are still on Spark 1.5x or 1.6x, so it doesn't make sense moving to Spark 2.x.

Feel free to submit a PR for Spark 2x support nevertheless.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Andrew Palumbo (JIRA)

2016-12-16 05:12:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Palumbo updated MAHOUT-1889:
-----------------------------------
Fix Version/s: 0.14.0

Post by Sergey Svinarchuk (JIRA)
Mahout doesn't work with Spark 2.0
----------------------------------
Key: MAHOUT-1889
URL: https://issues.apache.org/jira/browse/MAHOUT-1889
Project: Mahout
Issue Type: Bug
Reporter: Sergey Svinarchuk
Fix For: 0.14.0
In Spark 2.0 was changes path to libraries and classpath. If change classpath to correct for Spark 2.0, all Spark job failed with java.lang.NoSuchMethodError: , because Spark API was changed.
{code}
./bin/mahout spark-shell
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.repl.SparkILoop.setPrompt(Ljava/lang/String;)V
at org.apache.mahout.sparkbindings.shell.MahoutSparkILoop.<init>(MahoutSparkILoop.scala:58)
at org.apache.mahout.sparkbindings.shell.Main$.main(Main.scala:32)
at org.apache.mahout.sparkbindings.shell.Main.main(Main.scala)
{code}

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Andrew Palumbo (JIRA)

2016-12-16 05:12:58 UTC

Permalink

Post by Sergey Svinarchuk (JIRA)
Mahout doesn't work with Spark 2.0
----------------------------------
Key: MAHOUT-1889
URL: https://issues.apache.org/jira/browse/MAHOUT-1889
Project: Mahout
Issue Type: Bug
Reporter: Sergey Svinarchuk
Priority: Critical
Fix For: 0.14.0
In Spark 2.0 was changes path to libraries and classpath. If change classpath to correct for Spark 2.0, all Spark job failed with java.lang.NoSuchMethodError: , because Spark API was changed.
{code}
./bin/mahout spark-shell
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.repl.SparkILoop.setPrompt(Ljava/lang/String;)V
at org.apache.mahout.sparkbindings.shell.MahoutSparkILoop.<init>(MahoutSparkILoop.scala:58)
at org.apache.mahout.sparkbindings.shell.Main$.main(Main.scala:32)
at org.apache.mahout.sparkbindings.shell.Main.main(Main.scala)
{code}

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Andrew Palumbo (JIRA)

2016-12-16 05:13:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753483#comment-15753483 ]

Andrew Palumbo commented on MAHOUT-1889:
----------------------------------------

Marked this as fix for 0.14.0 but we will have to re-assess after 0.13.0 goes out, weather this will be included in 0.14 or later.

Post by Sergey Svinarchuk (JIRA)
Mahout doesn't work with Spark 2.0
----------------------------------
Key: MAHOUT-1889
URL: https://issues.apache.org/jira/browse/MAHOUT-1889
Project: Mahout
Issue Type: Bug
Reporter: Sergey Svinarchuk
Priority: Critical
Fix For: 0.14.0
In Spark 2.0 was changes path to libraries and classpath. If change classpath to correct for Spark 2.0, all Spark job failed with java.lang.NoSuchMethodError: , because Spark API was changed.
{code}
./bin/mahout spark-shell
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.repl.SparkILoop.setPrompt(Ljava/lang/String;)V
at org.apache.mahout.sparkbindings.shell.MahoutSparkILoop.<init>(MahoutSparkILoop.scala:58)
at org.apache.mahout.sparkbindings.shell.Main$.main(Main.scala:32)
at org.apache.mahout.sparkbindings.shell.Main.main(Main.scala)
{code}

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)