[jira] [Created] (MAHOUT-1885) Inital Implementation of VCL Bindings

Discussion:

Andrew Palumbo (JIRA)

2016-10-06 23:53:20 UTC

Andrew Palumbo created MAHOUT-1885:
--------------------------------------

Summary: Inital Implementation of VCL Bindings
Key: MAHOUT-1885
URL: https://issues.apache.org/jira/browse/MAHOUT-1885
Project: Mahout
Issue Type: Improvement
Components: Math
Affects Versions: 0.12.2
Reporter: Andrew Palumbo
Assignee: Andrew Palumbo
Fix For: 0.13.0

Push a working experimental branch of VCL bindings into master. There is still a lot of work to be done. All tests are passing, At the moment there am opening this JIRA mostly to get a number for PR and to test profiles against on travis.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-10 00:33:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560908#comment-15560908 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

GitHub user andrewpalumbo opened a pull request:

https://github.com/apache/mahout/pull/261

MAHOUT-1885 [WIP][FOR COMMENT]: Inital implementation of VCL bindings for mahout math.

This is an initial set of bindings for VCL to mahout matrices. Currently supports `DenseRowMatrix` and `SparseRowMatrix` for both GPU and multi-threaded CPU.

Note that there are two new modules: `vienniacl` and `viennacl-omp` there are new profiles to activate these: `-Pvienniacl` activates both and `-Pviennacl-omp` activates only the OpenMP version.

The default build should not require any new installs.

If activating either version you';; need to install vienniacl 1.7.2+

On ubuntu 16.04 this is simply:
```
sudo apt-get install libviennacl-dev
```
if running on GPU you'll also need, you'll need to have OpenCL 1.2+, which is likely installed with your video driver.

Hadoop2 is still in a profile which needs to be removed, so the command to install for GPU and CPU is:
```
mvn clean install -Pviennacl -Phadoop2 -DskipTests && mvn test
```
for OpenMP only (mac users) :
```
mvn clean install -Pviennacl -Phadoop2 -DskipTests && mvn test
```

The output is still very verbose and will let you know each time a `%*%` operatoion is made which device it is made on.

A few todos:

1. Not sure that we need two separate modules for omp and gpu. There may be a way to handle this with environment variables.
2. Currently a new object is being created each timw `%*%` is called. it would be best to create these once and then cache them for subsequent calls.
3. Currently using try/catch for `XXXMMul` creation, really bad style.
4. GPU Vector are not working and throw exceptions when trying to read data out of them
- Vector this we could implement at least one native linear solver. (most viennacl linear solvers require boost however this is slated for removal on the next release).
5. Have been noticing intermittent crashes on GPU.
6. Tests with CLI drivers are currently commented out. Was getting odd non-numeric failures with them. must be a config issue.
7. Currently each operation can only be run on one GPU /node.

I'd like to get this committed relatively quickly and then go back and fine tune.

Any feedback is much appreciated.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewpalumbo/mahout viennacl-opmmul-a

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/mahout/pull/261.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #261

----
commit dfc93684a3d17a0331504a45b68ad2dfb9520f5c
Author: Andrew Palumbo <***@apache.org>
Date: 2016-08-10T17:48:51Z

vienniacl back POC for mahout dense and sparse matrix multiplication. Compiling and testing as is with a couple of tests commented out. Having issues currently with Dense %*% Sparse, and reading data from GPU after Matrix %*% Vector multiplication.

commit 5d96a17bb828b2c2a887ddd48f9eede4e13eef92
Author: Andrew Palumbo <***@apache.org>
Date: 2016-08-16T20:32:47Z

fix imports for Sparse/Dense MMul

commit c2a548e239d31fee56428e7a02c8a0f76de01915
Author: Andrew Palumbo <***@apache.org>
Date: 2016-08-16T20:51:24Z

Add functions to LinalgFunctions.java for direct matrix prod expressions, remove import. Uncomment missed OpenCL test

commit 8de0fd03821a2bd08bd469177214567c6ae60b66
Author: Andrew Palumbo <***@apache.org>
Date: 2016-08-31T16:46:19Z

allow spark build in module. Add a linux-haswell properties file- this may need a property in viennacl/pom.xml to be used. As well will need one for mac. This is for the distribution artifact which may not be built by a 'native' machine. Currently leaving linux-x86_64 to use -march=native but as this will likely need to be changed as suneel does releases from a mac and I (AP) use an amdfam10 so we should have a general linux-x86_64 properties file.

commit 1525dbe389aebd3ea1244ec4468cff08fb683cda
Author: Andrew Palumbo <***@apache.org>
Date: 2016-08-31T20:19:43Z

[EXPIRMENTAL BRANCH] Split OpenCL and OpenMP modules up. Next steps: implement MMulGPU an MMulOMP and SolverFactory per doc, test on AWS GPU Cluster.

commit a6a81f8ced8f24dfbafc655e46822a9ddfd2c90a
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-07T17:21:49Z

[EXPIRMENTAL BRANCH] Hacked everything together in hopes of having a running branch to test on AWS byt Wed 9/7 Hangout... Compiling and auto detecting mahout-native-viennacl module. Getting a runtime error when trying to create the o.a.m.vcl.GPUMMul Object via reflection from the SolverFactory:

scala.reflect.runtime.ReflectError: value viennacl is not a package
at scala.reflect.runtime.JavaMirrors.scala14884makeScalaPackage(JavaMirrors.scala:912)

commit aac6adc2f08712fdcb5a0ce91aa2e1bccf9bdac4
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-08T18:27:38Z

Hardcode path to vcl jar for classloader. compiles.. getting null pointer.

commit 5024ad27fa296505a486ba32ca6645f7aa91e029
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-09T01:48:45Z

[EXPIRMENTAL BRANCH] everything is hacked together but tests r passing

commit 0a6d1855783dbcbf2a9809243759d9c2e0ba4a5e
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-09T02:06:42Z

[EXPIRMENTAL BRANCH] removed large core file. testing out some sparse/dense cases in viennacl/GPUMMul

commit c1b3f05b59cab146871b33526d1b0aa2a2e61a76
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-09T02:53:21Z

[EXPIRMENTAL BRANCH] gpuRWRW was failing with a missing allocator when directed from:

java.lang.UnsatisfiedLinkError: org.apache.mahout.viennacl.vcl.javacpp.Context.allocate(I)V
at org.apache.mahout.viennacl.vcl.javacpp.Context.allocate(Native Method)
at org.apache.mahout.viennacl.vcl.javacpp.Context.<init>(Context.scala:33)
at org.apache.mahout.viennacl.vcl.GPUMMul$.org4600gpuRWRW(GPUMMul.scala:146)
at org.apache.mahout.viennacl.vcl.GPUMMul$.org4600jvmCWRW(GPUMMul.scala:174)
at org.apache.mahout.viennacl.vcl.GPUMMul4600anonfun6.apply(GPUMMul.scala:79)

revert to jvm for now just to get some passing tests

commit d8abf28932a9c2216a41feff2e6cead4bd7a99d3
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-09T03:07:35Z

[EXPIRMENTAL BRANCH] GPUMMul using only Sparse %*% Sparse on GPU -> all spark tests pass

commit 8f1f716a7766e47ccda22dae2aeb16d4e54cff90
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-10T19:10:44Z

[EXPRIMENTAL BRANCH] wip: some VCL classes are not being picked up by spark.

Lost task 2.0 in stage 44.0 (TID 283, localhost): java.lang.ExceptionInInitializerError
at org.apache.mahout.viennacl.vcl.GPUMMul$.org5465gpuSparseRWRW(GPUMMul.scala:199)
at org.apache.mahout.viennacl.vcl.GPUMMul$.org5465jvmSparseCWRW(GPUMMul.scala:252)
at org.apache.mahout.viennacl.vcl.GPUMMul5465anonfun5.apply(GPUMMul.scala:111)
at org.apache.mahout.viennacl.vcl.GPUMMul5465anonfun5.apply(GPUMMul.scala:111)
at org.apache.mahout.viennacl.vcl.GPUMMul$.apply(GPUMMul.scala:122)
at org.apache.mahout.viennacl.vcl.GPUMMul$.apply(GPUMMul.scala:34)
at org.apache.mahout.math.scalabindings.RLikeMatrixOps.(RLikeMatrixOps.scala:36)
at org.apache.mahout.sparkbindings.blas.AtB5465anonfun15465anonfun.apply(AtB.scala:241)
at org.apache.mahout.sparkbindings.blas.AtB5465anonfun15465anonfun.apply(AtB.scala:239)
at scala.collection.Iterator5465anon.next(Iterator.scala:89)
at scala.collection.Iterator5465anon3.next(Iterator.scala:372)
at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:211)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.mahout.viennacl.vcl.javacpp.MemHandle

org.apache.mahout.viennacl.vcl.javacpp.MemHandle is clearly added to jar. jar must not be reacking back end properly other GPUMMul gpu algorithms are passing without issue

commit 07d0087dc49c0e14c128e45181622c31e8b60322
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-10T23:04:29Z

[EXPIREMENTAL BRANCH] in local mode, only 1 thread may be used per GPU so for tests to pass, change the Spark URL to local[1] in DistributedSparkSuite. Add mahout-native-niennack_2.10 module to Spark and remove hard coded path to jars, Tests are passing now except for those with 0 non-zero elements, an assertion made by vienniacl::compressed_matrix:

Solution will be to eith add a check for this wit a fallback to jvm MMul or implement:

try (GPUMMul) catch ex { try (OMPMMul) catch ex { jvmMMul}}

commit 1a37c5a3ee1dd8a483175fbcdb151699b3775601
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-11T00:28:42Z

[EXPIREMENTAL BRANCH] implement AAt in GPU MMul

commit 201bbdb2098cefdfffc1b006dd0d679d2ec5da88
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-11T01:28:31Z

[EXPIREMENTAL BRANCH] cleanup some fix an incorrect check for empty matrix in GPUMMul, still somehow getting error:

commit f55389cfe31fb8555ed1932024ae4a11562219eb
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-12T19:08:44Z

[EXPIREMENTAL BRANCH] All Spark teses pass with the exception of similiarity analysis and cf Drivers. All flink tests pass.

Can not get the backend to recogniz mahout-native-javacpp even when packed into the assembly jar. fails in Spark-shell in pseudo cluster mode (1) local worker

commit 971f6577c51b749cc520d3475c0f9656ec00def0
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-12T19:11:20Z

reomve flink from build. add test script

commit 5f6484846e40abe8ab8031c8a94853a32cb7cfc2
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-13T19:34:37Z

[EXPIREMENTAL BRANCH] fix check for empty SparseMatrices (possibly empty partitions) in GPUMMul and pass those cases off to jvm MMUl

commit 37ee67d40337c84650d6ef9bfabfd9e86ad9eba4
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-13T19:39:59Z

[EXPIREMENTAL BRANCH] set threads in DistributedSparkSuite back to 3

commit 8a7b747233756e57ce417d7f09da98b110ab99d3
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-13T20:49:39Z

Fixed typo in Spark-assembly that was keeping shell from working in standalone cluster mode with 1 worker. Tested drmA %*% drmB with different partition sizes:

timeSparseDRMMMul: (m: Int, n: Int, s: Int, para: Int, pctDense: Double)Long
mahout>
mahout>
mahout>
mahout> timeSparseDRMMMul(100,100,100,1,.1)
res0: Long = 2954
mahout> timeSparseDRMMMul(1000,1000,100,1,.1)
res1: Long = 3593
mahout> timeSparseDRMMMul(3000,3000,100,1,.1)
res2: Long = 27919
mahout> timeSparseDRMMMul(3000,3000,100,10,.1)
res3: Long = 11012
mahout> timeSparseDRMMMul(3000,3000,100,100,.1)
res4: Long = 24706
mahout> timeSparseDRMMMul(3000,3000,100,50,.1)
res5: Long = 10378
mahout> timeSparseDRMMMul(3000,3000,3000,50,.1)
res6: Long = 139161
mahout> timeSparseDRMMMul(3000,3000,3000,10,.1)
res7: Long = 122031

commit 5d6ff3c930d05acf5d4df2ea0af0bea0ec96615c
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-13T21:20:29Z

Supress output in SolverFactory, All tests passing except for viennacl

commit d855460ad3abe75cf6c0f6ccc5a8cc8c6466d6fe
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-14T16:31:16Z

clean up some of the comments From SolverFactory. Issue: Tests in VCL fail with java.lang.NullPointerException if there is only one card on the machine:

val oclCtx = new Context(Context.OPENCL_MEMORY) // creates context
...
val mDvecC = mxA %*% dvecB // creates context

commit 4ec1c651731b62c32ca55bf3d7907dc21272b1bc
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-14T16:39:44Z

debuging SolverFactory with println, add SparseSparseDrmTimer.mscala

commit 4441550557fa69d01c8db312d0dd648208b2d5cc
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-14T18:38:10Z

added MMul as default for classes n the case that more than one ocl context is set.

commit a843e3a459a19d2c9a850e07888300d1836a77c5
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-14T19:05:49Z

all tests passing (verbosely)

commit 09cc2d01f6eb113e7baa5078248e26f16dfc273a
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-15T00:38:44Z

[EXPIREMENTAL BRANCH(STILL)] Debugging GPUMMul.. SparseRowMatrix %*% SparseRowMatrix in DRM was geing passed off to default JVM method jvmRWCW.

commit 8fba0c888d8de2e8c41845227db026d772813d57
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-15T01:15:54Z

[EXPIREMENTAL BRANCH] changed spark ABt to use sparse marices. there should not be any Dense Matrices in the test calculation /home/andrew/sandbox/mahout/examples/bin/SparseSparseDrmTimer.mscala

we are getting dense matrices though it seems that it may be coming from the Kryo(de)Serializer:

169.1.102): java.lang.OutOfMemoryError: Java heap space
at org.apache.mahout.math.DenseMatrix.<init>(DenseMatrix.java:66)
at org.apache.mahout.math.scalabindings.package5989anonfun.apply(package.scala:163)
at org.apache.mahout.math.scalabindings.package5989anonfun.apply(package.scala:141)
at scala.collection.TraversableLike5989anonfun.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike5989anonfun.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at org.apache.mahout.math.scalabindings.package$.dense(package.scala:141)
at org.apache.mahout.common.io.GenericMatrixKryoSerializer.read(GenericMatrixKryoSerializer.scala:179)
at org.apache.mahout.common.io.GenericMatrixKryoSerializer.read(GenericMatrixKryoSerializer.scala:38)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
at com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:42)
at com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:33)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:192)
at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171)
at org.apache.spark.serializer.DeserializationStream5989anon.getNext(Serializer.scala:201)
at org.apache.spark.serializer.DeserializationStream5989anon.getNext(Serializer.scala:198)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
at scala.collection.Iterator5989anon3.hasNext(Iterator.scala:371)
at scala.collection.Iterator5989anon1.hasNext(Iterator.scala:327)
at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator5989anon1.hasNext(Iterator.scala:327)
at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:132)
at org.apache.spark.rdd.CoGroupedRDD5989anonfun.apply(CoGroupedRDD.scala:169)
at org.apache.spark.rdd.CoGroupedRDD5989anonfun.apply(CoGroupedRDD.scala:168)
at scala.collection.TraversableLike5989anonfun.apply(TraversableLike.scala:772)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

commit 3d759ba5f4513b3a4303176e8d3ddf382f026caf
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-15T22:31:20Z

[Expiremental Branch] Changed to use Matrix.zSum > 0 to identify Matrices that would trip up CSR in GPUMmul. Getting errors in flink and spark tests mostly from GPUMMul.gpuRWCW

commit 75cf32f75711df0939f3884056328fcd12f49469
Author: Andrew Palumbo <***@apache.org>
Date: 2016-09-15T22:52:05Z

removed debugging printout that was causing failures

commit 6bacd20e1a56d0c1a8c94f2f0ccf4e75444cf34a
Author: Andrew Palumbo <***@apache.org>
Date: 2016-10-07T00:49:06Z

commented out failing RowSimilarityDriver test temporarily

----

Post by Andrew Palumbo (JIRA)
Inital Implementation of VCL Bindings
-------------------------------------
Key: MAHOUT-1885
URL: https://issues.apache.org/jira/browse/MAHOUT-1885
Project: Mahout
Issue Type: Improvement
Components: Math
Affects Versions: 0.12.2
Reporter: Andrew Palumbo
Assignee: Andrew Palumbo
Fix For: 0.13.0
Push a working experimental branch of VCL bindings into master. There is still a lot of work to be done. All tests are passing, At the moment there am opening this JIRA mostly to get a number for PR and to test profiles against on travis.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-10 15:41:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15562616#comment-15562616 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user smarthi commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82629295

--- Diff: math-scala/src/main/scala/org/apache/mahout/math/scalabindings/RLikeMatrixOps.scala ---
@@ -17,14 +17,22 @@
package org.apache.mahout.math.scalabindings

import org.apache.mahout.math.function.Functions
-import org.apache.mahout.math.{Vector, Matrix}
+import org.apache.mahout.math.{Matrix, Vector}
import scala.collection.JavaConversions._
import RLikeOps._
+import org.apache.mahout.math.scalabindings._
+

class RLikeMatrixOps(m: Matrix) extends MatrixOps(m) {

/** Structure-optimized mmul */
- def %*%(that: Matrix) = MMul(m, that, None)
+
+ implicit var solverOperator: opMMulSolver =_
--- End diff --

leave a space after '='

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-10 15:42:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15562619#comment-15562619 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user smarthi commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82629450

--- Diff: pom.xml ---
@@ -803,34 +810,36 @@
<module>buildtools</module>
<module>math</module>
<module>hdfs</module>
- <module>mr</module>
- <module>integration</module>
- <module>examples</module>
+ 
+ 
+ 
<module>distribution</module>
<module>math-scala</module>
<module>spark</module>
<module>spark-shell</module>
<module>flink</module>
<module>h2o</module>
+ 
+ 
</modules>

<profiles>
+
<profile>
- <id>hadoop1</id>
- <properties>
- <hadoop.classifier>hadoop1</hadoop.classifier>
- <hadoop.version>1.2.1</hadoop.version>
- </properties>
- <dependencyManagement>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-core</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- </dependencies>
- </dependencyManagement>
+ <id>viennacl</id>
--- End diff --

hadoop1 profile has been removed, rebase ur code and update the PR

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 03:55:21 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564366#comment-15564366 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261

rebased on master

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 05:08:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564487#comment-15564487 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725138

--- Diff: LICENSE.txt ---
@@ -799,3 +799,32 @@ The following license applies to the H2O package
identification within third-party archives.

Copyright 2012 0xdata, Inc
+
+================================================================
+The following applies to the ViennaCL library and files in the mahout-native-viennacl module
+================================================================
+
+ Copyright (c) 2010-2016 Institute for Microelectronics,
+ Institute for Analysis and Scientific Computing, TU Wien.
+ Portions of this software are copyright by UChicago Argonne, LLC.
+ Argonne National Laboratory, with facilities in the state of Illinois,
+ is owned by The United States Government, and operated by UChicago Argonne, LLC
+ under provision of a contract with the Department of Energy.
+
+ Permission is hereby granted, free of charge, to any person obtaining a copy
+ of this software and associated documentation files (the "Software"), to deal
+ in the Software without restriction, including without limitation the rights
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ copies of the Software, and to permit persons to whom the Software is
+ furnished to do so, subject to the following conditions:
+
+ The above copyright notice and this permission notice shall be included in
+ all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ THE SOFTWARE.
--- End diff --

yes definitly..

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 05:22:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564517#comment-15564517 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725973

--- Diff: math/src/main/java/org/apache/mahout/math/DenseMatrix.java ---
@@ -65,6 +65,12 @@ public DenseMatrix(int rows, int columns) {
super(rows, columns);
this.values = new double[rows][columns];
}
+ /*
--- End diff --

put the comment on a single line and leave a space after ()

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 05:22:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564515#comment-15564515 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725812

--- Diff: examples/bin/SparseSparseDrmTimer.mscala ---
@@ -0,0 +1,37 @@
+
+
+def timeSparseDRMMMul(m: Int, n: Int, s: Int, para: Int, pctDense: Double = .20): Long = {
+
+
+
+ val drmA = drmParallelizeEmpty(m , s, para).mapBlock(){
+ case (keys,block:Matrix) =>
+ val R = scala.util.Random
+ R.setSeed(1234L)
--- End diff --

If we use hardcoded seeds, we should probably define them as a constant somewhere

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 05:22:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564513#comment-15564513 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82726045

--- Diff: spark/src/main/scala/org/apache/mahout/sparkbindings/package.scala ---
@@ -75,7 +79,8 @@ package object sparkbindings {

} else {
// In local mode we don't care about jars, do we?
- sparkConf.setJars(customJars.toSeq)
+ // yes adding jars always now since we are not including the artifacts
+ sparkConf.setJars(customJars.toSeq)
--- End diff --

one space too much here

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 05:22:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564514#comment-15564514 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725986

--- Diff: math/src/main/java/org/apache/mahout/math/DenseVector.java ---
@@ -253,6 +253,13 @@ public void addAll(Vector v) {
}
}

+ /*
+ * return thr backing double array.
+ */
+ public double[] getBackingDataStructure(){
--- End diff --

put the comment on a single line and leave a space after ()

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 05:22:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564518#comment-15564518 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725859

--- Diff: examples/bin/testVcl.scala ---
@@ -0,0 +1,30 @@
+import org.apache.spark.SparkContext._
+import org.apache.spark.SparkContext
+import org.apache.spark.SparkConf
+
+
+
+object TestVCL {
+
+ def main(args: Array[String]) {
+ run()
+ }
+
+ private def run() {
+ val sparkConf = new SparkConf()
+ sparkConf.setAppName("TestJob")
+ sparkConf.set("spark.cores.max", "8")
+ sparkConf.set("spark.storage.memoryFraction", "0.1")
+ sparkConf.set("spark.shuffle.memoryFracton", "0.2")
+ sparkConf.set("spark.executor.memory", "2g")
+ sparkConf.setJars(List("target/scala-2.10/spark-test-assembly-1.0.jar"))
+ sparkConf.setMaster(s"spark://dev1.dev.pulse.io:7077")
+ sparkConf.setSparkHome("/home/pulseio/spark/current")
--- End diff --

hardcoded path again

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 05:22:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564516#comment-15564516 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725845

--- Diff: examples/bin/testVcl.scala ---
@@ -0,0 +1,30 @@
+import org.apache.spark.SparkContext._
+import org.apache.spark.SparkContext
+import org.apache.spark.SparkConf
+
+
+
+object TestVCL {
+
+ def main(args: Array[String]) {
+ run()
+ }
+
+ private def run() {
+ val sparkConf = new SparkConf()
+ sparkConf.setAppName("TestJob")
+ sparkConf.set("spark.cores.max", "8")
+ sparkConf.set("spark.storage.memoryFraction", "0.1")
+ sparkConf.set("spark.shuffle.memoryFracton", "0.2")
+ sparkConf.set("spark.executor.memory", "2g")
+ sparkConf.setJars(List("target/scala-2.10/spark-test-assembly-1.0.jar"))
+ sparkConf.setMaster(s"spark://dev1.dev.pulse.io:7077")
--- End diff --

We should not have these hardcoded paths here probably

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 05:22:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564519#comment-15564519 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725898

--- Diff: math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala ---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.mahout.math.scalabindings
+
+import java.io.File
+
+import org.apache.mahout.logging._
+
+import scala.reflect.ClassTag
+import scala.reflect.runtime._
+import scala.reflect._
+
+
+class SolverFactory {
+
+ private final implicit val log = getLog(this.getClass)
+
+ // just temp for quick POC
+ val classMap: Map[String,String] =
+ Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
--- End diff --

classOf[...].getName

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 05:51:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564574#comment-15564574 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82727932

--- Diff: math/src/main/java/org/apache/mahout/math/DenseVector.java ---
@@ -253,6 +253,13 @@ public void addAll(Vector v) {
}
}

+ /*
+ * return thr backing double array.
+ */
+ public double[] getBackingDataStructure(){
--- End diff --

this is actually never ended up needing. Will revert this change.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 05:51:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564571#comment-15564571 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261

@sscdotopen thx for the review. I did just rebase it on the current master, which changes do you see that are not related to VuennaCL?

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 05:58:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564584#comment-15564584 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82728207

--- Diff: viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/OMPMMul.scala ---
@@ -0,0 +1,448 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mahout.viennacl.openmp
+
+import org.apache.mahout.logging._
+import org.apache.mahout.math
+import org.apache.mahout.math._
+import org.apache.mahout.math.flavor.{BackEnum, TraversingStructureEnum}
+import org.apache.mahout.math.function.Functions
+import org.apache.mahout.math.scalabindings.RLikeOps._
+import org.apache.mahout.math.scalabindings._
+import org.apache.mahout.viennacl.openmp.javacpp.Functions._
+import org.apache.mahout.viennacl.openmp.javacpp.LinalgFunctions._
+import org.apache.mahout.viennacl.openmp.javacpp.{CompressedMatrix, Context, DenseRowMatrix}
+
+import scala.collection.JavaConversions._
+
+object OMPMMul extends MMBinaryFunc {
+
+ private final implicit val log = getLog(OMPMMul.getClass)
+
+ override def apply(a: Matrix, b: Matrix, r: Option[Matrix]): Matrix = {
+
+ require(a.ncol == b.nrow, "Incompatible matrix sizes in matrix multiplication.")
+
+ val (af, bf) = (a.getFlavor, b.getFlavor)
+ val backs = (af.getBacking, bf.getBacking)
+ val sd = (af.getStructure, math.scalabindings.densityAnalysis(a), bf.getStructure, densityAnalysis(b))
+
+
+ try {
+
+ val alg: MMulAlg = backs match {
+
+ // Both operands are jvm memory backs.
+ case (BackEnum.JVMMEM, BackEnum.JVMMEM) ⇒
+
+ sd match {
+
+ // Multiplication cases by a diagonal matrix.
+ case (TraversingStructureEnum.VECTORBACKED, _, TraversingStructureEnum.COLWISE, _)
+ if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagCW
+ case (TraversingStructureEnum.VECTORBACKED, _, TraversingStructureEnum.SPARSECOLWISE, _)
+ if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagCW
+ case (TraversingStructureEnum.VECTORBACKED, _, TraversingStructureEnum.ROWWISE, _)
+ if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagRW
+ case (TraversingStructureEnum.VECTORBACKED, _, TraversingStructureEnum.SPARSEROWWISE, _)
+ if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagRW
+
+ case (TraversingStructureEnum.COLWISE, _, TraversingStructureEnum.VECTORBACKED, _)
+ if b.isInstanceOf[DiagonalMatrix] ⇒ jvmCWDiag
+ case (TraversingStructureEnum.SPARSECOLWISE, _, TraversingStructureEnum.VECTORBACKED, _)
+ if b.isInstanceOf[DiagonalMatrix] ⇒ jvmCWDiag
+ case (TraversingStructureEnum.ROWWISE, _, TraversingStructureEnum.VECTORBACKED, _)
+ if b.isInstanceOf[DiagonalMatrix] ⇒ jvmRWDiag
+ case (TraversingStructureEnum.SPARSEROWWISE, _, TraversingStructureEnum.VECTORBACKED, _)
+ if b.isInstanceOf[DiagonalMatrix] ⇒ jvmRWDiag
+
+ // Dense-dense cases
+ case (TraversingStructureEnum.ROWWISE, true, TraversingStructureEnum.COLWISE, true) if a eq b.t ⇒ ompDRWAAt
+ case (TraversingStructureEnum.ROWWISE, true, TraversingStructureEnum.COLWISE, true) if a.t eq b ⇒ ompDRWAAt
+ case (TraversingStructureEnum.ROWWISE, true, TraversingStructureEnum.COLWISE, true) ⇒ ompRWCW
+ case (TraversingStructureEnum.ROWWISE, true, TraversingStructureEnum.ROWWISE, true) ⇒ jvmRWRW
+ case (TraversingStructureEnum.COLWISE, true, TraversingStructureEnum.COLWISE, true) ⇒ jvmCWCW
+ case (TraversingStructureEnum.COLWISE, true, TraversingStructureEnum.ROWWISE, true) if a eq b.t ⇒ jvmDCWAAt
+ case (TraversingStructureEnum.COLWISE, true, TraversingStructureEnum.ROWWISE, true) if a.t eq b ⇒ jvmDCWAAt
+ case (TraversingStructureEnum.COLWISE, true, TraversingStructureEnum.ROWWISE, true) ⇒ jvmCWRW
+
+ // Sparse row matrix x sparse row matrix (array of vectors)
+ case (TraversingStructureEnum.ROWWISE, false, TraversingStructureEnum.ROWWISE, false) ⇒ ompSparseRWRW
+ case (TraversingStructureEnum.ROWWISE, false, TraversingStructureEnum.COLWISE, false) ⇒ jvmSparseRWCW
+ case (TraversingStructureEnum.COLWISE, false, TraversingStructureEnum.ROWWISE, false) ⇒ jvmSparseCWRW
+ case (TraversingStructureEnum.COLWISE, false, TraversingStructureEnum.COLWISE, false) ⇒ jvmSparseCWCW
+
+ // Sparse matrix x sparse matrix (hashtable of vectors)
+ case (TraversingStructureEnum.SPARSEROWWISE, false, TraversingStructureEnum.SPARSEROWWISE, false) ⇒
+ ompSparseRowRWRW
+ case (TraversingStructureEnum.SPARSEROWWISE, false, TraversingStructureEnum.SPARSECOLWISE, false) ⇒
+ jvmSparseRowRWCW
+ case (TraversingStructureEnum.SPARSECOLWISE, false, TraversingStructureEnum.SPARSEROWWISE, false) ⇒
+ jvmSparseRowCWRW
+ case (TraversingStructureEnum.SPARSECOLWISE, false, TraversingStructureEnum.SPARSECOLWISE, false) ⇒
+ jvmSparseRowCWCW
+
+ // Sparse matrix x non-like
+ case (TraversingStructureEnum.SPARSEROWWISE, false, TraversingStructureEnum.ROWWISE, _) ⇒ ompSparseRowRWRW
+ case (TraversingStructureEnum.SPARSEROWWISE, false, TraversingStructureEnum.COLWISE, _) ⇒ jvmSparseRowRWCW
+ case (TraversingStructureEnum.SPARSECOLWISE, false, TraversingStructureEnum.ROWWISE, _) ⇒ jvmSparseRowCWRW
+ case (TraversingStructureEnum.SPARSECOLWISE, false, TraversingStructureEnum.COLWISE, _) ⇒ jvmSparseCWCW
+ case (TraversingStructureEnum.ROWWISE, _, TraversingStructureEnum.SPARSEROWWISE, false) ⇒ ompSparseRWRW
+ case (TraversingStructureEnum.ROWWISE, _, TraversingStructureEnum.SPARSECOLWISE, false) ⇒ jvmSparseRWCW
+ case (TraversingStructureEnum.COLWISE, _, TraversingStructureEnum.SPARSEROWWISE, false) ⇒ jvmSparseCWRW
+ case (TraversingStructureEnum.COLWISE, _, TraversingStructureEnum.SPARSECOLWISE, false) ⇒ jvmSparseRowCWCW
+
+ // Everything else including at least one sparse LHS or RHS argument
+ case (TraversingStructureEnum.ROWWISE, false, TraversingStructureEnum.ROWWISE, _) ⇒ ompSparseRWRW
+ case (TraversingStructureEnum.ROWWISE, false, TraversingStructureEnum.COLWISE, _) ⇒ jvmSparseRWCW
+ case (TraversingStructureEnum.COLWISE, false, TraversingStructureEnum.ROWWISE, _) ⇒ jvmSparseCWRW
+ case (TraversingStructureEnum.COLWISE, false, TraversingStructureEnum.COLWISE, _) ⇒ jvmSparseCWCW2flips
+
+ // Sparse methods are only effective if the first argument is sparse, so we need to do a swap.
+ case (_, _, _, false) ⇒ (a, b, r) ⇒ apply(b.t, a.t, r.map {
+ _.t
+ }).t
+
+ // Default jvm-jvm case.
+ // for some reason a SrarseRowMatrix DRM %*% SrarseRowMatrix DRM was dumping off to here
+ case _ ⇒ ompRWCW
+ }
+ }
+
+ alg(a, b, r)
+ } catch {
+ // TODO FASTHACK: just revert to JVM if there is an exception..
+ // eg. java.lang.nullPointerException if more openCL contexts
+ // have been created than number of GPU cards.
+ // better option wuold be to fall back to OpenCl First.
+ case ex: Exception =>
+ println(ex.getMessage + "falling back to JVM MMUL")
+ return MMul(a, b, r)
+ }
+ }
+
+ type MMulAlg = MMBinaryFunc
+
+ @inline
+ private def ompRWCW(a: Matrix, b: Matrix, r: Option[Matrix] = None): Matrix = {
+ println("ompRWCW")
+ //
+ // require(r.forall(mxR ⇒ mxR.nrow == a.nrow && mxR.ncol == b.ncol))
+ // val (m, n) = (a.nrow, b.ncol)
+ //
+ // val mxR = r.getOrElse(if (densityAnalysis(a)) a.like(m, n) else b.like(m, n))
+ //
+ // for (row ← 0 until mxR.nrow; col ← 0 until mxR.ncol) {
+ // // this vector-vector should be sort of optimized, right?
+ // mxR(row, col) = a(row, ::) dot b(::, col)
+ // }
+ // mxR
+
+ val hasElementsA = a.zSum() > 0.0
+ val hasElementsB = b.zSum() > 0.0
+
+ // A has a sparse matrix structure of unknown size. We do not want to
+ // simply convert it to a Dense Matrix which may result in an OOM error.
+
+ // If it is empty use JVM MMul, since we can not convert it to a VCL CSR Matrix.
+ if (!hasElementsA) {
+ println("Matrix a has zero elements can not convert to CSR")
+ return MMul(a, b, r)
+ }
+
+ // CSR matrices are efficient up to 50% non-zero
+ if(b.getFlavor.isDense) {
--- End diff --

space after if

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 06:08:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564602#comment-15564602 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82729074

--- Diff: spark/src/test/scala/org/apache/mahout/cf/SimilarityAnalysisSuite.scala ---
@@ -17,16 +17,14 @@

package org.apache.mahout.cf

--- End diff --

Oh yeah- this cooccurance stuff- not sure how this made it in here...

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-11 14:43:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15565613#comment-15565613 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user smarthi commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82808490

--- Diff: spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala ---
@@ -162,7 +163,7 @@ class MahoutSparkILoop extends SparkILoop {
_ __ ___ __ _| |__ ___ _ _| |_
| '_ ` _ \ / _` | '_ \ / _ \| | | | __|
| | | | | | (_| | | | | (_) | |_| | |_
- |_| |_| |_|\__,_|_| |_|\___/ \__,_|\__| version 0.12.2
+ |_| |_| |_|\__,_|_| |_|\___/ \__,_|\__| version 0.13.0

--- End diff --

A question to the team for future releases - with Zeppelin now available, do we still want to maintain Mahout-Spark shell ?

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-12 03:58:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567475#comment-15567475 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82932654

--- Diff: math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala ---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.mahout.math.scalabindings
+
+import java.io.File
+
+import org.apache.mahout.logging._
+
+import scala.reflect.ClassTag
+import scala.reflect.runtime._
+import scala.reflect._
+
+
+class SolverFactory {
+
+ private final implicit val log = getLog(this.getClass)
+
+ // just temp for quick POC
+ val classMap: Map[String,String] =
+ Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
--- End diff --

not Sure that we can use classOf[..].getName, since the dependency is not guaranteed to be there. Eg if we build with the default `mvn clean install`, GPUMMul will be no where on the classpath. so `classOf[GPU]` would fail.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-12 04:36:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567547#comment-15567547 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82933150

--- Diff: math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala ---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.mahout.math.scalabindings
+
+import java.io.File
+
+import org.apache.mahout.logging._
+
+import scala.reflect.ClassTag
+import scala.reflect.runtime._
+import scala.reflect._
+
+
+class SolverFactory {
+
+ private final implicit val log = getLog(this.getClass)
+
+ // just temp for quick POC
+ val classMap: Map[String,String] =
+ Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
--- End diff --

no this class may not be present, there's no direct dependency on a concrete backend. This code is correct in its intent, but we will need a Backend interface (one for VCL, one for e.g., MKL etc.) that can probe & report availability of concrete solvers.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-12 04:36:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567548#comment-15567548 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82934828

--- Diff: viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/GPUMMul.scala ---
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mahout.viennacl.opencl
+
+import org.apache.mahout.logging._
+import org.apache.mahout.math
+import org.apache.mahout.math._
+import org.apache.mahout.math.flavor.{BackEnum, TraversingStructureEnum}
+import org.apache.mahout.math.function.Functions
+import org.apache.mahout.math.scalabindings.RLikeOps._
+import org.apache.mahout.math.scalabindings._
+import org.apache.mahout.viennacl.opencl.javacpp.Functions._
+import org.apache.mahout.viennacl.opencl.javacpp.LinalgFunctions._
+import org.apache.mahout.viennacl.opencl.javacpp.{CompressedMatrix, Context, DenseRowMatrix}
+
+import scala.collection.JavaConversions._
+
+object GPUMMul extends MMBinaryFunc {
+
+ private final implicit val log = getLog(GPUMMul.getClass)
+
+ override def apply(a: Matrix, b: Matrix, r: Option[Matrix]): Matrix = {
+
+ require(a.ncol == b.nrow, "Incompatible matrix sizes in matrix multiplication.")
+
+ val (af, bf) = (a.getFlavor, b.getFlavor)
+ val backs = (af.getBacking, bf.getBacking)
+ val sd = (af.getStructure, math.scalabindings.densityAnalysis(a), bf.getStructure, densityAnalysis(b))
+
+
+ try {
+
+ val alg: MMulAlg = backs match {
+
+ // Both operands are jvm memory backs.
+ case (BackEnum.JVMMEM, BackEnum.JVMMEM) ⇒
--- End diff --

Same comment here as for the OMP solver.

Also: I am still not convinced we need two actual different classes to support OMP and GPU. I thought we could use one concrete class that shares most of execution path for both cases, and just switch it to use OMP only or GPU.

We then potentially could report/instantiate two instances of essentially the same class as solvers for OMP and GPU.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-12 04:36:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567551#comment-15567551 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82932910

--- Diff: math-scala/src/main/scala/org/apache/mahout/math/scalabindings/RLikeMatrixOps.scala ---
@@ -17,14 +17,22 @@
package org.apache.mahout.math.scalabindings

import org.apache.mahout.math.function.Functions
-import org.apache.mahout.math.{Vector, Matrix}
+import org.apache.mahout.math.{Matrix, Vector}
import scala.collection.JavaConversions._
import RLikeOps._
+import org.apache.mahout.math.scalabindings._
+

class RLikeMatrixOps(m: Matrix) extends MatrixOps(m) {

/** Structure-optimized mmul */
- def %*%(that: Matrix) = MMul(m, that, None)
+
+ implicit var solverOperator: opMMulSolver =_
--- End diff --

Somehow i don't like implicit vars inside the decorator pattern. Clean decorator pattern (which is what it really is) assumes there's no new intrinsic state. so the only value instrinsic is `m`.

Instead, it should query solvers from a table as instantiated and saved in the current solver factory.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-12 04:36:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567546#comment-15567546 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82932953

--- Diff: math-scala/src/main/scala/org/apache/mahout/math/scalabindings/RLikeMatrixOps.scala ---
@@ -17,14 +17,22 @@
package org.apache.mahout.math.scalabindings

import org.apache.mahout.math.function.Functions
-import org.apache.mahout.math.{Vector, Matrix}
+import org.apache.mahout.math.{Matrix, Vector}
import scala.collection.JavaConversions._
import RLikeOps._
+import org.apache.mahout.math.scalabindings._
+

class RLikeMatrixOps(m: Matrix) extends MatrixOps(m) {

/** Structure-optimized mmul */
- def %*%(that: Matrix) = MMul(m, that, None)
+
+ implicit var solverOperator: opMMulSolver =_
--- End diff --

PS one can look at Breeze for one way of doing this.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-12 04:36:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567552#comment-15567552 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82933987

--- Diff: spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala ---
@@ -162,7 +163,7 @@ class MahoutSparkILoop extends SparkILoop {
_ __ ___ __ _| |__ ___ _ _| |_
| '_ ` _ \ / _` | '_ \ / _ \| | | | __|
| | | | | | (_| | | | | (_) | |_| | |_
- |_| |_| |_|\__,_|_| |_|\___/ \__,_|\__| version 0.12.2
+ |_| |_| |_|\__,_|_| |_|\___/ \__,_|\__| version 0.13.0

--- End diff --

if it is not an expensive maintenance, i would vote to keep it. I actually occasionally use it as a scratchpad or a quick demo env

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-12 04:36:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567549#comment-15567549 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82933830

--- Diff: math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala ---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.mahout.math.scalabindings
+
+import java.io.File
+
+import org.apache.mahout.logging._
+
+import scala.reflect.ClassTag
+import scala.reflect.runtime._
+import scala.reflect._
+
+
+class SolverFactory {
+
+ private final implicit val log = getLog(this.getClass)
+
+ // just temp for quick POC
+ val classMap: Map[String,String] =
+ Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
+ ("OMPMMul"->"org.apache.mahout.viennacl.openmp.OMPMMul"))
+}
+object SolverFactory extends SolverFactory {
+
+ // default is JVM
+ var clazz: MMBinaryFunc = MMul
+
+ // eventually match on implicit Classtag . for now. just take as is.
+ // this is a bit hacky, Shoud not be doing onlytry/catch here..
+ def getOperator[C: ClassTag]: MMBinaryFunc = {
+
+ try {
+ println("creating org.apache.mahout.viennacl.opencl.GPUMMul solver")
--- End diff --

I understand it is WIP. I imagined we should develop a concept of backends, rather than binding to a concrete solver like this.

So backend = collection of available solvers for specific library. So for VCL, there's just one Backend (if we ignore the fact that multiple platforms may be present in either one or many VCL backend jars).

Backends are enumerated and managed by SolverFactory. So SolverFactory has 3 functions :
(1) probe and manage backends
(2) probe individual solver support in each of backeds and decide which one to instantiate and return
(3) manage available solvers.

I imagine we won't do this initially, but we should leave enough flexibility in this class OOA so that we could insert rules allowing choosing among more than one active solver of the same type depending on additional factors.

Such things may include situations when e.g., multiple backends are available which do not share resources (i.e., gpu and host backends); or simple calibration estimates when for smaller problem sizes host backends are more efficient than device backends because they do not require problem repackaging and transfer between host and the device.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-12 04:36:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567554#comment-15567554 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82934105

--- Diff: spark/src/main/scala/org/apache/mahout/sparkbindings/package.scala ---
@@ -214,11 +223,11 @@ package object sparkbindings {
// During maven tests, the maven classpath also creeps in for some reason
!n.matches(".*/.m2/.*")
)
- /* verify jar passed to context
- log.info("\n\n\n")
- mcjars.foreach(j => log.info(j))
- log.info("\n\n\n")
- */
+ /* verify jar passed to context */
+// info("\n\n\n")
--- End diff --

this IMO is a useful info (perhaps should be downgraded to debug and be made a little bit more verbose explaining what it is it prints)

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-12 04:36:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567553#comment-15567553 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82933399

--- Diff: math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala ---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.mahout.math.scalabindings
+
+import java.io.File
+
+import org.apache.mahout.logging._
+
+import scala.reflect.ClassTag
+import scala.reflect.runtime._
+import scala.reflect._
+
+
+class SolverFactory {
+
+ private final implicit val log = getLog(this.getClass)
+
+ // just temp for quick POC
+ val classMap: Map[String,String] =
+ Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
--- End diff --

right, ok, but eventually it is probed and initialized on singleton init via (hopefully) a Backend's interface interrogations.
Typically, there are known (predefined) Backend set of classes + additional custom backends via a property.

The backend concrete implementations do not generally have to be a dependency of this module (e.g., because there are no native javacpp codes for this platform), so generally there's no concrete dependency and hense class name cannot be used directly, only via reflection.

On the other hand, in order to enable a custom (well-known) backend, it should be sufficient to drop in a version of such compiled for current platform for things to start moving around.

Consider, for example, a blas Backend. then solvers there generally would not be available if (1) blas backend jar is not in the classpath; or (2) blas backend probed -- and reported back as unavailable (usually due to failure to bind to libBlas.so in the system).

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-12 04:36:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567550#comment-15567550 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82934625

--- Diff: viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/OMPMMul.scala ---
@@ -0,0 +1,448 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mahout.viennacl.openmp
+
+import org.apache.mahout.logging._
+import org.apache.mahout.math
+import org.apache.mahout.math._
+import org.apache.mahout.math.flavor.{BackEnum, TraversingStructureEnum}
+import org.apache.mahout.math.function.Functions
+import org.apache.mahout.math.scalabindings.RLikeOps._
+import org.apache.mahout.math.scalabindings._
+import org.apache.mahout.viennacl.openmp.javacpp.Functions._
+import org.apache.mahout.viennacl.openmp.javacpp.LinalgFunctions._
+import org.apache.mahout.viennacl.openmp.javacpp.{CompressedMatrix, Context, DenseRowMatrix}
+
+import scala.collection.JavaConversions._
+
+object OMPMMul extends MMBinaryFunc {
--- End diff --

ok so likely it is WIP again, but we wanted a cache here. So i think on one hand it needs a specific proxy matrix type; on the other, it doesn't need all these mappings based on argument orientation.

On the latter point in particualr, since we are going to a native solver, we are rewriting (copyign) data anyway, so we already have ok'd the fact that rewrite is less expensive than mmul itself.

Many solvers will accept whatever orientation of arguments (can't immediately remember if it is the case with VCL OMP), and even if they do not, we just simply can copy it to whatever orientation is needed.

so i think all this matching eventually would be irrelevant really.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-16 20:59:20 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580526#comment-15580526 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261

I apologize - in the header and in the docs, I had VienniaCL 1.7.1+ this branch has been built for 1.7.0.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-17 01:13:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580863#comment-15580863 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261

Correcting myself again, This branch was built for *ViennaCL 1.7.1*.

There is an error in the ViennaCL Library that shows the wrong version:

https://github.com/viennacl/viennacl-dev/blob/master/viennacl/version.hpp

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-17 04:54:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581210#comment-15581210 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user rawkintrevo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r83577666

--- Diff: math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala ---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.mahout.math.scalabindings
+
+import java.io.File
+
+import org.apache.mahout.logging._
+
+import scala.reflect.ClassTag
+import scala.reflect.runtime._
+import scala.reflect._
+
+
+class SolverFactory {
+
+ private final implicit val log = getLog(this.getClass)
+
+ // just temp for quick POC
+ val classMap: Map[String,String] =
+ Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
+ ("OMPMMul"->"org.apache.mahout.viennacl.openmp.OMPMMul"))
+}
+object SolverFactory extends SolverFactory {
+
+ // default is JVM
+ var clazz: MMBinaryFunc = MMul
+
+ // eventually match on implicit Classtag . for now. just take as is.
+ // this is a bit hacky, Shoud not be doing onlytry/catch here..
+ def getOperator[C: ClassTag]: MMBinaryFunc = {
+
+ try {
+ println("creating org.apache.mahout.viennacl.opencl.GPUMMul solver")
--- End diff --

given the cyclical reference issues, maybe a solution here should come sooner rather than later (e.g. solving this, would also 'fix' the cyclical reference problem?)

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Suneel Marthi (JIRA)

2016-10-17 12:17:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582075#comment-15582075 ]

Suneel Marthi commented on MAHOUT-1885:
---------------------------------------

This has been fixed as of today in ViennaCL. Please update ur ViennaCL binaries.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-17 15:51:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582587#comment-15582587 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user dlyubimov commented on the issue:

https://github.com/apache/mahout/pull/261

IMO this needs more work and committing this is premature at this point due to architecture considerations. Not only it misses some things, but also IMO has things that should not be done the way they currently are.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-17 16:56:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582746#comment-15582746 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261

I'd really opened this PR so that we could discuss architecture and the correct entry points for the SolverFactory, hence the ridiculously verbose and confusing debugging output. I'd like to get the arch right before we commit. At its core it works as is, but there are alot of changes that need to be made.

Eg. We may be able to get away with only one Viennacl module by setting an environment variable and checking for it in Context.scala if it exists we "#define VIENNACL_WITH_OPENCL" which will make the module OpenMP only. The Problem With This Is That We Can't Test Gpu in this module.. without checking for the environment var.

The other big thing is that I don't know if the placement if the SolverFactory as it is is correct.. we may want to examine this, as well as cache a XXXMMUL Of both types the first time this class is hit so that we are not dynamically loading a class each time %*% is hit.

These are a couple of things that I'd like to get figured out before committing.

We've already identified a few pain points in the testing that everybody did over the weekend. So let's get it right before committing.

Sent from my Verizon Wireless 4G LTE smartphone

-------- Original message --------
From: Dmitriy Lyubimov <***@github.com>
Date: 10/17/2016 8:51 AM (GMT-08:00)
To: apache/mahout <***@noreply.github.com>
Cc: Andrew Palumbo <***@outlook.com>, Author <***@noreply.github.com>
Subject: Re: [apache/mahout] MAHOUT-1885 [WIP][FOR COMMENT]: Inital implementation of VCL bindings for mahout math. (#261)

IMO this needs more work and committing this is premature at this point due to architecture considerations. Not only it misses some things, but also IMO has things that should not be done the way they currently are.

-
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/apache/mahout/pull/261#issuecomment-254248310>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHU2HUmvxvJwfx7bWgzjiKJuNdmtei49ks5q05mMgaJpZM4KSKUm>.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-17 16:56:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582747#comment-15582747 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261

+1

Sent from my Verizon Wireless 4G LTE smartphone

-------- Original message --------
From: Trevor Grant <***@github.com>
Date: 10/16/2016 9:53 PM (GMT-08:00)
To: apache/mahout <***@noreply.github.com>
Cc: Andrew Palumbo <***@outlook.com>, Author <***@noreply.github.com>
Subject: Re: [apache/mahout] MAHOUT-1885 [WIP][FOR COMMENT]: Inital implementation of VCL bindings for mahout math. (#261)

@rawkintrevo commented on this pull request.

________________________________

Post by ASF GitHub Bot (JIRA)
@@ -162,7 +163,7 @@ class MahoutSparkILoop extends SparkILoop {

_ __ ___ __ _| |__ ___ _ _| |_
| '_ ` _ \ / _` | '_ \ / _ \| | | | __|
| | | | | | (_| | | | | (_) | |_| | |_
- |_| |_| |_|\__,_|_| |_|\___/ \__,_|\__| version 0.12.2
+ |_| |_| |_|\__,_|_| |_|\___/ \__,_|\__| version 0.13.0

I agree to keep. The Zeppelin integration has a few of its own little issues, esp when it comes to serialization, and some other stuff. The mahout-shell will always be way more 'bullet proof'. I also like to sanity check myself, to figure out is an issues 1) Zeppelin's Fault, 2) Mahout's Fault 3) My Fault, by running code in the mahout-spark shell (after a zeppelin failure).

-
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/apache/mahout/pull/261>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHU2HfXtW5iB3MHcYnchiUK-k3JTxFGFks5q0v76gaJpZM4KSKUm>.

Post by ASF GitHub Bot (JIRA)
Inital Implementation of VCL Bindings
-------------------------------------
Key: MAHOUT-1885
URL: https://issues.apache.org/jira/browse/MAHOUT-1885
Project: Mahout
Issue Type: Improvement
Components: Math
Affects Versions: 0.12.2
Reporter: Andrew Palumbo
Assignee: Andrew Palumbo
Fix For: 0.13.0
Push a working experimental branch of VCL bindings into master. There is still a lot of work to be done. All tests are passing, At the moment there am opening this JIRA mostly to get a number for PR and to test profiles against on travis.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-10-17 17:00:00 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582758#comment-15582758 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261

I don't think that cyclical referencing is an issue. There are other ways to get the class onto the path. They just got dropped somehow when I was creating/reason the pr. For now though adding the dependency at test time works for me just to make sure everything's working.

Sent from my Verizon Wireless 4G LTE smartphone

-------- Original message --------
From: Trevor Grant <***@github.com>
Date: 10/16/2016 9:54 PM (GMT-08:00)
To: apache/mahout <***@noreply.github.com>
Cc: Andrew Palumbo <***@outlook.com>, Author <***@noreply.github.com>
Subject: Re: [apache/mahout] MAHOUT-1885 [WIP][FOR COMMENT]: Inital implementation of VCL bindings for mahout math. (#261)

@rawkintrevo commented on this pull request.

________________________________

Post by ASF GitHub Bot (JIRA)
+ // just temp for quick POC

+ val classMap: Map[String,String] =
+ Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
+ ("OMPMMul"->"org.apache.mahout.viennacl.openmp.OMPMMul"))
+}
+object SolverFactory extends SolverFactory {
+
+ // default is JVM
+ var clazz: MMBinaryFunc = MMul
+
+ // eventually match on implicit Classtag . for now. just take as is.
+ // this is a bit hacky, Shoud not be doing onlytry/catch here..
+ def getOperator[C: ClassTag]: MMBinaryFunc = {
+
+ try {
+ println("creating org.apache.mahout.viennacl.opencl.GPUMMul solver")

given the cyclical reference issues, maybe a solution here should come sooner rather than later (e.g. solving this, would also 'fix' the cyclical reference problem?)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/apache/mahout/pull/261>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHU2HXNDs1M9yqQjTnKl6onfaOeAjtKQks5q0v-LgaJpZM4KSKUm>.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2016-11-21 04:38:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15682482#comment-15682482 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261

ok fixed up the poms and added profiles for viennacl/viennacl-omp in build pom.xml. There is still something wrong with the viennacl-omp module, I'm not sure what but it is not recognized by the other poms, and the jar is not gettting picked up when its being run for tests.

Should be an easy fix, and I'll come back to it soon.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Andrew Palumbo (JIRA)

2016-12-20 18:52:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Palumbo updated MAHOUT-1885:
-----------------------------------
Sprint: Jan/Feb-2017

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Andrew Palumbo (JIRA)

2017-01-11 02:46:58 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on MAHOUT-1885 started by Andrew Palumbo.
----------------------------------------------

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Andrew Palumbo (JIRA)

2017-01-16 02:28:28 UTC

Permalink

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Andrew Palumbo (JIRA)

2017-01-16 02:28:29 UTC

Permalink

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2017-01-16 21:33:26 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824609#comment-15824609 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261

I can now verify that from a fresh install of Ubuntu 16.04(.1)LTS, with the Ubuntu-Provided NVIDIA-367 Drivers that the GPU section of this PR works without Issue, following the above instructions:
```
- sparse mmul microbenchmark
+ Mahout Sparse multiplication time: 12314 ms.
+ ViennaCL/OpenCL Sparse multiplication time: 10959 ms.
+ ViennaCL/cpu/OpenMP Sparse multiplication time: 3690 ms.
- VCL Dense Matrix %*% Dense vector
+ Mahout dense matrix %*% dense vector multiplication time: 0 ms.
+ ViennaCL/cpu/OpenMP dense matrix %*% dense vector multiplication time: 4 ms.
```

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2017-01-17 01:30:26 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824821#comment-15824821 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261

Interesting.. After fixing a few small bugs in the `viennacl-omp` module (00fbe3e), all tests pass with the exception of `sparse mmul microbenchmark`, which fails with a
``` java.lang.OutOfMemoryError: Java heap space
at it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash
```

if I chage the dimensions of the matrices to :
```
val m = 300
val n = 300
val s = 10
```
(one order of magnitude lower on `m` and `n` two lower on `s`, all tests will pass:
```
Discovery completed in 296 milliseconds.
Run starting. Expected test count is: 6
ViennaCLSuiteOMP:
- row-major viennacl::matrix
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[WARN] Unable to create class GPUMMul: attempting OpenMP version
[INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.openmp.OMPMMul solver
jvmRWRW
ompRWCW
log4j:WARN No appenders could be found for logger (org.apache.mahout.viennacl.openmp.OMPMMul$).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
- mmul microbenchmark
+ Mahout multiplication time: 11444 ms.
+ ViennaCL/cpu/OpenMP multiplication time: 2820 ms.
- trans
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[WARN] Unable to create class GPUMMul: attempting OpenMP version
[INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.openmp.OMPMMul solver
ompSparseRWRW
- sparse mmul microbenchmark
+ Mahout Sparse multiplication time: 65 ms.
+ ViennaCL/cpu/OpenMP Sparse multiplication time: 5 ms.
- VCL Dense Matrix %*% Dense vector - no OpenCl
+ Mahout dense matrix %*% dense vector multiplication time: 30 ms.
+ ViennaCL/cpu/OpenMP dense matrix %*% dense vector multiplication time: 16 ms.
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[WARN] Unable to create class GPUMMul: attempting OpenMP version
[INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.openmp.OMPMMul solver
ompSparseRowRWRW
- Sparse %*% Dense mmul microbenchmark
+ Mahout multiplication time: 2583 ms.
+ ViennaCL/cpu/OpenMP multiplication time: 2639 ms.
Run completed in 21 seconds, 602 milliseconds.
Total number of tests run: 6
Suites: completed 2, aborted 0
Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 40.433 s
[INFO] Finished at: 2017-01-16T17:27:49-08:00
[INFO] Final Memory: 21M/327M
[INFO] ------------------------------------------------------------------------
```

What's confusing to me is why they would pass on the GPU module, and not on the OpenMP module, which use almost Identical methods to convert back and forth to interface with ViennaCL operations. As well, this same test is run for comparison in the GPU module. What an annoying little bug. Must be a maven test setting.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2017-01-17 02:39:26 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824862#comment-15824862 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261

All tests now pass in GPU and OpenMP mode.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2017-01-17 02:49:26 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824865#comment-15824865 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r96336102

--- Diff: viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/GPUMMul.scala ---
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mahout.viennacl.opencl
+
+import org.apache.mahout.logging._
+import org.apache.mahout.math
+import org.apache.mahout.math._
+import org.apache.mahout.math.flavor.{BackEnum, TraversingStructureEnum}
+import org.apache.mahout.math.function.Functions
+import org.apache.mahout.math.scalabindings.RLikeOps._
+import org.apache.mahout.math.scalabindings._
+import org.apache.mahout.viennacl.opencl.javacpp.Functions._
+import org.apache.mahout.viennacl.opencl.javacpp.LinalgFunctions._
+import org.apache.mahout.viennacl.opencl.javacpp.{CompressedMatrix, Context, DenseRowMatrix}
+
+import scala.collection.JavaConversions._
+
+object GPUMMul extends MMBinaryFunc {
+
+ private final implicit val log = getLog(GPUMMul.getClass)
+
+ override def apply(a: Matrix, b: Matrix, r: Option[Matrix]): Matrix = {
+
+ require(a.ncol == b.nrow, "Incompatible matrix sizes in matrix multiplication.")
+
+ val (af, bf) = (a.getFlavor, b.getFlavor)
+ val backs = (af.getBacking, bf.getBacking)
+ val sd = (af.getStructure, math.scalabindings.densityAnalysis(a), bf.getStructure, densityAnalysis(b))
+
+
+ try {
+
+ val alg: MMulAlg = backs match {
+
+ // Both operands are jvm memory backs.
+ case (BackEnum.JVMMEM, BackEnum.JVMMEM) ⇒
--- End diff --

We may be able to get away with only one Viennacl module by setting an environment variable and checking for it in Context.scala if it exists we "#define VIENNACL_WITH_OPENCL" which will make the module OpenMP only. The Problem With This Is That We Can't Test GPU in this module.. without checking for the environment var.

I guess the above is a possibility, but not especially simple. For no, though all tests are passing.. though i am seeing that some of the eg. spark tests are not running `GPUMmul` when Built for `-Pviennacl`. So I may need to fix some imports somewhere.

Either way I 'd like to get the base Architecture nailed down with this release.

Inital Implementation of VCL Bindings
-------------------------------------
Key: MAHOUT-1885
URL: https://issues.apache.org/jira/browse/MAHOUT-1885
Project: Mahout
Issue Type: Improvement
Components: Math
Affects Versions: 0.12.2
Reporter: Andrew Palumbo
Assignee: Andrew Palumbo
Fix For: 0.13.0
Push a working experimental branch of VCL bindings into master. There is still a lot of work to be done. All tests are passing, At the moment there am opening this JIRA mostly to get a number for PR and to test profiles against on travis.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2017-01-17 02:49:26 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824866#comment-15824866 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r96336137

--- Diff: spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala ---
@@ -162,7 +163,7 @@ class MahoutSparkILoop extends SparkILoop {
_ __ ___ __ _| |__ ___ _ _| |_
| '_ ` _ \ / _` | '_ \ / _ \| | | | __|
| | | | | | (_| | | | | (_) | |_| | |_
- |_| |_| |_|\__,_|_| |_|\___/ \__,_|\__| version 0.12.2
+ |_| |_| |_|\__,_|_| |_|\___/ \__,_|\__| version 0.13.0

--- End diff --

+1

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2017-01-17 02:52:26 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824869#comment-15824869 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r96336350

--- Diff: viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/OMPMMul.scala ---
@@ -0,0 +1,448 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mahout.viennacl.openmp
+
+import org.apache.mahout.logging._
+import org.apache.mahout.math
+import org.apache.mahout.math._
+import org.apache.mahout.math.flavor.{BackEnum, TraversingStructureEnum}
+import org.apache.mahout.math.function.Functions
+import org.apache.mahout.math.scalabindings.RLikeOps._
+import org.apache.mahout.math.scalabindings._
+import org.apache.mahout.viennacl.openmp.javacpp.Functions._
+import org.apache.mahout.viennacl.openmp.javacpp.LinalgFunctions._
+import org.apache.mahout.viennacl.openmp.javacpp.{CompressedMatrix, Context, DenseRowMatrix}
+
+import scala.collection.JavaConversions._
+
+object OMPMMul extends MMBinaryFunc {
+
+ private final implicit val log = getLog(OMPMMul.getClass)
+
+ override def apply(a: Matrix, b: Matrix, r: Option[Matrix]): Matrix = {
+
+ require(a.ncol == b.nrow, "Incompatible matrix sizes in matrix multiplication.")
+
+ val (af, bf) = (a.getFlavor, b.getFlavor)
+ val backs = (af.getBacking, bf.getBacking)
+ val sd = (af.getStructure, math.scalabindings.densityAnalysis(a), bf.getStructure, densityAnalysis(b))
+
+
+ try {
+
+ val alg: MMulAlg = backs match {
+
+ // Both operands are jvm memory backs.
+ case (BackEnum.JVMMEM, BackEnum.JVMMEM) ⇒
+
+ sd match {
+
+ // Multiplication cases by a diagonal matrix.
+ case (TraversingStructureEnum.VECTORBACKED, _, TraversingStructureEnum.COLWISE, _)
+ if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagCW
+ case (TraversingStructureEnum.VECTORBACKED, _, TraversingStructureEnum.SPARSECOLWISE, _)
+ if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagCW
+ case (TraversingStructureEnum.VECTORBACKED, _, TraversingStructureEnum.ROWWISE, _)
+ if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagRW
+ case (TraversingStructureEnum.VECTORBACKED, _, TraversingStructureEnum.SPARSEROWWISE, _)
+ if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagRW
+
+ case (TraversingStructureEnum.COLWISE, _, TraversingStructureEnum.VECTORBACKED, _)
+ if b.isInstanceOf[DiagonalMatrix] ⇒ jvmCWDiag
+ case (TraversingStructureEnum.SPARSECOLWISE, _, TraversingStructureEnum.VECTORBACKED, _)
+ if b.isInstanceOf[DiagonalMatrix] ⇒ jvmCWDiag
+ case (TraversingStructureEnum.ROWWISE, _, TraversingStructureEnum.VECTORBACKED, _)
+ if b.isInstanceOf[DiagonalMatrix] ⇒ jvmRWDiag
+ case (TraversingStructureEnum.SPARSEROWWISE, _, TraversingStructureEnum.VECTORBACKED, _)
+ if b.isInstanceOf[DiagonalMatrix] ⇒ jvmRWDiag
+
+ // Dense-dense cases
+ case (TraversingStructureEnum.ROWWISE, true, TraversingStructureEnum.COLWISE, true) if a eq b.t ⇒ ompDRWAAt
+ case (TraversingStructureEnum.ROWWISE, true, TraversingStructureEnum.COLWISE, true) if a.t eq b ⇒ ompDRWAAt
+ case (TraversingStructureEnum.ROWWISE, true, TraversingStructureEnum.COLWISE, true) ⇒ ompRWCW
+ case (TraversingStructureEnum.ROWWISE, true, TraversingStructureEnum.ROWWISE, true) ⇒ jvmRWRW
+ case (TraversingStructureEnum.COLWISE, true, TraversingStructureEnum.COLWISE, true) ⇒ jvmCWCW
+ case (TraversingStructureEnum.COLWISE, true, TraversingStructureEnum.ROWWISE, true) if a eq b.t ⇒ jvmDCWAAt
+ case (TraversingStructureEnum.COLWISE, true, TraversingStructureEnum.ROWWISE, true) if a.t eq b ⇒ jvmDCWAAt
+ case (TraversingStructureEnum.COLWISE, true, TraversingStructureEnum.ROWWISE, true) ⇒ jvmCWRW
+
+ // Sparse row matrix x sparse row matrix (array of vectors)
+ case (TraversingStructureEnum.ROWWISE, false, TraversingStructureEnum.ROWWISE, false) ⇒ ompSparseRWRW
+ case (TraversingStructureEnum.ROWWISE, false, TraversingStructureEnum.COLWISE, false) ⇒ jvmSparseRWCW
+ case (TraversingStructureEnum.COLWISE, false, TraversingStructureEnum.ROWWISE, false) ⇒ jvmSparseCWRW
+ case (TraversingStructureEnum.COLWISE, false, TraversingStructureEnum.COLWISE, false) ⇒ jvmSparseCWCW
+
+ // Sparse matrix x sparse matrix (hashtable of vectors)
+ case (TraversingStructureEnum.SPARSEROWWISE, false, TraversingStructureEnum.SPARSEROWWISE, false) ⇒
+ ompSparseRowRWRW
+ case (TraversingStructureEnum.SPARSEROWWISE, false, TraversingStructureEnum.SPARSECOLWISE, false) ⇒
+ jvmSparseRowRWCW
+ case (TraversingStructureEnum.SPARSECOLWISE, false, TraversingStructureEnum.SPARSEROWWISE, false) ⇒
+ jvmSparseRowCWRW
+ case (TraversingStructureEnum.SPARSECOLWISE, false, TraversingStructureEnum.SPARSECOLWISE, false) ⇒
+ jvmSparseRowCWCW
+
+ // Sparse matrix x non-like
+ case (TraversingStructureEnum.SPARSEROWWISE, false, TraversingStructureEnum.ROWWISE, _) ⇒ ompSparseRowRWRW
+ case (TraversingStructureEnum.SPARSEROWWISE, false, TraversingStructureEnum.COLWISE, _) ⇒ jvmSparseRowRWCW
+ case (TraversingStructureEnum.SPARSECOLWISE, false, TraversingStructureEnum.ROWWISE, _) ⇒ jvmSparseRowCWRW
+ case (TraversingStructureEnum.SPARSECOLWISE, false, TraversingStructureEnum.COLWISE, _) ⇒ jvmSparseCWCW
+ case (TraversingStructureEnum.ROWWISE, _, TraversingStructureEnum.SPARSEROWWISE, false) ⇒ ompSparseRWRW
+ case (TraversingStructureEnum.ROWWISE, _, TraversingStructureEnum.SPARSECOLWISE, false) ⇒ jvmSparseRWCW
+ case (TraversingStructureEnum.COLWISE, _, TraversingStructureEnum.SPARSEROWWISE, false) ⇒ jvmSparseCWRW
+ case (TraversingStructureEnum.COLWISE, _, TraversingStructureEnum.SPARSECOLWISE, false) ⇒ jvmSparseRowCWCW
+
+ // Everything else including at least one sparse LHS or RHS argument
+ case (TraversingStructureEnum.ROWWISE, false, TraversingStructureEnum.ROWWISE, _) ⇒ ompSparseRWRW
+ case (TraversingStructureEnum.ROWWISE, false, TraversingStructureEnum.COLWISE, _) ⇒ jvmSparseRWCW
+ case (TraversingStructureEnum.COLWISE, false, TraversingStructureEnum.ROWWISE, _) ⇒ jvmSparseCWRW
+ case (TraversingStructureEnum.COLWISE, false, TraversingStructureEnum.COLWISE, _) ⇒ jvmSparseCWCW2flips
+
+ // Sparse methods are only effective if the first argument is sparse, so we need to do a swap.
+ case (_, _, _, false) ⇒ (a, b, r) ⇒ apply(b.t, a.t, r.map {
+ _.t
+ }).t
+
+ // Default jvm-jvm case.
+ // for some reason a SrarseRowMatrix DRM %*% SrarseRowMatrix DRM was dumping off to here
+ case _ ⇒ ompRWCW
+ }
+ }
+
+ alg(a, b, r)
+ } catch {
+ // TODO FASTHACK: just revert to JVM if there is an exception..
+ // eg. java.lang.nullPointerException if more openCL contexts
+ // have been created than number of GPU cards.
+ // better option wuold be to fall back to OpenCl First.
+ case ex: Exception =>
+ println(ex.getMessage + "falling back to JVM MMUL")
+ return MMul(a, b, r)
+ }
+ }
+
+ type MMulAlg = MMBinaryFunc
+
+ @inline
+ private def ompRWCW(a: Matrix, b: Matrix, r: Option[Matrix] = None): Matrix = {
+ println("ompRWCW")
+ //
+ // require(r.forall(mxR ⇒ mxR.nrow == a.nrow && mxR.ncol == b.ncol))
+ // val (m, n) = (a.nrow, b.ncol)
+ //
+ // val mxR = r.getOrElse(if (densityAnalysis(a)) a.like(m, n) else b.like(m, n))
+ //
+ // for (row ← 0 until mxR.nrow; col ← 0 until mxR.ncol) {
+ // // this vector-vector should be sort of optimized, right?
+ // mxR(row, col) = a(row, ::) dot b(::, col)
+ // }
+ // mxR
+
+ val hasElementsA = a.zSum() > 0.0
+ val hasElementsB = b.zSum() > 0.0
+
+ // A has a sparse matrix structure of unknown size. We do not want to
+ // simply convert it to a Dense Matrix which may result in an OOM error.
+
+ // If it is empty use JVM MMul, since we can not convert it to a VCL CSR Matrix.
+ if (!hasElementsA) {
+ println("Matrix a has zero elements can not convert to CSR")
+ return MMul(a, b, r)
+ }
+
+ // CSR matrices are efficient up to 50% non-zero
+ if(b.getFlavor.isDense) {
--- End diff --

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2017-01-17 02:53:26 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824871#comment-15824871 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r96336410

--- Diff: LICENSE.txt ---
@@ -799,3 +799,32 @@ The following license applies to the H2O package
identification within third-party archives.

Copyright 2012 0xdata, Inc
+
+================================================================
+The following applies to the ViennaCL library and files in the mahout-native-viennacl module
+================================================================
+
+ Copyright (c) 2010-2016 Institute for Microelectronics,
+ Institute for Analysis and Scientific Computing, TU Wien.
+ Portions of this software are copyright by UChicago Argonne, LLC.
+ Argonne National Laboratory, with facilities in the state of Illinois,
+ is owned by The United States Government, and operated by UChicago Argonne, LLC
+ under provision of a contract with the Department of Energy.
+
+ Permission is hereby granted, free of charge, to any person obtaining a copy
+ of this software and associated documentation files (the "Software"), to deal
+ in the Software without restriction, including without limitation the rights
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ copies of the Software, and to permit persons to whom the Software is
+ furnished to do so, subject to the following conditions:
+
+ The above copyright notice and this permission notice shall be included in
+ all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ THE SOFTWARE.
--- End diff --

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

ASF GitHub Bot (JIRA)

2017-01-26 04:25:26 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839158#comment-15839158 ]

ASF GitHub Bot commented on MAHOUT-1885:
----------------------------------------

Github user asfgit closed the pull request at:

https://github.com/apache/mahout/pull/261

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Andrew Palumbo (JIRA)

2017-01-26 04:27:26 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Palumbo resolved MAHOUT-1885.
------------------------------------
Resolution: Done

Several related issues to clean up. A small bug introduced as well small but still a blocker.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Hudson (JIRA)

2017-01-26 04:57:26 UTC

Permalink

[ https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839198#comment-15839198 ]

Hudson commented on MAHOUT-1885:
--------------------------------

FAILURE: Integrated in Jenkins build Mahout-Quality #3410 (See [https://builds.apache.org/job/Mahout-Quality/3410/])
MAHOUT-1885: Inital commit of VCL bindings. closes apache/mahout#269 (apalumbo: rev 034790cce40fcee2b7a875c482345d35f7c0fa8d)
* (add) viennacl/pom.xml
* (edit) distribution/pom.xml
* (edit) spark/src/test/scala/org/apache/mahout/drivers/RowSimilarityDriverSuite.scala
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/VCLVector.scala
* (edit) spark/src/main/scala/org/apache/mahout/sparkbindings/package.scala
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/ProdExpression.scala
* (edit) spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/VectorBase.scala
* (edit) flink/pom.xml
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/VecMultExpression.scala
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/MatVecProdExpression.scala
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/SrMatDnMatProdExpression.scala
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/package.scala
* (edit) runtests.sh
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/Context.scala
* (add) math-scala/src/main/scala/org/apache/mahout/math/backend/jvm/JvmBackend.scala
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/GPUMMul.scala
* (add) viennacl-omp/runs
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/VectorBase.scala
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/CompressedMatrix.scala
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/ProdExpression.scala
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/SrMatDnMatProdExpression.scala
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/VCLVector.scala
* (edit) spark/src/test/scala/org/apache/mahout/sparkbindings/test/DistributedSparkSuite.scala
* (edit) spark/src/test/scala/org/apache/mahout/sparkbindings/SparkBindingsSuite.scala
* (add) viennacl-omp/src/main/java/org/apache/mahout/viennacl/openmp/javacpp/LinalgFunctions.java
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/MatMatProdExpression.scala
* (add) viennacl-omp/src/main/java/org/apache/mahout/viennacl/openmp/javacpp/MatrixTransExpression.scala
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/MatMatProdExpression.scala
* (edit) pom.xml
* (add) viennacl/src/main/java/org/apache/mahout/viennacl/opencl/javacpp/MatrixTransExpression.scala
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/OMPMMul.scala
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/MemHandle.scala
* (add) math-scala/src/main/scala/org/apache/mahout/math/backend/Backend.scala
* (edit) math-scala/src/test/scala/org/apache/mahout/math/scalabindings/RLikeMatrixOpsSuite.scala
* (add) math-scala/src/main/scala/org/apache/mahout/math/backend/RootSolverFactory.scala
* (edit) bin/mahout
* (edit) spark/pom.xml
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/VecMultExpression.scala
* (add) math-scala/src/main/scala/org/apache/mahout/math/backend/incore/package.scala
* (edit) spark-shell/pom.xml
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/DenseColumnMatrix.scala
* (add) viennacl-omp/linux-haswell.properties
* (add) viennacl/linux-x86_64-viennacl.properties
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/MatVecProdExpression.scala
* (edit) spark/src/main/assembly/dependency-reduced.xml
* (add) viennacl-omp/src/main/java/org/apache/mahout/viennacl/openmp/javacpp/Functions.java
* (add) viennacl/linux-haswell.properties
* (add) math-scala/src/test/scala/org/apache/mahout/math/backend/BackendSuite.scala
* (add) viennacl/src/test/scala/org/apache/mahout/viennacl/opencl/ViennaCLSuiteVCL.scala
* (edit) spark/src/test/scala/org/apache/mahout/drivers/TextDelimitedReaderWriterSuite.scala
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/Context.scala
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/MatrixBase.scala
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/DenseRowMatrix.scala
* (edit) .travis.yml
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/package.scala
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/MemHandle.scala
* (add) viennacl-omp/src/test/scala/org/apache/mahout/viennacl/omp/ViennaCLSuiteOMP.scala
* (edit) LICENSE.txt
* (add) math-scala/src/main/scala/org/apache/mahout/math/backend/SolverFactory.scala
* (add) examples/bin/SparseSparseDrmTimer.mscala
* (edit) spark/src/main/scala/org/apache/mahout/sparkbindings/blas/ABt.scala
* (edit) math-scala/src/main/scala/org/apache/mahout/math/scalabindings/package.scala
* (edit) math-scala/pom.xml
* (add) viennacl-omp/pom.xml
* (edit) spark/src/test/scala/org/apache/mahout/drivers/ItemSimilarityDriverSuite.scala
* (edit) math-scala/src/main/scala/org/apache/mahout/math/scalabindings/RLikeMatrixOps.scala
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/MatrixBase.scala
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/CompressedMatrix.scala
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/DenseRowMatrix.scala
* (add) viennacl/src/main/java/org/apache/mahout/viennacl/opencl/javacpp/LinalgFunctions.java
* (add) viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/DenseColumnMatrix.scala
* (edit) math-scala/src/main/scala/org/apache/mahout/math/scalabindings/MMul.scala
* (add) viennacl-omp/linux-x86_64-viennacl.properties
* (add) viennacl/src/main/java/org/apache/mahout/viennacl/opencl/javacpp/Functions.java

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)