[ https://issues.apache.org/jira/browse/MAHOUT-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546663#comment-15546663 ]
Dmitriy Lyubimov edited comment on MAHOUT-1884 at 10/4/16 9:10 PM:
-------------------------------------------------------------------
drmWrap is not internal in the least (which is why it is not package-private). it is public and intended for plugging external general sources into input barrier of the optimizer/
loading in memory would happen anyway. Caching is not necessarily -- but it is not guaranteed not to happen, there's no such contract.
Materially it only makes any difference if the input is larger than avaialble cluster capacity. Which is I am yet to encounter as algebraic tasks are CPU and io bound, but not memory. Usually we run out of IO and CPU much sooner that we run out of memory, which makes this situation pragmatically unrealistic.
note that optimizer should --and will -- retain control over caching. we don't have explicit caching api except for checkpoint "hints" but even that is only a hint, not guaranteed. Giving it some heuristics about dataset doesn't guarantee that it won't compute others or won't cache or sample for some other reason, now or in the future.
This siutation is fine as it is one of the function of optimizer, as much as choosing degrees of parallelization, product task sizes or operators to execute. Making those choices automatically is, actually, the point. As long as optimizer does right enough things, that should be ok.
Bottom line, i don't see harm in adding _optional_ ncol and nrow to drmDfsRead specifically. But I do not see a tangible benefit either. There's possibly only a slight benefit right now (no no-cache or no-sample guarantee), which likely only decrease in the future. I am fine with it as understood there's no "no-cache" contract anywhere.
was (Author: dlyubimov):
drmWrap is not internal in the least (which is why it is not package-private). it is public and intended for plugging external general sources into input barrier of the optimizer/
loading in memory would happen anyway. Caching is not necessarily -- but it is not guaranteed not to happen, there's no such contract.
Materially it only makes any difference if the input is larger than avaialble cluster capacity. Which is I am yet to encounter as algebraic tasks are CPU and io bound, but not memory. Usually we run out of IO and CPU much sooner that we run out of memory, which makes this situation pragmatically unrealistic.
note that optimizer should --and will -- retain control over caching. we don't have explicit caching api except for checkpoint "hints" but even that is only a hint, not guaranteed. Giving it some heuristics about dataset doesn't guarantee that it won't compute others or won't cache or sample for some other reason, now or in the future.
This siutation is fine as it is one of the function of optimizer, as much as choosing degrees of parallelization, product task sizes or operators to execute. Making those choices automatically is, actually, the point. As long as optimizer does right enough things, that should be ok.
Bottom line, i don't see harm in adding _optional_ ncol and nrow to drmDfsRead specifically. But I do not see a tangible benefit either. There's possibly only a slight benefit right now (no no-cache or no-sample guarantee), which likely only decrease in the future. I am fine with it as understood there's no "no-cache" contract anywhere.
Post by Sebastian Schelter (JIRA)Allow specification of dimensions of a DRM
------------------------------------------
Key: MAHOUT-1884
URL: https://issues.apache.org/jira/browse/MAHOUT-1884
Project: Mahout
Issue Type: Improvement
Affects Versions: 0.12.2
Reporter: Sebastian Schelter
Assignee: Sebastian Schelter
Priority: Minor
Currently, in many cases, a DRM must be read to compute its dimensions when a user calls nrow or ncol. This also implicitly caches the corresponding DRM.
In some cases, the user actually knows the matrix dimensions (e.g., when the matrices are synthetically generated, or when some metadata about them is known). In such cases, the user should be able to specify the dimensions upon creating the DRM and the caching should be avoided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)