On Wed, Nov 27, 2013 at 04:13:14PM +0000, Neil Williams wrote:
On Wed, 27 Nov 2013 13:01:36 -0300 Antonio Terceiro antonio.terceiro@linaro.org wrote:
For 3. I think it would make sense to have an API call that you could use from your data analysis node that would retrieve a given directory from the other nodes. Something like
lava-collect PATH DEST
Collects the contents of PATH in all other devices that are part of the multinode job and store them at DEST locally. For example, the call `lava-collect /var/lib/foobar /var/tmp` would result in
/var/tmp node01/ var/lib/foobar (stuff) node02/ var/lib/foobar (stuff) (...)
How does the receiving node authenticate with each other node to get read access?
I don't think we need to worry that much about security across on disposable test systems.
Data cannot go over the existing API connection, it has to be configured separately over something like TCP/IP and root on node01 does not necessarily have access to anything on node02 without node02 being explicitly configured in advance to either serve files anonymously or allow login.
good point. my idea was that such helper would be using lava-network under the hood, but I neglected the sending side. We probably also need a matching call on the sending side (`lava-serve PATH`)?
so the data analysis node does
lava-sync processing-done lava-collect RESULTS-PATH DEST # calculate ...
and the others
# run ... lava-sync processing-done lava-serve RESULTS-PATH
(sure we could find better names than lava-collect and lava-serve)
I'm not sure this is appropriate as a helper across all LAVA deployments and if restricted to the same deployments as lava-network, it would require particular services to be installed and running on all nodes which lava-network does not enforce.
with the matching call on the sending side, `tar | nc -` ought to be good enough on the sending side and `nc | tar` should do it for the receiving side.