On 2 August 2016 at 10:09, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Hi,
Chase did an excellent job and put together a piece of code that allows us local execution of lava-test-shell. This means we can use our 'lava' test definitions on the boards that are not deployed in any LAB. There are 2 main reasons for doing that:
- prototyping tests for LAVA
- semi-automated execution of tests on targets that are not deployed in any LAB
A few provisos on that. * This won't be able to support any Multinode API usage, it won't be able to support LXC usage in V2 either. * LAVA V2 has removed the ACK from lava_test_shell and has changes to the lava_test_shell snippets. * This re-implements out things like the deployment data which specify which shell to use and other items like directory layouts.
Major part of this code is taken directly from lava dispatcher. There are slight modifications but I would like to keep them to minimum or remove at all (if possible). So the question follows - is there a way to achieve the same goal with only LAVA code?
For the migration to V2, we kept compatibility with the lava test shell definitions - as far as runtime operation is concerned. Once the V1 code is removed, we do have plans for improving lava test shell in ways which will break compatibility, but changes cannot be made until then. In the meantime, the advice being written into the documentation is to prepare custom scripts in the test definition repository which can do all the parsing and other work in a self-contained way which can be tested locally against log files from the device. This is a more robust way of ensuring that operations happen as expected. lava-test-shell is primarily intended for very simple command operation. lava-test-case itself will always have problems with much more than a single command with arguments to be able to pass in parameters from the test job. When things become sufficiently complex that the test definition YAML actually needs to be debugged, it is better to move that logic into another script, written in a language with genuine support for conditionals, logic, arrays and dictionaries. The script/program can then call lava-test-case and other support scripts or output the results if not running in LAVA. Such scripts can improve the package installation work as well by only installing something if the parameters actually require it. The emphasis will shift to doing less in lava-test-shell in the same way that V2 has removed the assumptions and misguided helpers in the rest of the code, to allow the test shell to have easier support for alternative test platforms and utilities.
Lava test shell actually provides little benefit for targets not deployed in LAVA, apart from possibly familiarity for current LAVA test writers, over a common stand-alone script.
In LAVA V2, if you have a local instance, there is no need to replicate the code from lava_test_shell, the complete overlay is available as a tarball and that's exactly the same tarball - with any V2 changes and deployment_data changes - as was used in the test. We're still working out the best way to expose that overlay, currently it is only built on the worker. However, the advice remains the same that tests definitions which are sufficiently complex to need prototyping or debugging should go into custom scripts running against log files which can then be switched to read from stdin or similar.
One of the biggest problems was requirement for ACKing that lava-test-shell requires. This makes the tests 'interactive' which isn't best for semi-automated execution. This bit was removed and now we're able to use test shell locally in non-interactive mode.
This change is not compatible with LAVA V1. The ACK has been removed from V2 but it is not possible to do so reliably in V1.