On 28 April 2017 at 09:28, Neil Williams neil.williams@linaro.org wrote:
On 28 April 2017 at 08:34, Guillaume Tucker guillaume.tucker@collabora.com wrote:
Hi,
In LAVA v1, one could declare login commands to be run after logging in and before starting any of the tests. For example:
"actions": [ { "command": "deploy_image", "parameters": { "image": "https://some-url-to-a-rootfs.ext4.img.gz", "bootfstype": "ext2", "login_commands": [ "sudo su" ], "login_prompt": "login:", "username": "username", "password_prompt": "Password:", "password": "password" } },
In this case, "sudo su" is needed to open a regular user session and inherit the user's environment while also having root privileges.
That is test shell dependent, not device or test job. Some test shells will be quite happy to run without needing root and if such a test shell needs packages installed, that is up to the test writer to arrange, not LAVA. There is no actual requirement for LAVA to deliver a root shell on the DUT, it is convention.
V2 is making a direct effort to move away from reliance on LAVA test shell helpers. Once we drop V1, some of the existing helpers still used in V2 will be deprecated and then removed, including package installation helpers and VCS helpers.
The drive with V2 is to have portable test definition scripts which check that relevant support is installed and install it themselves (removing the duplication when the script changes dependencies but the test shell definition file does not). Such scripts also need to check that interfaces exist and create those if not, similarly for users or database configurations or apache config or build dependencies.
Once V2 delivers the test writer at a shell the job of V2 is only to execute the scripts to start the portable test scripts.
This change goes against that flow.
V2 does have explicit support for handling the environment within the test shell, where this is necessary.
In LAVA v2, there isn't the possibility to do anything like this directly. One could define a test with inline commands, but this is not ideal. The login commands are not a test but part of how the job sets up the environment in which the tests are run - i.e. it's part of the initial conditions.
sudo su is not a login command. username and password are the login commands. Anything after that is no longer handled by the login itself but by whatever shell the test job has configured.
The test writer also has the ability to modify the submitted image to ensure that whatever users are preferred are available in advance. Note: It is not an objective for LAVA V2 to test production images without alterations because many production images have security features which prevent automation. Some level of alteration is expected.
Also it's quite a convoluted and lengthy way of running some commands, and it relies on the side effects of that "login commands test" to persist when running subsequent tests.
Lots of test definitions include persistent actions (like udhcpc or ifup eth0). It is up to the test writer to construct test definitions which do setup and tear down where appropriate.
So I've made a couple of patches to see how this could be implemented in LAVA v2 with an extra parameter in auto_login:
I don't think this is the correct approach. It is replicating poor V1 behaviour for no good reason.
Unless there is a clearer use case than changing user or making persistent changes, I do not think this is suitable for V2.
Notes from the Gerrit reviews:
- The login commands can't be part of a device definition as they are not related to the device hardware or the boot configuration. For example,
when running Android, one would not want to run "sudo su" but maybe "setprop ..." or something else - to be defined in each job.
No, to be defined and managed in the portable test shell scripts.
- The login commands should not be fixed in a given distro / userspace configuration as each job may need to set up a different initial
environment. For example, some jobs may need to be run with a regular user and would not use the "sudo su" login command.
Again, this is a test shell element, not a test job element.
- Some documentation and unit tests would need to be added for this to be merged. This is to first discuss the idea and review the code changes.
Any thoughts? Would it make sense to add this feature or maybe implement it differently?
Take the opportunity provided by V2 to migrate your test shell definition files to portable scripts. Add a check for whether lava-test-case exists in $PATH to determine whether to use the remaining LAVA helpers or not and do everything else in the scripts. The ideal is to not have any install: section and make the script do all the setup it would need.
Portability is the primary objective - allowing test writers to run exactly the same scripts on their local box as will be run in LAVA with minimal detection of when to call lava-test-case and when to call echo or logging or whatever. Make no assumptions about what support exists in the image, let the script determine what can be done to initialise the test. Such scripts can also work in V1 but the aim with V2 is to make it much clearer that this is the direction.
Once V2 gets to the first shell prompt and starts executing lava-test-runner, that should be where a portable script takes over and does all the remaining initialisation, tear down and configuration. Then anyone can take that script and run it in a bare chroot or VM and it will sort itself out without needing special knowledge which is embedded in LAVA.
We've tried to indicate this direction in the docs:
https://staging.validation.linaro.org/static/docs/v2/writing-tests.html#best...
Let us know if those guidelines can be expanded or clarified.
Essentially, anything a developer would need to do outside LAVA to be able to run the scripts used in your LAVA tests should be within the scripts themselves. LAVA needs to provide certain pieces of data and some handlers to report test cases but apart from that, the drive is towards portability and away from hidden steps being done by LAVA.