Hi,
Is it possible in LAVA to force reboot between installing test case dependencies and running the test itself? I'm trying to run bootchart test, but the bootchart package is not included in the ubuntu image I'm running.
Best Regards, milosz
On Mon, Aug 26, 2013 at 06:22:22PM +0100, Milosz Wasilewski wrote:
Hi,
Is it possible in LAVA to force reboot between installing test case dependencies and running the test itself? I'm trying to run bootchart test, but the bootchart package is not included in the ubuntu image I'm running.
That's not possible. You can workaround this by having two lava_test_shell actions, one that installs the dependency, and one that actually runs the tests. It sounds horrible (and it probably is), but will do the job because the target is rebooted before each lava_test_shell action.
IMO the right fix for the problem is having bootchart already installed in the image though.
On 26 August 2013 21:32, Antonio Terceiro antonio.terceiro@linaro.org wrote:
On Mon, Aug 26, 2013 at 06:22:22PM +0100, Milosz Wasilewski wrote:
Hi,
Is it possible in LAVA to force reboot between installing test case dependencies and running the test itself? I'm trying to run bootchart test, but the bootchart package is not included in the ubuntu image I'm running.
That's not possible. You can workaround this by having two lava_test_shell actions, one that installs the dependency, and one that actually runs the tests. It sounds horrible (and it probably is), but will do the job because the target is rebooted before each lava_test_shell action.
IMO the right fix for the problem is having bootchart already installed in the image though.
it has been considered but I'm not convinced it's the right fix. It's the right workaround ;)
imo, the real problem: a test is considered finished when we reboot the image. bottom line, multiple phase tests (it could be a reboot or something) isn't supported by LAVA.
bootchart is just a use-case. it needs to set up the OS, reboot to collect data, finally test results are available.
n Tue, Aug 27, 2013 at 09:37:56AM +0300, Fathi Boudra wrote:
On 26 August 2013 21:32, Antonio Terceiro antonio.terceiro@linaro.org wrote:
On Mon, Aug 26, 2013 at 06:22:22PM +0100, Milosz Wasilewski wrote:
Hi,
Is it possible in LAVA to force reboot between installing test case dependencies and running the test itself? I'm trying to run bootchart test, but the bootchart package is not included in the ubuntu image I'm running.
That's not possible. You can workaround this by having two lava_test_shell actions, one that installs the dependency, and one that actually runs the tests. It sounds horrible (and it probably is), but will do the job because the target is rebooted before each lava_test_shell action.
IMO the right fix for the problem is having bootchart already installed in the image though.
it has been considered but I'm not convinced it's the right fix. It's the right workaround ;)
imo, the real problem: a test is considered finished when we reboot the image. bottom line, multiple phase tests (it could be a reboot or something) isn't supported by LAVA.
It is supported, just not with a single lava-test-shell action. lava-test-shell was designed to be black box (i.e. run from the target without intervention from the host), and you can't really reboot and pick it up after from any program running on the target.
bootchart is just a use-case. it needs to set up the OS, reboot to collect data, finally test results are available.
Maybe my solution with two test definitions is the right way to do it after all. Ideally we could also have a separate action to just install packages in the test image, but I think the overhead of using a separate lava-test-shell test definition is low enough.
On Tue, 27 Aug 2013 07:41:38 -0300 Antonio Terceiro antonio.terceiro@linaro.org wrote:
n Tue, Aug 27, 2013 at 09:37:56AM +0300, Fathi Boudra wrote:
On 26 August 2013 21:32, Antonio Terceiro antonio.terceiro@linaro.org wrote:
On Mon, Aug 26, 2013 at 06:22:22PM +0100, Milosz Wasilewski wrote:
Hi,
Is it possible in LAVA to force reboot between installing test case dependencies and running the test itself? I'm trying to run bootchart test, but the bootchart package is not included in the ubuntu image I'm running.
That's not possible. You can workaround this by having two lava_test_shell actions, one that installs the dependency, and one that actually runs the tests. It sounds horrible (and it probably is), but will do the job because the target is rebooted before each lava_test_shell action.
IMO the right fix for the problem is having bootchart already installed in the image though.
it has been considered but I'm not convinced it's the right fix. It's the right workaround ;)
I don't see it as a workaround. Having to install / build a program in the test image isn't ideal - it may be necessary to add the test support to an existing image or a previous image or a customer image. By installing or building the test support inside the image, the checksum of the image at the start of the test can be used to reliably indicate a valid test of that image, not a hand-modified version of that image.
imo, the real problem: a test is considered finished when we reboot the image.
Just to add to Antonio's comment - the test is considered complete but the results are not collected until *all* tests are complete for that job.
Installing the test support is a separate test, it deserves a separate result in the final bundle - the result of the installation / build. So the first test is to do that to the supplied image. That is one set of results in the bundle. The test then reboots (because of the second lava_test_shell block) and runs the second test. The final results bundle contains all the test results, separating the setup of the test support from the operation of the test itself.
There is one result bundle, it contains multiple sets of results and is only created once all tests are complete and it is preserved between reboots.
(This does need to be documented more clearly too.)
A reboot has to be one of the valid ways to end a test case - there is too much state within typical test cases about which LAVA knows nothing at all.
It is up to the test writer to put the reboots in the right places. What we do need to do is document that this is done by having more than one lava_test_shell blocks in the JSON instead of trying to do the reboot in the YAML.
Extend this to a use case where the test support needs to be compiled instead of installed and it is clear to me that the results need to distinguish between a failure to build the test support prior to the reboot from the failure of the test operation after the reboot. i.e. each operation is a different test with a different test result in the one result bundle. In this use case, I can also see a third result which is needed - the result of first installing the build dependencies of the thing to be compiled, the first result of the three. The bundle would have dep_install, compile & run results.
bootchart is just a use-case. it needs to set up the OS, reboot to collect data, finally test results are available.
That is just what LAVA does with two lava_test_shell blocks.
Maybe my solution with two test definitions is the right way to do it after all. Ideally we could also have a separate action to just install packages in the test image, but I think the overhead of using a separate lava-test-shell test definition is low enough.
That can just as easily be a single YAML file.
On 27 August 2013 14:17, Neil Williams codehelp@debian.org wrote:
On Tue, 27 Aug 2013 07:41:38 -0300 Antonio Terceiro antonio.terceiro@linaro.org wrote:
n Tue, Aug 27, 2013 at 09:37:56AM +0300, Fathi Boudra wrote:
On 26 August 2013 21:32, Antonio Terceiro antonio.terceiro@linaro.org wrote:
On Mon, Aug 26, 2013 at 06:22:22PM +0100, Milosz Wasilewski wrote:
Hi,
Is it possible in LAVA to force reboot between installing test case dependencies and running the test itself? I'm trying to run bootchart test, but the bootchart package is not included in the ubuntu image I'm running.
That's not possible. You can workaround this by having two lava_test_shell actions, one that installs the dependency, and one that actually runs the tests. It sounds horrible (and it probably is), but will do the job because the target is rebooted before each lava_test_shell action.
IMO the right fix for the problem is having bootchart already installed in the image though.
it has been considered but I'm not convinced it's the right fix. It's the right workaround ;)
I don't see it as a workaround. Having to install / build a program in the test image isn't ideal - it may be necessary to add the test support to an existing image or a previous image or a customer image. By installing or building the test support inside the image, the checksum of the image at the start of the test can be used to reliably indicate a valid test of that image, not a hand-modified version of that image.
imo, the real problem: a test is considered finished when we reboot the image.
Just to add to Antonio's comment - the test is considered complete but the results are not collected until *all* tests are complete for that job.
Installing the test support is a separate test, it deserves a separate result in the final bundle - the result of the installation / build. So the first test is to do that to the supplied image. That is one set of results in the bundle. The test then reboots (because of the second lava_test_shell block) and runs the second test. The final results bundle contains all the test results, separating the setup of the test support from the operation of the test itself.
There is one result bundle, it contains multiple sets of results and is only created once all tests are complete and it is preserved between reboots.
(This does need to be documented more clearly too.)
A reboot has to be one of the valid ways to end a test case - there is too much state within typical test cases about which LAVA knows nothing at all.
It is up to the test writer to put the reboots in the right places. What we do need to do is document that this is done by having more than one lava_test_shell blocks in the JSON instead of trying to do the reboot in the YAML.
Extend this to a use case where the test support needs to be compiled instead of installed and it is clear to me that the results need to distinguish between a failure to build the test support prior to the reboot from the failure of the test operation after the reboot. i.e. each operation is a different test with a different test result in the one result bundle. In this use case, I can also see a third result which is needed - the result of first installing the build dependencies of the thing to be compiled, the first result of the three. The bundle would have dep_install, compile & run results.
bootchart is just a use-case. it needs to set up the OS, reboot to collect data, finally test results are available.
That is just what LAVA does with two lava_test_shell blocks.
Maybe my solution with two test definitions is the right way to do it after all. Ideally we could also have a separate action to just install packages in the test image, but I think the overhead of using a separate lava-test-shell test definition is low enough.
That can just as easily be a single YAML file.
+1 for a single yaml file.
I think it's better to keep it at the test definition level. We produce a limited set of rootfs/images, we won't be able to produce test images on demand because there's unlimited possibilities.
It seems also more logical to me. Since the test doesn't know which image is used, it's up to the test to set up its prerequisite like dependencies and then run the test itself.
On Tue, 27 Aug 2013 14:34:09 +0300 Fathi Boudra fathi.boudra@linaro.org wrote:
On 27 August 2013 14:17, Neil Williams codehelp@debian.org wrote:
It is up to the test writer to put the reboots in the right places. What we do need to do is document that this is done by having more than one lava_test_shell blocks in the JSON instead of trying to do the reboot in the YAML.
bootchart is just a use-case. it needs to set up the OS, reboot to collect data, finally test results are available.
That is just what LAVA does with two lava_test_shell blocks.
Maybe my solution with two test definitions is the right way to do it after all. Ideally we could also have a separate action to just install packages in the test image, but I think the overhead of using a separate lava-test-shell test definition is low enough.
That can just as easily be a single YAML file.
+1 for a single yaml file.
I think it's better to keep it at the test definition level. We produce a limited set of rootfs/images, we won't be able to produce test images on demand because there's unlimited possibilities.
It seems also more logical to me. Since the test doesn't know which image is used, it's up to the test to set up its prerequisite like dependencies and then run the test itself.
So, for completeness, that works out as:
A YAML file called installer.yaml with install: deps: - bootchart
or, if the test image doesn't raise a usable network interface by default: run: steps: - lava-test-case network-up --shell ifconfig eth0 up - lava-test-case package-update --shell apt-get update - lava-test-case package-install --shell apt-get -y install bootchart
or similar.
A YAML file called parser.yaml with:
run: steps: - lava-test-case bootchart-run --shell ./run-bootchart-parser.sh
(Contents of run-bootchart-parser.sh TBD)
A single JSON file with a snippet:
{ "command": "lava_test_shell", "parameters": { "testdef_repos": [ { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "installer.yaml" } ], "timeout": 900 } }, { "command": "lava_test_shell", "parameters": { "testdef_repos": [ { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "parser.yaml" } ], "timeout": 900 } },
All of that is already supported.
The one thing which needs better documentation is that the use of more than one lava_test_shell command block in the JSON causes a reboot between actions when adding to the testdef_repos array does not:
{ "command": "lava_test_shell", "parameters": { "testdef_repos": [ { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "installer.yaml" }, { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "parser.yaml" } ], "timeout": 900 } },
On 27 August 2013 15:12, Neil Williams codehelp@debian.org wrote:
On Tue, 27 Aug 2013 14:34:09 +0300 Fathi Boudra fathi.boudra@linaro.org wrote:
On 27 August 2013 14:17, Neil Williams codehelp@debian.org wrote:
It is up to the test writer to put the reboots in the right places. What we do need to do is document that this is done by having more than one lava_test_shell blocks in the JSON instead of trying to do the reboot in the YAML.
bootchart is just a use-case. it needs to set up the OS, reboot to collect data, finally test results are available.
That is just what LAVA does with two lava_test_shell blocks.
Maybe my solution with two test definitions is the right way to do it after all. Ideally we could also have a separate action to just install packages in the test image, but I think the overhead of using a separate lava-test-shell test definition is low enough.
That can just as easily be a single YAML file.
+1 for a single yaml file.
I think it's better to keep it at the test definition level. We produce a limited set of rootfs/images, we won't be able to produce test images on demand because there's unlimited possibilities.
It seems also more logical to me. Since the test doesn't know which image is used, it's up to the test to set up its prerequisite like dependencies and then run the test itself.
So, for completeness, that works out as:
A YAML file called installer.yaml with install: deps: - bootchart
or, if the test image doesn't raise a usable network interface by default: run: steps: - lava-test-case network-up --shell ifconfig eth0 up - lava-test-case package-update --shell apt-get update - lava-test-case package-install --shell apt-get -y install bootchart
or similar.
A YAML file called parser.yaml with:
run: steps: - lava-test-case bootchart-run --shell ./run-bootchart-parser.sh
(Contents of run-bootchart-parser.sh TBD)
A single JSON file with a snippet:
{ "command": "lava_test_shell", "parameters": { "testdef_repos": [ { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "installer.yaml" } ], "timeout": 900 } }, { "command": "lava_test_shell", "parameters": { "testdef_repos": [ { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "parser.yaml" } ], "timeout": 900 } },All of that is already supported.
The one thing which needs better documentation is that the use of more than one lava_test_shell command block in the JSON causes a reboot between actions when adding to the testdef_repos array does not:
{ "command": "lava_test_shell", "parameters": { "testdef_repos": [ { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "installer.yaml" }, { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "parser.yaml" } ], "timeout": 900 } },
Thanks! Good to know.
Neil Williams codehelp@debian.org writes:
On Tue, 27 Aug 2013 14:34:09 +0300 Fathi Boudra fathi.boudra@linaro.org wrote:
On 27 August 2013 14:17, Neil Williams codehelp@debian.org wrote:
It is up to the test writer to put the reboots in the right places. What we do need to do is document that this is done by having more than one lava_test_shell blocks in the JSON instead of trying to do the reboot in the YAML.
bootchart is just a use-case. it needs to set up the OS, reboot to collect data, finally test results are available.
That is just what LAVA does with two lava_test_shell blocks.
Maybe my solution with two test definitions is the right way to do it after all. Ideally we could also have a separate action to just install packages in the test image, but I think the overhead of using a separate lava-test-shell test definition is low enough.
That can just as easily be a single YAML file.
+1 for a single yaml file.
I think it's better to keep it at the test definition level. We produce a limited set of rootfs/images, we won't be able to produce test images on demand because there's unlimited possibilities.
It seems also more logical to me. Since the test doesn't know which image is used, it's up to the test to set up its prerequisite like dependencies and then run the test itself.
So, for completeness, that works out as:
A YAML file called installer.yaml with install: deps: - bootchart
or, if the test image doesn't raise a usable network interface by default: run: steps: - lava-test-case network-up --shell ifconfig eth0 up - lava-test-case package-update --shell apt-get update - lava-test-case package-install --shell apt-get -y install bootchart
or similar.
A YAML file called parser.yaml with:
run: steps: - lava-test-case bootchart-run --shell ./run-bootchart-parser.sh
(Contents of run-bootchart-parser.sh TBD)
A single JSON file with a snippet:
{ "command": "lava_test_shell", "parameters": { "testdef_repos": [ { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "installer.yaml" } ], "timeout": 900 } }, { "command": "lava_test_shell", "parameters": { "testdef_repos": [ { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "parser.yaml" } ], "timeout": 900 } },All of that is already supported.
The one thing which needs better documentation is that the use of more than one lava_test_shell command block in the JSON causes a reboot between actions when adding to the testdef_repos array does not:
{ "command": "lava_test_shell", "parameters": { "testdef_repos": [ { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "installer.yaml" }, { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "parser.yaml" } ], "timeout": 900 } },
Is it possible to have a different timeout value per testdef? In the last example can 'installer.yaml' have a timeout of 60 and 'parser.yaml' a timeout of 900?
On Fri, 20 Sep 2013 11:41:45 +0100 Andrew McDermott andrew.mcdermott@linaro.org wrote:
All of that is already supported.
The one thing which needs better documentation is that the use of more than one lava_test_shell command block in the JSON causes a reboot between actions when adding to the testdef_repos array does not:
{ "command": "lava_test_shell", "parameters": { "testdef_repos": [ { "git-repo":"git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "installer.yaml" }, { "git-repo": "git://git.linaro.org/people/foobar/bootchart-yaml.git", "testdef": "parser.yaml" } ], "timeout": 900 } },
Is it possible to have a different timeout value per testdef? In the last example can 'installer.yaml' have a timeout of 60 and 'parser.yaml' a timeout of 900?
No. Only lava_test_shell parameters support the timeout, not the testdef_repos or testdef_url. To change the timeout, you need to have multiple command: lava_test_shell sections but those will still cause a reboot between each lava_test_shell operation.
linaro-validation@lists.linaro.org
-
Andrew McDermott -
Antonio Terceiro -
Fathi Boudra -
Milosz Wasilewski -
Neil Williams