Hello,
Currently I'm testing a board type which deploys OE via u-boot-ums. Before flashing the image into the device, LAVA slave modifies the image copying tests definition onto it. Once the image is modified with all the tests, this is dd'ed into the device, booted and tests will be running. This is OK so far but as soon as we enable signed images (roofs will be R/O but we will be having other R/W partitions) this won't work anymore as we are changing the image and it needs to be resigned. Moreover, this is very board specific.
I'm here to investigate alternative solutions which have a more "generic" approach which also a developer (without LAVA) can run. The only assumption is that *the DUT has always wired network connectivity with the SLAVE*.
The workflow I have in mind is something like: 1) I have a signed image which I deploy onto the DUT 2) Boot the DUT 3) Instruct the device to get tests from somewhere (either from the SLAVE or internet) 4) Run those tests
The step I'd like to solve is 3). I was thinking something like that: * download/compile all I need on the SLAVE (it is not possible to do it on the DUT due to limited resources/libraries/tooling) * setup some sort of server http on the SLAVE in order to serve those files * wget those files onto the DUT * setup and execute the tests.
The above approach should work WITHOUT LAVA as well. Basically, replace SLAVE with "developer computer"
Is it something I can architect with LAVA? Does LAVA give me this flexibility?
Thanks for your help
Regards
-- Diego Russo Staff Software Engineer - diego.russo@arm.com Direct Tel. no: +44 1223 405920 Main Tel. no: +44 1223 400400 ARM Ltd. CPC1, Capital Park, Cambridge Road, Fulbourn, CB21 5XE, United Kingdom http://www.diegor.co.uk - http://twitter.com/diegor http://www.linkedin.com/in/diegor
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On Mon, 8 Oct 2018 at 17:15, Diego Russo Diego.Russo@arm.com wrote:
Hello,
Currently I'm testing a board type which deploys OE via u-boot-ums. Before flashing the image into the device, LAVA slave modifies the image copying tests definition onto it. Once the image is modified with all the tests, this is dd'ed into the device, booted and tests will be running. This is OK so far but as soon as we enable signed images (roofs will be R/O but we will be having other R/W partitions) this won't work anymore as we are changing the image and it needs to be resigned. Moreover, this is very board specific.
I'm here to investigate alternative solutions which have a more "generic" approach which also a developer (without LAVA) can run. The only assumption is that *the DUT has always wired network connectivity with the SLAVE*.
Then you can use the existing transfer-overlay support. https://master.lavasoftware.org/static/docs/v2/actions-boot.html#transfer-ov...
The restriction you didn't mention but which must be implemented is: the DUT must have a static IPv4 address.
The workflow I have in mind is something like:
- I have a signed image which I deploy onto the DUT
- Boot the DUT
- Instruct the device to get tests from somewhere (either from the SLAVE
or internet)
That needs to be done by the automation or it will not scale. The DUT cannot know, in advance, where to find the specific "thing" for this test job as opposed to the slightly different "thing" which is meant to be used by the next or previous test job on another DUT of the same kind right next to it on the same shelf in the same lab - not without building an enormous matrix of almost identical images. It does not scale.
- Run those tests
The step I'd like to solve is 3). I was thinking something like that:
- download/compile all I need on the SLAVE (it is not possible to do it on
the DUT due to limited resources/libraries/tooling)
- setup some sort of server http on the SLAVE in order to serve those files
- wget those files onto the DUT
- setup and execute the tests.
The above approach should work WITHOUT LAVA as well. Basically, replace SLAVE with "developer computer"
It needs to be under the control of LAVA. Avoid thinking that the signed image on the DUT should go off and find this thing automatically - there are always temporary path names involved or it simply will not scale. All LAVA does is initiate the call to something like wget on the DUT with a known path name which includes a unique location for this test job (so that another test job on a second DUT in the same lab can simultaneously get it's own) , unpacks it with tar *and* knows where to find the script to start the process.
LAVA provides an apache service on the worker to support the download.
Is it something I can architect with LAVA? Does LAVA give me this flexibility?
Thanks for your help
Regards
-- Diego Russo Staff Software Engineer - diego.russo@arm.com Direct Tel. no: +44 1223 405920 Main Tel. no: +44 1223 400400 ARM Ltd. CPC1, Capital Park, Cambridge Road, Fulbourn, CB21 5XE, United Kingdom http://www.diegor.co.uk - http://twitter.com/diegor http://www.linkedin.com/in/diegor
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
From: Neil Williams neil.williams@linaro.org Date: Tuesday, 9 October 2018 at 08:31 To: Diego Russo Diego.Russo@arm.com Cc: Lava Users Mailman list lava-users@lists.linaro.org Subject: Re: [Lava-users] Getting tests on DUT: other solutions?
On Mon, 8 Oct 2018 at 17:15, Diego Russo <Diego.Russo@arm.commailto:Diego.Russo@arm.com> wrote: Hello,
Currently I'm testing a board type which deploys OE via u-boot-ums. Before flashing the image into the device, LAVA slave modifies the image copying tests definition onto it. Once the image is modified with all the tests, this is dd'ed into the device, booted and tests will be running. This is OK so far but as soon as we enable signed images (roofs will be R/O but we will be having other R/W partitions) this won't work anymore as we are changing the image and it needs to be resigned. Moreover, this is very board specific.
I'm here to investigate alternative solutions which have a more "generic" approach which also a developer (without LAVA) can run. The only assumption is that *the DUT has always wired network connectivity with the SLAVE*.
Then you can use the existing transfer-overlay support. https://master.lavasoftware.org/static/docs/v2/actions-boot.html#transfer-ov...
That’s a good suggestion, thanks.
The restriction you didn't mention but which must be implemented is: the DUT must have a static IPv4 address.
I guess you are right. It was implicit but yes, DUT and SLAVE need to be able to talk at IP level.
The workflow I have in mind is something like: 1) I have a signed image which I deploy onto the DUT 2) Boot the DUT 3) Instruct the device to get tests from somewhere (either from the SLAVE or internet)
That needs to be done by the automation or it will not scale. The DUT cannot know, in advance, where to find the specific "thing" for this test job as opposed to the slightly different "thing" which is meant to be used by the next or previous test job on another DUT of the same kind right next to it on the same shelf in the same lab - not without building an enormous matrix of almost identical images. It does not scale.
“That needs to be done by the automation” what do you mean by that? I mean I might understand the meaning but I’m not sure I understood your meaning. Are you able to give me an example?
4) Run those tests
The step I'd like to solve is 3). I was thinking something like that: * download/compile all I need on the SLAVE (it is not possible to do it on the DUT due to limited resources/libraries/tooling) * setup some sort of server http on the SLAVE in order to serve those files * wget those files onto the DUT * setup and execute the tests.
The above approach should work WITHOUT LAVA as well. Basically, replace SLAVE with "developer computer"
It needs to be under the control of LAVA. Avoid thinking that the signed image on the DUT should go off and find this thing automatically - there are always temporary path names involved or it simply will not scale. All LAVA does is initiate the call to something like wget on the DUT with a known path name which includes a unique location for this test job (so that another test job on a second DUT in the same lab can simultaneously get it's own) , unpacks it with tar *and* knows where to find the script to start the process.
This is exactly my intention, sort of. We need though to enable developers who don’t have LAVA access to be able to test their personal board they are developing on. That’s the main reason of having the *same* behaviour without LAVA, maybe scripting something out. The requirement/restriction is always the same: network connection with IPv4 working between DUT and computer. We haven’t decided how to do it yet as we are still in requirement phase. No implementation details have been decided.
LAVA provides an apache service on the worker to support the download.
Oh, that’s quite interesting.
Anyway in general it seems possible to do what I’d like to do and LAVA does provide tools for it. That’s a good start. Are you able to give me public LAVA pipeline which are similar to what I’d like to do?
Thanks
Is it something I can architect with LAVA? Does LAVA give me this flexibility?
Thanks for your help
Regards
-- Diego Russo Staff Software Engineer - diego.russo@arm.commailto:diego.russo@arm.com Direct Tel. no: +44 1223 405920 Main Tel. no: +44 1223 400400 ARM Ltd. CPC1, Capital Park, Cambridge Road, Fulbourn, CB21 5XE, United Kingdom http://www.diegor.co.uk - http://twitter.com/diegor http://www.linkedin.com/in/diegor
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ Lava-users mailing list Lava-users@lists.linaro.orgmailto:Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
--
Neil Williams ============= neil.williams@linaro.orgmailto:neil.williams@linaro.org http://www.linux.codehelp.co.uk/ IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On Tue, 9 Oct 2018 at 15:19, Diego Russo Diego.Russo@arm.com wrote:
*From: *Neil Williams neil.williams@linaro.org *Date: *Tuesday, 9 October 2018 at 08:31 *To: *Diego Russo Diego.Russo@arm.com *Cc: *Lava Users Mailman list lava-users@lists.linaro.org *Subject: *Re: [Lava-users] Getting tests on DUT: other solutions?
On Mon, 8 Oct 2018 at 17:15, Diego Russo Diego.Russo@arm.com wrote:
Hello,
Currently I'm testing a board type which deploys OE via u-boot-ums. Before flashing the image into the device, LAVA slave modifies the image copying tests definition onto it. Once the image is modified with all the tests, this is dd'ed into the device, booted and tests will be running. This is OK so far but as soon as we enable signed images (roofs will be R/O but we will be having other R/W partitions) this won't work anymore as we are changing the image and it needs to be resigned. Moreover, this is very board specific.
I'm here to investigate alternative solutions which have a more "generic" approach which also a developer (without LAVA) can run. The only assumption is that *the DUT has always wired network connectivity with the SLAVE*.
Then you can use the existing transfer-overlay support. https://master.lavasoftware.org/static/docs/v2/actions-boot.html#transfer-ov...
That’s a good suggestion, thanks.
The restriction you didn't mention but which must be implemented is: the DUT must have a static IPv4 address.
I guess you are right. It was implicit but yes, DUT and SLAVE need to be able to talk at IP level.
The workflow I have in mind is something like:
- I have a signed image which I deploy onto the DUT
- Boot the DUT
- Instruct the device to get tests from somewhere (either from the SLAVE
or internet)
That needs to be done by the automation or it will not scale. The DUT cannot know, in advance, where to find the specific "thing" for this test job as opposed to the slightly different "thing" which is meant to be used by the next or previous test job on another DUT of the same kind right next to it on the same shelf in the same lab - not without building an enormous matrix of almost identical images. It does not scale.
“That needs to be done by the automation” what do you mean by that? I mean I might understand the meaning but I’m not sure I understood your meaning. Are you able to give me an example?
Instructing the device to download the tests needs to be done by the automation or it will not scale. As well as coming from the worker (not a random internet location), the instruction to do the work needs to come from the automation as well, not baked into the image deployed to the DUT. The location needs to be unique for this test job and therefore temporary, for the life of this test job.
- Run those tests
The step I'd like to solve is 3). I was thinking something like that:
- download/compile all I need on the SLAVE (it is not possible to do it on
the DUT due to limited resources/libraries/tooling)
- setup some sort of server http on the SLAVE in order to serve those files
- wget those files onto the DUT
- setup and execute the tests.
The above approach should work WITHOUT LAVA as well. Basically, replace SLAVE with "developer computer"
It needs to be under the control of LAVA. Avoid thinking that the signed image on the DUT should go off and find this thing automatically - there are always temporary path names involved or it simply will not scale. All LAVA does is initiate the call to something like wget on the DUT with a known path name which includes a unique location for this test job (so that another test job on a second DUT in the same lab can simultaneously get it's own) , unpacks it with tar *and* knows where to find the script to start the process.
This is exactly my intention, sort of. We need though to enable developers who don’t have LAVA access to be able to test their personal board they are developing on. That’s the main reason of having the **same** behaviour without LAVA, maybe scripting something out.
However, you will still be relying on something outside LAVA providing the temporary file locations and something to initiate the operation and neither of those can be baked into the image.
So you won't be able to have the same behaviour without automation - all you can do is provide a different automation solution to do the tasks that LAVA does - or make it entirely manual and expect developers to get the files to run their own tests.
When you think about it, this is exactly the same as any other test in LAVA. Developers can TFTP boot their beaglebone-black but to be able to run the tests, the developer needs to login to the DUT, download or install their own scripts and start the execution of those scripts. Whatever gets scripted, the developer still has to get those scripts onto the DUT and that either happens manually or it happens by replicating some method of home-brew automation. This step cannot be baked into the image deployed onto the DUT, it simply does not scale. It's the my-one-board-my-one-machine mindset and it is incompatible with automation where there are lots of machines, lots of users and lots of devices.
transfer_overlay just runs a wget and a tar -xaf then LAVA calls a pre-determined script in a known location (specific to that one test job). The developer has to do something similar but the download differs. e.g. LAVA has support for metadata and parameters which change how the test definition is executed - the lone developer would have to replicate that.
What you do need to consider is that the *content* of your test defnitions also need to take the developer-outside-LAVA into account. This is recommended best practice in the LAVA docs and the Linaro QA team. Don't use install: deps: in the test shell definition - write a script which installs everything you need, put it in the test definition git repo so that the developer has access to it. Make your scripts check for lava-test-case in $PATH and only call it if it exists (use print or echo or whatever instead) so that the lone developer can call the same scripts and get output without needing to fake up lava-test case and other scripts from the LAVA overlay. All of this is up to the test writer to plan and implement in their own git repositories.
The requirement/restriction is always the same: network connection with IPv4 working between DUT and computer.
We haven’t decided how to do it yet as we are still in requirement phase. No implementation details have been decided.
LAVA provides an apache service on the worker to support the download.
Oh, that’s quite interesting.
Anyway in general it seems possible to do what I’d like to do and LAVA does provide tools for it. That’s a good start.
Are you able to give me public LAVA pipeline which are similar to what I’d like to do?
https://lava.codehelp.co.uk/scheduler/job/1741
Thanks
Is it something I can architect with LAVA? Does LAVA give me this flexibility?
Thanks for your help
Regards
-- Diego Russo Staff Software Engineer - diego.russo@arm.com Direct Tel. no: +44 1223 405920 Main Tel. no: +44 1223 400400 ARM Ltd. CPC1, Capital Park, Cambridge Road, Fulbourn, CB21 5XE, United Kingdom http://www.diegor.co.uk - http://twitter.com/diegor http://www.linkedin.com/in/diegor
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/ IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On 09/10/2018, 16:13, "Neil Williams" <neil.williams@linaro.orgmailto:neil.williams@linaro.org> wrote:
“That needs to be done by the automation” what do you mean by that? I mean I might understand the meaning but I’m not sure I understood your meaning. Are you able to give me an example?
Instructing the device to download the tests needs to be done by the automation or it will not scale. As well as coming from the worker (not a random internet location), the instruction to do the work needs to come from the automation as well, not baked into the image deployed to the DUT. The location needs to be unique for this test job and therefore temporary, for the life of this test job.
Ok, got it. We are well aligned on this. The image won’t contain anything about testing (that’s a requirement).
This is exactly my intention, sort of. We need though to enable developers who don’t have LAVA access to be able to test their personal board they are developing on. That’s the main reason of having the *same* behaviour without LAVA, maybe scripting something out.
However, you will still be relying on something outside LAVA providing the temporary file locations and something to initiate the operation and neither of those can be baked into the image.
Yes, what I had in mind was a simple “python -m SimpleHTTPServer 8000” on the developer computer and to run a wget on DUT. But yes, I agree with you.
So you won't be able to have the same behaviour without automation - all you can do is provide a different automation solution to do the tasks that LAVA does - or make it entirely manual and expect developers to get the files to run their own tests.
Agreed.
When you think about it, this is exactly the same as any other test in LAVA. Developers can TFTP boot their beaglebone-black but to be able to run the tests, the developer needs to login to the DUT, download or install their own scripts and start the execution of those scripts. Whatever gets scripted, the developer still has to get those scripts onto the DUT and that either happens manually or it happens by replicating some method of home-brew automation. This step cannot be baked into the image deployed onto the DUT, it simply does not scale. It's the my-one-board-my-one-machine mindset and it is incompatible with automation where there are lots of machines, lots of users and lots of devices.
Totally agreed. We don’t plan to bake *anything* into the image which is related to testing. The image is a generic one and we need tooling around it to get those tests onto it.
transfer_overlay just runs a wget and a tar -xaf then LAVA calls a pre-determined script in a known location (specific to that one test job). The developer has to do something similar but the download differs. e.g. LAVA has support for metadata and parameters which change how the test definition is executed - the lone developer would have to replicate that.
Agreed. This is exactly what I was expecting.
What you do need to consider is that the *content* of your test defnitions also need to take the developer-outside-LAVA into account. This is recommended best practice in the LAVA docs and the Linaro QA team. Don't use install: deps: in the test shell definition - write a script which installs everything you need, put it in the test definition git repo so that the developer has access to it. Make your scripts check for lava-test-case in $PATH and only call it if it exists (use print or echo or whatever instead) so that the lone developer can call the same scripts and get output without needing to fake up lava-test case and other scripts from the LAVA overlay. All of this is up to the test writer to plan and implement in their own git repositories.
Agreed with the script that acts as “entry point” for encapsulating the logic for setting up the environment and running tests. I think overall we are on the same page but of course I wanted a confirmation from more experienced people of what I had in mind. With this email you’ve just given it ☺ Are you able to give me public LAVA pipeline which are similar to what I’d like to do?
https://lava.codehelp.co.uk/scheduler/job/1741
Great, thanks!
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-
Diego Russo -
Neil Williams