Hi, list: I am working on Validation part for Android, in which mainly inclusive of integrating existing benchmark and testing suites into abrek, like 0xbench and android CTS. For now in my thought, if these benchmark/testing can integrate with abrek, then it should be no much problem or extra effort for integrating further into LAVA.
Though I have some questions regarding to validation.
How to detect the early fail through serial console? like detecting the failure during kernel booting stage also before running init or getting the shell. I think this will be a job issued by the dispatcher though I don't know how this checking mechanism will be done. Would there be anything here needed to do specifically for Android?
How to connect the devices in the farm? I checked the wiki page, Platform/Validation/Specs/HardwareSetup [1]. The network will be used. I am thinking what's needed for setting up a network environment for this? The device is needed to fetch a fixed ip by dhcp just after booting up? USB gadget is an alternative for Android and in most situation could be more convenient for personal testing. adb (Android Debug Bridge, running on host side) can connect to a device through USB or TCP/IP.
I suppose there are device context for describing how to connect certain device probably, like Android devices.
Regarding to remote device registration, adb can list all the attached device but we still need a way to distinguish from each device in the farm. IP is a way to distinguish between each of them, like:
$ adb devices List of devices attached emulator-5554 device 192.168.85.102:5555 device
Any feedback is appreciated. Thanks.
Regards, -Jeremy Chang
[1] https://wiki.linaro.org/Platform/Validation/Specs/HardwareSetup
On Mon, Mar 14, 2011 at 11:22 AM, Jeremy Chang jeremy.chang@linaro.orgwrote:
Hi, list: I am working on Validation part for Android, in which mainly inclusive of integrating existing benchmark and testing suites into abrek, like 0xbench and android CTS. For now in my thought, if these benchmark/testing can integrate with abrek, then it should be no much problem or extra effort for integrating further into LAVA.
As discussed before, I think abrek is probably not the right choice for running tests under android. Abrek was designed to work on linaro images running python in the client, specifically it was designed to run under Linaro or Ubuntu images, before we had even considered android at all. We probably need some kind of mechanism for running the tests from the server over adb instead, then outputting the results into a json bundle that the dashboard can understand.
Though I have some questions regarding to validation.
How to detect the early fail through serial console? like detecting the failure during kernel booting stage also before running init or getting the shell. I think this will be a job issued by the dispatcher though I don't know how this checking mechanism will be done. Would there be anything here needed to do specifically for Android?
Right, the dispatcher currently handles this. We are using pexpect to drive things over the serial console. So once the system is booting, we expect to see a valid shell prompt within some reasonable timeout period. If we don't, then we know the boot failed. I expect the bit that detects the shell prompt may need to be changed somewhat for android. Perhaps for android we need to wait for a valid adb devices output? or a working adb shell?
How to connect the devices in the farm? I checked the wiki page, Platform/Validation/Specs/HardwareSetup [1]. The network will be used. I am thinking what's needed for setting up a network environment for this? The device is needed to fetch a fixed ip by dhcp just after booting up? USB gadget is an alternative for Android and in most situation could be more convenient for personal testing. adb (Android Debug Bridge, running on host side) can connect to a device through USB or TCP/IP.
USB gadget is hard for us, if we have a lot of machines. Network would be better, but can we rely in it to be working? Maybe we just make that a requirement for running android tests. I don't currently have it set up, but my current thinking is that we want to define a new network for the validation machines, perhaps 10.1.x.x, and set up static assignments for them in a dhcp server running on the control node. This way, they could either be configured statically, or via dhcp.
Thanks, Paul Larson
On Tue, Mar 15, 2011 at 4:09 PM, Paul Larson paul.larson@linaro.org wrote:
On Mon, Mar 14, 2011 at 11:22 AM, Jeremy Chang jeremy.chang@linaro.org wrote:
Hi, list: I am working on Validation part for Android, in which mainly inclusive of integrating existing benchmark and testing suites into abrek, like 0xbench and android CTS. For now in my thought, if these benchmark/testing can integrate with abrek, then it should be no much problem or extra effort for integrating further into LAVA.
As discussed before, I think abrek is probably not the right choice for running tests under android. Abrek was designed to work on linaro images running python in the client, specifically it was designed to run under Linaro or Ubuntu images, before we had even considered android at all. We probably need some kind of mechanism for running the tests from the server over adb instead, then outputting the results into a json bundle that the dashboard can understand.
we talked about this on IRC. I pointed Jeremy to our lp:lava branch and explained to him the theory of how things work. We agreed to first write a few example job descriptions and then decide what features a generic testsuite framework would require (if we need this at all). Also we might do a prototype to learn how things would work if we implemented the actions for android deploy, boot and test run.
Though I have some questions regarding to validation.
How to detect the early fail through serial console? like detecting the failure during kernel booting stage also before running init or getting the shell. I think this will be a job issued by the dispatcher though I don't know how this checking mechanism will be done. Would there be anything here needed to do specifically for Android?
Right, the dispatcher currently handles this. We are using pexpect to drive things over the serial console. So once the system is booting, we expect to see a valid shell prompt within some reasonable timeout period. If we don't, then we know the boot failed. I expect the bit that detects the shell prompt may need to be changed somewhat for android. Perhaps for android we need to wait for a valid adb devices output? or a working adb shell?
yes, its a timeout thing. we should give the adb connect a reasonable timeout. In this way we can guess that the boot failed in a bad way as no network came up.
How to connect the devices in the farm? I checked the wiki page, Platform/Validation/Specs/HardwareSetup [1]. The network will be used. I am thinking what's needed for setting up a network environment for this? The device is needed to fetch a fixed ip by dhcp just after booting up? USB gadget is an alternative for Android and in most situation could be more convenient for personal testing. adb (Android Debug Bridge, running on host side) can connect to a device through USB or TCP/IP.
USB gadget is hard for us, if we have a lot of machines. Network would be better, but can we rely in it to be working? Maybe we just make that a requirement for running android tests. I don't currently have it set up, but my current thinking is that we want to define a new network for the validation machines, perhaps 10.1.x.x, and set up static assignments for them in a dhcp server running on the control node. This way, they could either be configured statically, or via dhcp.
yes, USB gadget is not that great for our current setup.
I like statically IP. How do we know which board would get which IP? One idea would be to set up a DNS for internal board names. e.g. panda01.internal.network would resolve to our preallocated static IP for board "panda01".
This could then be used by the dispatcher to resolve the IP and pass it to the board through kernel cmdline.
I like statically IP. How do we know which board would get which IP? One idea would be to set up a DNS for internal board names. e.g. panda01.internal.network would resolve to our preallocated static IP for board "panda01".
This could then be used by the dispatcher to resolve the IP and pass it to the board through kernel cmdline.
I hadn't really thought of passing it via cmdline. If we need to bring it
up statically, we could possibly do that, or possibly just make the necessary adjustments before booting the image. What I was proposing, if the image is already set up to dhcp by default, we could run a dhcp server on the control node with the static ip assignments already in place for each board. That way they always get the same address that we pre-allocated for that board. We'll also need to make sure that the control node has a valid ip on both the internal and the external network.
Thanks, Paul Larson
On Tue, 15 Mar 2011 11:17:14 -0500, Paul Larson paul.larson@linaro.org wrote:
I like statically IP. How do we know which board would get which IP? One idea would be to set up a DNS for internal board names. e.g. panda01.internal.network would resolve to our preallocated static IP for board "panda01".
This could then be used by the dispatcher to resolve the IP and pass it to the board through kernel cmdline.
I hadn't really thought of passing it via cmdline. If we need to bring it
up statically, we could possibly do that, or possibly just make the necessary adjustments before booting the image. What I was proposing, if the image is already set up to dhcp by default, we could run a dhcp server on the control node with the static ip assignments already in place for each board.
How would you do this? Most boards don't have a fixed MAC address. You can set up a dhcp-name but that (AIUI) means a different rootfs for each board and then you might as well just allocate a static IP...
That way they always get the same address that we pre-allocated for that board. We'll also need to make sure that the control node has a valid ip on both the internal and the external network.
Cheers, mwh
How would you do this? Most boards don't have a fixed MAC address. You can set up a dhcp-name but that (AIUI) means a different rootfs for each board and then you might as well just allocate a static IP...
I know this is a problem on some boards, but I thought it had been fixed on some, and some required something in uboot, which I hoped we could set in advance, to make it work. That's one reason I hadn't moved to this yet, I want to make sure it's actually possible first. But if you have a different idea, I'm open to suggestions. :)
Thanks, Paul Larson
On 16 March 2011 10:03, Michael Hudson-Doyle michael.hudson@canonical.comwrote:
On Tue, 15 Mar 2011 11:17:14 -0500, Paul Larson paul.larson@linaro.org wrote:
I like statically IP. How do we know which board would get which IP? One idea would be to set up a DNS for internal board names. e.g. panda01.internal.network would resolve to our preallocated static IP for board "panda01".
This could then be used by the dispatcher to resolve the IP and pass it to the board through kernel cmdline.
I hadn't really thought of passing it via cmdline. If we need to bring
it
up statically, we could possibly do that, or possibly just make the necessary adjustments before booting the image. What I was proposing, if the image is already set up to dhcp by default, we could run a dhcp
server
on the control node with the static ip assignments already in place for
each
board.
How would you do this? Most boards don't have a fixed MAC address. You can set up a dhcp-name but that (AIUI) means a different rootfs for each board and then you might as well just allocate a static IP...
Also, some boards of the same type have the same MAC as well..
That way they always get the same address that we pre-allocated for that board. We'll also need to make sure that the control node has a
valid
ip on both the internal and the external network.
Cheers, mwh
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
On Wed, Mar 16, 2011 at 6:24 AM, Sachin Kamat sachin.kamat@linaro.org wrote:
On 16 March 2011 10:03, Michael Hudson-Doyle michael.hudson@canonical.com wrote:
On Tue, 15 Mar 2011 11:17:14 -0500, Paul Larson paul.larson@linaro.org wrote:
I like statically IP. How do we know which board would get which IP? One idea would be to set up a DNS for internal board names. e.g. panda01.internal.network would resolve to our preallocated static IP for board "panda01".
This could then be used by the dispatcher to resolve the IP and pass it to the board through kernel cmdline.
I hadn't really thought of passing it via cmdline. If we need to bring it
up statically, we could possibly do that, or possibly just make the necessary adjustments before booting the image. What I was proposing, if the image is already set up to dhcp by default, we could run a dhcp server on the control node with the static ip assignments already in place for each board.
How would you do this? Most boards don't have a fixed MAC address. You can set up a dhcp-name but that (AIUI) means a different rootfs for each board and then you might as well just allocate a static IP...
Also, some boards of the same type have the same MAC as well..
Is setting a static IP for those boards actually enough or do we also need to set some reasonable MAC?
-
Alexander Sack -
Jeremy Chang -
Michael Hudson-Doyle -
Paul Larson -
Sachin Kamat