On Tue, Mar 15, 2011 at 4:09 PM, Paul Larson paul.larson@linaro.org wrote:
On Mon, Mar 14, 2011 at 11:22 AM, Jeremy Chang jeremy.chang@linaro.org wrote:
Hi, list: I am working on Validation part for Android, in which mainly inclusive of integrating existing benchmark and testing suites into abrek, like 0xbench and android CTS. For now in my thought, if these benchmark/testing can integrate with abrek, then it should be no much problem or extra effort for integrating further into LAVA.
As discussed before, I think abrek is probably not the right choice for running tests under android. Abrek was designed to work on linaro images running python in the client, specifically it was designed to run under Linaro or Ubuntu images, before we had even considered android at all. We probably need some kind of mechanism for running the tests from the server over adb instead, then outputting the results into a json bundle that the dashboard can understand.
we talked about this on IRC. I pointed Jeremy to our lp:lava branch and explained to him the theory of how things work. We agreed to first write a few example job descriptions and then decide what features a generic testsuite framework would require (if we need this at all). Also we might do a prototype to learn how things would work if we implemented the actions for android deploy, boot and test run.
Though I have some questions regarding to validation.
How to detect the early fail through serial console? like detecting the failure during kernel booting stage also before running init or getting the shell. I think this will be a job issued by the dispatcher though I don't know how this checking mechanism will be done. Would there be anything here needed to do specifically for Android?
Right, the dispatcher currently handles this. We are using pexpect to drive things over the serial console. So once the system is booting, we expect to see a valid shell prompt within some reasonable timeout period. If we don't, then we know the boot failed. I expect the bit that detects the shell prompt may need to be changed somewhat for android. Perhaps for android we need to wait for a valid adb devices output? or a working adb shell?
yes, its a timeout thing. we should give the adb connect a reasonable timeout. In this way we can guess that the boot failed in a bad way as no network came up.
How to connect the devices in the farm? I checked the wiki page, Platform/Validation/Specs/HardwareSetup [1]. The network will be used. I am thinking what's needed for setting up a network environment for this? The device is needed to fetch a fixed ip by dhcp just after booting up? USB gadget is an alternative for Android and in most situation could be more convenient for personal testing. adb (Android Debug Bridge, running on host side) can connect to a device through USB or TCP/IP.
USB gadget is hard for us, if we have a lot of machines. Network would be better, but can we rely in it to be working? Maybe we just make that a requirement for running android tests. I don't currently have it set up, but my current thinking is that we want to define a new network for the validation machines, perhaps 10.1.x.x, and set up static assignments for them in a dhcp server running on the control node. This way, they could either be configured statically, or via dhcp.
yes, USB gadget is not that great for our current setup.
I like statically IP. How do we know which board would get which IP? One idea would be to set up a DNS for internal board names. e.g. panda01.internal.network would resolve to our preallocated static IP for board "panda01".
This could then be used by the dispatcher to resolve the IP and pass it to the board through kernel cmdline.