Hi, All
Here just some thought about the implementation of black-box test. If you have any ideas, or something I missed, please give a comment. Anything will be appreciated.
------------------------------------------------------------ *Glue between lava and android* In android there is a directory /data/blackbox_tesxt/, under it are TODO, TESTING, DONE 3 direcories.
- TODO: the flags for test that need to run will be put here - TESTING: the flags for test that are running will be put here. normally, there should be only one entry.In the future will be more entries when we support test execution via thread - DONE: the flags for test that have been completed will be put here
About the entry format, will use JSON or just key/value pair. but need to have the following two features 1. one item to indicate the command to run 2. other items used for pass information between android test tool and lava job
*Black-box test framework on Android* On android, a test framework will check the entries in TODO, and run the command indicated in the entry. Before the test is start to run, the framework will put the entry to TESTING, and after test finished will put the entry to DONE. when run the test command, this framework will run the command and pass the entry file as parameter.
The black-box test framework in android mainly do: 1. invoked after boot up and home screen is displayed. also charge for prepare the test environment like unlock screen, disable suspend 2. charging for invoking test command and changing the status of the each test
*Framework on LAVA*
Will have 2 actions 1. install_black-box_test 2. wait_black_test_finish will loop to check until there is no entry in TODO and TESTING also will check if the test framework is running, if it is not in ps and there are still test to be run, will invoke it to run. show the output of the running test 3. collect_test_result parse and upload to lava ------------------------------------------------------------
*Related BPs* 1. lava side: https://blueprints.launchpad.net/lava-dispatcher/+spec/black-box-test-action... 2. android-side: https://blueprints.launchpad.net/linaro-android/+spec/support-blackbox-test
Thanks, Yongqin Liu
On 07/12/2012 02:42 AM, YongQin Liu wrote:
Hi, All
Here just some thought about the implementation of black-box test. If you have any ideas, or something I missed, please give a comment. Anything will be appreciated.
*Glue between lava and android* In android there is a directory /data/blackbox_tesxt/, under it are TODO, TESTING, DONE 3 direcories.
- TODO: the flags for test that need to run will be put here
- TESTING: the flags for test that are running will be put here. normally, there should be only one entry. In the future will be more entries when we support test execution via thread
- DONE: the flags for test that have been completed will be put here
About the entry format, will use JSON or just key/value pair. but need to have the following two features
- one item to indicate the command to run
- other items used for pass information between android test tool and
lava job
*Black-box test framework on Android* On android, a test framework will check the entries in TODO, and run the command indicated in the entry. Before the test is start to run, the framework will put the entry to TESTING, and after test finished will put the entry to DONE. when run the test command, this framework will run the command and pass the entry file as parameter.
The black-box test framework in android mainly do:
- invoked after boot up and home screen is displayed. also charge for prepare the test environment like unlock screen,
disable suspend 2. charging for invoking test command and changing the status of the each test
*Framework on LAVA*
Will have 2 actions
- install_black-box_test
I can see why you are proposing this. However, I'm wondering if this is what Zach was envisioning? ie - I thought the assumption was the android build would come pre-populated with these things installed. However, maybe we need this so we can support something like an AOSP build?
I guess I have two questions: 1) can you describe what the install_black-box_test action would look like?
2) Zach - are you okay with this approach?
- wait_black_test_finish will loop to check until there is no entry in TODO and TESTING also will check if the test framework is running, if it is not in
ps and there are still test to be run, will invoke it to run. show the output of the running test 3. collect_test_result parse and upload to lava
*Related BPs*
- lava side:
https://blueprints.launchpad.net/lava-dispatcher/+spec/black-box-test-action... 2. android-side: https://blueprints.launchpad.net/linaro-android/+spec/support-blackbox-test
Thanks, Yongqin Liu
linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation
YongQin Liu yongqin.liu@linaro.org writes:
Hi, All
Here just some thought about the implementation of black-box test. If you have any ideas, or something I missed, please give a comment. Anything will be appreciated.
Thanks for sending this.
*Glue between lava and android* In android there is a directory /data/blackbox_tesxt/, under it are TODO, TESTING, DONE 3 direcories.
- TODO: the flags for test that need to run will be put here
- TESTING: the flags for test that are running will be put here.
normally, there should be only one entry.In the future will be more entries when we support test execution via thread
Do we actually want to support running more than one test at once? It doesn't seem like a super good idea to me.
- DONE: the flags for test that have been completed will be put here
About the entry format, will use JSON or just key/value pair. but need to have the following two features
- one item to indicate the command to run
- other items used for pass information between android test tool and lava
job
*Black-box test framework on Android* On android, a test framework will check the entries in TODO, and run the command indicated in the entry. Before the test is start to run, the framework will put the entry to TESTING, and after test finished will put the entry to DONE. when run the test command, this framework will run the command and pass the entry file as parameter.
The black-box test framework in android mainly do:
- invoked after boot up and home screen is displayed. also charge for prepare the test environment like unlock screen,
disable suspend 2. charging for invoking test command and changing the status of the each test
*Framework on LAVA*
Will have 2 actions
- install_black-box_test
Will this action have a list of tests to install?
- wait_black_test_finish will loop to check until there is no entry in TODO and TESTING also will check if the test framework is running, if it is not in ps
and there are still test to be run, will invoke it to run. show the output of the running test
Do we want to reboot between tests?
- collect_test_result parse and upload to lava
*Related BPs*
- lava side:
https://blueprints.launchpad.net/lava-dispatcher/+spec/black-box-test-action... 2. android-side: https://blueprints.launchpad.net/linaro-android/+spec/support-blackbox-test
Thanks, Yongqin Liu _______________________________________________ linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation
Cheers, mwh
On 12 July 2012 09:58, Andy Doan andy.doan@linaro.org wrote:
On 07/12/2012 02:42 AM, YongQin Liu wrote:
Hi, All
Here just some thought about the implementation of black-box test. If you have any ideas, or something I missed, please give a comment. Anything will be appreciated.
*Glue between lava and android*
In android there is a directory /data/blackbox_tesxt/, under it are TODO, TESTING, DONE 3 direcories.
TODO: the flags for test that need to run will be put here
TESTING: the flags for test that are running will be put here.
normally, there should be only one entry. In the future will be more entries when we support test execution via thread
DONE: the flags for test that have been completed will
be put here
About the entry format, will use JSON or just key/value pair. but need to have the following two features
- one item to indicate the command to run
- other items used for pass information between android test tool and
lava job
*Black-box test framework on Android*
On android, a test framework will check the entries in TODO, and run the command indicated in the entry. Before the test is start to run, the framework will put the entry to TESTING, and after test finished will put the entry to DONE. when run the test command, this framework will run the command and pass the entry file as parameter.
The black-box test framework in android mainly do:
- invoked after boot up and home screen is displayed. also charge for prepare the test environment like unlock screen,
disable suspend 2. charging for invoking test command and changing the status of the each test
*Framework on LAVA*
Will have 2 actions
- install_black-box_test
I can see why you are proposing this. However, I'm wondering if this is what Zach was envisioning? ie - I thought the assumption was the android build would come pre-populated with these things installed. However, maybe we need this so we can support something like an AOSP build?
I guess I have two questions:
can you describe what the install_black-box_test action would look like?
Zach - are you okay with this approach?
I think these are interesting ideas. I like the idea of the files.
I think all of this is actually handled in the blackbox itself and the only thing that LAVA does is call a well-known function, then wait some time t, then reset the unit and read the output file at a well-known location. Everything else we'd handle on target.
- wait_black_test_finish will loop to check until there is no entry in TODO and TESTING also will check if the test framework is running, if it is not in
ps and there are still test to be run, will invoke it to run. show the output of the running test 3. collect_test_result parse and upload to lava
*Related BPs*
- lava side:
https://blueprints.launchpad.net/lava-dispatcher/+spec/black-box-test-action... 2. android-side:
https://blueprints.launchpad.net/linaro-android/+spec/support-blackbox-test
Thanks, Yongqin Liu
linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation
Zach Pfeffer zach.pfeffer@linaro.org writes:
On 12 July 2012 09:58, Andy Doan andy.doan@linaro.org wrote:
On 07/12/2012 02:42 AM, YongQin Liu wrote:
Hi, All
Here just some thought about the implementation of black-box test. If you have any ideas, or something I missed, please give a comment. Anything will be appreciated.
*Glue between lava and android*
In android there is a directory /data/blackbox_tesxt/, under it are TODO, TESTING, DONE 3 direcories.
TODO: the flags for test that need to run will be put here
TESTING: the flags for test that are running will be put here.
normally, there should be only one entry. In the future will be more entries when we support test execution via thread
DONE: the flags for test that have been completed will
be put here
About the entry format, will use JSON or just key/value pair. but need to have the following two features
- one item to indicate the command to run
- other items used for pass information between android test tool and
lava job
*Black-box test framework on Android*
On android, a test framework will check the entries in TODO, and run the command indicated in the entry. Before the test is start to run, the framework will put the entry to TESTING, and after test finished will put the entry to DONE. when run the test command, this framework will run the command and pass the entry file as parameter.
The black-box test framework in android mainly do:
- invoked after boot up and home screen is displayed. also charge for prepare the test environment like unlock screen,
disable suspend 2. charging for invoking test command and changing the status of the each test
*Framework on LAVA*
Will have 2 actions
- install_black-box_test
I can see why you are proposing this. However, I'm wondering if this is what Zach was envisioning? ie - I thought the assumption was the android build would come pre-populated with these things installed. However, maybe we need this so we can support something like an AOSP build?
I guess I have two questions:
can you describe what the install_black-box_test action would look like?
Zach - are you okay with this approach?
I think these are interesting ideas. I like the idea of the files.
I think all of this is actually handled in the blackbox itself and the only thing that LAVA does is call a well-known function, then wait some time t, then reset the unit and read the output file at a well-known location. Everything else we'd handle on target.
But if we want to run blackbox tests on AOSP, we'll have to do _something_ in lava, right? Can that be as simple as installing an apk or something with adb?
Cheers, mwh
On 15 July 2012 17:18, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
Zach Pfeffer zach.pfeffer@linaro.org writes:
On 12 July 2012 09:58, Andy Doan andy.doan@linaro.org wrote:
On 07/12/2012 02:42 AM, YongQin Liu wrote:
Hi, All
Here just some thought about the implementation of black-box test. If you have any ideas, or something I missed, please give a comment. Anything will be appreciated.
*Glue between lava and android*
In android there is a directory /data/blackbox_tesxt/, under it are TODO, TESTING, DONE 3 direcories.
TODO: the flags for test that need to run will be put here
TESTING: the flags for test that are running will be put here.
normally, there should be only one entry. In the future will be more entries when we support test execution via thread
DONE: the flags for test that have been completed will
be put here
About the entry format, will use JSON or just key/value pair. but need to have the following two features
- one item to indicate the command to run
- other items used for pass information between android test tool and
lava job
*Black-box test framework on Android*
On android, a test framework will check the entries in TODO, and run the command indicated in the entry. Before the test is start to run, the framework will put the entry to TESTING, and after test finished will put the entry to DONE. when run the test command, this framework will run the command and pass the entry file as parameter.
The black-box test framework in android mainly do:
- invoked after boot up and home screen is displayed. also charge for prepare the test environment like unlock screen,
disable suspend 2. charging for invoking test command and changing the status of the each test
*Framework on LAVA*
Will have 2 actions
- install_black-box_test
I can see why you are proposing this. However, I'm wondering if this is what Zach was envisioning? ie - I thought the assumption was the android build would come pre-populated with these things installed. However, maybe we need this so we can support something like an AOSP build?
I guess I have two questions:
can you describe what the install_black-box_test action would look like?
Zach - are you okay with this approach?
I think these are interesting ideas. I like the idea of the files.
I think all of this is actually handled in the blackbox itself and the only thing that LAVA does is call a well-known function, then wait some time t, then reset the unit and read the output file at a well-known location. Everything else we'd handle on target.
But if we want to run blackbox tests on AOSP, we'll have to do _something_ in lava, right? Can that be as simple as installing an apk or something with adb?
Not really. We build AOSP from a copy of their manifest. It wouldn't be hard to just add the test stuff in.
Cheers, mwh
On 12 July 2012 22:58, Andy Doan andy.doan@linaro.org wrote:
On 07/12/2012 02:42 AM, YongQin Liu wrote:
Hi, All
Here just some thought about the implementation of black-box test. If you have any ideas, or something I missed, please give a comment. Anything will be appreciated.
------------------------------**------------------------------ *Glue between lava and android*
In android there is a directory /data/blackbox_tesxt/, under it are TODO, TESTING, DONE 3 direcories.
TODO: the flags for test that need to run will be put here
TESTING: the flags for test that are running will be put here.
normally, there should be only one entry. In the future will be more entries when we support test execution via thread
DONE: the flags for test that have been completed will
be put hereAbout the entry format, will use JSON or just key/value pair. but need to have the following two features
- one item to indicate the command to run
- other items used for pass information between android test tool and
lava job
*Black-box test framework on Android*
On android, a test framework will check the entries in TODO, and run the command indicated in the entry. Before the test is start to run, the framework will put the entry to TESTING, and after test finished will put the entry to DONE. when run the test command, this framework will run the command and pass the entry file as parameter.
The black-box test framework in android mainly do:
- invoked after boot up and home screen is displayed. also charge for prepare the test environment like unlock screen,
disable suspend 2. charging for invoking test command and changing the status of the each test
*Framework on LAVA*
Will have 2 actions
- install_black-box_test
I can see why you are proposing this. However, I'm wondering if this is what Zach was envisioning? ie - I thought the assumption was the android build would come pre-populated with these things installed. However, maybe we need this so we can support something like an AOSP build?
Yes, I think it's better we support the install action. Then we can install the test intended when we deploy android. Of course as you said above, if the android to the same things during build, then it is unnecessary to specified the install action.
I guess I have two questions:
- can you describe what the install_black-box_test action would look like?
I think at first we can make the install similar to the lava_android_test_install action now. Like this: { "command": "install_android_black_box_test", "parameters": { "test": "monkey", "command": ["monkey", $(OPTION), "500"], "option": "-s 5", "timeout": XXX, "reboots_allowed": 3 } } I'd like the result directory and indicator directory are set in some configure file, not specified in the action. With the above information, we could generate a simple indicator file only has the following line:
command="monkey -s 5 500"
And about the status whether the test is finished, we can use the directory structure I described above. writing the completion signal to the serial console would not be easy for some test.
Thanks, Yongqin Liu
linaro-validation@lists.linaro.org
-
Andy Doan -
Michael Hudson-Doyle -
YongQin Liu -
Zach Pfeffer