Hey everyone.
I'm trying to figure out a way to get connectivity manager tests enabled and pushed to LAVA. I read the code briefly (please correct any misconceptions I may have) and I see a few steps that need to be taken to achieve that.
1) We need to build them into our images. That's the obvious bit. It may be done already, I'm still digging through the code and the build system.
2) We need to script them so that they can execute given the correct environment. This part feels rather simple as there are explicit ADB commands in the comments throughout that code. I will try to build a comprehensive list of tests with notes and update you as I go. Still is should be relatively easy to script the execution
3) We need to work with the LAVA team to prepare a proper environment. So far this seems to be the biggest technical challenge. From my shallow understanding of the code we'd have to setup a few wireless access points with various configurations (or alternate the configuration on a single access point but that seems harder and more error prone) and see if we can get repeatable outcome from those connection tests.
The thing that worries me is the susceptibility to background noise / other transmissions / etc. It seems that most "interesting" tests are related to wireless communication. I wonder what is your opinion on this. Should we just enable all the tests that we technically can and ignore the fact that our test environment is not isolated?
Thanks for the feedback ZK
On Wed, Jun 13, 2012 at 3:58 PM, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
I'm trying to figure out a way to get connectivity manager tests enabled and pushed to LAVA. I read the code briefly (please correct any misconceptions I may have) and I see a few steps that need to be taken to achieve that.
- We need to build them into our images. That's the obvious bit. It may be
done already, I'm still digging through the code and the build system.
On this, I did a build that just included _all_ AOSP tests a month ago or so. At connect Zach and me discussed to move all our images to include all AOSP tests default. For that we have to set TARGET_BUILD_VARIANT=test (check out: https://android-build.linaro.org/builds/~asac/asac-alltests/#build=2).
On 13 June 2012 09:08, Alexander Sack asac@linaro.org wrote:
On Wed, Jun 13, 2012 at 3:58 PM, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
I'm trying to figure out a way to get connectivity manager tests enabled and pushed to LAVA. I read the code briefly (please correct any misconceptions I may have) and I see a few steps that need to be taken to achieve that.
- We need to build them into our images. That's the obvious bit. It may be
done already, I'm still digging through the code and the build system.
On this, I did a build that just included _all_ AOSP tests a month ago or so. At connect Zach and me discussed to move all our images to include all AOSP tests default. For that we have to set TARGET_BUILD_VARIANT=test (check out: https://android-build.linaro.org/builds/~asac/asac-alltests/#build=2).
Yeah, I think the first step is to get the tests build workin on all baselines. Bero was taking a look at this. Bero, we're we pretty close on this?
-- Alexander Sack Technical Director, Linaro Platform Teams http://www.linaro.org | Open source software for ARM SoCs http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
Hi Zygmunt,
I may can provide some information here.
1. The Connectivity Manager APK has been integrated to all Android images, and you can launch it via command line, which is described on this link:
https://wiki.linaro.org/Platform/QA/TestCases/Android#android-net-Connectivi...
2. I remember that Yongqin did some experiments before, which proved that this test suite can be run in LAVA, he may would like to give you more details.
3. Some test functions need to be commented out according to our board feature status, for example, those 3G tests. Otherwise, it will block the rest of tests. To find out which test function should be deleted, you have to look into the code.
4. The source code need to be read carefully and understood well since we need to figure out what every test function does. Some of them may also need to be modified to adapt to our board.
5. Once 3 & 4 finished, then we can integrated this test suite with those test functions we want, and pass the test result to LAVA to display. The test result of this APK is already well-formatted, so it won't be a big deal.
FYI.
Thank you.
BR Botao Sun
On Wed, Jun 13, 2012 at 11:58 PM, Zygmunt Krynicki < zygmunt.krynicki@linaro.org> wrote:
Hey everyone.
I'm trying to figure out a way to get connectivity manager tests enabled and pushed to LAVA. I read the code briefly (please correct any misconceptions I may have) and I see a few steps that need to be taken to achieve that.
- We need to build them into our images. That's the obvious bit. It may
be done already, I'm still digging through the code and the build system.
- We need to script them so that they can execute given the correct
environment. This part feels rather simple as there are explicit ADB commands in the comments throughout that code. I will try to build a comprehensive list of tests with notes and update you as I go. Still is should be relatively easy to script the execution
- We need to work with the LAVA team to prepare a proper environment. So
far this seems to be the biggest technical challenge. From my shallow understanding of the code we'd have to setup a few wireless access points with various configurations (or alternate the configuration on a single access point but that seems harder and more error prone) and see if we can get repeatable outcome from those connection tests.
The thing that worries me is the susceptibility to background noise / other transmissions / etc. It seems that most "interesting" tests are related to wireless communication. I wonder what is your opinion on this. Should we just enable all the tests that we technically can and ignore the fact that our test environment is not isolated?
Thanks for the feedback ZK
-- Zygmunt Krynicki Linaro Validation Team s/Validation/Android/
______________________________**_________________ linaro-android mailing list linaro-android@lists.linaro.**org linaro-android@lists.linaro.org http://lists.linaro.org/**mailman/listinfo/linaro-**androidhttp://lists.linaro.org/mailman/listinfo/linaro-android
Also, I have some test result output files I saved before, which was run by Abhishek I remember. Refer to attachment to get them, then you may know a general picture of it.
Thanks.
BR Botao Sun
On Thu, Jun 14, 2012 at 12:20 AM, Botao Sun botao.sun@linaro.org wrote:
Hi Zygmunt,
I may can provide some information here.
- The Connectivity Manager APK has been integrated to all Android images,
and you can launch it via command line, which is described on this link:
https://wiki.linaro.org/Platform/QA/TestCases/Android#android-net-Connectivi...
- I remember that Yongqin did some experiments before, which proved that
this test suite can be run in LAVA, he may would like to give you more details.
- Some test functions need to be commented out according to our board
feature status, for example, those 3G tests. Otherwise, it will block the rest of tests. To find out which test function should be deleted, you have to look into the code.
- The source code need to be read carefully and understood well since we
need to figure out what every test function does. Some of them may also need to be modified to adapt to our board.
- Once 3 & 4 finished, then we can integrated this test suite with those
test functions we want, and pass the test result to LAVA to display. The test result of this APK is already well-formatted, so it won't be a big deal.
FYI.
Thank you.
BR Botao Sun
On Wed, Jun 13, 2012 at 11:58 PM, Zygmunt Krynicki < zygmunt.krynicki@linaro.org> wrote:
Hey everyone.
I'm trying to figure out a way to get connectivity manager tests enabled and pushed to LAVA. I read the code briefly (please correct any misconceptions I may have) and I see a few steps that need to be taken to achieve that.
- We need to build them into our images. That's the obvious bit. It may
be done already, I'm still digging through the code and the build system.
- We need to script them so that they can execute given the correct
environment. This part feels rather simple as there are explicit ADB commands in the comments throughout that code. I will try to build a comprehensive list of tests with notes and update you as I go. Still is should be relatively easy to script the execution
- We need to work with the LAVA team to prepare a proper environment. So
far this seems to be the biggest technical challenge. From my shallow understanding of the code we'd have to setup a few wireless access points with various configurations (or alternate the configuration on a single access point but that seems harder and more error prone) and see if we can get repeatable outcome from those connection tests.
The thing that worries me is the susceptibility to background noise / other transmissions / etc. It seems that most "interesting" tests are related to wireless communication. I wonder what is your opinion on this. Should we just enable all the tests that we technically can and ignore the fact that our test environment is not isolated?
Thanks for the feedback ZK
-- Zygmunt Krynicki Linaro Validation Team s/Validation/Android/
______________________________**_________________ linaro-android mailing list linaro-android@lists.linaro.**org linaro-android@lists.linaro.org http://lists.linaro.org/**mailman/listinfo/linaro-**androidhttp://lists.linaro.org/mailman/listinfo/linaro-android
On 13 June 2012 22:20, Botao Sun botao.sun@linaro.org wrote:
Hi Zygmunt,
I may can provide some information here.
- The Connectivity Manager APK has been integrated to all Android images,
and you can launch it via command line, which is described on this link:
https://wiki.linaro.org/Platform/QA/TestCases/Android#android-net-Connectivi...
- I remember that Yongqin did some experiments before, which proved that
this test suite can be run in LAVA, he may would like to give you more details.
I just verified that the wifi and the eth can coexist, so in theory it should be able to run on LAVA.
But when I run the test that time, there are many test errors, not only the test failures. So we don't know how to prepare the environment to make the test runs well (even with some failures) on lava. So I suggest that first make this test works well as a test tool on android, then consider to integrate it into lava. IMO, the integration into lava should be easy to do comparing to making the test itself runs well.
Thanks, Yongqin Liu
- Some test functions need to be commented out according to our board
feature status, for example, those 3G tests. Otherwise, it will block the rest of tests. To find out which test function should be deleted, you have to look into the code.
- The source code need to be read carefully and understood well since we
need to figure out what every test function does. Some of them may also need to be modified to adapt to our board.
- Once 3 & 4 finished, then we can integrated this test suite with those
test functions we want, and pass the test result to LAVA to display. The test result of this APK is already well-formatted, so it won't be a big deal.
FYI.
Thank you.
BR Botao Sun
On Wed, Jun 13, 2012 at 11:58 PM, Zygmunt Krynicki < zygmunt.krynicki@linaro.org> wrote:
Hey everyone.
I'm trying to figure out a way to get connectivity manager tests enabled and pushed to LAVA. I read the code briefly (please correct any misconceptions I may have) and I see a few steps that need to be taken to achieve that.
- We need to build them into our images. That's the obvious bit. It may
be done already, I'm still digging through the code and the build system.
- We need to script them so that they can execute given the correct
environment. This part feels rather simple as there are explicit ADB commands in the comments throughout that code. I will try to build a comprehensive list of tests with notes and update you as I go. Still is should be relatively easy to script the execution
- We need to work with the LAVA team to prepare a proper environment. So
far this seems to be the biggest technical challenge. From my shallow understanding of the code we'd have to setup a few wireless access points with various configurations (or alternate the configuration on a single access point but that seems harder and more error prone) and see if we can get repeatable outcome from those connection tests.
The thing that worries me is the susceptibility to background noise / other transmissions / etc. It seems that most "interesting" tests are related to wireless communication. I wonder what is your opinion on this. Should we just enable all the tests that we technically can and ignore the fact that our test environment is not isolated?
Thanks for the feedback ZK
-- Zygmunt Krynicki Linaro Validation Team s/Validation/Android/
______________________________**_________________ linaro-android mailing list linaro-android@lists.linaro.**org linaro-android@lists.linaro.org http://lists.linaro.org/**mailman/listinfo/linaro-**androidhttp://lists.linaro.org/mailman/listinfo/linaro-android
On Thu, Jun 14, 2012 at 3:42 AM, YongQin Liu yongqin.liu@linaro.org wrote:
On 13 June 2012 22:20, Botao Sun botao.sun@linaro.org wrote:
Hi Zygmunt,
I may can provide some information here.
- The Connectivity Manager APK has been integrated to all Android images,
and you can launch it via command line, which is described on this link:
https://wiki.linaro.org/Platform/QA/TestCases/Android#android-net-Connectivi...
- I remember that Yongqin did some experiments before, which proved that
this test suite can be run in LAVA, he may would like to give you more details.
I just verified that the wifi and the eth can coexist, so in theory it should be able to run on LAVA.
But when I run the test that time, there are many test errors, not only the test failures. So we don't know how to prepare the environment to make the test runs well (even with some failures) on lava. So I suggest that first make this test works well as a test tool on android, then consider to integrate it into lava. IMO, the integration into lava should be easy to do comparing to making the test itself runs well.
can you file a bug and post the error/log you see?
On 14 June 2012 09:48, Alexander Sack asac@linaro.org wrote:
On Thu, Jun 14, 2012 at 3:42 AM, YongQin Liu yongqin.liu@linaro.org wrote:
On 13 June 2012 22:20, Botao Sun botao.sun@linaro.org wrote:
Hi Zygmunt,
I may can provide some information here.
- The Connectivity Manager APK has been integrated to all Android
images,
and you can launch it via command line, which is described on this link:
https://wiki.linaro.org/Platform/QA/TestCases/Android#android-net-Connectivi...
- I remember that Yongqin did some experiments before, which proved
that
this test suite can be run in LAVA, he may would like to give you more details.
I just verified that the wifi and the eth can coexist, so in theory it should be able to run on LAVA.
But when I run the test that time, there are many test errors, not only
the
test failures. So we don't know how to prepare the environment to make the test runs
well
(even with some failures) on lava. So I suggest that first make this test works well as a test tool on
android,
then consider to integrate it into lava. IMO, the integration into lava should be easy to do comparing to making
the
test itself runs well.
can you file a bug and post the error/log you see?
OK, I will try to reproduce it today and report the bug.
Thanks, Yongqin Liu
On 14 June 2012 09:48, Alexander Sack asac@linaro.org wrote:
On Thu, Jun 14, 2012 at 3:42 AM, YongQin Liu yongqin.liu@linaro.org wrote:
On 13 June 2012 22:20, Botao Sun botao.sun@linaro.org wrote:
Hi Zygmunt,
I may can provide some information here.
- The Connectivity Manager APK has been integrated to all Android
images,
and you can launch it via command line, which is described on this link:
https://wiki.linaro.org/Platform/QA/TestCases/Android#android-net-Connectivi...
- I remember that Yongqin did some experiments before, which proved
that
this test suite can be run in LAVA, he may would like to give you more details.
I just verified that the wifi and the eth can coexist, so in theory it should be able to run on LAVA.
But when I run the test that time, there are many test errors, not only
the
test failures. So we don't know how to prepare the environment to make the test runs
well
(even with some failures) on lava. So I suggest that first make this test works well as a test tool on
android,
then consider to integrate it into lava. IMO, the integration into lava should be easy to do comparing to making
the
test itself runs well.
can you file a bug and post the error/log you see?
The bug is https://bugs.launchpad.net/linaro-android/+bug/1012983
Thanks, Yongqin Liu.
On 13 June 2012 08:58, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
I'm trying to figure out a way to get connectivity manager tests enabled and pushed to LAVA. I read the code briefly (please correct any misconceptions I may have) and I see a few steps that need to be taken to achieve that.
- We need to build them into our images. That's the obvious bit. It may be
done already, I'm still digging through the code and the build system.
- We need to script them so that they can execute given the correct
environment. This part feels rather simple as there are explicit ADB commands in the comments throughout that code. I will try to build a comprehensive list of tests with notes and update you as I go. Still is should be relatively easy to script the execution
When I looked through the Connectivity Manager tests it looked like we could execute parts of it, so it may be that we need to decompose the test suite a bit, i.e. only execute tests when we have infrastructure for the tests then run those tests manually to see what the infrastructure needs may be.
- We need to work with the LAVA team to prepare a proper environment. So
far this seems to be the biggest technical challenge. From my shallow understanding of the code we'd have to setup a few wireless access points with various configurations (or alternate the configuration on a single access point but that seems harder and more error prone) and see if we can get repeatable outcome from those connection tests.
The thing that worries me is the susceptibility to background noise / other transmissions / etc. It seems that most "interesting" tests are related to wireless communication. I wonder what is your opinion on this. Should we just enable all the tests that we technically can and ignore the fact that our test environment is not isolated?
Yeah, maybe we could mark off a few "wireless test only" LAVA targets.
Thanks for the feedback ZK
-- Zygmunt Krynicki Linaro Validation Team s/Validation/Android/
On Wed, Jun 13, 2012 at 11:16 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 13 June 2012 08:58, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
I'm trying to figure out a way to get connectivity manager tests enabled and pushed to LAVA. I read the code briefly (please correct any misconceptions I may have) and I see a few steps that need to be taken to achieve that.
- We need to build them into our images. That's the obvious bit. It may be
done already, I'm still digging through the code and the build system.
- We need to script them so that they can execute given the correct
environment. This part feels rather simple as there are explicit ADB commands in the comments throughout that code. I will try to build a comprehensive list of tests with notes and update you as I go. Still is should be relatively easy to script the execution
When I looked through the Connectivity Manager tests it looked like we could execute parts of it, so it may be that we need to decompose the test suite a bit, i.e. only execute tests when we have infrastructure for the tests then run those tests manually to see what the infrastructure needs may be.
- We need to work with the LAVA team to prepare a proper environment. So
far this seems to be the biggest technical challenge. From my shallow understanding of the code we'd have to setup a few wireless access points with various configurations (or alternate the configuration on a single access point but that seems harder and more error prone) and see if we can get repeatable outcome from those connection tests.
The thing that worries me is the susceptibility to background noise / other transmissions / etc. It seems that most "interesting" tests are related to wireless communication. I wonder what is your opinion on this. Should we just enable all the tests that we technically can and ignore the fact that our test environment is not isolated?
Yeah, maybe we could mark off a few "wireless test only" LAVA targets.
we can do that. For now let's get things going and once we see real noise problems figure out how to best solve the problem.
linaro-android@lists.linaro.org
-
Alexander Sack -
Botao Sun -
YongQin Liu -
Zach Pfeffer -
Zygmunt Krynicki