Hi
Marcus here has worked on CTS for 4.4 quite a bit so here is his reply.
We had to do some manual patching on 4.2 so that both the non-interactive mode and the disable-reboot options didn’t break the execution but it seems like those problems are gone now. Just in case you have any such issues we can dig through our old stuff...
From: Marcus Oakland [mailto:marcus.oakland@arm.com] Sent: 29 May 2014 12:13 To: Ramin Zaghi Subject: Re: Any comments about CTS !?
Invoke CTS tests with the
--disable-reboot
option to stop cts-tradefed rebooting.
For example:
$ echo | cts-tradefed run cts --class libcore.java.lang.reflect.ProxyTest --method testMethodsImplementedByFarIndirectInterface --disable-reboot
That runs a single CTS test method (libcore.java.lang.reflect.ProxyTest#testMethodsImplementedByFarIndirectInterface in this case) in "Non-interactive mode" (so that cts-tradefed exits when the test has completed (caused by the use of "echo | " at the start of the invocation line). Use of "Non-interactive mode" means that scripts can be used to invoke the tests.
You can run all methods in a CTS test by specifying the class (e.g. --class libcore.java.lang.reflect.ProxyTest) but omitting the method.
You can run all CTS test classes in a package by, for example:
$ echo | cts-tradefed run cts --package android.core.tests.libcore.package.libcore --disable-reboot
You can run all CTS test packages in a plan by, for example:
$ echo | cts-tradefed run cts --plan Java --disable-reboot
On the model running the CTS tests is sloooow. Figures I have from testing for 4.3 on the android64_2013Q4 branch, the Java plan for example took 9:21:34, the VM-TF plan took 16:00:10, and the Android plan took 26:32:31. However the model used to "die" fairly frequently and tests had to be restarted.
There are more CTS tests in the complete suite for 4.4 than there were in 4.3 (43,021 compared with 24,491).
From: linaro-android-bounces@lists.linaro.org [mailto:linaro-android-bounces@lists.linaro.org] On Behalf Of YongQin Liu Sent: 29 May 2014 02:11 To: John Stultz Cc: linaro-android@lists.linaro.org Subject: Re: CTS help?
Hi, John
On 29 May 2014 08:10, John Stultz <john.stultz@linaro.orgmailto:john.stultz@linaro.org> wrote: On 05/28/2014 05:04 PM, YongQin Liu wrote:
Hi, John
Since you are running in an emulated environment, you probably need some changing about timeout.
Here are the 2 packages we are using for juice with some timeout changes:
For 32 bit android userspace: http://testdata.validation.linaro.org/cts/android-cts-32bit-master.zip
For 64 bit android userspace: http://testdata.validation.linaro.org/cts/android-cts-aarch64-master.zip
After you got the cts-tf prompt via cts-tradefed, please run the "run cts" subcommands with "--disable-reboot" option specified. you can also have "--skip-device-info" specified since you probably already know the device information.
Thanks for the tips! Also any rough idea how long a basic CTS run should take? On emulator like fastmodels, it will take very longer time than with hardware. Here are some information about the time it will run I recorded before. https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AqtM2gThV-VudEh6R1...
Except the android.core.tests.libcore.package.orghttp://android.core.tests.libcore.package.org package, others seems still can be referenced. The android.core.tests.libcore.package.orghttp://android.core.tests.libcore.package.org will take 1 or 2 days now, so we split it into 7 smaller sub packages in the above 2 mentioned cts packages.
I'm hoping to find something that stresses the hardware/kernel side of things and finishes in a few short hours or less. you can run "list packages" to see all the packages supported. For hardware, you can select the packages from the name I think.(but for emulator...) For kernel, IMO, android.core.tests.libcore.package.libcore and android.core.tests.libcore.package.orghttp://android.core.tests.libcore.package.org are better to be used for kernel test, since they can cover most basic tests(but this is just from my own perspective), but they will take long time:(
-- Best Regards, Yongqin Liu --------------------------------------------------------------- #mailing list linaro-android@lists.linaro.orgmailto:linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-android
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590 ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782