Hi All,
So I’ve spent a little time checking into how one might take advantage of ‘useful’ test cases which are part of various distro packages.
Specifically this often runs the make check or akin target.
These are often wired into the various package build systems that distros have. The nice thing about using this mechanism is: 1) the package already has been base lined. 2) packages run a range of completely useless tests for testing the kernel (like checking internal apis to a package) to things which actually trigger kernel activity like running valgrind or driving some network activity.
So in my case I’ve been using gentoo to dork around since gentoo has a pretty nice process and happily is able to email testing results. More about why this is a good attribute in a moment.
The use scenario ends up looking like:
RC released -> build/deploy new kernel -> emerge package_foo; email package_foo result; emerge package_bar; email package_bar result; post process results and integrate into qa-reports.
Where this might be viewed as kind of weird is this class of testing really doesn’t belong in LAVA. One doesn’t need (or want) to do whole fresh image deployment. I know what you’re thinking, the system will be dirty after a run is made so one MUST do a redeploy or the results will be invalid.
That might be the case for some environments, but it’s not the case for gentoo. Package build and test is already ‘clean’ for the purposes of a system under test because in the land of gentoo we sandbox. If we didn’t distro package testing would be incredibly painful. So really the only ‘danger’ is that the local environment does need to build and boot a to the kernel under test but that’s not a big deal, since if something fails to build/boot, well kernelci will complain. This isn’t about testing that type of situation.
Anyway I should have some test test 4.14 results in a few hours as some examples. I’m thinking an upstream report is probably super super simple, unless a regression pops up.
Comments?
Tom
On 21 November 2017 at 18:39, Tom Gall tom.gall@linaro.org wrote:
Hi All,
So I’ve spent a little time checking into how one might take advantage of ‘useful’ test cases which are part of various distro packages.
Specifically this often runs the make check or akin target.
These are often wired into the various package build systems that distros have. The nice thing about using this mechanism is:
- the package already has been base lined.
- packages run a range of completely useless tests for testing the kernel (like checking internal apis to a package) to things which actually trigger kernel activity like running valgrind or driving some network activity.
So in my case I’ve been using gentoo to dork around since gentoo has a pretty nice process and happily is able to email testing results. More about why this is a good attribute in a moment.
The use scenario ends up looking like:
RC released -> build/deploy new kernel -> emerge package_foo; email package_foo result; emerge package_bar; email package_bar result; post process results and integrate into qa-reports.
Where this might be viewed as kind of weird is this class of testing really doesn’t belong in LAVA. One doesn’t need (or want) to do whole fresh image deployment. I know what you’re thinking, the system will be dirty after a run is made so one MUST do a redeploy or the results will be invalid.
That might be the case for some environments, but it’s not the case for gentoo. Package build and test is already ‘clean’ for the purposes of a system under test because in the land of gentoo we sandbox. If we didn’t distro package testing would be incredibly painful. So really the only ‘danger’ is that the local environment does need to build and boot a to the kernel under test but that’s not a big deal, since if something fails to build/boot, well kernelci will complain. This isn’t about testing that type of situation.
Anyway I should have some test test 4.14 results in a few hours as some examples. I’m thinking an upstream report is probably super super simple, unless a regression pops up.
Comments?
For OE RPB, we've recently enabled similar feature (ptest) on 410c builds. We still use LAVA in our case and the rootfs grows significantly (which could be an issue for boards with 4G eMMC). It's been done only on 410c to see how useful it could be and gather some data for discussion.
Tom
Lts-dev mailing list Lts-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lts-dev
On Nov 21, 2017, at 10:55 AM, Fathi Boudra fathi.boudra@linaro.org wrote:
On 21 November 2017 at 18:39, Tom Gall tom.gall@linaro.org wrote:
Hi All,
So I’ve spent a little time checking into how one might take advantage of ‘useful’ test cases which are part of various distro packages.
Specifically this often runs the make check or akin target.
These are often wired into the various package build systems that distros have. The nice thing about using this mechanism is:
- the package already has been base lined.
- packages run a range of completely useless tests for testing the kernel (like checking internal apis to a package) to things which actually trigger kernel activity like running valgrind or driving some network activity.
So in my case I’ve been using gentoo to dork around since gentoo has a pretty nice process and happily is able to email testing results. More about why this is a good attribute in a moment.
The use scenario ends up looking like:
RC released -> build/deploy new kernel -> emerge package_foo; email package_foo result; emerge package_bar; email package_bar result; post process results and integrate into qa-reports.
Where this might be viewed as kind of weird is this class of testing really doesn’t belong in LAVA. One doesn’t need (or want) to do whole fresh image deployment. I know what you’re thinking, the system will be dirty after a run is made so one MUST do a redeploy or the results will be invalid.
That might be the case for some environments, but it’s not the case for gentoo. Package build and test is already ‘clean’ for the purposes of a system under test because in the land of gentoo we sandbox. If we didn’t distro package testing would be incredibly painful. So really the only ‘danger’ is that the local environment does need to build and boot a to the kernel under test but that’s not a big deal, since if something fails to build/boot, well kernelci will complain. This isn’t about testing that type of situation.
Anyway I should have some test test 4.14 results in a few hours as some examples. I’m thinking an upstream report is probably super super simple, unless a regression pops up.
Comments?
For OE RPB, we've recently enabled similar feature (ptest) on 410c builds.
ptest is dirty tho. It doesn’t sandbox.
We still use LAVA in our case and the rootfs grows significantly (which could be an issue for boards with 4G eMMC). It's been done only on 410c to see how useful it could be and gather some data for discussion.
That’s cool. To be useful on dev boards I think we’d really need sata based root filesystems or pretty close to it.
Where’s that socionext developer workstation :-)
Tom
Lts-dev mailing list Lts-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lts-dev
On 21 November 2017 at 17:46, Tom Gall tom.gall@linaro.org wrote:
On Nov 21, 2017, at 10:55 AM, Fathi Boudra fathi.boudra@linaro.org
wrote:
On 21 November 2017 at 18:39, Tom Gall tom.gall@linaro.org wrote:
Hi All,
So I’ve spent a little time checking into how one might take advantage
of ‘useful’ test cases which are part of various distro packages.
Specifically this often runs the make check or akin target.
These are often wired into the various package build systems that
distros have. The nice thing about using this mechanism is:
- the package already has been base lined.
- packages run a range of completely useless tests for testing the
kernel (like checking internal apis to a package) to things which actually trigger kernel activity like running valgrind or driving some network activity.
So in my case I’ve been using gentoo to dork around since gentoo has a
pretty nice process and happily is able to email testing results. More about why this is a good attribute in a moment.
The use scenario ends up looking like:
RC released -> build/deploy new kernel -> emerge package_foo; email
package_foo result; emerge package_bar; email package_bar result; post process results and integrate into qa-reports.
The question here is whether this actually needs to be done on real hardware and if it does, does it actually matter *what* real hardware is used? Most distro-level in-build checks are agnostic about the specific kernel or specific hardware involved as long as the kernel is reasonably recent and the hardware is an architecture which is understood by the distro utilities like binutils, gcc and coreutils.
This kind of testing works nicely in tools like Jenkins and Travis simply because all the tests can be treated equally and run on an arbitrary machine running a stable kernel and stable rootfs, setup in a temporary directory and wrapped in a script that cleans things away at the end. There are questions about ensuring that the build dependencies are available but there are tools like sbuild and chroot (or containers) which can solve those problems. Distributions have been doing these things without needing to care about the specific hardware for years. So, yes, this isn't really something which plays to LAVA's strengths. We can do it there and we can just as easily do it elsewhere as what is being done is actually quite close to the idea of a build farm.
Where this might be viewed as kind of weird is this class of testing
really doesn’t belong in LAVA. One doesn’t need (or want) to do whole fresh image deployment. I know what you’re thinking, the system will be dirty after a run is made so one MUST do a redeploy or the results will be invalid.
That might be the case for some environments, but it’s not the case for
gentoo. Package build and test is already ‘clean’ for the purposes of a system under test because in the land of gentoo we sandbox. If we didn’t distro package testing would be incredibly painful. So really the only ‘danger’ is that the local environment does need to build and boot a to the kernel under test but that’s not a big deal, since if something fails to build/boot, well kernelci will complain. This isn’t about testing that type of situation.
Anyway I should have some test test 4.14 results in a few hours as some
examples. I’m thinking an upstream report is probably super super simple, unless a regression pops up.
Comments?
For OE RPB, we've recently enabled similar feature (ptest) on 410c builds.
ptest is dirty tho. It doesn’t sandbox.
We still use LAVA in our case and the rootfs grows significantly (which could be an issue for boards with 4G eMMC). It's been done only on 410c to see how useful it could be and gather some data for discussion.
That’s cool. To be useful on dev boards I think we’d really need sata based root filesystems or pretty close to it.
Where’s that socionext developer workstation :-)
Quite. These kinds of tests really do benefit from fast filesystems, lots of RAM and fast, multi-core CPUs. Generally, there is no effect on the results whether the build was done on a 4Gb eMMC db410c or a 128 core beast with 128Gb RAM and 3Tb of SATA. What does matter is the speed of getting those results.
Tom
Lts-dev mailing list Lts-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lts-dev
Lts-dev mailing list Lts-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lts-dev
-
Fathi Boudra -
Neil Williams -
Tom Gall