On 17 October 2017 at 01:14, Tom Gall <tom.gall@linaro.org> wrote:

This past RC cycle I think we exposed a weakness in what we have in LKFT where the ability to execute some key functional stacks in the system to drive the kernel would probably be useful for validation.

The networking bug involving dhclient for example.

So what if we used either Debian, Gentoo or akin that has a mechanism that has as part of it’s packaging system a test target for each package. Simplest build a package, runs ‘make test’ (or akin) for some key packages that exercises parts of the system that should help tickle the kernel in interesting ways to tease out regressions.

Debian has two types of test supported within the packaging:

* in-build tests - needs a full build environment on the buildd device and tests the binaries which have just been compiled rather than what is actually installed. (buildd.debian.org) - most of these tests are only run and maintained against Debian unstable (when the package is first built for the archive). Intermittent rebuilds are performed, again using Debian unstable. These tests are therefore only a snapshot against a particular set of packages and the focus is on ensuring that the package builds successfully. e.g. https://buildd.debian.org/status/package.php?p=dpkg

* autopkg-tests - designed to test that the installed package works against updated dependencies. Needs a bit of setup (generally a QEMU image or an LXC) (ci.debian.org) - these tests are also run against Debian unstable (as that is where newly updated dependencies turn up). Tests are run continually, whenever a dependency is updated. Rather than a build environment, these tests require the package to be installed with a few extra tools. e.g. https://ci.debian.net/packages/l/lava-server/unstable/amd64/

Neither test covers 100% of packages (separately or combined) - in-build tests tend to be present in many packages written in C or Perl, autopkgtests tend to be of more interest with packages with a wide range of dependencies. (e.g. LAVA uses autopkgtest.) Particular note is that each of these run against Debian unstable which is a constantly moving target, not the stable release which is what we will tend to deploy.

Thus wouldn’t work on modest boards but the socionext or Juno boards could probably work fine.

Thoughts?

Carefully chosen, these tests will be useful but there isn't going to be a blanket we can put over all of the functional stacks. I'm uncertain how many of these tests will exercise the kernel as many will try quite hard to isolate the build/test environment from the runtime environment in the interests of reproducibility.

Some work will be required to write new tests in the gaps - between packages and the kernel. It is likely that organisations like Debian would consider these tests useful, once created.

Neil Williams
=============
neil.williams@linaro.org
http://www.linux.codehelp.co.uk/