Platform tests for kernel CI-loop

[Deepak Saxena]

Hi all,

I have created a blueprint [1] for connect to discuss the
topic of platform level testing and want us to have some
sense of what we want before going into a face to face
discussion. I would like all the landing team leads
to attend this session, so I've marked you as essential.

The overall problem we are trying to solve is how
do we ensure that hardware enablement (USB,
MMC, etc) does not break across kernel versions.
We've had several cases where a patch got merged
into kernel.org that broke a device driver and
we didn't catch it until just before our release.
This causes us to scramble and reduces the
quality of the work we're delivering to our members.
With the CI loop in place, we now have the opportunity
to catch these type of issues early on, before they
even make it into the -rc kernels. My hope is that
by the end of the summit session, we have a enough
information to go back and develop a high level
roadmap of  test cases to deliver. The questions I'd
like us to think about before the connect session include:

1) What are the different devices on each board that
    we want to test and how do we test them?

   We need to go through board by board and determine
   which I/O devices can be tested and come up with
   common methods to test them across all our existing
   platforms. One of the questions that comes up for me
   is what level of testing do we want? We can do simple
   discovery tests such as look for sysfs device nodes and
   poke at values in there or we can right small applications
   that test specific functionality (mounting block devices
   and running benchmark tests and also testing ioctls for
   example).

2) Who develops these tests?

    I think the answer to this is to have Landing Team
    engineers developing the board-specific tests with a
    KWG engineer assigned to coordinate overall direction
    of the work.

3) Do we integrate those tests into an existing
    framework (LTP for example) or develop a
    separate framework for these tests? Related
    to this question is whether some of the vendors
    will provide us test cases we can just pull into
    Lava?

~Deepak

[1] https://blueprints.launchpad.net/linux-linaro/+spec/linaro-kernel-platform-testing