Hi all,
I have created a blueprint [1] for connect to discuss the topic of platform level testing and want us to have some sense of what we want before going into a face to face discussion. I would like all the landing team leads to attend this session, so I've marked you as essential.
The overall problem we are trying to solve is how do we ensure that hardware enablement (USB, MMC, etc) does not break across kernel versions. We've had several cases where a patch got merged into kernel.org that broke a device driver and we didn't catch it until just before our release. This causes us to scramble and reduces the quality of the work we're delivering to our members. With the CI loop in place, we now have the opportunity to catch these type of issues early on, before they even make it into the -rc kernels. My hope is that by the end of the summit session, we have a enough information to go back and develop a high level roadmap of test cases to deliver. The questions I'd like us to think about before the connect session include:
1) What are the different devices on each board that we want to test and how do we test them?
We need to go through board by board and determine which I/O devices can be tested and come up with common methods to test them across all our existing platforms. One of the questions that comes up for me is what level of testing do we want? We can do simple discovery tests such as look for sysfs device nodes and poke at values in there or we can right small applications that test specific functionality (mounting block devices and running benchmark tests and also testing ioctls for example).
2) Who develops these tests?
I think the answer to this is to have Landing Team engineers developing the board-specific tests with a KWG engineer assigned to coordinate overall direction of the work.
3) Do we integrate those tests into an existing framework (LTP for example) or develop a separate framework for these tests? Related to this question is whether some of the vendors will provide us test cases we can just pull into Lava?
~Deepak
[1] https://blueprints.launchpad.net/linux-linaro/+spec/linaro-kernel-platform-t...