Here's a list of features that I think would be useful for the arm-soc(?) auto-builder/tester systems. This is a big pie-in-the-sky wish-list so I'm well aware that I'm asking someone to implement something useful and complex for me for free, and hence I may well not get want I want:-) Still, there are some fun ideas below, I hope!

* Per-build/test email distribution list.

If there is a change to e.g. the tegra_defconfig build status (first broken build, first successful build, etc.), I'd like to be emailed about this specifically, likely in a separate email. That way, I can filter these actionable messages separately from the general list content, so that I generally don't need to read most of the list messages, unless I'm notified of a specific problem.

This could be implemented by adding me to To:/Cc: on the existing messages in addition to sending them to the list, rather than sending a separate notification. However, this still means that I need to read through the whole email to find the specific issue that I'm interested in.

* Web-based UI, with history and filtering.

Email is very useful for notification of changes. However, if I want to quickly check the current status, or research historic build status, I find a web UI much easier. Some public examples that I use:

http://nvt.pwsan.com/experimental/linux-next/testlogs/

http://kisskb.ellerman.id.au/kisskb/matrix/ http://kisskb.ellerman.id.au/kisskb/config/338/

I'd love the ability to restrict the history so I can see just one architecture, compiler, defconfig, branch, board, test-case, or somewhat arbitrary combinations thereof. Perhaps throw HW configuration into the mix too (e.g. single-CPU vs. SMP vs. a specific core count, different RAM-size bands, presence of certain HW such as SPI flash, etc.)

* Archival of full build/test logs

Perhaps not so much for build problems, but it can be useful to have an entire console log for test failures. Preferably, these would all be linked from the web UI filtered history view.

* More test-cases

Boot test catches a lot of issues, but I'd love to run tests such as the following:

+ Reboot + Shutdown + System suspend/resume + CPU hotplug + check that cpuidle is activating + force cpufreq to min/max to check it works, or similar + dd from SD or eMMC, perhaps check for performance changes + Play audio, and at least check that it takes the right amount of time; doesn't hang or cause kernel oops. Perhaps also loop it back and validate that capture roughly approximates playback sound (e.g. FFT a sine wave or similar?) + Test I2C transactions (likely tested by playing audio on many systems, but some explicit i2cget's wouldn't hurt. + USB, e.g. lsusb, ping over a USB NIC, dd from USB mass storage? + hexdump an SPI flash, validate against expected content. Perhaps do a write test too on a specific flash region? + Verify HDMI signal is generated. Perhaps use xrandr/... to test changing video modes + Test WiFi, Bluetooth connectivity + Test PCIe devices are detected, and work + crypto driver tests + kexec

I run all of those manually sometimes. Some more often than others. Not often enough though in many cases! It'd be awesome if they ran automatically e.g. nightly, or on every commit.

I would love each test to be a standalone script file that developers could just check out from git and run themselves to reproduce problems, without having to set up their build/target systems identically to the auto-build/test systems. Hopefully they would report status in a standard way to ease integration into the test framework.

Some ability to run a stress-test (manually, automatically but less often?) to weed out problems that happen with low frequency.

* Automated bisect

If a build or test fails, bisect to find the problematic commit, automatically, and email people about the result.

* "Virtual" submissions

Send the system a git URL, and have it run the same set of builds/tests as it would if those commits appeared in a branch it usually tested automatically. Very useful for testing something across a range of HW before sending it or applying it, so problems can be caught early rather than once something is in linux-next.

A command-line or email-based interface to do this, for ease-of-use, perhaps in addition to a web form, which would probably lower the barrier to usage for some people.

There may be some security issues here re: letting arbitrary people compile/run kernel code on the test systems. Implementing ACLs would probably be annoying. Perhaps branches would have to be pushed to some git server, which would implement the ACLs? Say, trust all kernel.org repos?

* Packaged project

The ability to install the system locally, e.g.:

+ My own system can run tests that the main system might not be so interested in. This way, I get to use the same test infra-structure for everything, to avoid duplicating it.

+ My system could run extreme levels of stress testing.

+ My system could be more open in who I accept "virtual" submissions from for Tegra HW (e.g anyone @nvidia.com or using specific internal git servers/branches perhaps).

+ My system can use boards that aren't available to the main system, or that the main system isn't interested in. This will give me more coverage. For example, I could test every single Tegra board to enable extreme paranoia, rather than a representative sample in the main system.

* Apply similar testing to U-Boot, other relevant bootloader, or other useful projects!

* The human and computer resources to back this up

Implementing much of the above is a lot of work to code, and support. It'll require quite a bit of HW to run on.

(Note that this not a comment on the current arm-soc builder or other public systems I linked to above, but some internal system...)