On 3 July 2017 at 21:50, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello Milosz,
I appreciate getting at least some response ;-). Some questions however could use a reply from LAVA team, I guess.
On Mon, 3 Jul 2017 13:34:49 +0100 Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
[]
jobs submit a number of tests to LAVA (via https://qa-reports.linaro.org/) for the following boards: arduino_101, frdm_k64f, frdm_kw41z, qemu_cortex_m3. Here's an example of cumulative test report for these platforms: https://qa-reports.linaro.org/lite/zephyr-upstream/tests/
That's really great! (Though the list of tests to run in LAVA seems to be hardcoded: https://git.linaro.org/ci/job/configs.git/tree/zephyr-upstream/submit_for_te...)
It is, as I wasn't really sure what to test. The build job needs to prepare the test templates to be submitted to LAVA. In case of zephyr each tests is a separate binary. So we end up with the number of file paths to substitute in the template. Hardcoding was the easiest thing to get things running. But I see no reason why it wouldn't be changed with some smarter code to discover the binaries. The problem with this approach is that some of these tests are just build time. They have no meaning when running on the board and need to be filter out somehow.
I see, that makes some sense. But thinking further, I'm not entirely sure about "build only" tests. Zephyr's sanitycheck test has such concept, but I'd imagine it comes from the following reasons: a) sanitycheck runs tests on QEMU, which has very bare hardware support, so many tests are not runnable; b) sanitycheck can operate on "samples", not just "tests", as sample can be interactive, etc. it makes sense to only build them, not run.
So, I'm not exactly sure about build-only tests on real HW boards. The "default" idea would be that they should run, but I imagine in reality, some may need to be filtered out. But then blacklisting would be better approach than whitelisting. And I'm not sure if Zephyr has concept of "skipped" tests which may be useful to handle hardware variations. (Well, I actually dunno if LAVA supports skipped tests!)
As far as I can tell they acutely run on the board, but usually output just 'Hello world!' or sth similar. As we discussed with Kumar, this is still OK. What Kumar requested (and I still didn't deliver) is that whenever the LAVA test job completes, the test should be considered 'passed'. So we wouldn't have to do any parsing of patterns. I'm not sure if that will work, but it's worth to try.
Anyway, these are rough ideas for the future. I've spent couple of weeks of munging with LITE CI setup, there're definitely some improvements, but also a Pandora box of other ideas and improvements to make. I'm wrapping up for now, but hope to look again in some time (definitely hope to look before the Connect, so we can discuss further steps there). In the meantime, I hope that more boards will be installed in the Lab and stability of them improves (so far they seem to be pretty flaky).
You're absolutely right. This is a pretty big task to work on and IMHO requires someone to work full time at least for couple of weeks. The second part is also true, the boards don't behave as they should. I guess Dave can elaborate more on that. I can only see the result - boards (frdm-kw41z) don't run the tests they're requested.
[]
- test: monitors:
- name: foo start: "" end: Hello, ZJS world! pattern: (?P<result>(PASS|FAIL))\s-\s(?P<test_case_id>\w+).
So, the "start" substring is empty, and perhaps matches a line output by a USB multiplexer or board bootloader. "End" substring is actually the expected single-line output. And "pattern" is unused (dunno if it can be dropped without def file syntax error). Is there a better way to handle single-line test output?
You're making a silent assumption that if there is a matching line, the test is passed. In case of other tests (zephyr unit tests), it's not the case. The 'start' matches some line which is displayed when zephyr is booting. End matches the line which is displayed after all testing is done. The pattern follows the unit test pattern.
Thanks, but I'm not sure I understand this response. I don't challenge that Zephyr unittests need this support, or the way they're handled. LITE however needs to test more things than "batch" Zephyr unittests. I present another usercase which albeit simple, barely supported by LAVA. (That's a question to LAVA team definitely.)
I probably misunderstood the question as well. So let's wait for response from someone else.
Well, beyond a simple output matching, it would be nice even for the initial "smoke testing" to actually make some input into the application and check the expected output (e.g., input: "2+2", expected output: "4"). Is this already supported for LAVA "v2" pipeline tests? I may imagine that would be the same kind of functionality required to test bootloaders like U-boot for Linux boards.
I didn't use anything like this in v2 so far, but you're probably best off doing sth like
test 2+2=4 PASS.
than you can easily create pattern that will filter the output. In case of zephyr pattern is the only way to filter things out as there is no shell (?) on the board.
So, the problem, for starters, is how to make LAVA *feed* the input, as specified in the test definition (like "2+2") into a board.
Right. What I proposed was coding all the inputs in the test itself.
As there were no reply from LAVA team (I may imagine they're busy with other things), I decided to create a user story in Jira for them, as I couldn't create a LAVA-* ticket, I created it as https://projects.linaro.org/browse/LITE-175 . Hopefully that won't go unnoticed and LAVA team would get to it eventually.
It's probably best to create CTT ticket here: https://projects.linaro.org/servicedesk/customer/portal/1 These tickets won't go unnoticed.
milosz