Lava-users

lava-users@lists.linaro.org

358 discussions

New LAVA user looking for help with Yocto with grub

by Elric Hindy

Hi, I have been investigating LAVA for use in our organisation and i'm stuck trying to get a hello world test case running on our hardware and looking for some help. We looked at the YOCTO test tools however it can only use devices with a fixed ip which we can't guarantee or want during our testing as we also test network settings. It's also limited in configuration, LAVA package seems to meet all our requirements however i'm still unsure on how to do a few things. We use Yotco images and boot with the grub bootloader. All our devices are connected via Ethernet only and power and peripheral switching is controlled via usb relays. After reading through all the documentation i'm still unsure of how to set up and actually run a test on our hardware. What tools do i need to install in the test image and how do i get it to communicate with grub? I assume a base image is one that includes nothing but the tools and grub. We have a recovery partition with tiny core which could facilitate that but it's not required for the automated testing. I've used the akbennet/lava-server docker image and it is up and running, although test jobs are scheduled but never run on the qemu devices so a little stuck there. Basically, I need help to get LAVA to connect to one of our devices to load the image and run tests? Choosing the image, writing tests and mostly configuring the pipeline I understand. After 2 weeks i'm posting here hoping someone can assist me. Regards, Elric Elric Hindy Test Engineer T +61 2 6103 4700 M +61 413 224 841 E elric.hindy(a)seeingmachines.com W www.seeingmachines.com<http://www.seeingmachines.com> [Seeing Machines]<https://www.seeingmachines.com/> This email is confidential. If you are not the intended recipient, destroy all copies and do not disclose or use the information within. No warranties are given that this email does not contain viruses or harmful code.

9 years, 2 months

Issue in adding lava-worker using R-Pi

by ankit gupta

Hi All, I am trying to setup a remote lab using Raspberry pi on my local network. I installed lava-server and a worker on my laptop and its working fine. I installed raspbian on R-Pi and follow the instruction given on lava site, but when slave is trying to connect to master its not getting any response, I am able to ping master from my R-pi board and default port 3079 is open on my machine. I used no encryption and use URL to connect master as follow. MASTER_URL="tcp://10.42.0.24:3079" LOGGER_URL="tcp://10.42.0.24:3079" I continuosly getting log messgaes like, DEBUG Sending new HELLO_RETRY message to the master (are they both running the same version?) INFO Waiting for the master to reply DEBUG Sending new HELLO_RETRY message to the master (are they both running the same version?) INFO Waiting for the master to reply DEBUG Sending new HELLO_RETRY message to the master (are they both running the same version?) Please, if any one have some idea why I am not able to connect please help. Thanks, Ankit

9 years, 2 months

Re: [Lava-users] [Linaro-validation] Further steps with Zephyr tests

by Neil Williams

On Mon, 3 Jul 2017 23:50:25 +0300 Paul Sokolovsky <paul.sokolovsky(a)linaro.org> wrote: > Hello Milosz, > > I appreciate getting at least some response ;-). Some questions > however could use a reply from LAVA team, I guess. > > On Mon, 3 Jul 2017 13:34:49 +0100 > Milosz Wasilewski <milosz.wasilewski(a)linaro.org> wrote: > > [] > > > > jobs submit a number of tests to LAVA (via > > > https://qa-reports.linaro.org/) for the following boards: > > > arduino_101, frdm_k64f, frdm_kw41z, qemu_cortex_m3. Here's an > > > example of cumulative test report for these platforms: > > > https://qa-reports.linaro.org/lite/zephyr-upstream/tests/ > > > > > > That's really great! (Though the list of tests to run in LAVA > > > seems to be hardcoded: > > > https://git.linaro.org/ci/job/configs.git/tree/zephyr-upstream/submit_for_t…) > > > > > > > It is, as I wasn't really sure what to test. The build job needs to > > prepare the test templates to be submitted to LAVA. In case of > > zephyr each tests is a separate binary. So we end up with the > > number of file paths to substitute in the template. Hardcoding was > > the easiest thing to get things running. But I see no reason why it > > wouldn't be changed with some smarter code to discover the > > binaries. The problem with this approach is that some of these > > tests are just build time. They have no meaning when running on the > > board and need to be filter out somehow. Running the build tests within the Jenkins build makes a lot of sense. Typically, the build tests will have a different command syntax to the runtime tests (otherwise Jenkins would attempt to run both), so filtering should be possible. If the build tests are just a different set of binary blobs from the runtime tests, that may need a fix upstream in Zephyr to distinguish between the two modes. > I see, that makes some sense. But thinking further, I'm not entirely > sure about "build only" tests. Zephyr's sanitycheck test has such > concept, but I'd imagine it comes from the following reasons: a) > sanitycheck runs tests on QEMU, which has very bare hardware support, > so many tests are not runnable; b) sanitycheck can operate on > "samples", not just "tests", as sample can be interactive, etc. it > makes sense to only build them, not run. > > So, I'm not exactly sure about build-only tests on real HW boards. The > "default" idea would be that they should run, but I imagine in > reality, some may need to be filtered out. But then blacklisting > would be better approach than whitelisting. And I'm not sure if > Zephyr has concept of "skipped" tests which may be useful to handle > hardware variations. (Well, I actually dunno if LAVA supports skipped > tests!) Yes, LAVA has support for pass, fail, skip, unknown. For POSIX shell tests, the test writer just calls lava-test-case name --result skip For monitor tests, like Zephyr, it's down to the pattern but skip is as valid as pass and fail (as is unknown) for the result of the matches within the pattern. > Anyway, these are rough ideas for the future. I've spent couple of > weeks of munging with LITE CI setup, there're definitely some > improvements, but also a Pandora box of other ideas and improvements > to make. I'm wrapping up for now, but hope to look again in some time > (definitely hope to look before the Connect, so we can discuss further > steps there). In the meantime, I hope that more boards will be > installed in the Lab and stability of them improves (so far they seem > to be pretty flaky). There are known limitations with the USB subsystem and associated hardware across all architectures, affecting test devices and the workers which run the automation. LAVA has to drive that subsystem very hard for both fastboot devices and IoT devices. There are also problems due to the design of methods like fastboot and some of the IoT support which result from a single-developer model, leading to buggy performance when used at scale and added complexity in deploying workarounds to isolate such protocols in order to prevent interference between tests. The protocols themselves often lack robust error handling or retry support. Other deployment methods which rely on TFTP/network deployments are massively more reliable at scale, so comparing reliability across different device types is problematic. > [] > > > > - test: > > > monitors: > > > - name: foo > > > start: "" > > > end: Hello, ZJS world! > > > pattern: (?P<result>(PASS|FAIL))\s-\s(?P<test_case_id>\w+)\. > > > > > > So, the "start" substring is empty, and perhaps matches a line > > > output by a USB multiplexer or board bootloader. "End" substring > > > is actually the expected single-line output. And "pattern" is > > > unused (dunno if it can be dropped without def file syntax > > > error). Is there a better way to handle single-line test > > > output? > > > > You're making a silent assumption that if there is a matching line, > > the test is passed. In case of other tests (zephyr unit tests), it's > > not the case. The 'start' matches some line which is displayed when > > zephyr is booting. End matches the line which is displayed after all > > testing is done. The pattern follows the unit test pattern. > > Thanks, but I'm not sure I understand this response. I don't challenge > that Zephyr unittests need this support, or the way they're handled. > LITE however needs to test more things than "batch" Zephyr unittests. > I present another usercase which albeit simple, barely supported by > LAVA. (That's a question to LAVA team definitely.) LAVA result handling is ultimately a pattern matching system. Patterns must have a unique and reliable start string and a unique and reliable end string. An empty start string is just going to cause misleading results and bad pattern matches as the reality is that most boards emit some level of random junk immediately upon connection which needs to be ignored. So there needs to be a reliable, unique, start string emitted by the test device. It is not enough to *assume* a start at line zero, doing so increases the problems with reliability. > > > > Well, beyond a simple output matching, it would be nice even for > > > the initial "smoke testing" to actually make some input into the > > > application and check the expected output (e.g., input: "2+2", > > > expected output: "4"). Is this already supported for LAVA "v2" > > > pipeline tests? I may imagine that would be the same kind of > > > functionality required to test bootloaders like U-boot for Linux > > > boards. > > > > I didn't use anything like this in v2 so far, but you're probably > > best off doing sth like > > > > test 2+2=4 PASS. > > > > than you can easily create pattern that will filter the output. In > > case of zephyr pattern is the only way to filter things out as there > > is no shell (?) on the board. > > So, the problem, for starters, is how to make LAVA *feed* the > input, as specified in the test definition (like "2+2") into a board. That will need code changes, so please make a formal request for this support at CTT https://projects.linaro.org/servicedesk/customer/portal/1 so that we can track exactly what is required. > > As there were no reply from LAVA team (I may imagine they're busy with > other things), I decided to create a user story in Jira for them, as I > couldn't create a LAVA-* ticket, I created it as > https://projects.linaro.org/browse/LITE-175 . Hopefully that won't go > unnoticed and LAVA team would get to it eventually. That JIRA story is in the LITE project. Nobody in the LAVA team can manage those stories. It needs a CTT issue which can then be linked to the LITE story and from which a LAVA story can also be linked. Sadly, any story in the LITE project would go completely unnoticed by the LAVA software team until it is linked to CTT so that the work can be prioritised and the relevant LAVA story created. That's just how JIRA works. > > > > > milosz > > Thanks! > > -- > Best Regards, > Paul > > Linaro.org | Open source software for ARM SoCs > Follow Linaro: http://www.facebook.com/pages/Linaro > http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog > _______________________________________________ > linaro-validation mailing list > linaro-validation(a)lists.linaro.org > https://lists.linaro.org/mailman/listinfo/linaro-validation -- Neil Williams ============= http://www.linux.codehelp.co.uk/

9 years, 2 months

Re: [Lava-users] Fwd: OE/Yocto testcase

by Robert Berger

Hi, On 2017-07-04 15:18, Agustin Benito Bethencourt wrote: > Dear CIP friends, > > please check below. This is a use case we will meet in a few > weeks/months. It is important to see others walking the same route. My simple tests are running now thanks to the help of the nice people from the #linaro-lava chat. 1) Crazy me decided to use upstream u-boot 2017.05 instead of running some ancient version from 2014;) 1.1) which happens to have a different AUTOBOOT_PROMPT than the one Lava expects "Press SPACE to abort autoboot in %d seconds\n" instead of "Hit any key to stop autoboot" so (since) I would like to be able to stay as close as possible to upstream LAVA 2017.06 I patched u-boot[1]. Note that this could be fixed with LAVA as well - interrupt_prompt: {{ interrupt_prompt|default('Hit any key to stop autoboot') }} 1.2) also the SYS_PROMPT of upstream u-boot is different than the one expected by LAVA and again I made a u-boot patch[2]. Note that this could be fixed with LAVA as well - {% set bootloader_prompt = bootloader_prompt|default('U-Boot') %} 2) After some searching it turned out that LAVA set some funny variables in u-boot which made my kernel crash. (Crazy me decided to use a 4.9.x uImage without baked in address). Adding this: {% set base_high_limits = false %} to my bbb03.jinja2 file fixed it. ... obviously ;) Regards, Robert [1] https://github.com/RobertBerger/meta-mainline/blob/pyro-training-v4.9.x/u-b… [2] https://github.com/RobertBerger/meta-mainline/blob/pyro-training-v4.9.x/u-b…

9 years, 2 months

Re: [Lava-users] [Linaro-validation] Further steps with Zephyr tests

by Paul Sokolovsky

Hello Milosz, Thanks for routing this thread to lava-users - when I made initial post to linaro-validation, I check my archive and so that e.g. Neil posts there frequently, but I missed that it's not official LAVA list. On Mon, 3 Jul 2017 22:25:31 +0100 Milosz Wasilewski <milosz.wasilewski(a)linaro.org> wrote: [] > > So, I'm not exactly sure about build-only tests on real HW boards. > > The "default" idea would be that they should run, but I imagine in > > reality, some may need to be filtered out. But then blacklisting > > would be better approach than whitelisting. And I'm not sure if > > Zephyr has concept of "skipped" tests which may be useful to handle > > hardware variations. (Well, I actually dunno if LAVA supports > > skipped tests!) > > As far as I can tell they acutely run on the board, but usually output > just 'Hello world!' or sth similar. As we discussed with Kumar, this > is still OK. What Kumar requested (and I still didn't deliver) is that > whenever the LAVA test job completes, the test should be considered > 'passed'. So we wouldn't have to do any parsing of patterns. I'm not > sure if that will work, but it's worth to try. Hmm, I wonder what would be criteria for being "failed" for such tests... Anyway, thanks for sharing - I'm not familiar with all Zephyr tests/samples myself, will keep in mind such issues when looking into them. [] > > more boards will be installed in the Lab and stability of them > > improves (so far they seem to be pretty flaky). > > > > You're absolutely right. This is a pretty big task to work on and IMHO > requires someone to work full time at least for couple of weeks. The > second part is also true, the boards don't behave as they should. I > guess Dave can elaborate more on that. I can only see the result - > boards (frdm-kw41z) don't run the tests they're requested. Matt Hart actually showed me a ticket on that, so at least it's confirmed/known issue in works. But even with arduino_101 and frdm_k64f, I hit cases more than once when board(s) were stuck for extended time, but still were routed jobs to (which either failed or timed out). So, there may be problem with health checks, which either don't run frequently enough or aren't robust enough. arduino_101 is all the lone one, so if something happens to it, there's no backup. Etc, etc. [] > > So, the problem, for starters, is how to make LAVA *feed* the > > input, as specified in the test definition (like "2+2") into a > > board. > > Right. What I proposed was coding all the inputs in the test itself. Well, that would require bunch of legwork, but the biggest problem is that it wouldn't test what's actually required. E.g. both JerryScript and MicroPython Zephyr ports are actually interactive apps working over serial connection. And functional testing of them would be feeding something over this serial connection and checking that results are as expected. I'll keep in mind idea of "builtin" tests though. Thanks! -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

9 years, 2 months

Re: [Lava-users] [Linaro-validation] Further steps with Zephyr tests

by Milosz Wasilewski

I was too quick to hit reply. CCing lava-users for comments from LAVA team. milosz On 3 July 2017 at 21:50, Paul Sokolovsky <paul.sokolovsky(a)linaro.org> wrote: > Hello Milosz, > > I appreciate getting at least some response ;-). Some questions however > could use a reply from LAVA team, I guess. > > On Mon, 3 Jul 2017 13:34:49 +0100 > Milosz Wasilewski <milosz.wasilewski(a)linaro.org> wrote: > > [] > >> > jobs submit a number of tests to LAVA (via >> > https://qa-reports.linaro.org/) for the following boards: >> > arduino_101, frdm_k64f, frdm_kw41z, qemu_cortex_m3. Here's an >> > example of cumulative test report for these platforms: >> > https://qa-reports.linaro.org/lite/zephyr-upstream/tests/ >> > >> > That's really great! (Though the list of tests to run in LAVA seems >> > to be hardcoded: >> > https://git.linaro.org/ci/job/configs.git/tree/zephyr-upstream/submit_for_t…) >> > >> >> It is, as I wasn't really sure what to test. The build job needs to >> prepare the test templates to be submitted to LAVA. In case of zephyr >> each tests is a separate binary. So we end up with the number of file >> paths to substitute in the template. Hardcoding was the easiest thing >> to get things running. But I see no reason why it wouldn't be changed >> with some smarter code to discover the binaries. The problem with this >> approach is that some of these tests are just build time. They have no >> meaning when running on the board and need to be filter out somehow. > > I see, that makes some sense. But thinking further, I'm not entirely > sure about "build only" tests. Zephyr's sanitycheck test has such > concept, but I'd imagine it comes from the following reasons: a) > sanitycheck runs tests on QEMU, which has very bare hardware support, > so many tests are not runnable; b) sanitycheck can operate on "samples", > not just "tests", as sample can be interactive, etc. it makes sense to > only build them, not run. > > So, I'm not exactly sure about build-only tests on real HW boards. The > "default" idea would be that they should run, but I imagine in reality, > some may need to be filtered out. But then blacklisting would be better > approach than whitelisting. And I'm not sure if Zephyr has concept of > "skipped" tests which may be useful to handle hardware variations. > (Well, I actually dunno if LAVA supports skipped tests!) > > Anyway, these are rough ideas for the future. I've spent couple of > weeks of munging with LITE CI setup, there're definitely some > improvements, but also a Pandora box of other ideas and improvements to > make. I'm wrapping up for now, but hope to look again in some time > (definitely hope to look before the Connect, so we can discuss further > steps there). In the meantime, I hope that more boards will be > installed in the Lab and stability of them improves (so far they seem > to be pretty flaky). > > [] > >> > - test: >> > monitors: >> > - name: foo >> > start: "" >> > end: Hello, ZJS world! >> > pattern: (?P<result>(PASS|FAIL))\s-\s(?P<test_case_id>\w+)\. >> > >> > So, the "start" substring is empty, and perhaps matches a line >> > output by a USB multiplexer or board bootloader. "End" substring is >> > actually the expected single-line output. And "pattern" is unused >> > (dunno if it can be dropped without def file syntax error). Is >> > there a better way to handle single-line test output? >> >> You're making a silent assumption that if there is a matching line, >> the test is passed. In case of other tests (zephyr unit tests), it's >> not the case. The 'start' matches some line which is displayed when >> zephyr is booting. End matches the line which is displayed after all >> testing is done. The pattern follows the unit test pattern. > > Thanks, but I'm not sure I understand this response. I don't challenge > that Zephyr unittests need this support, or the way they're handled. > LITE however needs to test more things than "batch" Zephyr unittests. I > present another usercase which albeit simple, barely supported by LAVA. > (That's a question to LAVA team definitely.) > >> > Well, beyond a simple output matching, it would be nice even for the >> > initial "smoke testing" to actually make some input into the >> > application and check the expected output (e.g., input: "2+2", >> > expected output: "4"). Is this already supported for LAVA "v2" >> > pipeline tests? I may imagine that would be the same kind of >> > functionality required to test bootloaders like U-boot for Linux >> > boards. >> >> I didn't use anything like this in v2 so far, but you're probably best >> off doing sth like >> >> test 2+2=4 PASS. >> >> than you can easily create pattern that will filter the output. In >> case of zephyr pattern is the only way to filter things out as there >> is no shell (?) on the board. > > So, the problem, for starters, is how to make LAVA *feed* the > input, as specified in the test definition (like "2+2") into a board. > > As there were no reply from LAVA team (I may imagine they're busy with > other things), I decided to create a user story in Jira for them, as I > couldn't create a LAVA-* ticket, I created it as > https://projects.linaro.org/browse/LITE-175 . Hopefully that won't go > unnoticed and LAVA team would get to it eventually. > >> >> milosz > > Thanks! > > -- > Best Regards, > Paul > > Linaro.org | Open source software for ARM SoCs > Follow Linaro: http://www.facebook.com/pages/Linaro > http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

9 years, 2 months

OE/Yocto testcase

by Robert Berger

Hi, I succeeded in running the example Lava tests[1][2] on a Beagle Bone Black in my lab. Now I would like to test a core-image-minimal I built with Yocto and cooked up something like this[3][4]. The result is unfortunately this[5]. I guess one of the problems I have is this: lava-test: # ls -l /lava-38/ export SHELL=/bin/sh /lava-38/bin/lava-test-runner /lava-38/0ls: cannot access '/lava-38/': No such file or directory lava-test: # export SHELL=/bin/sh lava-test: # /lava-38/bin/lava-test-runner /lava-38/0 -sh: /lava-38/bin/lava-test-runner: No such file or directory Regards, Robert [1] https://validation.linaro.org/static/docs/v2/examples/test-jobs/standard-ar… [2] https://validation.linaro.org/static/docs/v2/examples/test-jobs/standard-ar… [3] https://github.com/RobertBerger/git.linaro.org-lava-team-refactoring/blob/m… [4] https://github.com/RobertBerger/git.linaro.org-qa-test-definitions/blob/mas… [5] https://pastebin.com/14pjxhiN

9 years, 2 months

job yaml

by Li, Xuelian

Hi,all: If devive have boot successed，can I skip deploy and boot step in job.yaml? This email is intended only for the named addressee. It may contain information that is confidential/private, legally privileged, or copyright-protected, and you should handle it accordingly. If you are not the intended recipient, you do not have legal rights to retain, copy, or distribute this email or its contents, and should promptly delete the email and all electronic copies in your system; do not retain copies in any media. If you have received this email in error, please notify the sender promptly. Thank you.

9 years, 2 months

Is it possible to get data in device dictionary inside a job?

by Li, Yifan2

Hello LAVA experts, I am currently working with LAVA and was asked to find is there a way to get data from device dictionary inside a running job. Details like below: I have a device name demo-01, and its device-dictionary have one line like "{% set my_property = 'my_prop' %}". Then I have a job running on demo-01 device, and I would like to use string 'my_prop' to passed into a script during running. Is it possible to get device-dictionary data directly from job definition(Job Submitter webpage) or test definition(yaml file)? If yes, how could I do this? If not, is there any good way you would like to share to solve this problem? Thanks and Best Regards, Yifan

9 years, 2 months

About Test Duration

by kingboli

Hi all, Is there any limitation on test duration when using LAVA to do tests? Recently, I found my tests were automatically canceled after running for 24 hours and I could not find any clue about why the job was canceled. Can anybody give me some help? (Very sorry my LAVA version is V1.) 12794.0 <LAVA_DISPATCHER>2017-06-25 02:31:42 PM INFO: Cancel operation 12794.1 <LAVA_DISPATCHER>2017-06-25 02:31:42 PM DEBUG: finally status fail 12794.2 <LAVA_DISPATCHER>2017-06-25 02:31:42 PM WARNING: [ACTION-E] boot_image is finished with error (Cancel). 12794.3 <LAVA_DISPATCHER>2017-06-25 02:31:42 PM WARNING: Target : power_off being call Thanks.

9 years, 3 months

← Newer
1
...
24
25
26
27
28
29
30
...
36
Older →

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Lava-users