linaro-validation

linaro-validation@lists.linaro.org

818 discussions

lava-test-shell tweaks and audio capture

by Michael Hudson-Doyle

I was recently making some additions to the lava-test-shell code and in the process more or less had to write some documentation on how things work. Here is what I wrote (actually what I wrote includes stuff about test result attachments but that's not in trunk yet): # LAVA Test Shell implementation details # ====================================== # # The idea of lava-test-shell is a YAML test definition is "compiled" into a # job that is run when the device under test boots and then the output of this # job is retrieved and analyzed and turned into a bundle of results. # # In practice, this means a hierarchy of directories and files is created # during test installation, a sub-hierarchy is created during execution to # hold the results and these latter sub-hierarchy whole lot is poked at on the # host during analysis. # # On Ubuntu and OpenEmbedded, the hierarchy is rooted at /lava. / is mounted # read-only on Android, so there we root the hierarchy at /data/lava. I'll # assume Ubuntu paths from here for simplicity. # # The directory tree that is created during installation looks like this: # # /lava/ # bin/ This directory is put on the path when the # test code is running -- these binaries can # be viewed as a sort of device-side "API" # for test authors. # lava-test-runner The job that runs the tests on boot. # lava-test-shell A helper to run a test suite. # tests/ # ${IDX}_${TEST_ID}/ One directory per test to be executed. # testdef.yml The test definition. # install.sh The install steps. # run.sh The run steps. # [repos] The test definition can specify bzr or git # repositories to clone into this directory. # # In addition, a file /etc/lava-test-runner.conf is created containing the # names of the directories in /lava/tests/ to execute. # # During execution, the following files are created: # # /lava/ # results/ # cpuinfo.txt Hardware info. # meminfo.txt Ditto. # build.txt Software info. # pkgs.txt Ditto # ${IDX}_${TEST_ID}-${TIMESTAMP}/ # testdef.yml Attached to the test run in the bundle for # archival purposes. # install.sh Ditto. # run.sh Ditto. # stdout.log The standard output of run.sh. # stderr.log The standard error of run.sh (actually not # created currently) # return_code The exit code of run.sh. # # After the test run has completed, the /lava/results directory is pulled over # to the host and turned into a bundle for submission to the dashboard. Now clearly the work Senthil is doing on getting testdefs from a repo is going to make some changes to the files that get created during installation (I had envisioned that the repo that the job specifies would be cloned to /lava/tests/${IDX}_${TEST_ID} but that might not be completely safe -- if the repo contains a file called run.sh we don't want to clobber that! Not sure what to do here). But the tweaks I want to propose are more to do with what goes into the /lava/bin and /lava/results directories. For a start, I don't think it's especially useful that the tests run with lava-test-runner or lava-test-shell on $PATH -- there's no reason for a test author to call either of those functions! However I want to add helpers -- called lava-test-case and lava-test-case-attach in my brain currently: * lava-test-case will send the test case started signal, run a shell command, interpret its exit code as a test result and send the test case stopped signal * lava-test-case-attach arranges to attach a file to the test result for a particular test case id So you could imagine some run steps for an audio capture test: run: steps: - lava-fft -t 3 > generated.wav - lava-test-case-attach sine-wave generated.wav audio/wav - lava-test-case sine-wave aplay generated.wav and appropriate hooks on the host side: * test case start/stop hooks that would capture audio * an "analysis hook" that would compare the captured sample with the attached wav file (and attach the captured wav) Semi-relatedly, I would like to (at least optionally) store the test result data more explicitly in the /lava/results/${IDX}_${TEST_ID}-${TIMESTAMP} directory. Maybe something like this (in the above style): # /lava/ # results/ # ... As before # ${IDX}_${TEST_ID}-${TIMESTAMP}/ # ... All the stuff we had before. # cases/ # ${TEST_CASE_ID}/ # result This would contain pass/fail/skip/unknown # units Mb/s, V, W, Hz, whatever # measurement Self explanatory I expect. # attachments/ # ${FILENAME} The content of the attachment # ${FILENAME}.mimetype The mime-type for the attachment # attributes/ # ${KEY} The content of the file would be the # value of the attachment. # ... other things you can stick on test results ... Basically this would be defining another representation for test results: on the file system, in addition to the existing two: as JSON or in a postgres DB. The reason for doing this is twofold: 1) it's more amenable than JSON to being incrementally built up by a bunch of shell scripts as a lava-test-shell test runs and 2) this directory could be presented to an "analysis hook" written in shell (earlier on today I told Andy Doan that I though writing hooks in shell would be impractical; now I'm not so sure). Also: 3) (noone expects the spanish inquisition!) it would allow us to write a lava-test-shell test that does not depend on parsing stdout.log. Apologies for the log ramble! Cheers, mwh

13 years, 10 months

VExpress A5

by Dave Pigott

Hi all, I've been soak testing A5 in staging for the last few days, and I've seen 6 failures out of 26 runs. They are all down to "MciReadBlockData(): TIMEOUT! Response:0xA404012 Status:0x48" when trying to boot a test image. The only sensible way to fix this is to add a retry on booting the test image, so until that's done I don't think we can deploy it in production. Thanks Dave

13 years, 10 months

Out of office

by Dave Pigott

Hi all, I have some odds and ends to sort out tomorrow, so short of a major problem in the lab, I'll be working from home. Thanks Dave

13 years, 10 months

Health check failures

by Dave Pigott

None! \o/

13 years, 10 months

stupid daylight savings

by Andy Doan

Hey Guys Sorry I missed the weekly meeting. I had forgotten that it had changed from 8am to 7am for me due to daylight savings.

13 years, 10 months

Re: [Linaro-validation] [Bug 1079642] [NEW] "parse" section is hardcoded-required for lava_test_shell

by Senthil Kumaran S

On Friday 16 November 2012 05:01 PM, Paul Sokolovsky wrote: > Public bug reported: > > It might seems that result parsing support should be optional, for the > case it's really needed (because otherwise results can already be in > format LAVA expects). However, trying testdef without "parse" section > leads to: > > <LAVA_DISPATCHER>2012-11-16 01:25:05 PM INFO: Downloading image: http://192.168.15.225:38713/fs.tgz > <LAVA_DISPATCHER>2012-11-16 01:25:06 PM ERROR: error processing results for: 0_cbuild-gcc-1353061077 > Traceback (most recent call last): > File "/home/pfalcon/devel/linaro/lava/lava-dispatcher/lava_dispatcher/lava_test_shell.py", line 207, in get_bundle > testruns.append(_get_test_run(results_dir, d, hwctx, swctx)) > File "/home/pfalcon/devel/linaro/lava/lava-dispatcher/lava_dispatcher/lava_test_shell.py", line 172, in _get_test_run > 'test_results': _get_test_results(testdef, stdout), > File "/home/pfalcon/devel/linaro/lava/lava-dispatcher/lava_dispatcher/lava_test_shell.py", line 121, in _get_test_results > pattern = re.compile(testdef['parse']['pattern']) > KeyError: 'parse' I just pushed a branch with code changes required to fix the above. I request someone to review the following: https://code.launchpad.net/~stylesen/lava-dispatcher/fix-empty-parse-patter… Thank You. -- Senthil Kumaran S http://www.stylesen.org/ http://www.sasenthilkumaran.com/

13 years, 10 months

Board failure

by Dave Pigott

Just the one since Saturday: ---------------- snowball06 ---------------- http://validation.linaro.org/lava-server/scheduler/job/38435 As noted by Andy, this is our most flakey snowball. I'm proposing retiring it. It failed three times to reboot and get the network up. I went on the board and had to hard reset to get the network to come up, which suggests it gets into a weird state that needs a completely clean boot. If it's not reliable it should be taken out. We can move it over to staging for more validation of reliability tests. Thanks Dave

13 years, 10 months

dispatcher timeout system.tar.bz2

by Andy Doan

I was just looking at a recent change: <http://bazaar.launchpad.net/~linaro-validation/lava-dispatcher/trunk/revisi…> This happens everywhere, but maybe this is a chance to help stop madness. Rather than making this a comment constant in code that also affects other boards. How about we make this a device config option. We could then keep the default to 600 for all boards except for this? A completely untested alternative would be: http://paste.ubuntu.com/1358331/

13 years, 10 months

Health check

by Dave Pigott

Just one (again): --------------------------------- rtsm_ve-a15x4-a7x4_01 --------------------------------- http://validation.linaro.org/lava-server/scheduler/device/rtsm_ve-a15x4-a7x… Not quite sure what I'm looking at, other than l-m-c bailed for some reason. I've put it back online to check if it was just a glitch. Dave

13 years, 10 months

lava_test_shell YAML syntax example?

by Paul Sokolovsky

Hello, I see that YAML syntax for lava_test_shell has landed in linaro-dispatcher trunk, and that old JSON syntax is no longer (automagically) supported. I tried to do a straightforward conversion of http://people.linaro.org/~doanac/lava/lava_test_shell/testdefs/lava-test.js… , but it throws exceptions trying to parse it. So, can you share examples of the new syntax? Thanks, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

13 years, 10 months

← Newer
1
...
51
52
53
54
55
56
57
...
82
Older →

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

linaro-validation