linaro-validation

linaro-validation@lists.linaro.org

818 discussions

lava-test-shell tweaks and audio capture

by Michael Hudson-Doyle

I was recently making some additions to the lava-test-shell code and in the process more or less had to write some documentation on how things work. Here is what I wrote (actually what I wrote includes stuff about test result attachments but that's not in trunk yet): # LAVA Test Shell implementation details # ====================================== # # The idea of lava-test-shell is a YAML test definition is "compiled" into a # job that is run when the device under test boots and then the output of this # job is retrieved and analyzed and turned into a bundle of results. # # In practice, this means a hierarchy of directories and files is created # during test installation, a sub-hierarchy is created during execution to # hold the results and these latter sub-hierarchy whole lot is poked at on the # host during analysis. # # On Ubuntu and OpenEmbedded, the hierarchy is rooted at /lava. / is mounted # read-only on Android, so there we root the hierarchy at /data/lava. I'll # assume Ubuntu paths from here for simplicity. # # The directory tree that is created during installation looks like this: # # /lava/ # bin/ This directory is put on the path when the # test code is running -- these binaries can # be viewed as a sort of device-side "API" # for test authors. # lava-test-runner The job that runs the tests on boot. # lava-test-shell A helper to run a test suite. # tests/ # ${IDX}_${TEST_ID}/ One directory per test to be executed. # testdef.yml The test definition. # install.sh The install steps. # run.sh The run steps. # [repos] The test definition can specify bzr or git # repositories to clone into this directory. # # In addition, a file /etc/lava-test-runner.conf is created containing the # names of the directories in /lava/tests/ to execute. # # During execution, the following files are created: # # /lava/ # results/ # cpuinfo.txt Hardware info. # meminfo.txt Ditto. # build.txt Software info. # pkgs.txt Ditto # ${IDX}_${TEST_ID}-${TIMESTAMP}/ # testdef.yml Attached to the test run in the bundle for # archival purposes. # install.sh Ditto. # run.sh Ditto. # stdout.log The standard output of run.sh. # stderr.log The standard error of run.sh (actually not # created currently) # return_code The exit code of run.sh. # # After the test run has completed, the /lava/results directory is pulled over # to the host and turned into a bundle for submission to the dashboard. Now clearly the work Senthil is doing on getting testdefs from a repo is going to make some changes to the files that get created during installation (I had envisioned that the repo that the job specifies would be cloned to /lava/tests/${IDX}_${TEST_ID} but that might not be completely safe -- if the repo contains a file called run.sh we don't want to clobber that! Not sure what to do here). But the tweaks I want to propose are more to do with what goes into the /lava/bin and /lava/results directories. For a start, I don't think it's especially useful that the tests run with lava-test-runner or lava-test-shell on $PATH -- there's no reason for a test author to call either of those functions! However I want to add helpers -- called lava-test-case and lava-test-case-attach in my brain currently: * lava-test-case will send the test case started signal, run a shell command, interpret its exit code as a test result and send the test case stopped signal * lava-test-case-attach arranges to attach a file to the test result for a particular test case id So you could imagine some run steps for an audio capture test: run: steps: - lava-fft -t 3 > generated.wav - lava-test-case-attach sine-wave generated.wav audio/wav - lava-test-case sine-wave aplay generated.wav and appropriate hooks on the host side: * test case start/stop hooks that would capture audio * an "analysis hook" that would compare the captured sample with the attached wav file (and attach the captured wav) Semi-relatedly, I would like to (at least optionally) store the test result data more explicitly in the /lava/results/${IDX}_${TEST_ID}-${TIMESTAMP} directory. Maybe something like this (in the above style): # /lava/ # results/ # ... As before # ${IDX}_${TEST_ID}-${TIMESTAMP}/ # ... All the stuff we had before. # cases/ # ${TEST_CASE_ID}/ # result This would contain pass/fail/skip/unknown # units Mb/s, V, W, Hz, whatever # measurement Self explanatory I expect. # attachments/ # ${FILENAME} The content of the attachment # ${FILENAME}.mimetype The mime-type for the attachment # attributes/ # ${KEY} The content of the file would be the # value of the attachment. # ... other things you can stick on test results ... Basically this would be defining another representation for test results: on the file system, in addition to the existing two: as JSON or in a postgres DB. The reason for doing this is twofold: 1) it's more amenable than JSON to being incrementally built up by a bunch of shell scripts as a lava-test-shell test runs and 2) this directory could be presented to an "analysis hook" written in shell (earlier on today I told Andy Doan that I though writing hooks in shell would be impractical; now I'm not so sure). Also: 3) (noone expects the spanish inquisition!) it would allow us to write a lava-test-shell test that does not depend on parsing stdout.log. Apologies for the log ramble! Cheers, mwh

12 years, 9 months

VExpress A5

by Dave Pigott

Hi all, I've been soak testing A5 in staging for the last few days, and I've seen 6 failures out of 26 runs. They are all down to "MciReadBlockData(): TIMEOUT! Response:0xA404012 Status:0x48" when trying to boot a test image. The only sensible way to fix this is to add a retry on booting the test image, so until that's done I don't think we can deploy it in production. Thanks Dave

12 years, 9 months

Out of office

by Dave Pigott

Hi all, I have some odds and ends to sort out tomorrow, so short of a major problem in the lab, I'll be working from home. Thanks Dave

12 years, 9 months

Health check failures

by Dave Pigott

None! \o/

12 years, 9 months

stupid daylight savings

by Andy Doan

Hey Guys Sorry I missed the weekly meeting. I had forgotten that it had changed from 8am to 7am for me due to daylight savings.

12 years, 9 months

Re: [Linaro-validation] [Bug 1079642] [NEW] "parse" section is hardcoded-required for lava_test_shell

by Senthil Kumaran S

On Friday 16 November 2012 05:01 PM, Paul Sokolovsky wrote: > Public bug reported: > > It might seems that result parsing support should be optional, for the > case it's really needed (because otherwise results can already be in > format LAVA expects). However, trying testdef without "parse" section > leads to: > > <LAVA_DISPATCHER>2012-11-16 01:25:05 PM INFO: Downloading image: http://192.168.15.225:38713/fs.tgz > <LAVA_DISPATCHER>2012-11-16 01:25:06 PM ERROR: error processing results for: 0_cbuild-gcc-1353061077 > Traceback (most recent call last): > File "/home/pfalcon/devel/linaro/lava/lava-dispatcher/lava_dispatcher/lava_test_shell.py", line 207, in get_bundle > testruns.append(_get_test_run(results_dir, d, hwctx, swctx)) > File "/home/pfalcon/devel/linaro/lava/lava-dispatcher/lava_dispatcher/lava_test_shell.py", line 172, in _get_test_run > 'test_results': _get_test_results(testdef, stdout), > File "/home/pfalcon/devel/linaro/lava/lava-dispatcher/lava_dispatcher/lava_test_shell.py", line 121, in _get_test_results > pattern = re.compile(testdef['parse']['pattern']) > KeyError: 'parse' I just pushed a branch with code changes required to fix the above. I request someone to review the following: https://code.launchpad.net/~stylesen/lava-dispatcher/fix-empty-parse-patter… Thank You. -- Senthil Kumaran S http://www.stylesen.org/ http://www.sasenthilkumaran.com/

12 years, 9 months

Board failure

by Dave Pigott

Just the one since Saturday: ---------------- snowball06 ---------------- http://validation.linaro.org/lava-server/scheduler/job/38435 As noted by Andy, this is our most flakey snowball. I'm proposing retiring it. It failed three times to reboot and get the network up. I went on the board and had to hard reset to get the network to come up, which suggests it gets into a weird state that needs a completely clean boot. If it's not reliable it should be taken out. We can move it over to staging for more validation of reliability tests. Thanks Dave

12 years, 9 months

dispatcher timeout system.tar.bz2

by Andy Doan

I was just looking at a recent change: <http://bazaar.launchpad.net/~linaro-validation/lava-dispatcher/trunk/revisi…> This happens everywhere, but maybe this is a chance to help stop madness. Rather than making this a comment constant in code that also affects other boards. How about we make this a device config option. We could then keep the default to 600 for all boards except for this? A completely untested alternative would be: http://paste.ubuntu.com/1358331/

12 years, 9 months

Health check

by Dave Pigott

Just one (again): --------------------------------- rtsm_ve-a15x4-a7x4_01 --------------------------------- http://validation.linaro.org/lava-server/scheduler/device/rtsm_ve-a15x4-a7x… Not quite sure what I'm looking at, other than l-m-c bailed for some reason. I've put it back online to check if it was just a glitch. Dave

12 years, 9 months

lava_test_shell YAML syntax example?

by Paul Sokolovsky

Hello, I see that YAML syntax for lava_test_shell has landed in linaro-dispatcher trunk, and that old JSON syntax is no longer (automagically) supported. I tried to do a straightforward conversion of http://people.linaro.org/~doanac/lava/lava_test_shell/testdefs/lava-test.js… , but it throws exceptions trying to parse it. So, can you share examples of the new syntax? Thanks, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

12 years, 9 months

← Newer
1
...
51
52
53
54
55
56
57
...
82
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

linaro-validation