On Fri, Feb 10, 2012 at 6:16 AM, Benjamin Gaignard
last time I have ask the cable was on panda 03 :-) Anyway e2eaudiotest itself isn't mature enough, I believe that we still have issue on audio paths settings, but Kurt will work on that (Tom can you confirm ?)
I have submit lava e2e test to check if the parser was working well even
on error cases.
That could be, we need to add tags to the boards that have it so that you can be sure, but for the audio cables, they are cheap enough that we should really just put them everywhere. And yes, even if the the test fails currently, we should still run it if it's ready to go. Is it ready to go? If so, I'd like to get it merged, and either in the daily runs or in some special multimedia run if that is preferred.
About cmatest: I have try to test the latest version (v21) but it failed
http://validation.linaro.org/lava-server/scheduler/job/11852 with this strange error "ERROR: There is no test with the specified ID". I don't understand where is the problem, I have run the test on my side without any issues. How the test ID could be wrong ? The same test ID has been used to install the test itself .... strange.
We'll try to look into this today.
The good news is that realvideo test is working well: http://validation.linaro.org/lava-server/scheduler/job/1%3Ba=blob_plain%3Bf=... HEAD1619 http://validation.linaro.org/lava-server/scheduler/job/11619
Great! It's done in json, so we can't merge it, but it would be possible to get into regular runs anyway. Looks like it takes maybe 45 min. to run. Any special requirements here?
I have lot of meetings this morning, and I leave at 12:30.... What I wanted to talk with you and Tom, is the numbers of automatic tests done daily on lava. They take so much time than I'm not able to run/validate my own tests before 3-4 pm. Do you think you can reduce the number of daily tests ? for example is it really needed to do android and ubuntu lab health test daily on each board ? Maybe we can also reduce lava test time setup by having rootfs with pre-installed lava-tool package.
Have dedicated lava rootfs could also help me the speed up test creation because it is always difficult to found the correct match between hwpack and rootfs.
I'm not sure what you mean here, you don't necessarily have to have the rootfs from the exact same day. As for the backload on the boards and not getting your job in quickly, there are a couple of things to be aware of: First, we had an issue with the scheduler earlier in the week and needed to shut it down for a bit. Short story is that it's back up now, processing jobs, and it's mostly cleared the queue that had built up. There are now lots of boards sitting idle (even snowball). Next, on snowball, we only have 3 at the moment. We're supposed to get some more, but that means for now they are going to be pretty busy.
The health checks are part of an effort to try to detect when there's something wrong with the board, or infrastructure that would prevent jobs from passing when they normally should. They aren't likely to go away, but we just merged something that should help us make them faster.
Thanks, Paul Larson
Regards, Benjamin
2012/2/10 Paul Larson paul.larson@linaro.org
Let's talk on Friday, sorry I missed today. I didn't forget you though, just got pulled into an unexpected meeting late in the day after my tsc talk. It never fails that at connects I never manage to talk to everyone I want to, but I wanted to definitely sync with the two of you and see how things were going with cmatest and e2eaudio.
One quick point - on the e2e audio tests, I noticed that you seem to have got a failed result (but a result none-the-less) on panda03. I *think* the only board we've managed to connect with the loopback audio cable is panda01. Dave can confirm that. The company we ordered the audio cables from delayed and we didn't get them before the connect. :( In the meantime, if you set the target for the job to look at panda01, you should be able to get to a board with the loopback actually plugged in.
Thanks, Paul Larson
--
Benjamin Gaignard
Multimedia Working Group
Linaro.org http://www.linaro.org/* **│ *Open source software for ARM SoCs
**
Follow *Linaro: *Facebook http://www.facebook.com/pages/Linaro | Twitterhttp://twitter.com/#%21/linaroorg | Blog http://www.linaro.org/linaro-blog/
On Fri, Feb 10, 2012 at 8:13 AM, Paul Larson paul.larson@linaro.org wrote:
On Fri, Feb 10, 2012 at 6:16 AM, Benjamin Gaignard
last time I have ask the cable was on panda 03 :-) Anyway e2eaudiotest itself isn't mature enough, I believe that we still have issue on audio paths settings, but Kurt will work on that (Tom can you confirm ?)
I have submit lava e2e test to check if the parser was working well even on error cases.
That could be, we need to add tags to the boards that have it so that you can be sure, but for the audio cables, they are cheap enough that we should really just put them everywhere. And yes, even if the the test fails
Yes! I have said that a few times: add it and use lava to track the fixing of audio and the test itself. It's a nature that if a test fails it can be that the feature it tests is broken or the test itself needs to be adjusted.
Dave, what's the status of the audio cables? Did we finally get more than just the one you had there? We could go two different ways with this: 1. use tags, Submit a separate audio-testing job to run only on boards with the tag for having an audio loopback 2. submit this test for *every* board, the intent being that every board should have these cables. Are there any that don't have the capability to hook up this loopback cable? I know there are some that default to using hdmi audio, and they will fail (as will every other board for now)
Thanks, Paul Larson
On Fri, Feb 10, 2012 at 3:48 PM, Alexander Sack asac@linaro.org wrote:
On Fri, Feb 10, 2012 at 8:13 AM, Paul Larson paul.larson@linaro.org wrote:
On Fri, Feb 10, 2012 at 6:16 AM, Benjamin Gaignard
last time I have ask the cable was on panda 03 :-) Anyway e2eaudiotest itself isn't mature enough, I believe that we still have issue on audio paths settings, but Kurt will work on that (Tom can
you
confirm ?)
I have submit lava e2e test to check if the parser was working well even on error cases.
That could be, we need to add tags to the boards that have it so that you can be sure, but for the audio cables, they are cheap enough that we
should
really just put them everywhere. And yes, even if the the test fails
Yes! I have said that a few times: add it and use lava to track the fixing of audio and the test itself. It's a nature that if a test fails it can be that the feature it tests is broken or the test itself needs to be adjusted.
-- Alexander Sack Technical Director, Linaro Platform Teams http://www.linaro.org | Open source software for ARM SoCs http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
Hi Paul,
All cables in, just not installed. I started the installations on Tuesday, but had to pick boards that were available. Didn't want to upgrade while boards were running tests. I created an "audio-loopback" tag and associated with the boards that currently have it installed. Intend to start the upgrade with the others on Monday.
Dave
On 14 Feb 2012, at 21:48, Paul Larson wrote:
Dave, what's the status of the audio cables? Did we finally get more than just the one you had there? We could go two different ways with this:
- use tags, Submit a separate audio-testing job to run only on boards with the tag for having an audio loopback
- submit this test for *every* board, the intent being that every board should have these cables. Are there any that don't have the capability to hook up this loopback cable? I know there are some that default to using hdmi audio, and they will fail (as will every other board for now)
Thanks, Paul Larson
On Fri, Feb 10, 2012 at 3:48 PM, Alexander Sack asac@linaro.org wrote: On Fri, Feb 10, 2012 at 8:13 AM, Paul Larson paul.larson@linaro.org wrote:
On Fri, Feb 10, 2012 at 6:16 AM, Benjamin Gaignard
last time I have ask the cable was on panda 03 :-) Anyway e2eaudiotest itself isn't mature enough, I believe that we still have issue on audio paths settings, but Kurt will work on that (Tom can you confirm ?)
I have submit lava e2e test to check if the parser was working well even on error cases.
That could be, we need to add tags to the boards that have it so that you can be sure, but for the audio cables, they are cheap enough that we should really just put them everywhere. And yes, even if the the test fails
Yes! I have said that a few times: add it and use lava to track the fixing of audio and the test itself. It's a nature that if a test fails it can be that the feature it tests is broken or the test itself needs to be adjusted.
-- Alexander Sack Technical Director, Linaro Platform Teams http://www.linaro.org | Open source software for ARM SoCs http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
On Sat, Feb 18, 2012 at 5:42 PM, Dave Pigott dave.pigott@linaro.org wrote:
Hi Paul,
All cables in, just not installed. I started the installations on Tuesday, but had to pick boards that were available. Didn't want to upgrade while boards were running tests. I created an "audio-loopback" tag and associated with the boards that currently have it installed. Intend to start the upgrade with the others on Monday.
Great. How to set requirement tags for jobs submitted? Is this something we can do inside the test definition? Or is this decided by the job submitter atm?
As far as I understand it, the test definition specifies the device tags that it requires, and the scheduler pushes the job to an appropriately tagged board. That said, I haven't tried it myself. We now have several boards with audio-loopback and usb-flash-drive tags enabled, more will be added as they come offline.
Dave
Dave Pigott Validation Engineer T: +44 1223 45 00 24 | M +44 7940 45 93 44 Linaro.org │ Open source software for ARM SoCs Follow Linaro: Facebook | Twitter | Blog
On 20 Feb 2012, at 09:08, Alexander Sack wrote:
On Sat, Feb 18, 2012 at 5:42 PM, Dave Pigott dave.pigott@linaro.org wrote: Hi Paul,
All cables in, just not installed. I started the installations on Tuesday, but had to pick boards that were available. Didn't want to upgrade while boards were running tests. I created an "audio-loopback" tag and associated with the boards that currently have it installed. Intend to start the upgrade with the others on Monday.
Great. How to set requirement tags for jobs submitted? Is this something we can do inside the test definition? Or is this decided by the job submitter atm?
-- Alexander Sack Technical Director, Linaro Platform Teams http://www.linaro.org | Open source software for ARM SoCs http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
On Mon, Feb 20, 2012 at 6:47 AM, Dave Pigott dave.pigott@linaro.org wrote:
As far as I understand it, the test definition specifies the device tags that it requires, and the scheduler pushes the job to an appropriately tagged board. That said, I haven't tried it myself. We now have several boards with audio-loopback and usb-flash-drive tags enabled, more will be added as they come offline.
Great, need to give the usb test a try!
Should we also add tags like wifi and bluetooth for the boards we have? I know these features are locked per board type, but it might also make sense to add the specific capabilities as tags as well.
Thanks,
On 23 Feb 2012, at 19:38, Ricardo Salveti wrote:
On Mon, Feb 20, 2012 at 6:47 AM, Dave Pigott dave.pigott@linaro.org wrote:
As far as I understand it, the test definition specifies the device tags that it requires, and the scheduler pushes the job to an appropriately tagged board. That said, I haven't tried it myself. We now have several boards with audio-loopback and usb-flash-drive tags enabled, more will be added as they come offline.
Great, need to give the usb test a try!
Should we also add tags like wifi and bluetooth for the boards we have? I know these features are locked per board type, but it might also make sense to add the specific capabilities as tags as well.
I bought this up at Connect, but the general feeling was that you wouldn't target a test at a board type that didn't support that functionality.
Anyone re-thinking that?
Dave
On Fri, Feb 24, 2012 at 2:29 AM, Dave Pigott dave.pigott@linaro.org wrote:
On 23 Feb 2012, at 19:38, Ricardo Salveti wrote:
On Mon, Feb 20, 2012 at 6:47 AM, Dave Pigott dave.pigott@linaro.org wrote:
As far as I understand it, the test definition specifies the device tags that it requires, and the scheduler pushes the job to an appropriately tagged board. That said, I haven't tried it myself. We now have several boards with audio-loopback and usb-flash-drive tags enabled, more will be added as they come offline.
Great, need to give the usb test a try!
Should we also add tags like wifi and bluetooth for the boards we have? I know these features are locked per board type, but it might also make sense to add the specific capabilities as tags as well.
I bought this up at Connect, but the general feeling was that you wouldn't target a test at a board type that didn't support that functionality.
But then we would not cover the test case where someone would just like to run a bluetooth test, without actually depending on a board type. Without the tags, it'd be hard to know if the test case should run on an imx53 board or not.
As it'd be easy to add and maintain such tags, I don't see why we shouldn't allow that.
Cheers,
On Fri, Feb 24, 2012 at 7:24 AM, Ricardo Salveti <ricardo.salveti@linaro.org
wrote:
On Fri, Feb 24, 2012 at 2:29 AM, Dave Pigott dave.pigott@linaro.org wrote:
On 23 Feb 2012, at 19:38, Ricardo Salveti wrote:
On Mon, Feb 20, 2012 at 6:47 AM, Dave Pigott dave.pigott@linaro.org
wrote:
As far as I understand it, the test definition specifies the device
tags
that it requires, and the scheduler pushes the job to an appropriately tagged board. That said, I haven't tried it myself. We now have several boards with audio-loopback and usb-flash-drive tags enabled, more will
be
added as they come offline.
Great, need to give the usb test a try!
Should we also add tags like wifi and bluetooth for the boards we have? I know these features are locked per board type, but it might also make sense to add the specific capabilities as tags as well.
I bought this up at Connect, but the general feeling was that you
wouldn't target a test at a board type that didn't support that functionality.
But then we would not cover the test case where someone would just like to run a bluetooth test, without actually depending on a board type. Without the tags, it'd be hard to know if the test case should run on an imx53 board or not.
As it'd be easy to add and maintain such tags, I don't see why we shouldn't allow that.
I like the idea of explicitely tagging our board types with a default set of hardware features available...
why wouldn't we do that?
On Fri, Feb 24, 2012 at 6:21 AM, Alexander Sack asac@linaro.org
I like the idea of explicitely tagging our board types with a default set of hardware features available...
why wouldn't we do that?
Of course we're doing that, that's the whole point of adding the tagging support. However I think we should sort out what kinds of tags people want. For instance, at this point, all our boards have ethernet (or at least have it through a dongle). In fact, they won't work without it. Should we have a tag for it? I think the audio-loopback is an obvious one, usb-storage or somthing like for indicating that it has a usb stick attached, but what others would you make use of in your jobs? Adding the tag is not hard at all, just needs to be maintained over time.
Thanks, Paul Larson
On 24 Feb 2012, at 15:17, Paul Larson wrote:
On Fri, Feb 24, 2012 at 6:21 AM, Alexander Sack asac@linaro.org I like the idea of explicitely tagging our board types with a default set of hardware features available...
why wouldn't we do that? Of course we're doing that, that's the whole point of adding the tagging support. However I think we should sort out what kinds of tags people want. For instance, at this point, all our boards have ethernet (or at least have it through a dongle). In fact, they won't work without it. Should we have a tag for it? I think the audio-loopback is an obvious one, usb-storage or somthing like for indicating that it has a usb stick attached, but what others would you make use of in your jobs? Adding the tag is not hard at all, just needs to be maintained over time.
Thanks, Paul Larson
What Ricardo was proposing was WiFi and Bluetooth
Dave
On Fri, Feb 24, 2012 at 9:32 AM, Dave Pigott dave.pigott@linaro.org wrote:
What Ricardo was proposing was WiFi and Bluetooth
Sure, let's add those. But while we're adding them, can we just find out if there are any more that we know right now that anyone wants to use?
On 24 February 2012 18:00, Paul Larson paul.larson@linaro.org wrote:
On Fri, Feb 24, 2012 at 9:32 AM, Dave Pigott dave.pigott@linaro.org wrote:
What Ricardo was proposing was WiFi and Bluetooth
Sure, let's add those. But while we're adding them, can we just find out if there are any more that we know right now that anyone wants to use?
We can take a look to the testing spreadsheet we're using as they provide a list of tags for our QA test cases and bugs. WiFi and Bluetooth are part of them.
Cheers
On Mon, 20 Feb 2012 10:08:42 +0100, Alexander Sack asac@linaro.org wrote:
On Sat, Feb 18, 2012 at 5:42 PM, Dave Pigott dave.pigott@linaro.org wrote:
Hi Paul,
All cables in, just not installed. I started the installations on Tuesday, but had to pick boards that were available. Didn't want to upgrade while boards were running tests. I created an "audio-loopback" tag and associated with the boards that currently have it installed. Intend to start the upgrade with the others on Monday.
Great. How to set requirement tags for jobs submitted? Is this something we can do inside the test definition? Or is this decided by the job submitter atm?
It's done in the job submission. "device_tags": ["audio-loopback"] is the syntax to use.
Cheers, mwh
On Tue, Feb 21, 2012 at 09:02:16AM +1300, Michael Hudson-Doyle wrote:
On Mon, 20 Feb 2012 10:08:42 +0100, Alexander Sack asac@linaro.org wrote:
On Sat, Feb 18, 2012 at 5:42 PM, Dave Pigott dave.pigott@linaro.org wrote:
Hi Paul,
All cables in, just not installed. I started the installations on Tuesday, but had to pick boards that were available. Didn't want to upgrade while boards were running tests. I created an "audio-loopback" tag and associated with the boards that currently have it installed. Intend to start the upgrade with the others on Monday.
Great. How to set requirement tags for jobs submitted? Is this something we can do inside the test definition? Or is this decided by the job submitter atm?
It's done in the job submission. "device_tags": ["audio-loopback"] is the syntax to use.
This does raise an interesting question. Perhaps test definitions could require certain tags and then the scheduler could automatically assemble the superset of required tags.
I guess it depends on why we want to use tags: 1) Because the test will fail on devices without them 2) Because we want to anrrow the possible selection of devices.
Best regards ZK
On Mon, 20 Feb 2012 20:30:23 +0000, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
On Tue, Feb 21, 2012 at 09:02:16AM +1300, Michael Hudson-Doyle wrote:
On Mon, 20 Feb 2012 10:08:42 +0100, Alexander Sack asac@linaro.org wrote:
On Sat, Feb 18, 2012 at 5:42 PM, Dave Pigott dave.pigott@linaro.org wrote:
Hi Paul,
All cables in, just not installed. I started the installations on Tuesday, but had to pick boards that were available. Didn't want to upgrade while boards were running tests. I created an "audio-loopback" tag and associated with the boards that currently have it installed. Intend to start the upgrade with the others on Monday.
Great. How to set requirement tags for jobs submitted? Is this something we can do inside the test definition? Or is this decided by the job submitter atm?
It's done in the job submission. "device_tags": ["audio-loopback"] is the syntax to use.
This does raise an interesting question. Perhaps test definitions could require certain tags and then the scheduler could automatically assemble the superset of required tags.
Well, the scheduler doesn't know anything about the tests currently... I guess this is another thing that could change, or we could add some kind of nice interface for building job definition files that does know about tests. In general I like having a the job json as a low-level thing that we can pass around to very explicitly describe a job but that doesn't mean it's how our users should see a job.
Cheers, mwh
Hi all,
I'm able to create a branch with bzr to deliver correctly the test definition file, anyway I can find it here: http://git.linaro.org/gitweb?p=people/kurt-r-taylor/e2eaudiotest.git%3Ba=blo...
http://git.linaro.org/gitweb?p=people/kurt-r-taylor/e2eaudiotest.git;a=blob;f=e2eaudiotest.py;h=4a737875d27b19521869e48395797d8204ea0656;hb=HEADPaul can you merge this file in lava-test ?
Regards, Benjamin
2012/2/20 Michael Hudson-Doyle michael.hudson@canonical.com
On Mon, 20 Feb 2012 10:08:42 +0100, Alexander Sack asac@linaro.org wrote:
On Sat, Feb 18, 2012 at 5:42 PM, Dave Pigott dave.pigott@linaro.org
wrote:
Hi Paul,
All cables in, just not installed. I started the installations on
Tuesday,
but had to pick boards that were available. Didn't want to upgrade
while
boards were running tests. I created an "audio-loopback" tag and
associated
with the boards that currently have it installed. Intend to start the upgrade with the others on Monday.
Great. How to set requirement tags for jobs submitted? Is this something
we
can do inside the test definition? Or is this decided by the job
submitter
atm?
It's done in the job submission. "device_tags": ["audio-loopback"] is the syntax to use.
Cheers, mwh
Great! by any chance have you run this through lava as an out of tree test already? Also, it looks like https://code.launchpad.net/~benjamin-gaignard/lava-test/e2eaudiotest hasn't been updated in a while, and there is not merge proposal for it. Is the wrapper code for lava-test in there good?
Thanks, Paul Larson
On Thu, Feb 23, 2012 at 8:35 AM, Benjamin Gaignard < benjamin.gaignard@linaro.org> wrote:
Hi all,
I'm able to create a branch with bzr to deliver correctly the test definition file, anyway I can find it here: http://git.linaro.org/gitweb?p=people/kurt-r-taylor/e2eaudiotest.git%3Ba=blo...
http://git.linaro.org/gitweb?p=people/kurt-r-taylor/e2eaudiotest.git;a=blob;f=e2eaudiotest.py;h=4a737875d27b19521869e48395797d8204ea0656;hb=HEADPaul can you merge this file in lava-test ?
Regards, Benjamin
2012/2/20 Michael Hudson-Doyle michael.hudson@canonical.com
On Mon, 20 Feb 2012 10:08:42 +0100, Alexander Sack asac@linaro.org wrote:
On Sat, Feb 18, 2012 at 5:42 PM, Dave Pigott dave.pigott@linaro.org
wrote:
Hi Paul,
All cables in, just not installed. I started the installations on
Tuesday,
but had to pick boards that were available. Didn't want to upgrade
while
boards were running tests. I created an "audio-loopback" tag and
associated
with the boards that currently have it installed. Intend to start the upgrade with the others on Monday.
Great. How to set requirement tags for jobs submitted? Is this
something we
can do inside the test definition? Or is this decided by the job
submitter
atm?
It's done in the job submission. "device_tags": ["audio-loopback"] is the syntax to use.
Cheers, mwh
--
Benjamin Gaignard
Multimedia Working Group
Linaro.org http://www.linaro.org/* **│ *Open source software for ARM SoCs
**
Follow *Linaro: *Facebook http://www.facebook.com/pages/Linaro | Twitterhttp://twitter.com/#%21/linaroorg | Blog http://www.linaro.org/linaro-blog/
linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation
linaro-validation@lists.linaro.org
-
Alexander Sack -
Benjamin Gaignard -
Dave Pigott -
Fathi Boudra -
Michael Hudson-Doyle -
Paul Larson -
Ricardo Salveti -
Zygmunt Krynicki