Apologies if this mail is a little incomprehensible. I'm brain dumping in a hurry :-)
The lava dispatcher currently communicates where the results it has uploaded to the dashboard have ended up to the scheduler by having an extra file descriptor open to the scheduler monitor process -- both stdout and stderr are pretty messy and I didn't want to take the risk of e.g. a boot message being mis-interpreted as structured data.
As well as being silly and over-engineered, this is implicated in my debugging of the "jobs don't always exit properly" bug. So let's stop using it.
What I'd like to do instead is add an --output-dir option to the dispatcher, pass it from the scheduler, have the scheduler write the location of the uploaded results to a known location in this output directory. In fact, I've done this:
http://bazaar.launchpad.net/~mwhudson/lava-dispatcher/output-dir/revision/53... http://bazaar.launchpad.net/~mwhudson/lava-scheduler/use-dispatcher-output-d...
Doesn't seem so bad really.
This fixed the immediate problem, but I want to go further: I want to store more structured log output (filtering boot logs away, for example) in this output directory and gradually teach the scheduler web bits how to interpret this structured data.
Thoughts?
Cheers, mwh
On 01/14/2013 08:32 PM, Michael Hudson-Doyle wrote:
Apologies if this mail is a little incomprehensible. I'm brain dumping in a hurry :-)
The lava dispatcher currently communicates where the results it has uploaded to the dashboard have ended up to the scheduler by having an extra file descriptor open to the scheduler monitor process -- both stdout and stderr are pretty messy and I didn't want to take the risk of e.g. a boot message being mis-interpreted as structured data.
As well as being silly and over-engineered, this is implicated in my debugging of the "jobs don't always exit properly" bug. So let's stop using it.
What I'd like to do instead is add an --output-dir option to the dispatcher, pass it from the scheduler, have the scheduler write the location of the uploaded results to a known location in this output directory. In fact, I've done this:
http://bazaar.launchpad.net/~mwhudson/lava-dispatcher/output-dir/revision/53... http://bazaar.launchpad.net/~mwhudson/lava-scheduler/use-dispatcher-output-d...
Doesn't seem so bad really.
This fixed the immediate problem, but I want to go further: I want to store more structured log output (filtering boot logs away, for example) in this output directory and gradually teach the scheduler web bits how to interpret this structured data.
Thoughts?
It simplifies things and looks better, so I'm a big fan. I have a couple of minor nitpicks with the actual patches, but that can wait until the MP's are created if you prefer.
Andy Doan andy.doan@linaro.org writes:
On 01/14/2013 08:32 PM, Michael Hudson-Doyle wrote:
Apologies if this mail is a little incomprehensible. I'm brain dumping in a hurry :-)
The lava dispatcher currently communicates where the results it has uploaded to the dashboard have ended up to the scheduler by having an extra file descriptor open to the scheduler monitor process -- both stdout and stderr are pretty messy and I didn't want to take the risk of e.g. a boot message being mis-interpreted as structured data.
As well as being silly and over-engineered, this is implicated in my debugging of the "jobs don't always exit properly" bug. So let's stop using it.
What I'd like to do instead is add an --output-dir option to the dispatcher, pass it from the scheduler, have the scheduler write the location of the uploaded results to a known location in this output directory. In fact, I've done this:
http://bazaar.launchpad.net/~mwhudson/lava-dispatcher/output-dir/revision/53... http://bazaar.launchpad.net/~mwhudson/lava-scheduler/use-dispatcher-output-d...
Doesn't seem so bad really.
This fixed the immediate problem, but I want to go further: I want to store more structured log output (filtering boot logs away, for example) in this output directory and gradually teach the scheduler web bits how to interpret this structured data.
Thoughts?
It simplifies things and looks better, so I'm a big fan.
Cool.
I have a couple of minor nitpicks with the actual patches, but that can wait until the MP's are created if you prefer.
Feel free to elaborate, especially your nitpicks are structural things, but the patches were pretty much hacks so I'm not super surprised they're not perfect :)
Cheers, mwh
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Andy Doan andy.doan@linaro.org writes:
On 01/14/2013 08:32 PM, Michael Hudson-Doyle wrote:
Apologies if this mail is a little incomprehensible. I'm brain dumping in a hurry :-)
The lava dispatcher currently communicates where the results it has uploaded to the dashboard have ended up to the scheduler by having an extra file descriptor open to the scheduler monitor process -- both stdout and stderr are pretty messy and I didn't want to take the risk of e.g. a boot message being mis-interpreted as structured data.
As well as being silly and over-engineered, this is implicated in my debugging of the "jobs don't always exit properly" bug. So let's stop using it.
What I'd like to do instead is add an --output-dir option to the dispatcher, pass it from the scheduler, have the scheduler write the location of the uploaded results to a known location in this output directory. In fact, I've done this:
http://bazaar.launchpad.net/~mwhudson/lava-dispatcher/output-dir/revision/53... http://bazaar.launchpad.net/~mwhudson/lava-scheduler/use-dispatcher-output-d...
Doesn't seem so bad really.
This fixed the immediate problem, but I want to go further: I want to store more structured log output (filtering boot logs away, for example) in this output directory and gradually teach the scheduler web bits how to interpret this structured data.
Thoughts?
It simplifies things and looks better, so I'm a big fan.
Cool.
I have a couple of minor nitpicks with the actual patches, but that can wait until the MP's are created if you prefer.
Feel free to elaborate, especially your nitpicks are structural things, but the patches were pretty much hacks so I'm not super surprised they're not perfect :)
I've pushed the dispatcher further in what I think is a sort of interesting direction. Branch at https://code.launchpad.net/~mwhudson/lava-dispatcher/moar-output-dir, diff to trunk at http://paste.ubuntu.com/1540064/.
The basic idea is to create a few files in the specified output directory and funnel different sorts of output to different files. So in this branch we have a file for the serial output, we have a file for log messages and we have a file for "everything". There is also a way for code to say that output is not very interesting and shouldn't be sprayed to stdout (although the use of this feature in the branch is a bit silly). These choices are perhaps not very exciting, but I think the facility could be useful.
I'd like to extend this to some kind of structured output file -- which would do things like point to another file that contained the master boot logs and then the scheduler could interpret this and allow the master boot logs to be hidden by default but expanded on request. Similar for things like lmc output.
This is your chance to say that this is all a bit crazy and I should stop (as I'm off until next Tuesday) but I do actually like where this is going (I've been wanting to do something like this for over a year, I think).
Cheers, mwh
On 01/16/2013 08:23 PM, Michael Hudson-Doyle wrote:
This is your chance to say that this is all a bit crazy and I should stop (as I'm off until next Tuesday) but I do actually like where this is going (I've been wanting to do something like this for over a year, I think).
I basically like it. I should read closer, but it almost felt like things might break if you didn't pass a --output-dir to the command line.
The other thing that seemed odd was:
with target.runner() as runner: - runner.wait_for_prompt(timeout) + with target.context.outputter.diverted(): + runner.wait_for_prompt(timeout)
Doesn't this mean we wouldn't get the boot log of the device to stdout? I can see doing that for booting to master images, but why this?
Andy Doan andy.doan@linaro.org writes:
On 01/16/2013 08:23 PM, Michael Hudson-Doyle wrote:
This is your chance to say that this is all a bit crazy and I should stop (as I'm off until next Tuesday) but I do actually like where this is going (I've been wanting to do something like this for over a year, I think).
I basically like it. I should read closer, but it almost felt like things might break if you didn't pass a --output-dir to the command line.
Ah yes, possibly. Will check that.
The other thing that seemed odd was:
with target.runner() as runner:
runner.wait_for_prompt(timeout)
with target.context.outputter.diverted():runner.wait_for_prompt(timeout)Doesn't this mean we wouldn't get the boot log of the device to stdout? I can see doing that for booting to master images, but why this?
Because I was writing this in a cafe without access to a master image device :-)
Cheers, mwh
linaro-validation@lists.linaro.org
-
Andy Doan -
Michael Hudson-Doyle -
Michael Hudson-Doyle