dependencies between actions in the dispatcher

Michael Hudson-Doyle michael.hudson at linaro.org
Mon Nov 14 22:06:42 UTC 2011


On Mon, 14 Nov 2011 13:17:25 -0600, Paul Larson <paul.larson at linaro.org> wrote:
> > 1) Change submit results to not be an action.
> >
> So it would be implicit that we always want to submit results? I know there
> are times that we do jobs that do not submit results.  For instance, when
> testing the scheduler, or other pieces of LAVA.

Well, that's interesting data then :-)

> Also, where would the results be submitted from? Still from the
> dispatcher?

Err, I was assuming so.  I guess it doesn't have to be that way, but I
think for the use of people who run the dispatcher by hand, without the
scheduler, it will be much easier if the dispatcher submits the jobs.

> There's an unrelated issue we've discussed before with submitting
> results to streams that must be authenticated to.  One of the earlier
> options that came up for this was submitting results from the
> scheduler, however I'm not sure I like that any better.

I think we came up with a plan for this didn't we?  Have the scheduler
generate a token when the job starts, stuff it in the job given to the
dispatcher and invalidate it when the job finishes.

> > 2) Add a result_locations list and action_data dictionary to
> >   LavaContext.  My half-thought through idea is that actions will use
> >   the action name as a prefix, e.g. deploy_linaro_image for a qemu
> >   client might set 'deploy_linaro_image.qemu_img_path' in this dict.
> >
> Those would be parameters to the action?

Not in the sense that those words mean things today no.  What I'm
proposing is that actions record on the context where the result bundles
are...

> The way it works right now is that we have a default location where
> all the test results go, and then they are gathered at the end.

... rather than assuming they all end up in the same place.

> I'm not sure I understand how this helps to let the job submitter
> specify the location.

That's not surprising because that's not what I was proposing :-)
Apologies for lack of clarity on my side.

> > 3) Change the lava-test and lava-android-test to store into
> >   result_locations and the submit step to read from there.
> >
> See #2, we already do this I believe, we just don't give an option to
> change that location at runtime.  This location is _inside_ the image
> because thats where the parsing takes place.  I'm starting to wonder if you
> want it to be stored outside the image for qemu testing purposes?  Since
> lava-test is the piece that parses it, it needs to be inside the image, but
> with lava-android-test it could certainly be outside.

Again, it's not so much that I want to prescribe where the test results
go, but rather not assuming so much about where they are, or perhaps how
to get at them -- for qemu, we're not going to boot a known good image,
mount the tested rootfs, copy files around and then run an HTTP server
to get the results onto the host (well, we _could_ I guess, but that
would be properly crazy).

> > 4) Use action data to have deploy_linaro_image and boot_linaro_image
> >   (and maybe lava_test_install and lava_test_run) talk to each other.
> >
> Maybe if you showed how this might look in json it would make sense to
> me.

The stuff I was trying to propose does not imply changing the json at
all.

> I don't think we should be too restrictive with how it's used though.  One
> thing that's come up before as a possible user of this is to have outputs
> that come out of some actions used in other actions.

This sort of question is _precisely_ what I'm talking about! :)

And a point I was trying to make, probably not very clearly, is that we
_already_ have outputs of actions that are used in other actions[1] and
ideally any framework we build to have more of this sort of this thing
should incorporate what we have now, not be some parallel system.

[1] the bundles produced by lava_test_run and also the state of the
    testrootfs and testboot partitions, although that last one is a bit
    hidden and implicit

> For instance, if we had a build_foo action that you pointed at a
> source tree and produced output, how could you point a subsequent step
> at the artifacts of that build for consumption?

I did half-hearted think of inventing some kind of syntax for writing
actions that could have outputs that could be referred to by later
actions, something like:

    {
      "command": "build_kernel",
      "parameters":
        {
          "git_location": "git:..."
        },
      "outputs": ["kernel_deb_location"]
    },
    {
      "command": "inject_kernel",
      "parameters":
        {
          "deb_location": "${build_kernel.kernel_deb_location}"
        },
      "outputs": ["hwpack_location"]
    },
    {
      "command": "deploy_linaro_image",
      "parameters":
        {
          "rootfs": "...",
          "hwpack": "${inject_kernel.hwpack_location}"
        }
    },

... but this seems a bit over-engineered somehow?

Cheers,
mwh



More information about the linaro-dev mailing list