On 17/01/2018 12:02, Neil Williams wrote:
> On 17 January 2018 at 10:47, Loys Ollivier <lollivier@baylibre.com> wrote:
>
>>
>>
>> On 17/01/2018 11:28, Neil Williams wrote:
>>> On 17 January 2018 at 10:04, Loys Ollivier <lollivier@baylibre.com>
>> wrote:
>>>
>>>>
>>>>
>>>> On 17/01/2018 10:52, Neil Williams wrote:
>>>>> On 17 January 2018 at 09:48, Loys Ollivier <lollivier@baylibre.com>
>>>> wrote:
>>>>>
>>>>>>
>>>>>> Hi Stevan, all,
>>>>>>
>>>>>> Following on that here's the output of the lava-tool using version
>>>>>> 0.24-1. Seems like there's a new bug:
>>>>>>
>>>>>> --8>
>>>>>>
>>>>>> $ lava-tool wait-job-events http://loys@lava-test:10080
>>>> --job-definition
>>>>>> job.yaml
>>>>>> submitted as job(s):
>>>>>> http://lava-test:10080/scheduler/job/7528
>>>>>
>>>>>
>>>>> This isn't a typical way to run LAVA, there's no guarantee that it will
>>>>> work on a random port - make sure you're not trying to run LAVA as if
>> it
>>>>> was just another django project, it is not. The django internal support
>>>> is
>>>>> not supported for LAVA. It needs to be packaged (using the developer
>>>> build
>>>>> scripts if you want to build latest) and have apache properly
>> configured.
>>>>> (It's also advisable to use https:// to protect the authentication
>>>> tokens.)
>>>>>
>>>>>
>>>>
>>>> Thank you for your feedback. Although this is not a typical way, I am
>>>> using it on a local network and it has been working properly for several
>>>> months.
>>>
>>>
>>> There is no guarantee that it will continue to work and bugs or issues in
>>> that setup *must* be replicated on a supported system first.
>>>
>>>
>>>> As it is my testing instance (to not break production) the
>>>> tokens are not sensitive and anyway the network is not connected to the
>>>> outside world.
>>>>
>>>
>>> That answers for the https:// but there is still a problem with the test
>>> instance setup using a random port. Make sure it is installed from
>> packages
>>> and is the latest release, 2018.1.
>>>
>>> Are you trying to run the test instance from the django testproject
>> support?
>>>
>>>
>>
>> I am running a docker container which redirects port 10080 -> 80. I
>> don't think it changes the lava behavior within the container.
>>
>>
> Right, that makes sense. OK - it was a throwback to another issue where the
> test instance was poorly configured. The redirection happened to use the
> same kind of URL.
>
>
>>>> Regarding the latest bug mentioned, it seems like the key name has
>>>> changed in the python dictionary. Playing around with packaging, random
>>>> port number or auth tokens won't help.
>>>>
>>>>
>>> There is a code change in the scheduler - your test instance must be on
>>> 2018.1 for the call in lava-tool 0.24 to work.
>>>
>>> The code as-is does work with those in combination:
>>>
>>> $ lava-tool wait-job-events --job-definition
>>> ~/code/lava/pipeline/git/lxc-ubuntu.yaml local
>>> submitted as job(s):
>>> http://localhost/scheduler/job/12502
>>> Now waiting for job events...
>>> {'pipeline': True, 'description': 'LXC Ubuntu Xenial', 'job': '
>>> http://localhost/scheduler/12502', 'visibility': 'Publicly visible',
>>> 'priority': 50, 'device': 'lxc-01', 'state': 'Scheduled', 'health':
>>> 'Unknown', 'device_type': 'lxc', 'submit_time':
>>> '2018-01-17T10:25:03.621066+00:00', 'submitter': 'neil.williams'}
>>> {'pipeline': True, 'description': 'LXC Ubuntu Xenial', 'start_time':
>>> '2018-01-17T10:25:09.425483+00:00', 'job': 'http://localhost/scheduler/
>> 12502',
>>> 'visibility': 'Publicly visible', 'priority': 50, 'device': 'lxc-01',
>>> 'state': 'Running', 'health': 'Unknown', 'device_type': 'lxc',
>>> 'submit_time': '2018-01-17T10:25:03.621066+00:00', 'submitter':
>>> 'neil.williams'}
>>> {'pipeline': True, 'description': 'LXC Ubuntu Xenial', 'start_time':
>>> '2018-01-17T10:25:09.425483+00:00', 'job': 'http://localhost/scheduler/
>> 12502',
>>> 'visibility': 'Publicly visible', 'priority': 50, 'device': 'lxc-01',
>>> 'state': 'Finished', 'health': 'Incomplete', 'end_time':
>>> '2018-01-17T10:25:12.979031+00:00', 'device_type': 'lxc', 'submit_time':
>>> '2018-01-17T10:25:03.621066+00:00', 'submitter': 'neil.williams'}
>>>
>>> The problem is with the data structure returned by your test instance,
>> it's
>>> not a problem in lava-tool which has been updated for those changes.
>>>
>>> $ dpkg -l lava-server lava-dispatcher
>>> Desired=Unknown/Install/Remove/Purge/Hold
>>> |
>>> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/
>> trig-aWait/Trig-pend
>>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
>>> ||/ Name                          Version             Architecture
>>> Description
>>> +++-=============================-===================-======
>> =============-==============================================
>> ==================
>>> ii  lava-dispatcher               2018.1+4769.7a93650 amd64
>>>  Linaro Automated Validation Architecture dispatcher
>>> ii  lava-server                   2018.1+7019.756d2e6 all
>>>  Linaro Automated Validation Architecture server
>>>
>>>
>>>
>>>
>>
>> Ok, thank you for the pointer. I wasn't aware that lava-tool version
>> requires a certain version of lava-server. Maybe this could be checked
>> in lava-tool to avoid this bug ? Like a message reporting that the
>> lava-server version must be >= 2018.01 for instance ?
>>
>
> Normally, we would do that on the server side through the XML-RPC interface
> - deprecate the old call and add a new one. lava-tool is beginning to show
> problems from a lot of historical legacy code and it's not at all easy to
> remove as we did with the removal of V1. We're looking at a new tool with a
> clean Python3 start as a replacement. Once we have moved forward a bit more
> with that one, I'll make the announcement.
>

I understand thanks.

> In the meantime, it's only this particular functionality in lava-tool which
> is affected. Are you using this particular support frequently? We can look
> at using the XML-RPC call system.api_version from lava-tool to check the
> particular instance being used for the wait-job-events support.
>

Here's what I am using in my CI loop:
# lava-tool submit-job http://loys@lava-test:10080 job.yaml --block
submitted as job: http://lava-test:10080/scheduler/job/75

This kind of polling is deprecated and will be removed in the next
release. Please use "wait-for-job" command.

Hence my question about wait-job-events. As I did not want to use a
deprecated command. I guess I will stick to that one until the new
announcement. I do not have particular issues with the submit-job
command.

>
>> I will try to upgrade lava and test with the new instance.
>>
>
> Thanks, sorry for any confusion - let us know the results either way.
>
>
>>
>>>>>
>>>>>> Now waiting for job events...
>>>>>> Traceback (most recent call last):
>>>>>>   File "/usr/bin/lava-tool", line 11, in <module>
>>>>>>     load_entry_point('lava-tool==0.24', 'console_scripts',
>>>> 'lava-tool')()
>>>>>>   File "/usr/lib/python2.7/dist-packages/lava_tool/dispatcher.py",
>> line
>>>>>> 49, in main
>>>>>>     LavaDispatcher.run()
>>>>
>>>>>   File "/usr/lib/python2.7/dist-packages/lava/tool/dispatcher.py",
>> line
>>>>>> 150, in run
>>>>>>     raise SystemExit(cls().dispatch(args))
>>>>>>   File "/usr/lib/python2.7/dist-packages/lava/tool/dispatcher.py",
>> line
>>>>>> 140, in dispatch
>>>>>>     return command.invoke()
>>>>>>   File
>>>>>> "/usr/lib/python2.7/dist-packages/lava_scheduler_tool/commands.py",
>>>> line
>>>>>> 820, in invoke
>>>>>>     if data["state"] == self.FINISHED_JOB_STATE:
>>>>>> KeyError: 'state'
>>>>>>
>>>>>
>>>>> Make sure the LAVA instance itself is running 2018.1 as there have been
>>>>> changes to the scheduler state machine.
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> On 16/12/2017 10:28, Stevan Radakovic wrote:
>>>>>>> Hi Loys,
>>>>>>>
>>>>>>> After some more testing I realized it's a bug in lava-tool which
>> causes
>>>>>>> a problem when you're using a specific port for the lava-server
>>>>>>> endpoint. It's a pretty straightforward fix, here's the tracking
>> ticket
>>>>>>> for it: https://projects.linaro.org/servicedesk/customer/portal/1/
>>>>>> CTT-819
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> On 12/14/2017 05:56 PM, Loys Ollivier wrote:
>>>>>>>>
>>>>>>>> On 14/12/2017 16:18, Stevan Radakovic wrote:
>>>>>>>>> Hi Loys,
>>>>>>>>>
>>>>>>>>> There's auth-add command which is used to add credentials for
>>>>>>>>> authenticatation of lava-tool to the lava-server instance. But if
>> you
>>>>>>>>> can submit a job via lava-tool you've probably already done that.
>>>>>>>>>
>>>>>>>>> Are you receiving no output after submitting the job via lava-tool
>>>>>>>>> wait-job-events?
>>>>>>>>>
>>>>>>>> Exactly.
>>>>>>>>> It's possibly a bug in lava-tool I'm also seeing job completed
>> event
>>>>>>>>> missing when trying it out locally. I'll investigate some more and
>>>> file
>>>>>>>>> a bug.
>>>>>>>>>
>>>>>>>>> lava-tool exits when it receives job completed event, that's the
>> most
>>>>>>>>> likely reason that it hangs.
>>>>>>>> Not sure to get your point here. But I think we're on the same page:
>>>>>>>> lava-tool hangs...
>>>>>>>>
>>>>>>>> Thanks for help !
>>>>>>>>
>>>>>>>> Is there any way I can follow the evolution / resolution of this
>> bug ?
>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> On 12/14/2017 03:35 PM, Loys Ollivier wrote:
>>>>>>>>>> Hello Stevan,
>>>>>>>>>>
>>>>>>>>>> Thanks for the help on that. I enabled the server side
>>>> notifications,
>>>>>>>>>> I can see them using a publisher + python on my host computer.
>>>>>>>>>> Unfortunately lava-tool still doesn't return even when the job
>>>>>>>>>> finishes....
>>>>>>>>>>
>>>>>>>>>> Also, my server is running on a specific port so to access it I am
>>>>>>>>>> using: lava-tool wait-job-events http://user@server:port/RPC2
>>>>>>>>>> It works perfectly well to submit jobs. I also checked that the
>> port
>>>>>>>>>> number 5500 is opened.
>>>>>>>>>>
>>>>>>>>>> Is there anything else I should do on the lava-tool side to get
>> this
>>>>>>>>>> notifications ? Again I can receive them on the host using python
>>>> but
>>>>>>>>>> for some reason lava-tool doesn't work.
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>>
>>>>>>>>>> Loys
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 13, 2017 at 4:47 PM, Stevan Radakovic
>>>>>>>>>> <stevan.radakovic@linaro.org <mailto:stevan.radakovic@linaro.org
>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>      Hi Loys,
>>>>>>>>>>
>>>>>>>>>>      There's definitely bug with the message you get from the
>>>>>>>>>>      "submit-job --block" and we will open a ticket for this -
>>>>>>>>>>      wait-for-job does not exist it was replaced with
>>>> wait-job-events
>>>>>>>>>>      long time ago.
>>>>>>>>>>
>>>>>>>>>>      For using it, you need to enable event notification on your
>>>>>> server
>>>>>>>>>>      first. Please follow these instructions:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://staging.validation.linaro.org/static/docs/v2/
>>>>>> advanced-installation.html#configuring-event-notifications
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <https://staging.validation.linaro.org/static/docs/v2/
>>>>>> advanced-installation.html#configuring-event-notifications>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>      After that, using lava-tool, you can either submit and wait
>> for
>>>>>>>>>>      the job events or track the events of the existing job.
>> Example
>>>>>>>>>>      you provided is good for the latter example.
>>>>>>>>>>
>>>>>>>>>>      To submit and wait you can use:
>>>>>>>>>>      lava-tool wait-job-events --job-definition /path/to/job.yaml
>>>>>>>>>>      http://your.server/RPC2/
>>>>>>>>>>
>>>>>>>>>>      In either case, lava-tool should exit when it receives job
>>>>>>>>>>      'completed' event from the server.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>      HTH,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>      On 12/13/2017 02:54 PM, Loys Ollivier wrote:
>>>>>>>>>>>      Hello everyone,
>>>>>>>>>>>
>>>>>>>>>>>      I am using lava-tool to monitor my jobs. Previously I used:
>>>>>>>>>>>      $ lava-tool submit-job --block
>>>>>>>>>>>
>>>>>>>>>>>      Using version of lava-tool 0.23 I now have this message:
>>>>>>>>>>>      --> This kind of polling is deprecated and will be removed
>> in
>>>>>> the
>>>>>>>>>>>      next release. Please use "wait-for-job" command.
>>>>>>>>>>>
>>>>>>>>>>>      But "wait-for-job" doesn't exist.
>>>>>>>>>>>      There is a "wait-job-events" option though. I tried this one
>>>> and
>>>>>>>>>>>      it doesn't return even once the job has finished. If I
>>>> manually
>>>>>>>>>>>      stop it and restart it with the same job number I get as
>>>> output:
>>>>>>>>>>>      --> Job already finished with status Complete.
>>>>>>>>>>>
>>>>>>>>>>>      Command I'm using:
>>>>>>>>>>>      $ lava-tool wait-job-events --job-id 20
>>>> http://user@lava-server
>>>>>>>>>>>
>>>>>>>>>>>      Is there anything I'm doing incorrectly ? Or are you aware
>> of
>>>>>>>>>>>      this bug ?
>>>>>>>>>>>
>>>>>>>>>>>      Thanks !
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>      --     Loys OLLIVIER
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>      _______________________________________________
>>>>>>>>>>>      Lava-users mailing list
>>>>>>>>>>>      Lava-users@lists.linaro.org <mailto:Lava-users@lists.
>>>> linaro.org
>>>>>>>
>>>>>>>>>>>      https://lists.linaro.org/mailman/listinfo/lava-users
>>>>>>>>>>>      <https://lists.linaro.org/mailman/listinfo/lava-users>
>>>>>>>>>>      --     Stevan Radaković | LAVA Engineer
>>>>>>>>>>      Linaro.org <www.linaro.org <http://www.linaro.org>> │ Open
>>>>>> source
>>>>>>>>>> software for ARM SoCs
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Loys OLLIVIER
>>>>>>>>>> Baylibre
>>>>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Lava-users mailing list
>>>>>> Lava-users@lists.linaro.org
>>>>>> https://lists.linaro.org/mailman/listinfo/lava-users
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>