If you are sure that the jobs is finished (the corresponding lava-run is not running), you can force a job to finish by using "lava-server manage jobs fail <job_id>".

This will change the job and device state in the database directly. This is useful if something went wrong and you are sure that lava-run has terminated.


Regards

2018-04-30 13:04 GMT+02:00 Tim Jaacks <tim.jaacks@garz-fricke.com>:
Hi Remi,

thanks for your quick reply.

>Hello,
>
>rebooting the server will not fix anything as the status is saved into the database (so kept between reboots).
>
>1/ Are lava-master, lava-slave and lava-logs all running?

I think so, yes:

tim.jaacks@A048:~$ ps -ax | grep lava
  637 ?        Ssl    0:01 /usr/bin/python /usr/bin/lava-server manage lava-publisher --level DEBUG
  673 ?        Ssl    0:41 /usr/bin/python /usr/bin/lava-server manage lava-logs --level DEBUG
  678 ?        Ssl    1:26 /usr/bin/python /usr/bin/lava-slave --level DEBUG --master tcp://localhost:5556 --socket-addr tcp://localhost:5555
  824 ?        S      0:00 /usr/bin/python /usr/bin/lava-coordinator --loglevel=DEBUG
 1017 ?        S      9:30 /usr/bin/python /usr/sbin/lavapdu-runner --loglevel=INFO
 1018 ?        S      0:51 /usr/bin/python /usr/sbin/lavapdu-listen --loglevel=INFO
 1473 ?        Ss     0:00 postgres: 9.6/main: lavaserver lavaserver ::1(35670) idle
 2476 ?        Ss     1:19 gunicorn: master [lava_server.wsgi]
 2482 ?        S      0:12 gunicorn: worker [lava_server.wsgi]
 2483 ?        S      0:11 gunicorn: worker [lava_server.wsgi]
 2484 ?        S      0:13 gunicorn: worker [lava_server.wsgi]
 2489 ?        S      0:13 gunicorn: worker [lava_server.wsgi]
 3077 ?        Ssl   24:59 /usr/bin/python /usr/bin/lava-server manage lava-master --level DEBUG
 3079 ?        Ss     8:13 postgres: 9.6/main: lavaserver lavaserver ::1(38208) idle
 3418 pts/0    S+     0:00 grep lava

>2/ Do you see anything in the corresponding logs? (/var/log/lava-server/lava-master.log, /var/log/lava-server/lava-logs.log, /var/log/lava-dispatcher/lava-slave.log)

Oh yes, there is something in lava-logs.log:

2018-04-30 11:00:52,046   DEBUG PING => master (last message 20s ago)
2018-04-30 11:00:52,106    INFO master => CANCEL(357)
2018-04-30 11:00:52,106   DEBUG [357] Unknown job, sending END
2018-04-30 11:00:52,107   ERROR [357] Unable to read 'description.yaml'
2018-04-30 11:00:52,107   ERROR [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/slave/tmp/357/description.yaml'
Traceback (most recent call last):
  File "/usr/bin/lava-slave", line 180, in description
    data = open(filename, 'r').read()
IOError: [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/slave/tmp/357/description.yaml'
2018-04-30 11:00:52,107   ERROR [357] lava-run crashed
2018-04-30 11:00:52,107   DEBUG master => PONG(20)
2018-04-30 11:00:52,114    INFO master => END_OK(357)
2018-04-30 11:00:52,114   DEBUG [357] Job END acked
2018-04-30 11:00:52,114   DEBUG [357] Removing /var/lib/lava/dispatcher/slave/tmp/357

Can you tell what this means?

>3/ Which version of lava re you using?

2018.2

>
>Regards
>
>2018-04-30 10:14 GMT+02:00 Tim Jaacks <tim.jaacks@garz-fricke.com>:
>Hey everyone,
>
>I still have this problem. Can anyone assist me? Rebooting the server does not help. :-/
>
>Regards,
>Tim
>
>
>>Hi Neil,
>>
>>thanks for your quick help.
>>
>>>> Hello everyone,
>>>>
>>>> I added an email notification to a test job but forgot to configure
>>>> an SMTP server first.
>>>
>>>
>>>There will likely be an exception logged in
>>>/var/log/lava-server/lava-master.log
>>>
>>>You will need to restart the lava-master
>>>
>>>$ sudo service lava-master restart
>>>
>>>Then check the logs.
>>
>>My log says:
>>
>>2018-04-24 14:22:08,025    INFO [POLL] Received a signal, leaving
>>2018-04-24 14:22:08,025    INFO [CLOSE] Closing the controler socket and dropping messages
>>2018-04-24 14:22:09,183    INFO [INIT] Dropping privileges
>>2018-04-24 14:22:09,184   DEBUG Switching to (lavaserver(114), lavaserver(118))
>>2018-04-24 14:22:09,184    INFO [INIT] Marking all workers as offline
>>2018-04-24 14:22:09,191    INFO [INIT] LAVA master has started.
>>2018-04-24 14:22:09,191    INFO [INIT] Using protocol version 3
>>2018-04-24 14:22:23,300   DEBUG lava-logs => PING(20)
>>2018-04-24 14:22:25,485   DEBUG A048.HAMBURG => PING(20)
>>2018-04-24 14:22:25,485 WARNING Unknown dispatcher <A048.HAMBURG> (server crashed)
>>2018-04-24 14:22:29,195    INFO scheduling health checks:
>>2018-04-24 14:22:29,234   DEBUG -> disabled on: lxc, qemu
>>2018-04-24 14:22:29,235    INFO scheduling jobs:
>>2018-04-24 14:22:29,236   DEBUG - beaglebone-black
>>2018-04-24 14:22:29,246   DEBUG - imx6guf
>>2018-04-24 14:22:29,255   DEBUG - lxc
>>2018-04-24 14:22:29,263   DEBUG - qemu
>>2018-04-24 14:22:29,284    INFO [357] CANCEL => A048.HAMBURG
>>2018-04-24 14:22:29,287    INFO [357] A048.HAMBURG => END (duplicated), skipping
>>
>>What does this mean? FYI: Master and dispatcher are on the same machine. Can I restart the dispatcher somehow?
>>
>>>You don't need an SMTP server to be configured in LAVA but you do need
>>>a working mail configuration on the master so that you can send email
>>>to external addresses using the command line program, mail. The SMTP
>>>settings in lava-server are an alternative. If you use the external
>>>mail support, mail errors would remain in the email system of the master.
>>>
>>>
>>>
>>>> The job reported:
>>>>
>>>> "JobError: Your job cannot terminate cleanly."
>>>>
>>>> Afterwards nothing happened. The job was still running, even after
>>>> all timeouts had been passed, so I tried to cancel it. Now the job
>>>> remains in "Cancelling" state and I have no idea why and how to fix this. Any hints?
>>>>
>>>> Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT
>>>> ENGINEER Garz & Fricke GmbH Tempowerkring 2
>>>> 21079 Hamburg
>>>> Direct: +49 40 791 899 - 55
>>>> Fax: +49 40 791899 - 39
>>>> tim.jaacks at garz-fricke.com
>>>> www.garz-fricke.com
>>>> SOLUTIONS THAT COMPLETE!
>>>>
>>>> Sitz der Gesellschaft: D-21079 Hamburg
>>>> Registergericht: Amtsgericht Hamburg, HRB 60514
>>>> Geschäftsführer: Matthias Fricke, Manfred Garz
>>>>
>>>> _______________________________________________
>>>> Lava-users mailing list
>>>> Lava-users at lists.linaro.org
>>>> https://lists.linaro.org/mailman/listinfo/lava-users
>>>>
>>>
>>>
>>>
>>>--
>>>
>>>Neil Williams
>>>=============
>>>neil.williams at linaro.org
>>>http://www.linux.codehelp.co.uk/
>>>-------------- next part -------------- An HTML attachment was
>>>scrubbed...
>>>URL:
>>><http://lists.linaro.org/pipermail/lava-users/attachments/20180424/e7a0
>>>46b0/attachment.html>
>>
>>
>>Mit freundlichen Grüßen / Best regards
>>Tim Jaacks
>>DEVELOPMENT ENGINEER
>>Garz & Fricke GmbH
>>Tempowerkring 2
>>21079 Hamburg
>>Direct: +49 40 791 899 - 55
>>Fax: +49 40 791899 - 39
>>tim.jaacks@garz-fricke.com
>>www.garz-fricke.com
>>SOLUTIONS THAT COMPLETE!
>>
>>Sitz der Gesellschaft: D-21079 Hamburg
>>Registergericht: Amtsgericht Hamburg, HRB 60514
>>Geschäftsführer: Matthias Fricke, Manfred Garz _______________________________________________
>>Lava-users mailing list
>>Lava-users@lists.linaro.org
>>https://lists.linaro.org/mailman/listinfo/lava-users
>>
>_______________________________________________
>Lava-users mailing list
>Lava-users@lists.linaro.org
>https://lists.linaro.org/mailman/listinfo/lava-users
>
>
>
>
>--
>Rémi Duraffort
>LAVA Team
>
_______________________________________________
Lava-users mailing list
Lava-users@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lava-users



--
Rémi Duraffort
LAVA Team