Hi,
I'm trying to use a situation where a large bulk of similar jobs are being sent to several devices of the same device type. These devices require the 'transfer_overlay' option in order for the tests to work due to their setup. We currently are using a single master and worker on the same server.
It seems that the same transfer overlay file location (i.e., /var/lib/lava/dispatcher/tmp/overlay-1.3.2.5.tar.gz) is used regardless of the job id and that some of our jobs are experiencing a collision trying to use different copies of that file at the same time. Thus, one of our jobs is deleting the transfer overlay file before another job tries to delete it and the second gets errors.
This is the error LAVA produces when viewing the job in the GUI:
File "/usr/bin/lava", line 11, in <module> load_entry_point('lava-tool==0.25', 'console_scripts', 'lava')() File "/usr/lib/python2.7/dist-packages/lava/tool/dispatcher.py", line 150, in run raise SystemExit(cls().dispatch(args)) File "/usr/lib/python2.7/dist-packages/lava/tool/dispatcher.py", line 140, in dispatch return command.invoke() File "/usr/lib/python2.7/dist-packages/lava/dispatcher/commands.py", line 247, in invoke self.args.validate) File "/usr/lib/python2.7/dist-packages/lava/dispatcher/commands.py", line 91, in run_pipeline_job exitcode = job.run() File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/job.py", line 340, in run self.cleanup(self.connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/job.py", line 375, in cleanup self.pipeline.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 251, in cleanup child.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 649, in cleanup self.internal_pipeline.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 251, in cleanup child.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 649, in cleanup self.internal_pipeline.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 251, in cleanup child.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/actions/boot/__init__.py", line 458, in cleanup os.unlink(self.url) OSError: [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/tmp/overlay-1.3.2.5.tar.gz'
We are working on the 2017.7 build of LAVA, so if this has been changed in a later build, please let me know. If this has not been updated, is there another way to change things so that each job uses a different name and/or location for the overlay file?
Thanks,
Jorgen Peddersen
Jorgen Peddersen Researcher t +61 2 6103 4700 m e jorgen.peddersen@seeingmachines.com a 80 Mildura Street, Fyshwick ACT 2609, Australia SEEINGMACHINES.COMhttp://seeingmachines.com
[SEEINGMACHINES]
This email may contain confidential information. If you are not the intended recipient, destroy all copies and do not disclose or use the information within. Any content that is not concerned with the business of the Seeing Machines Group reflects the views of the sender only, not those of Seeing Machines. No warranties are given that this email does not contain viruses or harmful code. [http://www.seeingmachines.com/wp-content/uploads/2018/04/sm-email-twitter-ic...] https://twitter.com/seeingmachines [http://www.seeingmachines.com/wp-content/uploads/2018/04/sm-email-linkedin-i...] https://www.linkedin.com/company/258949/
Hi Jorgen,
On Mon, Aug 13, 2018 at 02:23:34AM +0000, Jorgen Peddersen wrote:
I'm trying to use a situation where a large bulk of similar jobs are being sent to several devices of the same device type. These devices require the 'transfer_overlay' option in order for the tests to work due to their setup. We currently are using a single master and worker on the same server.
It seems that the same transfer overlay file location (i.e., /var/lib/lava/ dispatcher/tmp/overlay-1.3.2.5.tar.gz) is used regardless of the job id and that some of our jobs are experiencing a collision trying to use different copies of that file at the same time. Thus, one of our jobs is deleting the transfer overlay file before another job tries to delete it and the second gets errors.
This is the error LAVA produces when viewing the job in the GUI:
File "/usr/bin/lava", line 11, in <module> load_entry_point('lava-tool==0.25', 'console_scripts', 'lava')() File "/usr/lib/python2.7/dist-packages/lava/tool/dispatcher.py", line 150, in run raise SystemExit(cls().dispatch(args)) File "/usr/lib/python2.7/dist-packages/lava/tool/dispatcher.py", line 140, in dispatch return command.invoke() File "/usr/lib/python2.7/dist-packages/lava/dispatcher/commands.py", line 247, in invoke self.args.validate) File "/usr/lib/python2.7/dist-packages/lava/dispatcher/commands.py", line 91, in run_pipeline_job exitcode = job.run() File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/job.py", line 340, in run self.cleanup(self.connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/job.py", line 375, in cleanup self.pipeline.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 251, in cleanup child.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 649, in cleanup self.internal_pipeline.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 251, in cleanup child.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 649, in cleanup self.internal_pipeline.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 251, in cleanup child.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/actions/boot/__init__.py", line 458, in cleanup os.unlink(self.url) OSError: [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/tmp/overlay-1.3.2.5.tar.gz'
We are working on the 2017.7 build of LAVA, so if this has been changed in a later build, please let me know. If this has not been updated, is there another way to change things so that each job uses a different name and/or location for the overlay file?
2017.7 is quite old, and as far as I can see there was a fix for this issue some time ago, in commit 004371066fe7f13cb75d0d344852ad1863aece22. That fix was included for the first time in the 2018.4 release.
Please try upgrading...
Cheers,
Hi Steve,
Thank you for the reply and information. We've experienced difficulties with the upgrade in the past given our setup, but this provides the impetus to figure out the issues and upgrade.
Regards,
Jorgen
________________________________ From: Steve McIntyre steve.mcintyre@linaro.org Sent: Tuesday, 14 August 2018 11:31 PM To: Jorgen Peddersen Cc: lava-users@lists.linaro.org Subject: Re: [Lava-users] transfer_overlay file in the same location regardless of test
Hi Jorgen,
2017.7 is quite old, and as far as I can see there was a fix for this issue some time ago, in commit 004371066fe7f13cb75d0d344852ad1863aece22. That fix was included for the first time in the 2018.4 release.
Please try upgrading...
Cheers, -- Steve McIntyre steve.mcintyre@linaro.org http://www.linaro.org/ Linaro.org | Open source software for ARM SoCs
Jorgen Peddersen Researcher t +61 2 6103 4700 m e jorgen.peddersen@seeingmachines.com a 80 Mildura Street, Fyshwick ACT 2609, Australia SEEINGMACHINES.COMhttp://seeingmachines.com
[SEEINGMACHINES]
This email may contain confidential information. If you are not the intended recipient, destroy all copies and do not disclose or use the information within. Any content that is not concerned with the business of the Seeing Machines Group reflects the views of the sender only, not those of Seeing Machines. No warranties are given that this email does not contain viruses or harmful code. [http://www.seeingmachines.com/wp-content/uploads/2018/04/sm-email-twitter-ic...] https://twitter.com/seeingmachines [http://www.seeingmachines.com/wp-content/uploads/2018/04/sm-email-linkedin-i...] https://www.linkedin.com/company/258949/