Hi All,
I have recently carried out an upgrade of LAVA and I am now seeing an issue, where I am unable to trigger any jobs. The error listed in /srv/lava/instances/production/var/log/lava-scheduler.log can be seen below.
I have checked the database column in question (admin_notifications in the lava_scheduler_app_testjob table?) and the contents is as it says null. I have tried populating this column with a non-null string in an attempt to make Django happy, but I am still seeing the problem.
I am not sure where the corruption happened, I presume something went wrong in the upgrade stage. Would it be possible to give me an example of what should be in this column and I will add the data manually to try and resolve the problem.
Thanks Dean
###############################
2013-10-25 11:51:55,364 [ERROR] [lava_scheduler_daemon.service.JobQueue] IntegrityError: null value in column "admin_notifications" violates not-null constraint
Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 504, in run self.__target(*self.__args, **self.__kwargs) --- <exception caught here> --- File "/srv/lava/.cache/eggs/Twisted-12.1.0-py2.7-linux-x86_64.egg/twisted/python/threadpool.py", line 167, in _worker result = context.call(ctx, function, *args, **kwargs) File "/srv/lava/.cache/eggs/Twisted-12.1.0-py2.7-linux-x86_64.egg/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/srv/lava/.cache/eggs/Twisted-12.1.0-py2.7-linux-x86_64.egg/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File "/srv/lava/.cache/git-cache/exports/lava-server/2013-10-17-97c7da5/lava_scheduler_daemon/dbjobsource.py", line 70, in wrapper return func(*args, **kw) File "/srv/lava/.cache/git-cache/exports/lava-server/2013-10-17-97c7da5/lava_scheduler_daemon/dbjobsource.py", line 242, in getJobList_impl job_list = self._assign_jobs(job_list) File "/srv/lava/.cache/git-cache/exports/lava-server/2013-10-17-97c7da5/lava_scheduler_daemon/dbjobsource.py", line 205, in _assign_jobs job_list = self._get_health_check_jobs() File "/srv/lava/.cache/git-cache/exports/lava-server/2013-10-17-97c7da5/lava_scheduler_daemon/dbjobsource.py", line 121, in _get_health_check_jobs job_list.append(self._getHealthCheckJobForBoard(device)) File "/srv/lava/.cache/git-cache/exports/lava-server/2013-10-17-97c7da5/lava_scheduler_daemon/dbjobsource.py", line 286, in _getHealthCheckJobForBoard return TestJob.from_json_and_user(job_json, user, True) File "/srv/lava/.cache/git-cache/exports/lava-server/2013-10-17-97c7da5/lava_scheduler_app/models.py", line 622, in from_json_and_user job.save() File "/srv/lava/.cache/eggs/django_restricted_resource-0.2.7-py2.7.egg/django_restricted_resource/models.py", line 71, in save return super(RestrictedResource, self).save(*args, **kwargs) File "/srv/lava/.cache/eggs/Django-1.4.2-py2.7.egg/django/db/models/base.py", line 463, in save self.save_base(using=using, force_insert=force_insert, force_update=force_update) File "/srv/lava/.cache/eggs/Django-1.4.2-py2.7.egg/django/db/models/base.py", line 551, in save_base result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw) File "/srv/lava/.cache/eggs/Django-1.4.2-py2.7.egg/django/db/models/manager.py", line 203, in _insert return insert_query(self.model, objs, fields, **kwargs) File "/srv/lava/.cache/eggs/Django-1.4.2-py2.7.egg/django/db/models/query.py", line 1593, in insert_query return query.get_compiler(using=using).execute_sql(return_id) File "/srv/lava/.cache/eggs/Django-1.4.2-py2.7.egg/django/db/models/sql/compiler.py", line 910, in execute_sql cursor.execute(sql, params) File "/srv/lava/.cache/eggs/Django-1.4.2-py2.7.egg/django/db/backends/postgresql_psycopg2/base.py", line 52, in execute return self.cursor.execute(query, args) django.db.utils.IntegrityError: null value in column "admin_notifications" violates not-null constraint
2013-10-25 11:51:55,365 [ERROR] [sentry.errors] No servers configured, and sentry not installed. Cannot send message
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590 ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782
On Fri, 25 Oct 2013 13:01:28 +0100 Dean Arnold Dean.Arnold@arm.com wrote:
Hi All,
I have recently carried out an upgrade of LAVA and I am now seeing an issue, where I am unable to trigger any jobs. The error listed in /srv/lava/instances/production/var/log/lava-scheduler.log can be seen below.
This could be as simple as LAVA not being fully restarted.
Ensure you've run:
sudo service lava restart sudo service apache2 restart
Make sure lava is also restarted on any remote workers. (You should always use upgradeworker on each worker whenever you run upgrade on the server.)
This is normally part of the upgrade process, so maybe something went wrong at that stage and the upgrade was incomplete.
I am not sure where the corruption happened, I presume something went wrong in the upgrade stage. Would it be possible to give me an example of what should be in this column and I will add the data manually to try and resolve the problem.
The queries being sent need to handle the new field, it's not missing data because the field is typically empty.
linaro-validation@lists.linaro.org