2017.4 will deliver support for migrating V2 health checks out of the
database and into /etc/lava-server/dispatcher-config/health-checks/ on
the master. This has several benefits for admins.
* Health checks from 2017.4 onwards will be based on the device
configuration, not the device type. This applies to V1 devices and V2
devices.
* Multiple instances can all have the same V2 health checks controlled
in VCS using configuration management tools.
* V2 devices of the same device-type can have different V2 health
checks according to the Jinja2 template specified in the extends field
of the device dictionary. Submissions to such devices would need to be
managed through device tags.
* Devices which are V1 only (is_pipeline is False in the django admin
interface) will continue to use V1 health checks from the database,
based on the device-type.
* Devices which support V2 (whether these devices are exclusive to V2
or not) will only use V2 health checks from the filesystem, based on
the device dictionary of each device.
This does mean that devices which are currently both V1 and V2 will
need to have a V2 health check from 2017.4 onwards or the device will
no longer run a health check at all. Admins may choose to submit and
monitor regular V1 testjobs until all test jobs for those devices are
V2 but those test jobs will not be able to take the device offline
automatically.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
There is a bug in python-lockfile available from jessie-backports:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858835
This affects fresh installs of LAVA when LAVA is installed directly
from jessie-backports without first being installed in jessie.
The underlying problem is with apt which pulls in the updated version
of packages which have a backport available even if the requested
package does not explicitly ask for the backport of that package.
The problem shows up with lava-coordinator first - the package fails
to install cleanly because the daemon cannot be started correctly.
To fix this problem, simply instruct apt to install python-lockfile
from jessie, not from jessie-backports:
apt install python-lockfile=1:0.8-2
lava-coordinator will then install correctly.
This change will persist over later upgrades as apt does not consider
jessie-backports as upgrades of jessie - except in the case where the
package is not yet installed.
Other daemons also now need to be restarted:
$ sudo service lava-server restart
$ sudo service lava-master restart
$ sudo service lava-slave restart
Future builds for the staging-repo and production-repo will force only
version 1:0.8-2 of python-lockfile as these builds are done on and for
jessie.
Systems already running Stretch or unstable do not seem to be affected
and can safely use python-lockfile at version 1:0.12.2-2 - this is a
particular problem inside jessie-backports. However, this does mean
that until this bug is fixed, users running Stretch or unstable should
build their own developer packages instead of using the LAVA
repositories on images.validation.linaro.org. We currently do not have
capacity to build separate packages for jessie and stretch.
(LAVA does not recommend using Stretch or unstable for production
systems until Stretch has been released and stretch-backports becomes
available.)
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Reminder about the LAVA repositories
====================================
Until Debian Stretch has been released and the new testing suite
(called buster) is open for new uploads, new production releases of
LAVA will only be available from the
images.validation.linaro.org/production-repo repository.
See https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
See also: https://lists.linaro.org/pipermail/lava-announce/2016-December/000023.html
2017.2 release
==============
More updates to the jinja2 templates in this release and some new templates.
diffstat output:
lava_scheduler_app/tests/device-types/apq8016-sbc-uboot.jinja2 | 3
lava_scheduler_app/tests/device-types/arduino101.jinja2 | 5
lava_scheduler_app/tests/device-types/b2260.jinja2 | 8 -
lava_scheduler_app/tests/device-types/base-uboot.jinja2 | 26 ++--
lava_scheduler_app/tests/device-types/base.jinja2 | 11 +-
lava_scheduler_app/tests/device-types/d02.jinja2 | 4
lava_scheduler_app/tests/device-types/d03.jinja2 | 1
lava_scheduler_app/tests/device-types/frdm-k64f.jinja2 | 30 +++++
lava_scheduler_app/tests/device-types/hi6220-hikey.jinja2 | 7 -
lava_scheduler_app/tests/device-types/highbank.jinja2 | 2
lava_scheduler_app/tests/device-types/juno-uboot.jinja2 | 1
lava_scheduler_app/tests/device-types/mustang-uefi.jinja2 | 3
lava_scheduler_app/tests/device-types/nexus10.jinja2 | 1
lava_scheduler_app/tests/device-types/nexus4.jinja2 | 1
lava_scheduler_app/tests/device-types/nexus9.jinja2 | 1
lava_scheduler_app/tests/device-types/nrf52-nitrogen.jinja2 | 9 +
lava_scheduler_app/tests/device-types/nucleo-l476rg.jinja2 | 23 ++++
lava_scheduler_app/tests/device-types/nxp-k64f.jinja2 | 1
lava_scheduler_app/tests/device-types/nxp-ls2088.jinja2 | 12 ++
lava_scheduler_app/tests/device-types/overdrive.jinja2 | 1
lava_scheduler_app/tests/device-types/r8a7791-porter.jinja2 | 11 ++
lava_scheduler_app/tests/device-types/stm32-carbon.jinja2 | 11 +-
lava_scheduler_app/tests/device-types/x15.jinja2 | 55
++++++++++
lava_scheduler_app/tests/device-types/x86.jinja2 | 1
24 files changed, 178 insertions(+), 50 deletions(-)
Changes in the 2017.2 release are described below, including the
short git commit hash, include:
Use the git commit hash to go directly to that commit using a URL like
https://git.linaro.org/lava/lava-server.git/commit/?id=266b2e633
Commit messages including a reference to LAVA-NNN relate to projects.linaro.org
append the reference to https://projects.linaro.org/browse/
e.g. https://projects.linaro.org/browse/LAVA-854
lava-server - https://git.linaro.org/lava/lava-server.git/
-----------
9df8b354b Remove whitespace from case names
9354cd585 Uboot parameters fixes
297f27900 Fix documentation for auto_login
64b549e9a Fix group visibility and hidden device types
b211434de Document the use of a list for bug 2870
eb8a7bec3 LAVA-854 - Device type template for x15
ad81f33fb Add device template for renesas r8a7791-porter board
8c4d44ae4 Add device template for nucleo-l476rg
112ec6487 Remove send_char, and fix some broken character_delay settings
8f615cd6b Update reviewers handling
0fe482ffb b2260: remove duplicate ip_args from extra_kernel_args
a72c86732 LAVA-847 protect metadata store IOError
53a3c778e If a text_offset is used, quote it to ensure it is a string.
51351e18c Add documentation of the dispatcher-config support
7a1bfb649 Tidy up the Device-type templates for IoT devices
0b5de4dcb Allow lava_scheduler_app to unittest logs to stdout
bc807def4 Fix sizelimit warning handling for V2
13b691615 Summarise job timeouts with example and notes
b5c2ea1a7 Improve notes on portability.
65ae7d644 Fix internal server error due to unpacking TestJobUser.
83e8fa075 Migration doc improvements
9eb145050 Add a unit test for the ethaddr support in b2260
91eb07a36 Creating and restoring backups for V2
479a6295f Add support for setting ethaddr
52b5cf06d Add triage documentation for power failures
526a6e298 LAVA-840 - Remove documentation for USB_DEVICE_WAIT
3415d5319 Allow .py files in examples/sources to be packaged
be33389bf Allow notifications for jobs in Running status.
4d6afd607 Fix blacklisted bug in notifications.
58b48a8f0 Catch all errors during resubmission.
40b010238 Tag admin clean support must return name
14b15f40b LAVA-734 compatibility docs
dd18c26e1 scheduler: do not cascade deletion for some objects
5415eafa7 Do not create unused TestJobUser
2f3cd8389 Bug 2796. Add submitter username to job_details API.
08cd6c01f Revert "master: set the master identity"
be28c8c7b LAVA-814 - Explore pyudev for usb device wait
9150c9973 LAVA-832: Rework device-types management command
70e81493e LAVA-762 link level in timing to pipeline definition
c276df83f LAVA-728 extend ZMQ example to submit and wait
f6a0456a1 LAVA-731 document remote worker issues
41c67ae3b Add device template for NXP-ls2088
ee8964c7b Remove spurious call to save()
2a79a622d Ensure base_ip_args can be overridden
affa68b23 Add a command to add and list workers
10e6ea970 tokens: allow format as CSV
b0b5a1d30 Remove unused config value LAVA_CONFIG_VERSION
69138ac89 Add a command to manage tokens
e09beb1a4 Remove unused custom commands
905a0f9be Fix multinode job definition page
bb0fdc7a8 job logs: Highlight failing results
e1a125aa8 master: set the master identity
b7cb28e49 Fix device schema and test for arduino template
d411416bf LAVA-707 update result case and metadata views
15fc5c62f LAVA-140 - update running job calculation
9fcf92cfd LAVA-515 restore line numbers to YAML definitions
b7266712b LAVA-824 skip unused device types in running table
de6a141aa Expand docs on namespace
c54a5e262 events: fix documentation
lava-dispatcher - https://git.linaro.org/lava/lava-dispatcher.git/
---------------
ba779908 testdef: set "set +x" to avoid duplicating SIGNALS
ae2ab424 Enforce a download order to produce predictable pipelines
55a39b92 Remove unnecessary boot action in fastboot x15
81b178fc Add a unit test for db410c pipeline generation.
aa96dd75 LAVA-839 - Integrate X15 into LAVA V2
14acd2e2 Fixup u-boot prompts sanity
829fa380 Support CMSIS-DAP devices that do not reset after being flashed
d09a7a74 Simplify send_char support
031ba8e8 Ensure dispatcher.yaml is available for admins
aa6142dd Detect pdu controlled device using availability of power command.
c54891b9 Properly isolate stdout and stderr in unit tests
2d8b3110 Ensure lava-test-reference is available for release
555b6be0 Skip one test if lxc-info is not installed
fc42bd89 Fix support for persistent NFS
57c96b4b Remove duplicate prompt handling.
b0d39204 LAVA-844 allow string and dict for install: git-repos
76998b75 LAVA-830 Use the configured TFTP directory
1f01283f Partially revert a1658c by using a DEALER socket
9045292b Allow boot commands in the job definition
af34dfcb Extend the timeout unit tests to boot actions
9135c5aa Add support for CMSIS-DAP IoT devices and cleanup PyOCD
96dcc7d7 Only match udev events when device is added
a1658c7e slave: use a ROUTER socket to connect to master
35937826 Fix prospector warnings
e4e2fb97 etc/lava-slave: fix documentation of WARN log level
b45cb005 Remove unsed imports
93a282bf Job: log known error status when leaving
ee404039 lava-dispatch: allow to run without zmq logging
26daca49 lava-dispatch: work around a variable clash
c62e1ee9 LAVA-814 - Explore pyudev for usb device wait
989cd97b LAVA-827 ensure the case name is a valid URL
570c1b1e Do not reset the timeout before each connection
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Reminder about the LAVA repositories
===========================
Until Debian Stretch has been released and the new testing suite
(called buster) is open for new uploads, new production releases of
LAVA will only be available from the
images.validation.linaro.org/production-repo repository.
See https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
See also: https://lists.linaro.org/pipermail/lava-announce/2016-December/000023.html
2017.1 release
===========
This release includes substantial updates to the jinja2 templates. If
you have made local changes to these files, you will be prompted
during installation.
/etc/lava-server/dispatcher-config/device-types/apq8016-sbc-uboot.jinja2
/etc/lava-server/dispatcher-config/device-types/arndale.jinja2
/etc/lava-server/dispatcher-config/device-types/b2120h410.jinja2
/etc/lava-server/dispatcher-config/device-types/base-uboot.jinja2
/etc/lava-server/dispatcher-config/device-types/base.jinja2
/etc/lava-server/dispatcher-config/device-types/beaglebone-black.jinja2
/etc/lava-server/dispatcher-config/device-types/cubietruck.jinja2
/etc/lava-server/dispatcher-config/device-types/highbank.jinja2
/etc/lava-server/dispatcher-config/device-types/juno-uboot.jinja2
/etc/lava-server/dispatcher-config/device-types/mustang.jinja2
/etc/lava-server/dispatcher-config/device-types/panda.jinja2
/etc/lava-server/dispatcher-config/device-types/qemu.jinja2
Changes in the 2017.1 release are described below, including the
short git commit hash, include:
Use the git commit hash to go directly to that commit using a URL like
https://git.linaro.org/lava/lava-server.git/commit/?id=266b2e633
lava-server - https://git.linaro.org/lava/lava-server.git/
--------------
266b2e633 LAVA-795 - Expose lava-publisher event socket settings.
f4b76e48e Fix ordereddict failure
9d3ae86cc Use dpkg-query which is available on all systems
080c3184f Update the device-type context help page
46a055a3b Fixup some jinja2 templates
cecefbfef Timing: improve the description
5bef1d877 LAVA-821 - add support for lava-test-reference
3b26fb234 LAVA-820 record lava-server version in job metadata
ff71a42a6 Append to existing metadata store results
0c9abf643 Fix RST whitespace typo.
05b96af3c Merge timing page into the job page
edc4431f1 Restore access to chart Action buttons
69709881e Improve the timing page and port to the new log format
a00c02aa8 LAVA-535 Make devicetags case-insensitive
36617b955 LAVA-743 - handling complex test operations
50084f1b5 Omit summary button for V2 definitions.
5c8bf4643 Upgrade the dispatcher v2 parser interface.
22409d543 master: send the dispatcher config to the slave
lava-dispatcher - https://git.linaro.org/lava/lava-dispatcher.git/
----------------------
64ae9131 Report the bootloader load addresses used
2085ffb8 Skip parsing of kernel messages for ssh connections
00fa2d24 Improve allow_fail fault handling
dce680f3 Simplify the connection session handling
ea18594a Use dpkg-query which is available on all systems
2cc64b70 Reinstate git version_tag in version.py
5c05bdae Improve logging
50236943 LAVA-821 - lava-test-reference support
a118e943 Add missing wait definition in base class
2dc955f6 Allow timeout during Linux Kernel message parsing
0404a419 LAVA-819 - record lava-dispatcher version in logs
542effdf Raise an Infra error when download fail in run()
29a232c1 Fix Download retries crashes
12e5789f monitor: make test a bit faster
a90a5184 Fix computation of action duration
ffd1d176 Fix retry logic (partial revert of 7bbd706d)
36ca3c21 LAVA-818 - errors should set the job as Incomplete
f5ac0328 Skip generation of invalid test_case_id
9e5e56c6 Re-enable support for nose tests
bb89772f Merge and re-use BootloaderCommandOverlay and
BootloaderCommandAction
f3864c2f job: fix cleanup when validation failed
8dd450c5 run_actions: clean error messages and exceptions
ec44db64 Properly compute the timeout
f3058a12 atexit is not needed anymore for cleanup
7bbd706d Improve error handling and reporting
a5e0e678 Fix issue when downloading two files with the same name
0d3773ea Call finalize directly in job.cleanup
af608eb2 Remove unused setter, properties and __call__
f946a21d validate: do not extend action.errors
4504aeb9 slave: use the dispatcher configuration
925019c8 Use the dispatcher config to get the dispatcher ip
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Enable the LAVA repository
====================
2016.12 will be the last chance to update of lava-server and
lava-dispatcher into Debian Stretch before the release as Debian 9.0.
This means that users of jessie-backports should now enable the LAVA
production repository - see
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…https://lists.linaro.org/pipermail/lava-announce/2016-November/000019.html
Until Debian makes the 9.0 release, jessie-backports cannot receive
updates of the LAVA releases in 2017, so all updates will only be
available from the LAVA repositories. Shortly after the release, a new
stretch-backports suite will become available and jessie-backports
will be opened again.
In accordance with the Debian freeze policy, it will take 10 days for
the new uploads to migrate into stretch, not the previous 5, so an
upload to jessie-backports will be made at that time.
Note: this may be an issue for armhf or arm64 workers - if you are
using such workers, let us know so that we can prioritise the changes
required to build lava-dispatcher for those architectures.
Updates from jessie-backports
======================
A few updates are required from jessie-backports with 2016.12: lxc and
python-pyudev
The lxc update ensures that jessie and stretch containers can be
correctly created on jessie.
The pyudev support enables lava-dispatcher to monitor udev activity
for certain operations.
These updates can be applied before installing 2016.12, if desired:
$ sudo apt install -t jessie-backports lxc python-pyudev
lava-tool
======
A new version of lava-tool is also available, including a new way of
handling the authentication tokens and the ability to use shortcuts
(like "local" instead of "http://localhost/RPC2").
https://tracker.debian.org/pkg/lava-toolhttps://tracker.debian.org/news/825902
The new version of lava-tool is available via the LAVA staging and
production repositories and the package available in jessie-backports
will be updated in due course.
Unfortunately, it proved not to be possible to export the existing
authentication secrets into the new backend or to support both
backends simultaneously. All users will need to re-add all their
authentication tokens to use lava-tool after this upgrade. The change
has several benefits, including the ability to list and remove
authentication configurations using the new backend and the removal of
a whole host of problems related to using the old backend in
environments where lava-tool was not running in a full terminal. In
addition, this new backend allows lava-tool to support shortcuts which
prevent problems with repetitively entering long and complex URLs.
On installing the new upgrade, users will see the NEWS message:
https://github.com/Linaro/pkg-lava-tool/blob/master/debian/NEWS
lava-tool has changed the backend for token storage.
This means that existing authentications will not be usable to this
new version. This change has been made to fix persistent problems with
the python-keyring support, including:
* DBus errors when used over SSH
* Interaction with gnome-keyring causing authentication failure
* Inability to list or remove authentications added to the keyring
* Inability to work with other keyring solutions.
The new backend is able to list and remove authentications. The new
support also removes the need for a default password on the user
keyring, so lava-tool will no longer pause waiting for password
entry.
2016.12 release
============
Changes in the 2016.12 release are described below, including the
short git commit hash, include:
Use the git commit hash to go directly to that commit using a URL like
https://git.linaro.org/lava/lava-server.git/commit/?id=b19b9648a
lava-server - https://git.linaro.org/lava/lava-server.git/
-----------
541ba7b0a Enable ssh secondary connections on juno-uboot
2d6bb2fcc Handle errors which cause invalid description data
b19b9648a Fix template_mismatch check to use extends
d7f71eae3 Update ipxe x86 nfs to use base_ip_args
fa6b8b3ab Add a note on release-notes on lava-announce
d007d1819 Fix a crash when a device does not have a device dict
d46d6e4fd Protect against invalid parameter submissions
431fd2fc0 Allow pipeline jobs to be used with XML-RPC job_output
f3bf855ba Add Copy to clipboard functionality to MultiNode
19d814e0c Fix some failing tests due to new namespace changes
86950e985 LAVA-745 - expose device_path for all types
083d67d54 Add device-type template for arduino101
fd99c576e Fix UI submission error.
708a38a23 Add SSH support for overdrive
e838de3ba Update Linaro lab links to actual site
d8e127b74 Allow flash_cmds_order to be set
92eefe4aa Adding extra boot commands for secondary media on Juno
74310353c LAVA-792 - add shorthash to changelog
8f860a1e9 Keep result blocks on a single line with MultiNode
3b1d6654e LAVA-499 - Device type template for dragonboard-410c
5c46ea4ab Provide a default fastboot flash commands order.
a61ab2501 Download results XMLRPC support.
347fb3fd3 Allow to forward the messages to additional sockets
4d0753c3a Allow validate_pipeline devices to filter by device type name.
a859e5ede publisher: improve logging and fix logfile owner
f0002efb1 Fix sphinx indentation error
9f9bd71b9 Correctly expose TestSet data in results
c6f9b1df8 Handle decimal conversion error in test case
1769d8b6b Expand hints on new device-types
eec1ca1d0 master: expect a protocol version with HELLO messages
cd8d109f7 master: add the job id to the log messages
a0ac2d66a Add absolute_url to job_details API.
951ab9f68 master: device configuration is empty for secondary media
c0575503d master: improve error reporting when reading env file
f551daa49 Add notes on unattended upgrades
1d2e25d1d Allow override of nfsroot options, not arguments
edaad5419 Update docs to reflect reality of from: support
09503a482 Manually revert d31a8462: errors are already saved
3516b856d add some notes on packages needed for LXC usage
b5ee513bb Rename 'usb-showup' to 'usb-device-wait'
406832582 Drop misleading hostname from device metadata
6a6ee6f4d Handle change to git.linaro.org frontend
d85f99e1b Fix crash when description.yaml is empty
37a94aafc Fix prospector warnings
080499d18 Remove references to wsgi
267d22302 Paste job definition to clipboard as text and not as HTML.
ef09a9698 add a link to the device tag glossary entry
08bd011a4 Update aliases migration for review change
7010775b6 Move metadata extra content to files
95ed9a3ba Expand notes on debugging test jobs and test cases
6d772bb0d Update for pycodestyle pep8 failures
dd58a1182 LAVA-798 - lookup device-types by alias
104feb593 Enable LXC for existing device-types.
376a80021 LAVA-797 - Document changing USB_SHOW_UP_TIMEOUT in job
3e8dbfeca all_devices() api call to do one big query instead of many small ones.
d27f7a4d5 Fix for Debian reproducible builds.
ac8bb8e2a Fix bug #2596 - No result collected for multinode jobs
7b3adad0a All jobs will have a definition, so show it by default.
1e6992f6b Set boot_character_delay in x86 template
5cc0a4746 Extend x86 unit test to check character_delay support
ad4f8780f Combine interfaces block for juno and vland
bb2ecf716 Allow changing the timeout for u-boot-interrupt
19de59978 Add stm32-carbon jinja2
lava-dispatcher - https://git.linaro.org/lava/lava-dispatcher.git/
---------------
19fd79a9 Update preseed support for namespace changes
ef352abf Update zephyr sample jobs to use LAVA functional test urls
1301258a LAVA-498 - Support dragonboard-410c with lxc
1e0b60d2 Ensure pyudev version is >= 0.21
7d6a8134 Update the DFU boot code for namespace changes
161e8b47 Ensure installer tests share the final connection
567ddf0d remove duplicate call
e49bd971 Fix namespace error in flash_cmds_order retrieval.
cb04549d Set connection for secondary media deployments
dcd75e7b Use correct suffix for secondary media deployments
2a7b386f Declare the original ramdisk as compressed
fd73e937 Support DFU devices
6aa7280c LAVA-504 - Dynamic fastboot partition support in pipeline
9228901a LAVA-809 - Show HiKey kernel boot messages
774c7557 Extend namespace fixes for secondary connections
dc5bec22 Action: Always call super() in validate()
9b983cdf GitRepoAction: do not call validate twice
bd112aab Make namespaces common to all dynamic data operations
2e83e2ca Prevent invalid measurements being reported
1768c181 Fix testlevels for multi-test LXC jobs
0a49da8e Give namespaces to all actions and support simpler code flow
2423a123 slave: send the protocol version with HELLO message
983d3775 Ensure fixupdict is checked in test_case_result
08f79918 slave: writing an empty string to a file is valid
c6f382ef Move cleanup_actions from Action to Job
bc56ca24 Remove unused Action, Pipeline and Job members
1256f31e Run.sh should use set -x
a726be8f Allow install-steps to use the git-repo directly
b0ca553a Create a useful exception message
077aca26 slave: don't crash when lava-dispatch can't start
543ed4af Honor USB device wait timeout and add device before returning
connection
0e07089f Fix problem with duplicate test shells failing
f14747c1 Use dnf instead of yum on Fedora version greater than or equal to 22.
efdaabc1 More fix for usage of LXC with a device.
2d188baa Limit repetitive kernel message log entries
c3455a11 Allow newline line separator to be overriden per bootloader
01a482f6 Fix for Debian reproducible builds.
62c07465 Indicate that character delay is in milliseconds
c1f79680 Remove hardcoded character delay in ipxe
e43552b3 Fixing secondary media deployment
e885c881 device-types: add Renesas r8a7795 Salvator-X board
1374289b device-types: add Amlogic meson-gxbb-odroidc2
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Everyone using LAVA should be aware of V1 and V2 and that V1 is going
to be turned off. The plan for how to do that has been discussed
within the team and within Linaro. The most recent presentation on the
migration can be seen here:
http://connect.linaro.org/resource/las16/las16-503/
(Note: the video will auto-start, be aware.)
The slides are available on slideshare:
http://www.slideshare.net/linaroorg/las16503-lava-v2-migration
The dates for each step in the migration are not finalised but will
occur during 2017, this is advance notice.
The highlights of the migration are:
0: During 2017, regular announcements will be made on this list before
each major change. In addition, the regular release announcements will
include details of the major changes in that release.
1: The first major change will be that a new release of lava-server
and lava-dispatcher will *disable* V1 submissions entirely. Any queued
V1 test jobs will become invalid upon this upgrade,
2: A later release will remove V1 documentation and start removing the
V1 source code.
3: A release will be made which *forcibly deletes V1 test jobs,
bundles, filters and image reports*. This is required to complete the
removal of the V1 source code.
4: The final step of the migration is a release containing no V1
source code at all. Only V2 support will remain. This is essential as
the V1 source code is currently blocking some useful additions to V2
as well as plans for future development. The data cannot remain
accessible without the V1 source code. There is not and can not be any
support for converting V1 data to V2.
Please take this chance to read up on the V2 documentation, including
how Results, Queries and Charts replace the deprecated V1 support from
filters and image reports.
Work has started upstream on making sure the migration runs smoothly,
especially when deleting large numbers of V1 test cases.
Remember: Once the first major step is released, upgrades will stop
running any V1 test jobs. Subsequent releases will include changes
which force the deletion of your V1 test data.
If you want to preserve your V1 data, consider setting up a read-only
reference instance which is then pinned at a particular release of
LAVA to prevent deletion of the V1 data.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
There is a known bug with QEMU 2.7 which is affecting installations of
LAVA using QEMU from jessie-backports.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=840853
The problem manifests in the apply-overlay-guest action because QEMU
is unable to process the GuestFS operations due to a missing file
which is in the seabios 1.9 package but not in seabios 1.7 - so V2
QEMU test jobs hang at this point.
If you are using QEMU from jessie (version 1:2.1+dfsg-12+deb8u6) then
there is no problem and nothing needs to be done.
If you are using QEMU from jessie-backports (version
1:2.7+dfsg-3~bpo8+2 currently) then there are two simple remedies.
0: You can find the updated seabios package in the LAVA production
repository. See
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
If you have the production repository already available, you can
simply apt update and apt upgrade to install the new seabios package.
1: If you are not using aarch64 QEMU tests or the QEMU UEFI support
for LuvOS, then the version of qemu in jessie will work just fine. To
downgrade, specify the version string to apt:
$ sudo apt install qemu-system-x86=1:2.1+dfsg-12+deb8u6
qemu-system-common=1:2.1+dfsg-12+deb8u6
It is also worth downgrading other qemu packages at the same time.
Check your current install with:
$ dpkg -l | grep qemu
The relevant bug has been re-opened and a backport of seabios has been
uploaded. An updated backport of qemu has also been requested which
would require the installation of the backported seabios 1.9. So if
this bug is not troubling you immediately, you can simply wait for
seabios 1.9 to be available in jessie-backports and install it.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
2016.11 includes a move away from libapache2-mod-uwsgi and
libapache2-mod-wsgi to gunicorn support. This has been done to make it
possible for lava-server and lava-dispatcher to migrate into Stretch
ahead of the Debian 9 Stretch release. (see
https://lists.linaro.org/pipermail/lava-announce/2016-November/000019.html).
This should allow us to upload 2016.11 to jessie-backports in due
course.
See: https://lists.linaro.org/pipermail/lava-announce/2016-October/000017.html
for details of the changes for gunicorn support. In particular, if you
are using apache, before installing this upgrade, ensure that these
commands have been run on the master:
$ sudo a2enmod proxy
$ sudo a2enmod proxy_http
$ sudo service apache2 restart
These commands are safe to run before upgrading to 2016.11 but will
need to be run before 2016.11 will actually run.
This release includes all changes in master since the 2016.9 release.
The release has been uploaded to Debian unstable, it should appear on
the official mirrors shortly. It has also been uploaded to the
production repo.
We are anticipating that there will be an interim release ahead of
2016.12 for additional LXC support, more details will follow via this
list.
lava-server (2016.11-1) unstable; urgency=medium
.
* New production release aimed at Stretch.
Release details available at:
https://projects.linaro.org/browse/LAVA-789?filter=12078
* Drop dependency on libapache2-mod-uwsgi and libapache2-mod-wsgi.
Replaced by dependency on gunicorn.
* Require sphinx 1.4 or later for correct theme support.
* Downgrade linaro-image-tools and fuse to Recommends
* Update compat version.
* Highlights of upstream changes:
Note about removing adb and fastboot packages in LXC admin doc.
LAVA-789 - Document LXC support in V2
LAVA-788 - link to sub_jobs_list from results
Expand user notification documentation.
Move jinja template tests to unittest
Adding target interface entries to juno-uboot.jinja2
Add docs on notifications
Update home page documentation links
Improve documentation on metadata and job_name
Fix up glossary page
Tweak docs handling standard test jobs
Example jobs need auto_login support
Fix missing create_device_database ref
Port the documentation change for essential roles
Tweak the doc build options
Add notes on multiple hacking sessions
Update debugging docs for tmp file changes
LAVA-65 document recording measurements
LAVA-780 stop referring to wheezy images
Avoid crash in master on early failure
Remove beautify.js and fix yaml URL submission.
Update screenshot for cancel button fix
Fix visibility of the Cancel and Admin buttons
Update docs to reinstate web UI submission
Add a helper to add devices from cmd line
Remove old link about LAVA packaging which contains stale information.
LAVA-749 - Add actions column to results pages.
Ensure lava-server-gunicorn restarts cleanly
results: do not crash if the description is empty
Job submission UI for v2.
Add copy to clipboard button for job definition.
Turn all action durations into result measurements
Updating juno-uboot.jinja2
Add a log file for gunicorn logs.
Fix the LAVA logo
Ensure the V2 layout is included on jessie.
bootstrap: do not use the minified CSS
Results: fix HTML syntax
Use sensible default for interrupt string in HiKey jinja.
LAVA-749 - Improve query pages
Fix unit test to have unique test definition names
Add pre_* commands to base.jinja2 that will be applied conditionally.
Remove unused dependendies on lava_dispatcher
Fix a variable redefinition in list comprehension
Update contents and organise toctree
Allow changing the timeouts for auto-login-action
Add documentation on QEMU standard kernel tests
Remove old mention of JSON
master: set the failure_comment
Add a template for highbank using base-uboot
Remove hardcoded values from base
Fix profile page error due to wrong device health history url reference.
metadata: fix a crash if the description is invalid
Cover lava-coordinator in changes for V1 workers
LAVA-522 Link in existing docs on replacing vmgroups
Extend the qemu options docs
Fix index links and debugging doc links
LAVA-735 describe anonymous access to JIRA
qemu.jinja2: use qemu-system-i386 for i386
Extend docs for disabling V1 on a master.
Fix HTTP 500 on query views when a group is used.
LAVA-765 V1 MultiNode sub_id correction
Fix jinja2 syntax issues in base-uboot
Fix sub_id assignment in V2
Improved debugging docs
Add content for growing your lab.
Improve first device and first job pages
Update apache docs for gunicorn
Update hacking sessions and lava-network docs
Enable vland for overdrive
Fix uWSGI configuration file
Add notes on removing V1 from dispatchers
Add a helper to add device types from cmd line
DeviceType admin: fix has_health_check computation
Replace ip=dhcp with {{ base_ip_args }}
timing: add the corresponding timeout
Fix typo in the title of the result download link
Add ip=dhcp for juno with a base default
Fix page to show health history of my devices and not my device type.
Fix error on 'Devices Health History' page in django 1.10
Fix user profile page error in django 1.10
Create a base uboot jinja template
Use Gunicorn instead of uWSGI
wsgi: use the Django public API
index: add the missing title
Mark support for LAVA in Debian testing.
Expand the context schema for extra_options
Add the new LAVA logo to the front page of the V2 docs
Warn if a pipeline device does not have a template
master: don't generate the description
Add nxp-k64f jinja2
Add nrf52-nitrogen jinja2
Add drafts of new logos with SVG.
Extend base timeouts and convert to minutes
Handle inline definitions using only install steps
job details: removed unused template variable
master: don't print too many logs in debug
Fix build errors and simplify index pages
Add support for overdrive device type
Fix bad lookups into the glossary
Switch ASCII art to the existing SVG
Update dispatcher actions
Improve timing warning message
Document the principles of a CI Loop
master: remove the need for the ERROR message
Expand job_details doc string with available keys
Replace 1.10 deprecated get_field_by_name with get_field method.
Add support for extra QEMU options
LAVA-747 - Report parameters as result metadata
Switch from using png to svg for the architecture diagram
Add a page to display the pipeline timings
Expand glossary and add sections on parameters
Add D03 device type to pipeline
Add an introduction to results in LAVA.
Add docs on the standard test jobs
Simplify and clarify the "first install" docs
LAVA-584 - expand for sub-types of device-types
LAVA-706 - codebase structure
Port submission priority support to V2
Adopt a similar contents layout to the django docs
Device dictionary output
Extend qemu jinja template for cortex-m3
Pipeline log: use a better icon for download
Update bootstrap from 3.1.1 to 3.3.7
Fix relative url for suite results
Add nbdroot as optional parameter to the schema
Update qemu/kvm templates to allow overriding of guestfs interface
LAVA-93 mark some roles as essential
lava-dispatcher (2016.11-1) unstable; urgency=medium
.
* New production release aimed at Stretch.
Release details available at:
https://projects.linaro.org/browse/LAVA-789?filter=12078
* Update compat
* Highlights of upstream changes:
Ensure namespace is available to the test shell.
cpio output is not an error
Downgrade message match failure to a warning
Fix up usage of LXC with a device
Only attempt lxc-destroy if the container exists based on exitstatus.
Drop ShellLogger __del__ command
Improve accepts for monitored QEMU
Strategy needs to check methods, not device_type
Support parallel (pbzip2) compressed bzip2 files
Allow vendor U-Boot builds using Ctrl interrupts
Improve cpio extract command line and error message
slave: do not send None as error code
slave: send a SIGINT to lava-dispatch to cancel
Making target IP and MAC addresses available in lava test shell
TestShell: allow multiple test shells
Fix kernel message detection for panic
testdef: remove unused common data
Fix the logic regarding the pipeline building
shell: make testdef_index common data a list
device/target: allow multiple boot control chars
Add 943907AEVAL1F to pyocd deploy supported devices
Check if container exists before destroying.
Do not generate results.yaml
Fix uefi_menu pipeline for fastboot.
slave: move tmp files to /var/
Add device to lxc only when it is available.
device-types: add Amlogic meson8b-odroidc1 board
LAVA-755 - use UUID for guest, not label.
slave: send an error code when err is not empty
Fix git-repos reference in inline test definition.
Allow a different bootloader interrupt character
Make pipeline logging python3 compatible
slave: fix a crash under python3
device-types: add Amlogic meson-gxbb-p200 board
device-types: add TI da850-lcdk
bootloader: add ramdisk_raw option
Detect invalid characters in testdefinition name
Fix timeout inheritance from job definition blocks
Set the right permis when creating the ramdisk dir
ci-run: exclude the .eggs directory
tests: Close the file descriptors
Fix root directory of V2 apache config
LAVA-760 - Trim LXC log output
Require a summary and description to validate
Additional changes for removable media usage
Fix extra_options to always be a list of strings
Do not set kernel type for all 'kernel' YAML keys
Action: re-raised the exception without modification
LAVA-748 ensure test definition names are unique
Change sample jobs urls to images.v.l.o
Allow testdefinitions without run steps
Generate the pipeline description on the slave
slave: fix the END message parameters
Use the QEMU extra options where available
slave: resend the END message if the ack was lost
slave: send the error file along with the END message
Move tmp files to /var/lib/lava/dispatcher/job_id/
Only extract the nfsrootfs once.
Use the uboot interrupt prompt from templates
Improve LXC HiKey support.
Don't end test shell at invalid testcase name
SignalMatch: simplify and fix exception message
Shell: improve report formatting
Shell: factorize handling of measurements
RetryAction: log errors
Shell: factorize signal handling
Move override and log_results functions into Action
Initial support for PyOCD
Use \r\n for newlines
Allow guest drive interface to be specified
allow-modify functionality added
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The 2016.11 release is currently in final preparation and is targeted
at Debian 9 Stretch. We've tried to get the documentation and support
up to a point where users of Stretch have a usable platform before
later upgrading via stretch-backports.
I expect to get 2016.11 into jessie-backports about 5 days after that release.
A few reminders on how the Debian release process (which starts in
January 2017) will affect things, most of which are the same as during
the jessie release process. More details will be given on this list as
the Debian release process continues.
0: New lava-server and lava-dispatcher uploads to Debian won't migrate
into Stretch once the freeze starts. This means that these will not be
available in jessie-backports either as the presence of the same
version of the package in Stretch is required for an upload to
jessie-backports. I expect to get 2016.11 and 2016.12 into
jessie-backports.
1: Once Stretch is released, stretch-backports will be opened and
uploads to jessie-backports can resume. Debian makes the release when
the release is ready, so there are only estimates of when Stretch will
actually be released. The key information is that the freeze to
prepare the release starts in January 2017.
2: I expect to keep jessie-backports updated at least for the first
few releases. This list will carry more information about when there
may have to be changes in this area. Essentially it depends on
movement within Debian after the Stretch release.
3: During this period, uploads will remain available on
images.validation.linaro.org built for jessie - lava-dispatcher will
only be available for amd64 from those repositories. If you are using
armhf or arm64 for the dispatcher, please let us know. Use the
production-repo to get updates of lava-server and lava-dispatcher
during the Stretch release process until jessie-backports is open
again.
4: At some point after the release, the production-repo will start to
be built on and for Stretch. stretch-backports and jessie-backports
will need to be open before this change is made. More details of that
will appear on this list before it happens.
5: Once Stretch becomes Debian stable, Jessie becomes Debian
oldstable. Debian testing gains a new name: Buster. Get used to the
names.
6: It is worth checking *now* that all your apt sources on all your
machines (including VMs) specify jessie and not stable. Otherwise, as
soon as you do an apt upgrade after the Stretch release, your machine
will start the upgrade from jessie to stretch. Make sure you are ready
for that upgrade and that it doesn't happen until you are ready for
it. Also check the images you use in test jobs and the build tools you
use to prepare those images.
7: We have been building images based on Stretch ahead of the release
for test jobs using QEMU and similar. We have also updated the
documentation in the next release to use stretch images.
8: At some point after the Stretch release, there will come a point
where support for jessie is dropped and uploads to jessie-backports
will cease. The upgrade to stretch will be required at some point to
continue getting updates to LAVA. Prepare for it ahead of time and
everything will go more smoothly. Further details of the timing will
be announced on this list.
9: The LAVA software team have been testing with stretch for some time
and will continue to do so. We do not expect any issues with LAVA
packages when upgrading from jessie to stretch. Nonetheless, all
admins are recommended to prepare for the new Debian release and to
perform the upgrade under their own control. Make sure your other
services will work after those machines upgrade to Debian 9 Stretch.
More information: https://release.debian.org/stretch/freeze_policy.html
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
$ sudo apt-get install -t jessie-backports python-sphinx
To solve some problems with the build of the documentation, we have
had to move to the version of python-sphinx available in
jessie-backports. (Essentially, our documentation is now so large that
it is causing problems for the theme we've chosen and we've had to
make some changes to keep improve the navigation within the pages.)
If you are building lava-server in sbuild, there is an addition option
required to sbuild:
--build-dep-resolver=aptitude
The build script in the lava-dev package uses this option already.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
