A change was sent a while ago to add support for the Coreboot / Depthcharge bootloader which is used on Chromebook devices. This is useful in particular to avoid having to install U-Boot on Chromebook devices. See this Gerrit review here for previous history:
https://review.linaro.org/#/c/15203/
I'm now opening this case again to try and get this resolved, there seem to be several issues with the original patch that would need to be clarified. Also, some things might have changed since then in LAVA or Coreboot which could potentially lead to a different approach - any feedback on this would be welcome.
To start with, I understand that running mkimage on the dispatcher is not a valid thing to do, it should receive a FIT (flattened image tree) kernel image ready to be booted. This complicates things a bit for projects like kernelci.org where only a plain kernel image is built and ramdisks are served separately, but it's fair enough to say that LAVA is not meant to be packaging kernel images on the fly.
Then I believe creating the command line file in LAVA should be fine, although it probably makes more sense to have both the FIT image and cmdline file generated by the same build system. In any case, both files would need to be served from the dispatcher TFTP server to the target device running Coreboot / Depthcharge.
So the idea was basically to have an option in Coreboot / Depthcharge to interactively tell it where to find these files for the current job to run, say:
<JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/vmlinuz <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/cmdline
It looks like the current patch in Gerrit relies on this location to be hard-coded in the bootloader, which works fine for a private development set-up but not for LAVA.
To recap, my understanding is that the "depthcharge" boot support code in LAVA would need to:
* maybe create the cmdline file with basically the kernel command line split up with one argument per line
* or just download the cmdline file along with the vmlinuz FIT
* place both the cmdline and vmlinuz FIT files in the job's TFTP directory on the dispatcher
* turn on the device and open the serial console...
* interactively pass at least the path to the job TFTP directory on the serial console (and if possible the server IP address as well, and maybe even the individual file names rather than hard-coded vmlinuz and cmdline)
* look for a bootloader message to know when the kernel starts to load and hand over to the next action (login...)
Please let me know if this sounds reasonable or if we should be doing anything differently. I think it would be good to have some agreement and a clear understanding of how this is going to be implemented before starting to work on the code again.
Best wishes, Guillaume
On 7 December 2017 at 16:20, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
A change was sent a while ago to add support for the Coreboot / Depthcharge bootloader which is used on Chromebook devices. This is useful in particular to avoid having to install U-Boot on Chromebook devices. See this Gerrit review here for previous history:
https://review.linaro.org/#/c/15203/I'm now opening this case again to try and get this resolved, there seem to be several issues with the original patch that would need to be clarified. Also, some things might have changed since then in LAVA or Coreboot which could potentially lead to a different approach - any feedback on this would be welcome.
Thanks for picking this up.
To start with, I understand that running mkimage on the dispatcher is not a valid thing to do, it should receive a FIT (flattened image tree) kernel image ready to be booted. This complicates things a bit for projects like kernelci.org where only a plain kernel image is built and ramdisks are served separately, but it's fair enough to say that LAVA is not meant to be packaging kernel images on the fly.
We've come up with a method in the meantime, although it does mean using LXC but that makes it completely generic. It's principally designed for boards which need to munge a kernel and other files into an image to be transferred to the device using tools like fastboot. This is how KernelCI will be able to submit boot tests on devices like HiKey and db410c. Sadly, the example test job is suffering because the db410c devices have a different problem which is keeping them offline. Matt has been looking into this.
https://staging.validation.linaro.org/scheduler/job/203317/definition
https://staging.validation.linaro.org/static/docs/v2/actions-deploy.html#ind...
Then I believe creating the command line file in LAVA should be fine, although it probably makes more sense to have both the FIT image and cmdline file generated by the same build system. In any case, both files would need to be served from the dispatcher TFTP server to the target device running Coreboot / Depthcharge.
That bit is fine, the problem is why this cannot use the existing temporary paths which all the other TFTP devices use. Is it just to do some mangling of the files?
So the idea was basically to have an option in Coreboot / Depthcharge to interactively tell it where to find these files for the current job to run, say:
<JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/vmlinuz <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/cmdlineIt looks like the current patch in Gerrit relies on this location to be hard-coded in the bootloader, which works fine for a private development set-up but not for LAVA.
That makes very little sense because the whole point of TFTP is that everything after the SERVER_IP is just a relative path from the TFTP base directory which is handled by the TFTP daemon itself.
To recap, my understanding is that the "depthcharge" boot support code in LAVA would need to:
- maybe create the cmdline file with basically the kernel command line split up with one argument per line
Alternatively, do whatever operations are required in a test shell in the LXC and then pass those files to the device - entirely within the test shell support.
or just download the cmdline file along with the vmlinuz FIT
place both the cmdline and vmlinuz FIT files in the job's TFTP directory on the dispatcher
turn on the device and open the serial console...
interactively pass at least the path to the job TFTP directory on the serial console (and if possible the server IP address as well, and maybe even the individual file names rather than hard-coded vmlinuz and cmdline)
Isn't this equivalent to what U-Boot already does with TFTP?
- look for a bootloader message to know when the kernel starts to load and hand over to the next action (login...)
Please let me know if this sounds reasonable or if we should be doing anything differently. I think it would be good to have some agreement and a clear understanding of how this is going to be implemented before starting to work on the code again.
Best wishes, Guillaume _______________________________________________ Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
On 07/12/17 17:16, Neil Williams wrote:
On 7 December 2017 at 16:20, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
A change was sent a while ago to add support for the Coreboot / Depthcharge bootloader which is used on Chromebook devices. This is useful in particular to avoid having to install U-Boot on Chromebook devices. See this Gerrit review here for previous history:
https://review.linaro.org/#/c/15203/I'm now opening this case again to try and get this resolved, there seem to be several issues with the original patch that would need to be clarified. Also, some things might have changed since then in LAVA or Coreboot which could potentially lead to a different approach - any feedback on this would be welcome.
Thanks for picking this up.
You're welcome. I've now uploaded a new version which generates the command line file but not the FIT image, it expects the kernel image to be already in this format. Still the same Gerrit number:
https://review.linaro.org/#/c/15203/
I've also made a patch to add the rk3288-veyron-jaq as a "depthcharge" device type:
https://review.linaro.org/#/c/22992/
So as a next step, it would be convenient to find a way to have the FIT image generated as part of the LAVA job with a given kernel image, dtb, maybe the .its file and optionally a ramdisk.
For the reference:
http://git.denx.de/?p=u-boot.git%3Ba=blob%3Bf=doc/uImage.FIT/howto.txt%3Bhb=...
To start with, I understand that running mkimage on the dispatcher is not a valid thing to do, it should receive a FIT (flattened image tree) kernel image ready to be booted. This complicates things a bit for projects like kernelci.org where only a plain kernel image is built and ramdisks are served separately, but it's fair enough to say that LAVA is not meant to be packaging kernel images on the fly.
We've come up with a method in the meantime, although it does mean using LXC but that makes it completely generic. It's principally designed for boards which need to munge a kernel and other files into an image to be transferred to the device using tools like fastboot. This is how KernelCI will be able to submit boot tests on devices like HiKey and db410c. Sadly, the example test job is suffering because the db410c devices have a different problem which is keeping them offline. Matt has been looking into this.
https://staging.validation.linaro.org/scheduler/job/203317/definition
https://staging.validation.linaro.org/static/docs/v2/actions-deploy.html#ind...
Thanks for the pointers, seems worth investigating.
On the other hand, creating the FIT image is a similar process to that of uImage, which is currently being done directly on the dispatcher:
https://git.linaro.org/lava/lava-dispatcher.git/tree/lava_dispatcher/actions...
So would it make sense to add some code there to support FIT?
Then I believe creating the command line file in LAVA should be fine, although it probably makes more sense to have both the FIT image and cmdline file generated by the same build system. In any case, both files would need to be served from the dispatcher TFTP server to the target device running Coreboot / Depthcharge.
That bit is fine, the problem is why this cannot use the existing temporary paths which all the other TFTP devices use. Is it just to do some mangling of the files?
This is resolved now with the version I sent yesterday.
So the idea was basically to have an option in Coreboot / Depthcharge to interactively tell it where to find these files for the current job to run, say:
<JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/vmlinuz <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/cmdlineIt looks like the current patch in Gerrit relies on this location to be hard-coded in the bootloader, which works fine for a private development set-up but not for LAVA.
That makes very little sense because the whole point of TFTP is that everything after the SERVER_IP is just a relative path from the TFTP base directory which is handled by the TFTP daemon itself.
Ditto.
To recap, my understanding is that the "depthcharge" boot support code in LAVA would need to:
- maybe create the cmdline file with basically the kernel command line split up with one argument per line
Alternatively, do whatever operations are required in a test shell in the LXC and then pass those files to the device - entirely within the test shell support.
That, or maybe run mkimage on the dispatcher like for uImage...
The cmdline file is now generated on the dispatcher.
- or just download the cmdline file along with the vmlinuz FIT
The ready-made FIT kernel image is now downloaded with the version I sent yesterday.
place both the cmdline and vmlinuz FIT files in the job's TFTP directory on the dispatcher
turn on the device and open the serial console...
interactively pass at least the path to the job TFTP directory on the serial console (and if possible the server IP address as well, and maybe even the individual file names rather than hard-coded vmlinuz and cmdline)
Isn't this equivalent to what U-Boot already does with TFTP?
Almost. This part is now all implemented in the last patch I sent. One thing though is that the NFS rootfs parameters are stored in the kernel cmdline file and not set interactively in the bootloader shell. The only command sent is to start the tftp boot with the server IP and the relative paths to the kernel and cmdline files.
- look for a bootloader message to know when the kernel starts to load and hand over to the next action (login...)
Done as well, I've now got the veyron-jaq device booting fine with NFS rootfs. There was an issue with adding a ramdisk to the FIT image as it was to big to boot on the device, will investigate this part to add "ramdisk" boot commands.
Please let me know if this sounds reasonable or if we should be doing anything differently. I think it would be good to have some agreement and a clear understanding of how this is going to be implemented before starting to work on the code again.
Best wishes, Guillaume
On 14 December 2017 at 09:47, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 07/12/17 17:16, Neil Williams wrote:
On 7 December 2017 at 16:20, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
A change was sent a while ago to add support for the Coreboot /
Depthcharge bootloader which is used on Chromebook devices. This is useful in particular to avoid having to install U-Boot on Chromebook devices. See this Gerrit review here for previous history:
https://review.linaro.org/#/c/15203/I'm now opening this case again to try and get this resolved, there seem to be several issues with the original patch that would need to be clarified. Also, some things might have changed since then in LAVA or Coreboot which could potentially lead to a different approach - any feedback on this would be welcome.
Thanks for picking this up.
You're welcome. I've now uploaded a new version which generates the command line file but not the FIT image, it expects the kernel image to be already in this format. Still the same Gerrit number:
https://review.linaro.org/#/c/15203/
I've also made a patch to add the rk3288-veyron-jaq as a "depthcharge" device type:
https://review.linaro.org/#/c/22992/
So as a next step, it would be convenient to find a way to have the FIT image generated as part of the LAVA job with a given kernel image, dtb, maybe the .its file and optionally a ramdisk.
For the reference:
http://git.denx.de/?p=u-boot.git%3Ba=blob%3Bf=doc/uImage.FIT/how to.txt;hb=master
To start with, I understand that running mkimage on the
dispatcher is not a valid thing to do, it should receive a FIT (flattened image tree) kernel image ready to be booted. This complicates things a bit for projects like kernelci.org where only a plain kernel image is built and ramdisks are served separately, but it's fair enough to say that LAVA is not meant to be packaging kernel images on the fly.
We've come up with a method in the meantime, although it does mean using LXC but that makes it completely generic. It's principally designed for boards which need to munge a kernel and other files into an image to be transferred to the device using tools like fastboot. This is how KernelCI will be able to submit boot tests on devices like HiKey and db410c. Sadly, the example test job is suffering because the db410c devices have a different problem which is keeping them offline. Matt has been looking into this.
https://staging.validation.linaro.org/scheduler/job/203317/definition
https://staging.validation.linaro.org/static/docs/v2/actions -deploy.html#index-25
Thanks for the pointers, seems worth investigating.
On the other hand, creating the FIT image is a similar process to that of uImage, which is currently being done directly on the dispatcher:
https://git.linaro.org/lava/lava-dispatcher.git/tree/lava_di spatcher/actions/deploy/prepare.py#n79
So would it make sense to add some code there to support FIT?
What is an example command line to mkimage to do this?
Are any external configuration files required?
Then I believe creating the command line file in LAVA should be
fine, although it probably makes more sense to have both the FIT image and cmdline file generated by the same build system. In any case, both files would need to be served from the dispatcher TFTP server to the target device running Coreboot / Depthcharge.
That bit is fine, the problem is why this cannot use the existing temporary paths which all the other TFTP devices use. Is it just to do some mangling of the files?
This is resolved now with the version I sent yesterday.
That makes this review much better, thanks.
So the idea was basically to have an option in Coreboot /
Depthcharge to interactively tell it where to find these files for the current job to run, say:
<JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/vmlinuz <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/cmdlineIt looks like the current patch in Gerrit relies on this location to be hard-coded in the bootloader, which works fine for a private development set-up but not for LAVA.
That makes very little sense because the whole point of TFTP is that everything after the SERVER_IP is just a relative path from the TFTP base directory which is handled by the TFTP daemon itself.
Ditto.
To recap, my understanding is that the "depthcharge" boot support
code in LAVA would need to:
- maybe create the cmdline file with basically the kernel command line split up with one argument per line
Alternatively, do whatever operations are required in a test shell in the LXC and then pass those files to the device - entirely within the test shell support.
That, or maybe run mkimage on the dispatcher like for uImage...
The cmdline file is now generated on the dispatcher.
- or just download the cmdline file along with the vmlinuz FIT
The ready-made FIT kernel image is now downloaded with the version I sent yesterday.
- place both the cmdline and vmlinuz FIT files in the job's
TFTP directory on the dispatcher
turn on the device and open the serial console...
interactively pass at least the path to the job TFTP directory on the serial console (and if possible the server IP address as well, and maybe even the individual file names rather than hard-coded vmlinuz and cmdline)
Isn't this equivalent to what U-Boot already does with TFTP?
Almost. This part is now all implemented in the last patch I sent. One thing though is that the NFS rootfs parameters are stored in the kernel cmdline file and not set interactively in the bootloader shell.
How can these be extended by test writers? We do see requests to add arguments to the NFS parameters but adding options to the kernel command line itself is all but essential for most testing.
The only command sent is to start the tftp boot with the server IP and the relative paths to the kernel and cmdline files.
- look for a bootloader message to know when the kernel starts
to load and hand over to the next action (login...)Done as well, I've now got the veyron-jaq device booting fine with NFS rootfs. There was an issue with adding a ramdisk to the FIT image as it was to big to boot on the device, will investigate this part to add "ramdisk" boot commands.
Please let me know if this sounds reasonable or if we should be
doing anything differently. I think it would be good to have some agreement and a clear understanding of how this is going to be implemented before starting to work on the code again.
Best wishes,
Guillaume
On 18/12/17 11:45, Neil Williams wrote:
On 14 December 2017 at 09:47, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 07/12/17 17:16, Neil Williams wrote:
On 7 December 2017 at 16:20, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
A change was sent a while ago to add support for the Coreboot /
Depthcharge bootloader which is used on Chromebook devices. This is useful in particular to avoid having to install U-Boot on Chromebook devices. See this Gerrit review here for previous history:
https://review.linaro.org/#/c/15203/I'm now opening this case again to try and get this resolved, there seem to be several issues with the original patch that would need to be clarified. Also, some things might have changed since then in LAVA or Coreboot which could potentially lead to a different approach - any feedback on this would be welcome.
Thanks for picking this up.
You're welcome. I've now uploaded a new version which generates the command line file but not the FIT image, it expects the kernel image to be already in this format. Still the same Gerrit number:
https://review.linaro.org/#/c/15203/
I've also made a patch to add the rk3288-veyron-jaq as a "depthcharge" device type:
https://review.linaro.org/#/c/22992/
So as a next step, it would be convenient to find a way to have the FIT image generated as part of the LAVA job with a given kernel image, dtb, maybe the .its file and optionally a ramdisk.
For the reference:
http://git.denx.de/?p=u-boot.git%3Ba=blob%3Bf=doc/uImage.FIT/how to.txt;hb=master
To start with, I understand that running mkimage on the
dispatcher is not a valid thing to do, it should receive a FIT (flattened image tree) kernel image ready to be booted. This complicates things a bit for projects like kernelci.org where only a plain kernel image is built and ramdisks are served separately, but it's fair enough to say that LAVA is not meant to be packaging kernel images on the fly.
We've come up with a method in the meantime, although it does mean using LXC but that makes it completely generic. It's principally designed for boards which need to munge a kernel and other files into an image to be transferred to the device using tools like fastboot. This is how KernelCI will be able to submit boot tests on devices like HiKey and db410c. Sadly, the example test job is suffering because the db410c devices have a different problem which is keeping them offline. Matt has been looking into this.
https://staging.validation.linaro.org/scheduler/job/203317/definition
https://staging.validation.linaro.org/static/docs/v2/actions -deploy.html#index-25
Thanks for the pointers, seems worth investigating.
On the other hand, creating the FIT image is a similar process to that of uImage, which is currently being done directly on the dispatcher:
https://git.linaro.org/lava/lava-dispatcher.git/tree/lava_di spatcher/actions/deploy/prepare.py#n79
So would it make sense to add some code there to support FIT?
What is an example command line to mkimage to do this?
mkimage -D "-I dts -O dtb -p 2048" -f rk3288-veyron-jaq.its arch/arm/boot/vmlinuz
Are any external configuration files required?
Everything should be in the .its file, and it should also be possible to generate it on the fly using a template and the LAVA device properties (kernel load address etc...). If this proves to not be flexible enough in practice, then I suppose the .its file could be downloaded although I think we should avoid doing this if we can.
Then I believe creating the command line file in LAVA should be
fine, although it probably makes more sense to have both the FIT image and cmdline file generated by the same build system. In any case, both files would need to be served from the dispatcher TFTP server to the target device running Coreboot / Depthcharge.
That bit is fine, the problem is why this cannot use the existing temporary paths which all the other TFTP devices use. Is it just to do some mangling of the files?
This is resolved now with the version I sent yesterday.
That makes this review much better, thanks.
Great, thanks for confirming.
So the idea was basically to have an option in Coreboot /
Depthcharge to interactively tell it where to find these files for the current job to run, say:
<JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/vmlinuz <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/cmdlineIt looks like the current patch in Gerrit relies on this location to be hard-coded in the bootloader, which works fine for a private development set-up but not for LAVA.
That makes very little sense because the whole point of TFTP is that everything after the SERVER_IP is just a relative path from the TFTP base directory which is handled by the TFTP daemon itself.
Ditto.
To recap, my understanding is that the "depthcharge" boot support
code in LAVA would need to:
* maybe create the cmdline file with basically the kernel command line split up with one argument per lineAlternatively, do whatever operations are required in a test shell in the LXC and then pass those files to the device - entirely within the test shell support.
That, or maybe run mkimage on the dispatcher like for uImage...
The cmdline file is now generated on the dispatcher.
* or just download the cmdline file along with the vmlinuz FITThe ready-made FIT kernel image is now downloaded with the version I sent yesterday.
* place both the cmdline and vmlinuz FIT files in the job'sTFTP directory on the dispatcher * turn on the device and open the serial console... * interactively pass at least the path to the job TFTP directory on the serial console (and if possible the server IP address as well, and maybe even the individual file names rather than hard-coded vmlinuz and cmdline)Isn't this equivalent to what U-Boot already does with TFTP?
Almost. This part is now all implemented in the last patch I sent. One thing though is that the NFS rootfs parameters are stored in the kernel cmdline file and not set interactively in the bootloader shell.
How can these be extended by test writers? We do see requests to add arguments to the NFS parameters but adding options to the kernel command line itself is all but essential for most testing.
This can be done using the {{ extra_kernel_args }} template variable, see the other change to add base-depthcharge.jinja2:
https://review.linaro.org/#/c/22992/1/lava_scheduler_app/tests/device-types/...
If anything more special ever needs to be done with some parameters such as inserting some IP address, it can be done in DepthchargeCommandOverlay where the command line file is generated.
The only command sent is to start the tftp boot with the server IP and the relative paths to the kernel and cmdline files.
On this topic, the changes to add the tftpboot command in Depthcharge are still under review:
https://chromium-review.googlesource.com/c/chromiumos/platform/depthcharge/+...
So I think it would actually be wiser to not merge base-depthcharge.jinja2 until the review above has been merged in case the command line syntax needs to be adjusted.
* look for a bootloader message to know when the kernel startsto load and hand over to the next action (login...)Done as well, I've now got the veyron-jaq device booting fine with NFS rootfs. There was an issue with adding a ramdisk to the FIT image as it was to big to boot on the device, will investigate this part to add "ramdisk" boot commands.
Please let me know if this sounds reasonable or if we should be
doing anything differently. I think it would be good to have some agreement and a clear understanding of how this is going to be implemented before starting to work on the code again.
Best wishes, Guillaume
he On 18 December 2017 at 14:44, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 18/12/17 11:45, Neil Williams wrote:
On 14 December 2017 at 09:47, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 07/12/17 17:16, Neil Williams wrote:
On 7 December 2017 at 16:20, Guillaume Tucker <
guillaume.tucker@collabora.com> wrote:
A change was sent a while ago to add support for the Coreboot /
Depthcharge bootloader which is used on Chromebook devices. This is useful in particular to avoid having to install U-Boot on Chromebook devices. See this Gerrit review here for previous history:
https://review.linaro.org/#/c/15203/I'm now opening this case again to try and get this resolved, there seem to be several issues with the original patch that would need to be clarified. Also, some things might have changed since then in LAVA or Coreboot which could potentially lead to a different approach - any feedback on this would be welcome.
Thanks for picking this up.
You're welcome. I've now uploaded a new version which generates the command line file but not the FIT image, it expects the kernel image to be already in this format. Still the same Gerrit number:
https://review.linaro.org/#/c/15203/
I've also made a patch to add the rk3288-veyron-jaq as a "depthcharge" device type:
https://review.linaro.org/#/c/22992/
So as a next step, it would be convenient to find a way to have the FIT image generated as part of the LAVA job with a given kernel image, dtb, maybe the .its file and optionally a ramdisk.
For the reference:
http://git.denx.de/?p=u-boot.git%3Ba=blob%3Bf=doc/uImage.FIT/how to.txt;hb=master
To start with, I understand that running mkimage on the
dispatcher is not a valid thing to do, it should receive a
FIT (flattened image tree) kernel image ready to be booted. This complicates things a bit for projects like kernelci.org where only a plain kernel image is built and ramdisks are served separately, but it's fair enough to say that LAVA is not meant to be packaging kernel images on the fly.
We've come up with a method in the meantime, although it does mean
using LXC but that makes it completely generic. It's principally designed for boards which need to munge a kernel and other files into an image to be transferred to the device using tools like fastboot. This is how KernelCI will be able to submit boot tests on devices like HiKey and db410c. Sadly, the example test job is suffering because the db410c devices have a different problem which is keeping them offline. Matt has been looking into this.
https://staging.validation.linaro.org/scheduler/job/203317/definition
https://staging.validation.linaro.org/static/docs/v2/actions -deploy.html#index-25
Thanks for the pointers, seems worth investigating.
On the other hand, creating the FIT image is a similar process to that of uImage, which is currently being done directly on the dispatcher:
https://git.linaro.org/lava/lava-dispatcher.git/tree/lava_di spatcher/actions/deploy/prepare.py#n79
So would it make sense to add some code there to support FIT?
What is an example command line to mkimage to do this?
mkimage -D "-I dts -O dtb -p 2048" -f rk3288-veyron-jaq.its arch/arm/boot/vmlinuz
No --arch argument? Is this operation agnostic about 32bit vs 64bit or armv7 vs armv8?
-p 2048 - the position presumably comes from the device configuration too, just like a load address?
(mkimage_arch is an available value, so it would seem sensible to use it if it's supportable. Someone, somewhere will create a 64bit machine with 32bit firmware - as happened with the mustang - so avoid any assumptions here.)
Are any external configuration files required?
Everything should be in the .its file, and it should also be possible to generate it on the fly using a template and the LAVA device properties (kernel load address etc...). If this proves to not be flexible enough in practice, then I suppose the .its file could be downloaded although I think we should avoid doing this if we can.
Building the .its file in the per-job temporary directory from data already available in the device configuration sounds ideal.
The LXC support remains available if particular tests need customised handling or options (or if a patched version of mkimage is needed).
Then I believe creating the command line file in LAVA should be
fine, although it probably makes more sense to have both the FIT
image and cmdline file generated by the same build system. In any case, both files would need to be served from the dispatcher TFTP server to the target device running Coreboot / Depthcharge.
That bit is fine, the problem is why this cannot use the existing
temporary paths which all the other TFTP devices use. Is it just to do some mangling of the files?
This is resolved now with the version I sent yesterday.
That makes this review much better, thanks.
Great, thanks for confirming.
So the idea was basically to have an option in Coreboot /
Depthcharge to interactively tell it where to find these files
for the current job to run, say:
<JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/vmlinuz <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/cmdlineIt looks like the current patch in Gerrit relies on this location to be hard-coded in the bootloader, which works fine for a private development set-up but not for LAVA.
That makes very little sense because the whole point of TFTP is that
everything after the SERVER_IP is just a relative path from the TFTP base directory which is handled by the TFTP daemon itself.
Ditto.
To recap, my understanding is that the "depthcharge" boot support
code in LAVA would need to:
* maybe create the cmdline file with basically the kernel command line split up with one argument per lineAlternatively, do whatever operations are required in a test shell in
the LXC and then pass those files to the device - entirely within the test shell support.
That, or maybe run mkimage on the dispatcher like for uImage...
The cmdline file is now generated on the dispatcher.
* or just download the cmdline file along with the vmlinuz FITThe ready-made FIT kernel image is now downloaded with the
version I sent yesterday.
* place both the cmdline and vmlinuz FIT files in the job'sTFTP directory on the dispatcher* turn on the device and open the serial console... * interactively pass at least the path to the job TFTP directory on the serial console (and if possible the server IP address as well, and maybe even the individual file names rather than hard-coded vmlinuz and cmdline)Isn't this equivalent to what U-Boot already does with TFTP?
Almost. This part is now all implemented in the last patch I sent. One thing though is that the NFS rootfs parameters are stored in the kernel cmdline file and not set interactively in the bootloader shell.
How can these be extended by test writers? We do see requests to add arguments to the NFS parameters but adding options to the kernel command line itself is all but essential for most testing.
This can be done using the {{ extra_kernel_args }} template variable, see the other change to add base-depthcharge.jinja2:
https://review.linaro.org/#/c/22992/1/lava_scheduler_app/tes ts/device-types/base-depthcharge.jinja2
If anything more special ever needs to be done with some parameters such as inserting some IP address, it can be done in DepthchargeCommandOverlay where the command line file is generated.
That's what I wanted to check, OK. Generating the command line is the right way to do it.
The only command sent is to start the tftp
boot with the server IP and the relative paths to the kernel and cmdline files.
On this topic, the changes to add the tftpboot command in Depthcharge are still under review:
https://chromium-review.googlesource.com/c/chromiumos/platfo rm/depthcharge/+/451382
So I think it would actually be wiser to not merge base-depthcharge.jinja2 until the review above has been merged in case the command line syntax needs to be adjusted.
OK.
The one remaining issue is that this support (like NBD) doesn't have supported devices available in the staging instance, so we need to do as much validation as possible in the unit tests to ensure that future changes elsewhere do not cause issues with depthcharge. e.g. Do as much of the command line generation as possible in validate() so that this can be checked in the unit test. Then use the substitute support to replace {KERNEL} or {LOAD_ADDR} type fields in the run() function to allow for the temporary locations. Only you know how much variability there may be in the final command lines - each variant should be checked, that if the incoming device configuration is correct (static, so easy to check), then the validate() produces a command line which is precisely what is expected, prior to running substitute. We won't be able to run functional tests in the run up to a production release, we need the unit test to have coverage which is as wide as possible.
* look for a bootloader message to know when the kernel startsto load and hand over to the next action (login...)Done as well, I've now got the veyron-jaq device booting fine
with NFS rootfs. There was an issue with adding a ramdisk to the FIT image as it was to big to boot on the device, will investigate this part to add "ramdisk" boot commands.
Seems to be a perennial problem with U-Boot - the initial load addresses are always too conservative and as soon as people start building fully functional kernels / ramdisks, there isn't enough space between the load addresses.
Please let me know if this sounds reasonable or if we should be
doing anything differently. I think it would be good to have
some agreement and a clear understanding of how this is going to be implemented before starting to work on the code again.
Best wishes, Guillaume
On 18/12/17 15:06, Neil Williams wrote:
he On 18 December 2017 at 14:44, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 18/12/17 11:45, Neil Williams wrote:
On 14 December 2017 at 09:47, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 07/12/17 17:16, Neil Williams wrote:
On 7 December 2017 at 16:20, Guillaume Tucker <
guillaume.tucker@collabora.com> wrote:
A change was sent a while ago to add support for the Coreboot /
Depthcharge bootloader which is used on Chromebook devices. This is useful in particular to avoid having to install U-Boot on Chromebook devices. See this Gerrit review here for previous history:
https://review.linaro.org/#/c/15203/I'm now opening this case again to try and get this resolved, there seem to be several issues with the original patch that would need to be clarified. Also, some things might have changed since then in LAVA or Coreboot which could potentially lead to a different approach - any feedback on this would be welcome.
Thanks for picking this up.
You're welcome. I've now uploaded a new version which generates the command line file but not the FIT image, it expects the kernel image to be already in this format. Still the same Gerrit number:
https://review.linaro.org/#/c/15203/I've also made a patch to add the rk3288-veyron-jaq as a "depthcharge" device type:
https://review.linaro.org/#/c/22992/So as a next step, it would be convenient to find a way to have the FIT image generated as part of the LAVA job with a given kernel image, dtb, maybe the .its file and optionally a ramdisk.
For the reference:
http://git.denx.de/?p=u-boot.git;a=blob;f=doc/uImage.FIT/howto.txt;hb=master
To start with, I understand that running mkimage on the
dispatcher is not a valid thing to do, it should receive a
FIT (flattened image tree) kernel image ready to be booted. This complicates things a bit for projects like kernelci.org where only a plain kernel image is built and ramdisks are served separately, but it's fair enough to say that LAVA is not meant to be packaging kernel images on the fly.
We've come up with a method in the meantime, although it does mean
using LXC but that makes it completely generic. It's principally designed for boards which need to munge a kernel and other files into an image to be transferred to the device using tools like fastboot. This is how KernelCI will be able to submit boot tests on devices like HiKey and db410c. Sadly, the example test job is suffering because the db410c devices have a different problem which is keeping them offline. Matt has been looking into this.
https://staging.validation.linaro.org/scheduler/job/203317/definition
https://staging.validation.linaro.org/static/docs/v2/actions -deploy.html#index-25
Thanks for the pointers, seems worth investigating.
On the other hand, creating the FIT image is a similar process to that of uImage, which is currently being done directly on the dispatcher:
https://git.linaro.org/lava/lava-dispatcher.git/tree/lava_dispatcher/actions/deploy/prepare.py#n79
So would it make sense to add some code there to support FIT?
What is an example command line to mkimage to do this?
mkimage -D "-I dts -O dtb -p 2048" -f rk3288-veyron-jaq.its arch/arm/boot/vmlinuz
No --arch argument? Is this operation agnostic about 32bit vs 64bit or armv7 vs armv8?
The CPU arch is also part of the .its file, here's an example:
https://people.collabora.com/~gtucker/tmp/rk3288-veyron-jaq-ramdisk.its
-p 2048 - the position presumably comes from the device configuration too, just like a load address?
This is the dtc -p padding option, not mkimage -p position for external data. It's actually more related to the kernel image and what the bootloader does to it than any platform in particular. So I think a large enough padding value like 2048 should work with all cases. It may still make sense to have a template variable to override this on some devices.
(mkimage_arch is an available value, so it would seem sensible to use it if it's supportable. Someone, somewhere will create a 64bit machine with 32bit firmware - as happened with the mustang - so avoid any assumptions here.)
Sure.
Are any external configuration files required?
Everything should be in the .its file, and it should also be possible to generate it on the fly using a template and the LAVA device properties (kernel load address etc...). If this proves to not be flexible enough in practice, then I suppose the .its file could be downloaded although I think we should avoid doing this if we can.
Building the .its file in the per-job temporary directory from data already available in the device configuration sounds ideal.
Great.
The LXC support remains available if particular tests need customised handling or options (or if a patched version of mkimage is needed).
The FIT format has been around for long enough for it to work with standard mkimage, I would hope. But yes, if some LAVA labs use an older distro with an incompatible version then LXC would be a way to keep the version of mkimage under control. It would however make the test definitions a bit more complicated and depthcharge-specific, which is something we would prefer to avoid for kernelci.org in particular.
So I'll see how it goes with running mkimage on the dispatcher and with various versions of mkimage.
Then I believe creating the command line file in LAVA should be
fine, although it probably makes more sense to have both the FIT
image and cmdline file generated by the same build system. In any case, both files would need to be served from the dispatcher TFTP server to the target device running Coreboot / Depthcharge.
That bit is fine, the problem is why this cannot use the existing
temporary paths which all the other TFTP devices use. Is it just to do some mangling of the files?
This is resolved now with the version I sent yesterday.
That makes this review much better, thanks.
Great, thanks for confirming.
So the idea was basically to have an option in Coreboot /
Depthcharge to interactively tell it where to find these files
for the current job to run, say:
<JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/vmlinuz <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/cmdlineIt looks like the current patch in Gerrit relies on this location to be hard-coded in the bootloader, which works fine for a private development set-up but not for LAVA.
That makes very little sense because the whole point of TFTP is that
everything after the SERVER_IP is just a relative path from the TFTP base directory which is handled by the TFTP daemon itself.
Ditto.
To recap, my understanding is that the "depthcharge" boot support
code in LAVA would need to:
* maybe create the cmdline file with basically the kernel command line split up with one argument per lineAlternatively, do whatever operations are required in a test shell in
the LXC and then pass those files to the device - entirely within the test shell support.
That, or maybe run mkimage on the dispatcher like for uImage...
The cmdline file is now generated on the dispatcher.
* or just download the cmdline file along with the vmlinuz FITThe ready-made FIT kernel image is now downloaded with the
version I sent yesterday.
* place both the cmdline and vmlinuz FIT files in the job'sTFTP directory on the dispatcher* turn on the device and open the serial console... * interactively pass at least the path to the job TFTP directory on the serial console (and if possible the server IP address as well, and maybe even the individual file names rather than hard-coded vmlinuz and cmdline)Isn't this equivalent to what U-Boot already does with TFTP?
Almost. This part is now all implemented in the last patch I sent. One thing though is that the NFS rootfs parameters are stored in the kernel cmdline file and not set interactively in the bootloader shell.
How can these be extended by test writers? We do see requests to add arguments to the NFS parameters but adding options to the kernel command line itself is all but essential for most testing.
This can be done using the {{ extra_kernel_args }} template variable, see the other change to add base-depthcharge.jinja2:
https://review.linaro.org/#/c/22992/1/lava_scheduler_app/tes ts/device-types/base-depthcharge.jinja2
If anything more special ever needs to be done with some parameters such as inserting some IP address, it can be done in DepthchargeCommandOverlay where the command line file is generated.
That's what I wanted to check, OK. Generating the command line is the right way to do it.
Great.
The only command sent is to start the tftpboot with the server IP and the relative paths to the kernel and cmdline files.
On this topic, the changes to add the tftpboot command in Depthcharge are still under review:
https://chromium-review.googlesource.com/c/chromiumos/platfo rm/depthcharge/+/451382
So I think it would actually be wiser to not merge base-depthcharge.jinja2 until the review above has been merged in case the command line syntax needs to be adjusted.
OK.
The one remaining issue is that this support (like NBD) doesn't have supported devices available in the staging instance, so we need to do as much validation as possible in the unit tests to ensure that future changes elsewhere do not cause issues with depthcharge. e.g. Do as much of the command line generation as possible in validate() so that this can be checked in the unit test. Then use the substitute support to replace {KERNEL} or {LOAD_ADDR} type fields in the run() function to allow for the temporary locations. Only you know how much variability there may be in the final command lines - each variant should be checked, that if the incoming device configuration is correct (static, so easy to check), then the validate() produces a command line which is precisely what is expected, prior to running substitute. We won't be able to run functional tests in the run up to a production release, we need the unit test to have coverage which is as wide as possible.
Understood.
In addition to adding good unit test coverage, I'll see if I can keep a spare Chromebook outside of our lab to use with my dev set-up as I'm using now (i.e. laptop running latest LAVA). It won't be automated like on your staging instance but would be better than nothing if I can just run some tests each month.
* look for a bootloader message to know when the kernel startsto load and hand over to the next action (login...)Done as well, I've now got the veyron-jaq device booting fine
with NFS rootfs. There was an issue with adding a ramdisk to the FIT image as it was to big to boot on the device, will investigate this part to add "ramdisk" boot commands.
Seems to be a perennial problem with U-Boot - the initial load addresses are always too conservative and as soon as people start building fully functional kernels / ramdisks, there isn't enough space between the load addresses.
I've fixed this now, it's a slightly different issue here. There's a build-time configuration in Depthcharge to reserve some memory to receive the FIT image. The default value is fine for Chromebook products which don't have a ramdisk. I've extended it and now it gets the full FIT with ramdisk included. I still need to fix a few more things though as it's not quite booting with ramdisk yet...
Best wishes, Guillaume
On 18/12/17 17:26, Guillaume Tucker wrote:
On 18/12/17 15:06, Neil Williams wrote:
he On 18 December 2017 at 14:44, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 18/12/17 11:45, Neil Williams wrote:
On 14 December 2017 at 09:47, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 07/12/17 17:16, Neil Williams wrote:
On 7 December 2017 at 16:20, Guillaume Tucker <
guillaume.tucker@collabora.com> wrote:
A change was sent a while ago to add support for the Coreboot /
> Depthcharge bootloader which is used on Chromebook devices. This > is useful in particular to avoid having to install U-Boot on > Chromebook devices. See this Gerrit review here for previous > history: > > https://review.linaro.org/#/c/15203/ > > I'm now opening this case again to try and get this resolved, > there seem to be several issues with the original patch that > would need to be clarified. Also, some things might have changed > since then in LAVA or Coreboot which could potentially lead to a > different approach - any feedback on this would be welcome. > > > Thanks for picking this up.
You're welcome. I've now uploaded a new version which generates the command line file but not the FIT image, it expects the kernel image to be already in this format. Still the same Gerrit number:
https://review.linaro.org/#/c/15203/
I've also made a patch to add the rk3288-veyron-jaq as a "depthcharge" device type:
https://review.linaro.org/#/c/22992/
So as a next step, it would be convenient to find a way to have the FIT image generated as part of the LAVA job with a given kernel image, dtb, maybe the .its file and optionally a ramdisk.
For the reference:
http://git.denx.de/?p=u-boot.git%3Ba=blob%3Bf=doc/uImage.FIT/how to.txt;hb=master
To start with, I understand that running mkimage on the
dispatcher is not a valid thing to do, it should receive a > FIT (flattened image tree) kernel image ready to be booted. This > complicates things a bit for projects like kernelci.org where > only a plain kernel image is built and ramdisks are served > separately, but it's fair enough to say that LAVA is not meant to > be packaging kernel images on the fly. > > > We've come up with a method in the meantime, although it does mean using LXC but that makes it completely generic. It's principally designed for boards which need to munge a kernel and other files into an image to be transferred to the device using tools like fastboot. This is how KernelCI will be able to submit boot tests on devices like HiKey and db410c. Sadly, the example test job is suffering because the db410c devices have a different problem which is keeping them offline. Matt has been looking into this.
https://staging.validation.linaro.org/scheduler/job/203317/definition
https://staging.validation.linaro.org/static/docs/v2/actions -deploy.html#index-25
Thanks for the pointers, seems worth investigating.
On the other hand, creating the FIT image is a similar process to that of uImage, which is currently being done directly on the dispatcher:
https://git.linaro.org/lava/lava-dispatcher.git/tree/lava_di spatcher/actions/deploy/prepare.py#n79
So would it make sense to add some code there to support FIT?
What is an example command line to mkimage to do this?
mkimage -D "-I dts -O dtb -p 2048" -f rk3288-veyron-jaq.its arch/arm/boot/vmlinuz
No --arch argument? Is this operation agnostic about 32bit vs 64bit or armv7 vs armv8?
The CPU arch is also part of the .its file, here's an example:
https://people.collabora.com/~gtucker/tmp/rk3288-veyron-jaq-ramdisk.its
-p 2048 - the position presumably comes from the device configuration too, just like a load address?
This is the dtc -p padding option, not mkimage -p position for external data. It's actually more related to the kernel image and what the bootloader does to it than any platform in particular. So I think a large enough padding value like 2048 should work with all cases. It may still make sense to have a template variable to override this on some devices.
(mkimage_arch is an available value, so it would seem sensible to use it if it's supportable. Someone, somewhere will create a 64bit machine with 32bit firmware - as happened with the mustang - so avoid any assumptions here.)
Sure.
Are any external configuration files required?
Everything should be in the .its file, and it should also be possible to generate it on the fly using a template and the LAVA device properties (kernel load address etc...). If this proves to not be flexible enough in practice, then I suppose the .its file could be downloaded although I think we should avoid doing this if we can.
Building the .its file in the per-job temporary directory from data already available in the device configuration sounds ideal.
Great.
I've now added some patches to generate the FIT image using mkimage using a similar approach as for uImage, see all the current reviews here:
https://review.linaro.org/#/q/topic:T6321-depthcharge
As Tomeu pointed out, the "mkimage -f auto" option can be used to automatically generate a .its structure without actually using any intermediate file. There are a few things that can't be done with this simple mode but as far as I can tell it should cover our needs for Depthcharge.
The LXC support remains available if particular tests need customised handling or options (or if a patched version of mkimage is needed).
The FIT format has been around for long enough for it to work with standard mkimage, I would hope. But yes, if some LAVA labs use an older distro with an incompatible version then LXC would be a way to keep the version of mkimage under control. It would however make the test definitions a bit more complicated and depthcharge-specific, which is something we would prefer to avoid for kernelci.org in particular.
So I'll see how it goes with running mkimage on the dispatcher and with various versions of mkimage.
I still need to compare various versions of mkimage.
Then I believe creating the command line file in LAVA should be
fine, although it probably makes more sense to have both the FIT > image and cmdline file generated by the same build system. In > any case, both files would need to be served from the dispatcher > TFTP server to the target device running Coreboot / Depthcharge. > > > That bit is fine, the problem is why this cannot use the existing temporary paths which all the other TFTP devices use. Is it just to do some mangling of the files?
This is resolved now with the version I sent yesterday.
That makes this review much better, thanks.
Great, thanks for confirming.
So the idea was basically to have an option in Coreboot /
Depthcharge to interactively tell it where to find these files > for the current job to run, say: > > <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/vmlinuz > <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/cmdline > > It looks like the current patch in Gerrit relies on this location > to be hard-coded in the bootloader, which works fine for a > private development set-up but not for LAVA. > > > That makes very little sense because the whole point of TFTP is that everything after the SERVER_IP is just a relative path from the TFTP base directory which is handled by the TFTP daemon itself.
Ditto.
To recap, my understanding is that the "depthcharge" boot support
code in LAVA would need to: > > * maybe create the cmdline file with basically the kernel > command line split up with one argument per line > > > Alternatively, do whatever operations are required in a test shell in the LXC and then pass those files to the device - entirely within the test shell support.
That, or maybe run mkimage on the dispatcher like for uImage...
The cmdline file is now generated on the dispatcher.
* or just download the cmdline file along with the vmlinuz FIT
> The ready-made FIT kernel image is now downloaded with the
version I sent yesterday.
* place both the cmdline and vmlinuz FIT files in the job's
TFTP directory on the dispatcher > > * turn on the device and open the serial console... > > * interactively pass at least the path to the job TFTP > directory on the serial console (and if possible the server > IP address as well, and maybe even the individual file names > rather than hard-coded vmlinuz and cmdline) > > > Isn't this equivalent to what U-Boot already does with TFTP?
Almost. This part is now all implemented in the last patch I sent. One thing though is that the NFS rootfs parameters are stored in the kernel cmdline file and not set interactively in the bootloader shell.
How can these be extended by test writers? We do see requests to add arguments to the NFS parameters but adding options to the kernel command line itself is all but essential for most testing.
This can be done using the {{ extra_kernel_args }} template variable, see the other change to add base-depthcharge.jinja2:
https://review.linaro.org/#/c/22992/1/lava_scheduler_app/tes ts/device-types/base-depthcharge.jinja2
If anything more special ever needs to be done with some parameters such as inserting some IP address, it can be done in DepthchargeCommandOverlay where the command line file is generated.
That's what I wanted to check, OK. Generating the command line is the right way to do it.
Great.
The only command sent is to start the tftp
boot with the server IP and the relative paths to the kernel and cmdline files.
On this topic, the changes to add the tftpboot command in Depthcharge are still under review:
https://chromium-review.googlesource.com/c/chromiumos/platfo rm/depthcharge/+/451382
So I think it would actually be wiser to not merge base-depthcharge.jinja2 until the review above has been merged in case the command line syntax needs to be adjusted.
OK.
The one remaining issue is that this support (like NBD) doesn't have supported devices available in the staging instance, so we need to do as much validation as possible in the unit tests to ensure that future changes elsewhere do not cause issues with depthcharge. e.g. Do as much of the command line generation as possible in validate() so that this can be checked in the unit test. Then use the substitute support to replace {KERNEL} or {LOAD_ADDR} type fields in the run() function to allow for the temporary locations. Only you know how much variability there may be in the final command lines - each variant should be checked, that if the incoming device configuration is correct (static, so easy to check), then the validate() produces a command line which is precisely what is expected, prior to running substitute. We won't be able to run functional tests in the run up to a production release, we need the unit test to have coverage which is as wide as possible.
Understood.
In addition to adding good unit test coverage, I'll see if I can keep a spare Chromebook outside of our lab to use with my dev set-up as I'm using now (i.e. laptop running latest LAVA). It won't be automated like on your staging instance but would be better than nothing if I can just run some tests each month.
* look for a bootloader message to know when the kernel starts
to load and hand over to the next action (login...) > > Done as well, I've now got the veyron-jaq device booting fine
with NFS rootfs. There was an issue with adding a ramdisk to the FIT image as it was to big to boot on the device, will investigate this part to add "ramdisk" boot commands.
Seems to be a perennial problem with U-Boot - the initial load addresses are always too conservative and as soon as people start building fully functional kernels / ramdisks, there isn't enough space between the load addresses.
I've fixed this now, it's a slightly different issue here. There's a build-time configuration in Depthcharge to reserve some memory to receive the FIT image. The default value is fine for Chromebook products which don't have a ramdisk. I've extended it and now it gets the full FIT with ramdisk included. I still need to fix a few more things though as it's not quite booting with ramdisk yet...
Now I do have the device booting with ramdisk, but not a compressed one and it shouldn't be too big... So I think it's just about good enough but would be good to enable gzip in Depthcharge next. This isn't strictly a LAVA issue though, I don't think the depthcharge boot method should be validating the type of ramdisk compression used in LAVA.
Guillaume
On 19/12/17 19:02, Guillaume Tucker wrote: [...]
I've now added some patches to generate the FIT image using mkimage using a similar approach as for uImage, see all the current reviews here:
https://review.linaro.org/#/q/topic:T6321-depthcharge
As Tomeu pointed out, the "mkimage -f auto" option can be used to automatically generate a .its structure without actually using any intermediate file. There are a few things that can't be done with this simple mode but as far as I can tell it should cover our needs for Depthcharge.
The LXC support remains available if particular tests need customised handling or options (or if a patched version of mkimage is needed).
The FIT format has been around for long enough for it to work with standard mkimage, I would hope. But yes, if some LAVA labs use an older distro with an incompatible version then LXC would be a way to keep the version of mkimage under control. It would however make the test definitions a bit more complicated and depthcharge-specific, which is something we would prefer to avoid for kernelci.org in particular.
So I'll see how it goes with running mkimage on the dispatcher and with various versions of mkimage.
I still need to compare various versions of mkimage.
So Debian Stretch has mkimage 2016.11 which supports the "-f auto" option. However, Debian Jessie has mkimage 2014.10 which does not and really needs an its file to generate an FIT image.
Is it OK to require a recent enough version of mkimage as in Stretch in order to generate these FIT images, or should we generate the its file to support older versions?
I started to look at options to generate the its file, it can be a bit cumbersome but should not be technically challenging... Would be nice to avoid it if possible to keep the code simpler.
Alternatively, it should hopefully be easy enough to backport u-boot-tools from Stretch to a local Jessie repository.
Guillaume
On 21/12/17 14:49, Guillaume Tucker wrote:
On 19/12/17 19:02, Guillaume Tucker wrote: [...]
I've now added some patches to generate the FIT image using mkimage using a similar approach as for uImage, see all the current reviews here:
https://review.linaro.org/#/q/topic:T6321-depthcharge
As Tomeu pointed out, the "mkimage -f auto" option can be used to automatically generate a .its structure without actually using any intermediate file. There are a few things that can't be done with this simple mode but as far as I can tell it should cover our needs for Depthcharge.
The LXC support remains available if particular tests need customised handling or options (or if a patched version of mkimage is needed).
The FIT format has been around for long enough for it to work with standard mkimage, I would hope. But yes, if some LAVA labs use an older distro with an incompatible version then LXC would be a way to keep the version of mkimage under control. It would however make the test definitions a bit more complicated and depthcharge-specific, which is something we would prefer to avoid for kernelci.org in particular.
So I'll see how it goes with running mkimage on the dispatcher and with various versions of mkimage.
I still need to compare various versions of mkimage.
So Debian Stretch has mkimage 2016.11 which supports the "-f auto" option. However, Debian Jessie has mkimage 2014.10 which does not and really needs an its file to generate an FIT image.
Is it OK to require a recent enough version of mkimage as in Stretch in order to generate these FIT images, or should we generate the its file to support older versions?
I started to look at options to generate the its file, it can be a bit cumbersome but should not be technically challenging... Would be nice to avoid it if possible to keep the code simpler.
Alternatively, it should hopefully be easy enough to backport u-boot-tools from Stretch to a local Jessie repository.
Also, dtc is not installed on ci.linaro.org so the Gerrit automated unit tests are now failing since I added a check:
14:35:27 JobError: Invalid job data: ["Cannot find command 'dtc' in $PATH"]
Guillaume
On 21/12/17 14:51, Guillaume Tucker wrote:
On 21/12/17 14:49, Guillaume Tucker wrote:
On 19/12/17 19:02, Guillaume Tucker wrote: [...]
I've now added some patches to generate the FIT image using mkimage using a similar approach as for uImage, see all the current reviews here:
https://review.linaro.org/#/q/topic:T6321-depthcharge
As Tomeu pointed out, the "mkimage -f auto" option can be used to automatically generate a .its structure without actually using any intermediate file. There are a few things that can't be done with this simple mode but as far as I can tell it should cover our needs for Depthcharge.
The LXC support remains available if particular tests need customised handling or options (or if a patched version of mkimage is needed).
The FIT format has been around for long enough for it to work with standard mkimage, I would hope. But yes, if some LAVA labs use an older distro with an incompatible version then LXC would be a way to keep the version of mkimage under control. It would however make the test definitions a bit more complicated and depthcharge-specific, which is something we would prefer to avoid for kernelci.org in particular.
So I'll see how it goes with running mkimage on the dispatcher and with various versions of mkimage.
I still need to compare various versions of mkimage.
So Debian Stretch has mkimage 2016.11 which supports the "-f auto" option. However, Debian Jessie has mkimage 2014.10 which does not and really needs an its file to generate an FIT image.
Is it OK to require a recent enough version of mkimage as in Stretch in order to generate these FIT images, or should we generate the its file to support older versions?
I started to look at options to generate the its file, it can be a bit cumbersome but should not be technically challenging... Would be nice to avoid it if possible to keep the code simpler.
Alternatively, it should hopefully be easy enough to backport u-boot-tools from Stretch to a local Jessie repository.
Also, dtc is not installed on ci.linaro.org so the Gerrit automated unit tests are now failing since I added a check:
14:35:27 JobError: Invalid job data: ["Cannot find command 'dtc' in $PATH"]
Actually, dtc is not required with mkimage 2016.11 but it is with 2014.10, so I'll remove the check as the current code which uses "-f auto" only works with mkimage 2016.11 anyway.
</rambling>
Guillaume
On 21 December 2017 at 14:51, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 21/12/17 14:49, Guillaume Tucker wrote:
On 19/12/17 19:02, Guillaume Tucker wrote: [...]
I've now added some patches to generate the FIT image using mkimage using a similar approach as for uImage, see all the current reviews here:
https://review.linaro.org/#/q/topic:T6321-depthcharge
As Tomeu pointed out, the "mkimage -f auto" option can be used to automatically generate a .its structure without actually using any intermediate file. There are a few things that can't be done with this simple mode but as far as I can tell it should cover our needs for Depthcharge.
The LXC support remains available if particular tests need customised
handling or options (or if a patched version of mkimage is needed).
The FIT format has been around for long enough for it to work with standard mkimage, I would hope. But yes, if some LAVA labs use an older distro with an incompatible version then LXC would be a way to keep the version of mkimage under control. It would however make the test definitions a bit more complicated and depthcharge-specific, which is something we would prefer to avoid for kernelci.org in particular.
So I'll see how it goes with running mkimage on the dispatcher and with various versions of mkimage.
I still need to compare various versions of mkimage.
So Debian Stretch has mkimage 2016.11 which supports the "-f auto" option. However, Debian Jessie has mkimage 2014.10 which does not and really needs an its file to generate an FIT image.
There's no backport to jessie-backports currently.
Add a check the validate() based on infrastructure_check() and see if the version can be checked for compliance - raise InfrastructureError exception if not. Then it will be up to admins to base their dispatchers on Stretch. We'll be moving to Stretch being the recommended base for LAVA in the New Year and dropping support for Jessie at some point in 2018.
Is it OK to require a recent enough version of mkimage as in Stretch in order to generate these FIT images, or should we generate the its file to support older versions?
I started to look at options to generate the its file, it can be a bit cumbersome but should not be technically challenging... Would be nice to avoid it if possible to keep the code simpler.
Alternatively, it should hopefully be easy enough to backport u-boot-tools from Stretch to a local Jessie repository.
Also, dtc is not installed on ci.linaro.org so the Gerrit
Compare with existing unit tests - if you need a tool in the path which is not always installed, use a @skipIf decorator with the infrastructure_check on that binary.
automated unit tests are now failing since I added a check:
14:35:27 JobError: Invalid job data: ["Cannot find command 'dtc' in $PATH"]
Guillaume
On 21/12/17 15:02, Neil Williams wrote:
On 21 December 2017 at 14:51, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 21/12/17 14:49, Guillaume Tucker wrote:
On 19/12/17 19:02, Guillaume Tucker wrote: [...]
I've now added some patches to generate the FIT image using mkimage using a similar approach as for uImage, see all the current reviews here:
https://review.linaro.org/#/q/topic:T6321-depthcharge
As Tomeu pointed out, the "mkimage -f auto" option can be used to automatically generate a .its structure without actually using any intermediate file. There are a few things that can't be done with this simple mode but as far as I can tell it should cover our needs for Depthcharge.
The LXC support remains available if particular tests need customised
handling or options (or if a patched version of mkimage is needed).
The FIT format has been around for long enough for it to work with standard mkimage, I would hope. But yes, if some LAVA labs use an older distro with an incompatible version then LXC would be a way to keep the version of mkimage under control. It would however make the test definitions a bit more complicated and depthcharge-specific, which is something we would prefer to avoid for kernelci.org in particular.
So I'll see how it goes with running mkimage on the dispatcher and with various versions of mkimage.
I still need to compare various versions of mkimage.
So Debian Stretch has mkimage 2016.11 which supports the "-f auto" option. However, Debian Jessie has mkimage 2014.10 which does not and really needs an its file to generate an FIT image.
There's no backport to jessie-backports currently.
Add a check the validate() based on infrastructure_check() and see if the version can be checked for compliance - raise InfrastructureError exception if not. Then it will be up to admins to base their dispatchers on Stretch. We'll be moving to Stretch being the recommended base for LAVA in the New Year and dropping support for Jessie at some point in 2018.
Is it OK to require a recent enough version of mkimage as in Stretch in order to generate these FIT images, or should we generate the its file to support older versions?
I started to look at options to generate the its file, it can be a bit cumbersome but should not be technically challenging... Would be nice to avoid it if possible to keep the code simpler.
Alternatively, it should hopefully be easy enough to backport u-boot-tools from Stretch to a local Jessie repository.
Also, dtc is not installed on ci.linaro.org so the Gerrit
Compare with existing unit tests - if you need a tool in the path which is not always installed, use a @skipIf decorator with the infrastructure_check on that binary.
automated unit tests are now failing since I added a check:
14:35:27 JobError: Invalid job data: ["Cannot find command 'dtc' in $PATH"]
OK so it should all be good now, I've sent new patch sets that addressed all the existing comments in Gerrit. I'm hoping to get the Depthcharge tftpboot command merged upstream soon, probably in January though. Meanwhile I guess I can start updating the LAVA documentation.
Guillaume
On 21 December 2017 at 15:37, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 21/12/17 15:02, Neil Williams wrote:
On 21 December 2017 at 14:51, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 21/12/17 14:49, Guillaume Tucker wrote:
On 19/12/17 19:02, Guillaume Tucker wrote:
[...]
I've now added some patches to generate the FIT image using
mkimage using a similar approach as for uImage, see all the current reviews here:
https://review.linaro.org/#/q/topic:T6321-depthcharge
As Tomeu pointed out, the "mkimage -f auto" option can be used to automatically generate a .its structure without actually using any intermediate file. There are a few things that can't be done with this simple mode but as far as I can tell it should cover our needs for Depthcharge.
The LXC support remains available if particular tests need customised
handling or options (or if a patched version of mkimage is needed). > > The FIT format has been around for long enough for it to work with standard mkimage, I would hope. But yes, if some LAVA labs use an older distro with an incompatible version then LXC would be a way to keep the version of mkimage under control. It would however make the test definitions a bit more complicated and depthcharge-specific, which is something we would prefer to avoid for kernelci.org in particular.
So I'll see how it goes with running mkimage on the dispatcher and with various versions of mkimage.
I still need to compare various versions of mkimage.
So Debian Stretch has mkimage 2016.11 which supports the "-f auto" option. However, Debian Jessie has mkimage 2014.10 which does not and really needs an its file to generate an FIT image.
There's no backport to jessie-backports currently.
Add a check the validate() based on infrastructure_check() and see if the version can be checked for compliance - raise InfrastructureError exception if not. Then it will be up to admins to base their dispatchers on Stretch. We'll be moving to Stretch being the recommended base for LAVA in the New Year and dropping support for Jessie at some point in 2018.
Is it OK to require a recent enough version of mkimage as in
Stretch in order to generate these FIT images, or should we generate the its file to support older versions?
I started to look at options to generate the its file, it can be a bit cumbersome but should not be technically challenging... Would be nice to avoid it if possible to keep the code simpler.
Alternatively, it should hopefully be easy enough to backport u-boot-tools from Stretch to a local Jessie repository.
Also, dtc is not installed on ci.linaro.org so the Gerrit
Compare with existing unit tests - if you need a tool in the path which is not always installed, use a @skipIf decorator with the infrastructure_check on that binary.
automated unit tests are now failing since I added a check:
14:35:27 JobError: Invalid job data: ["Cannot find command 'dtc' in $PATH"]
OK so it should all be good now, I've sent new patch sets that addressed all the existing comments in Gerrit. I'm hoping to get the Depthcharge tftpboot command merged upstream soon, probably in January though. Meanwhile I guess I can start updating the LAVA documentation.
OK, so I remember you saying not to merge until the support is upstream. Do you need to rely on u-boot-tools from Stretch with the latest changes?
We're looking into the issues around Stretch because:
0: The Linaro lab are moving to Stretch in January 2018 1: Jessie is losing security support in July 2018 2: A number of packages needed for python3 support of lava-server are only currently in Stretch 3: Django 2.0 is on the horizon without python2 support.
So, as announced, we will be moving to Stretch during 2018 - and ending support for Jessie before July 2018. That may help you decide on what support you need from mkimage and friends.
On 21/12/17 15:53, Neil Williams wrote:
On 21 December 2017 at 15:37, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 21/12/17 15:02, Neil Williams wrote:
On 21 December 2017 at 14:51, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 21/12/17 14:49, Guillaume Tucker wrote:
On 19/12/17 19:02, Guillaume Tucker wrote:
[...]
I've now added some patches to generate the FIT image using
mkimage using a similar approach as for uImage, see all the current reviews here:
https://review.linaro.org/#/q/topic:T6321-depthcharge
As Tomeu pointed out, the "mkimage -f auto" option can be used to automatically generate a .its structure without actually using any intermediate file. There are a few things that can't be done with this simple mode but as far as I can tell it should cover our needs for Depthcharge.
The LXC support remains available if particular tests need customised
> handling or options (or if a patched version of mkimage is needed). >> >> > The FIT format has been around for long enough for it to work > with standard mkimage, I would hope. But yes, if some LAVA labs > use an older distro with an incompatible version then LXC would > be a way to keep the version of mkimage under control. It would > however make the test definitions a bit more complicated and > depthcharge-specific, which is something we would prefer to avoid > for kernelci.org in particular. > > So I'll see how it goes with running mkimage on the dispatcher > and with various versions of mkimage. > > I still need to compare various versions of mkimage.
So Debian Stretch has mkimage 2016.11 which supports the "-f auto" option. However, Debian Jessie has mkimage 2014.10 which does not and really needs an its file to generate an FIT image.
There's no backport to jessie-backports currently.
Add a check the validate() based on infrastructure_check() and see if the version can be checked for compliance - raise InfrastructureError exception if not. Then it will be up to admins to base their dispatchers on Stretch. We'll be moving to Stretch being the recommended base for LAVA in the New Year and dropping support for Jessie at some point in 2018.
Is it OK to require a recent enough version of mkimage as in
Stretch in order to generate these FIT images, or should we generate the its file to support older versions?
I started to look at options to generate the its file, it can be a bit cumbersome but should not be technically challenging... Would be nice to avoid it if possible to keep the code simpler.
Alternatively, it should hopefully be easy enough to backport u-boot-tools from Stretch to a local Jessie repository.
Also, dtc is not installed on ci.linaro.org so the Gerrit
Compare with existing unit tests - if you need a tool in the path which is not always installed, use a @skipIf decorator with the infrastructure_check on that binary.
automated unit tests are now failing since I added a check:
14:35:27 JobError: Invalid job data: ["Cannot find command 'dtc' in$PATH"]
OK so it should all be good now, I've sent new patch sets that addressed all the existing comments in Gerrit. I'm hoping to get the Depthcharge tftpboot command merged upstream soon, probably in January though. Meanwhile I guess I can start updating the LAVA documentation.
OK, so I remember you saying not to merge until the support is upstream. Do you need to rely on u-boot-tools from Stretch with the latest changes?
Yes, the changes I've sent require the mkimage (u-boot-tools) version from Stretch. This one does not depend on dtc.
We're looking into the issues around Stretch because:
0: The Linaro lab are moving to Stretch in January 2018 1: Jessie is losing security support in July 2018 2: A number of packages needed for python3 support of lava-server are only currently in Stretch 3: Django 2.0 is on the horizon without python2 support.
So, as announced, we will be moving to Stretch during 2018 - and ending support for Jessie before July 2018. That may help you decide on what support you need from mkimage and friends.
I guess it's good timing to do this while LAVA support for Jessie is being dropped. I would prefer to see this merged before July, but if need be we can keep patching our lab for a few releases.
Guillaume
On 21/12/17 16:09, Guillaume Tucker wrote:
On 21/12/17 15:53, Neil Williams wrote:
On 21 December 2017 at 15:37, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 21/12/17 15:02, Neil Williams wrote:
On 21 December 2017 at 14:51, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 21/12/17 14:49, Guillaume Tucker wrote:
On 19/12/17 19:02, Guillaume Tucker wrote:
[...]
I've now added some patches to generate the FIT image using > mkimage using a similar approach as for uImage, see all the > current reviews here: > > https://review.linaro.org/#/q/topic:T6321-depthcharge > > As Tomeu pointed out, the "mkimage -f auto" option can be used to > automatically generate a .its structure without actually using > any intermediate file. There are a few things that can't be done > with this simple mode but as far as I can tell it should cover > our needs for Depthcharge. > > The LXC support remains available if particular tests need customised > >> handling or options (or if a patched version of mkimage is needed). >>> >>> >> The FIT format has been around for long enough for it to work >> with standard mkimage, I would hope. But yes, if some LAVA labs >> use an older distro with an incompatible version then LXC would >> be a way to keep the version of mkimage under control. It would >> however make the test definitions a bit more complicated and >> depthcharge-specific, which is something we would prefer to avoid >> for kernelci.org in particular. >> >> So I'll see how it goes with running mkimage on the dispatcher >> and with various versions of mkimage. >> >> > I still need to compare various versions of mkimage. > > So Debian Stretch has mkimage 2016.11 which supports the "-f auto" option. However, Debian Jessie has mkimage 2014.10 which does not and really needs an its file to generate an FIT image.
There's no backport to jessie-backports currently.
Add a check the validate() based on infrastructure_check() and see if the version can be checked for compliance - raise InfrastructureError exception if not. Then it will be up to admins to base their dispatchers on Stretch. We'll be moving to Stretch being the recommended base for LAVA in the New Year and dropping support for Jessie at some point in 2018.
Is it OK to require a recent enough version of mkimage as in
Stretch in order to generate these FIT images, or should we generate the its file to support older versions?
I started to look at options to generate the its file, it can be a bit cumbersome but should not be technically challenging... Would be nice to avoid it if possible to keep the code simpler.
Alternatively, it should hopefully be easy enough to backport u-boot-tools from Stretch to a local Jessie repository.
Also, dtc is not installed on ci.linaro.org so the Gerrit
Compare with existing unit tests - if you need a tool in the path which is not always installed, use a @skipIf decorator with the infrastructure_check on that binary.
automated unit tests are now failing since I added a check:
14:35:27 JobError: Invalid job data: ["Cannot find command 'dtc' in $PATH"]
OK so it should all be good now, I've sent new patch sets that addressed all the existing comments in Gerrit. I'm hoping to get the Depthcharge tftpboot command merged upstream soon, probably in January though. Meanwhile I guess I can start updating the LAVA documentation.
OK, so I remember you saying not to merge until the support is upstream. Do you need to rely on u-boot-tools from Stretch with the latest changes?
Yes, the changes I've sent require the mkimage (u-boot-tools) version from Stretch. This one does not depend on dtc.
We're looking into the issues around Stretch because:
0: The Linaro lab are moving to Stretch in January 2018 1: Jessie is losing security support in July 2018 2: A number of packages needed for python3 support of lava-server are only currently in Stretch 3: Django 2.0 is on the horizon without python2 support.
So, as announced, we will be moving to Stretch during 2018 - and ending support for Jessie before July 2018. That may help you decide on what support you need from mkimage and friends.
I guess it's good timing to do this while LAVA support for Jessie is being dropped. I would prefer to see this merged before July, but if need be we can keep patching our lab for a few releases.
A quick update: all the changes have now been merged and are in the LAVA 2018.1 release. Thanks!
I've backported the changes on top of 2017.9 as that's what we're using in our lab and added a veryon-jaq device. It's now being added to kernelci.org as well.
As we're currently running Jessie on the dispatcher, I rebuilt the Stretch package version of u-boot-tools (2016.11) for Jessie. This was trivial, no porting effort was required and it's working fine.
Here's an example run:
https://lava.collabora.co.uk/scheduler/job/1040962
Guillaume
On 11 January 2018 at 13:16, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 21/12/17 16:09, Guillaume Tucker wrote:
On 21/12/17 15:53, Neil Williams wrote:
On 21 December 2017 at 15:37, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 21/12/17 15:02, Neil Williams wrote:
On 21 December 2017 at 14:51, Guillaume Tucker <
guillaume.tucker@collabora.com> wrote:
On 21/12/17 14:49, Guillaume Tucker wrote:
On 19/12/17 19:02, Guillaume Tucker wrote:
> [...] > > I've now added some patches to generate the FIT image using > >> mkimage using a similar approach as for uImage, see all the >> current reviews here: >> >> https://review.linaro.org/#/q/topic:T6321-depthcharge >> >> As Tomeu pointed out, the "mkimage -f auto" option can be used to >> automatically generate a .its structure without actually using >> any intermediate file. There are a few things that can't be done >> with this simple mode but as far as I can tell it should cover >> our needs for Depthcharge. >> >> The LXC support remains available if particular tests need >> customised >> >> handling or options (or if a patched version of mkimage is needed). >>> >>>> >>>> >>>> The FIT format has been around for long enough for it to work >>> with standard mkimage, I would hope. But yes, if some LAVA labs >>> use an older distro with an incompatible version then LXC would >>> be a way to keep the version of mkimage under control. It would >>> however make the test definitions a bit more complicated and >>> depthcharge-specific, which is something we would prefer to avoid >>> for kernelci.org in particular. >>> >>> So I'll see how it goes with running mkimage on the dispatcher >>> and with various versions of mkimage. >>> >>> >>> I still need to compare various versions of mkimage. >> >> >> So Debian Stretch has mkimage 2016.11 which supports the "-f > auto" option. However, Debian Jessie has mkimage 2014.10 which > does not and really needs an its file to generate an FIT image. > > > There's no backport to jessie-backports currently.
Add a check the validate() based on infrastructure_check() and see if the version can be checked for compliance - raise InfrastructureError exception if not. Then it will be up to admins to base their dispatchers on Stretch. We'll be moving to Stretch being the recommended base for LAVA in the New Year and dropping support for Jessie at some point in 2018.
Is it OK to require a recent enough version of mkimage as in
> Stretch in order to generate these FIT images, or should we > generate the its file to support older versions? > > I started to look at options to generate the its file, it can be > a bit cumbersome but should not be technically challenging... > Would be nice to avoid it if possible to keep the code simpler. > > Alternatively, it should hopefully be easy enough to backport > u-boot-tools from Stretch to a local Jessie repository. > > > Also, dtc is not installed on ci.linaro.org so the Gerrit
Compare with existing unit tests - if you need a tool in the path
which is not always installed, use a @skipIf decorator with the infrastructure_check on that binary.
automated unit tests are now failing since I added a check:
14:35:27 JobError: Invalid job data: ["Cannot find command 'dtc'in $PATH"]
OK so it should all be good now, I've sent new patch sets that
addressed all the existing comments in Gerrit. I'm hoping to get the Depthcharge tftpboot command merged upstream soon, probably in January though. Meanwhile I guess I can start updating the LAVA documentation.
OK, so I remember you saying not to merge until the support is upstream. Do you need to rely on u-boot-tools from Stretch with the latest changes?
Yes, the changes I've sent require the mkimage (u-boot-tools) version from Stretch. This one does not depend on dtc.
We're looking into the issues around Stretch because:
0: The Linaro lab are moving to Stretch in January 2018 1: Jessie is losing security support in July 2018 2: A number of packages needed for python3 support of lava-server are only currently in Stretch 3: Django 2.0 is on the horizon without python2 support.
So, as announced, we will be moving to Stretch during 2018 - and ending support for Jessie before July 2018. That may help you decide on what support you need from mkimage and friends.
I guess it's good timing to do this while LAVA support for Jessie is being dropped. I would prefer to see this merged before July, but if need be we can keep patching our lab for a few releases.
A quick update: all the changes have now been merged and are in the LAVA 2018.1 release. Thanks!
Good, I'm glad it is working!
I've backported the changes on top of 2017.9 as that's what we're using in our lab and added a veryon-jaq device. It's now being added to kernelci.org as well.
If there are particular issues holding you on 2017.9, let us know to see what needs to be done.
Note that work is ongoing to optimise the bootloader commands operation ( https://review.linaro.org/#/c/23195/)
As we're currently running Jessie on the dispatcher, I rebuilt the Stretch package version of u-boot-tools (2016.11) for Jessie. This was trivial, no porting effort was required and it's working fine.
Here's an example run:
... with two kernel exceptions logged: https://lava.collabora.co.uk/results/testcase/4847624
Hopefully, this kind of testing will help someone debug the kernel behaviour on that device.
Guillaume
On 11/01/18 13:31, Neil Williams wrote:
On 11 January 2018 at 13:16, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 21/12/17 16:09, Guillaume Tucker wrote:
On 21/12/17 15:53, Neil Williams wrote:
On 21 December 2017 at 15:37, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 21/12/17 15:02, Neil Williams wrote:
On 21 December 2017 at 14:51, Guillaume Tucker <
guillaume.tucker@collabora.com> wrote:
On 21/12/17 14:49, Guillaume Tucker wrote:
> > On 19/12/17 19:02, Guillaume Tucker wrote: > >> [...] >> >> I've now added some patches to generate the FIT image using >> >>> mkimage using a similar approach as for uImage, see all the >>> current reviews here: >>> >>> https://review.linaro.org/#/q/topic:T6321-depthcharge >>> >>> As Tomeu pointed out, the "mkimage -f auto" option can be used to >>> automatically generate a .its structure without actually using >>> any intermediate file. There are a few things that can't be done >>> with this simple mode but as far as I can tell it should cover >>> our needs for Depthcharge. >>> >>> The LXC support remains available if particular tests need >>> customised >>> >>> handling or options (or if a patched version of mkimage is needed). >>>> >>>>> >>>>> >>>>> The FIT format has been around for long enough for it to work >>>> with standard mkimage, I would hope. But yes, if some LAVA labs >>>> use an older distro with an incompatible version then LXC would >>>> be a way to keep the version of mkimage under control. It would >>>> however make the test definitions a bit more complicated and >>>> depthcharge-specific, which is something we would prefer to avoid >>>> for kernelci.org in particular. >>>> >>>> So I'll see how it goes with running mkimage on the dispatcher >>>> and with various versions of mkimage. >>>> >>>> >>>> I still need to compare various versions of mkimage. >>> >>> >>> So Debian Stretch has mkimage 2016.11 which supports the "-f >> auto" option. However, Debian Jessie has mkimage 2014.10 which >> does not and really needs an its file to generate an FIT image. >> >> >> There's no backport to jessie-backports currently. >
Add a check the validate() based on infrastructure_check() and see if the version can be checked for compliance - raise InfrastructureError exception if not. Then it will be up to admins to base their dispatchers on Stretch. We'll be moving to Stretch being the recommended base for LAVA in the New Year and dropping support for Jessie at some point in 2018.
Is it OK to require a recent enough version of mkimage as in > >> Stretch in order to generate these FIT images, or should we >> generate the its file to support older versions? >> >> I started to look at options to generate the its file, it can be >> a bit cumbersome but should not be technically challenging... >> Would be nice to avoid it if possible to keep the code simpler. >> >> Alternatively, it should hopefully be easy enough to backport >> u-boot-tools from Stretch to a local Jessie repository. >> >> >> Also, dtc is not installed on ci.linaro.org so the Gerrit >
> Compare with existing unit tests - if you need a tool in the path which is not always installed, use a @skipIf decorator with the infrastructure_check on that binary.
automated unit tests are now failing since I added a check:
> > 14:35:27 JobError: Invalid job data: ["Cannot find command 'dtc' > in > $PATH"] > > OK so it should all be good now, I've sent new patch sets that
addressed all the existing comments in Gerrit. I'm hoping to get the Depthcharge tftpboot command merged upstream soon, probably in January though. Meanwhile I guess I can start updating the LAVA documentation.
OK, so I remember you saying not to merge until the support is upstream. Do you need to rely on u-boot-tools from Stretch with the latest changes?
Yes, the changes I've sent require the mkimage (u-boot-tools) version from Stretch. This one does not depend on dtc.
We're looking into the issues around Stretch because:
0: The Linaro lab are moving to Stretch in January 2018 1: Jessie is losing security support in July 2018 2: A number of packages needed for python3 support of lava-server are only currently in Stretch 3: Django 2.0 is on the horizon without python2 support.
So, as announced, we will be moving to Stretch during 2018 - and ending support for Jessie before July 2018. That may help you decide on what support you need from mkimage and friends.
I guess it's good timing to do this while LAVA support for Jessie is being dropped. I would prefer to see this merged before July, but if need be we can keep patching our lab for a few releases.
A quick update: all the changes have now been merged and are in the LAVA 2018.1 release. Thanks!
Good, I'm glad it is working!
I've backported the changes on top of 2017.9 as that's what we're using in our lab and added a veryon-jaq device. It's now being added to kernelci.org as well.
If there are particular issues holding you on 2017.9, let us know to see what needs to be done.
Thanks, we just still have one last device in our lab that only runs v1 jobs... As soon as this is resolved we'll upgrade to the latest release.
Note that work is ongoing to optimise the bootloader commands operation ( https://review.linaro.org/#/c/23195/)
Looking good.
As we're currently running Jessie on the dispatcher, I rebuilt the Stretch package version of u-boot-tools (2016.11) for Jessie. This was trivial, no porting effort was required and it's working fine.
Here's an example run:
... with two kernel exceptions logged: https://lava.collabora.co.uk/results/testcase/4847624
Hopefully, this kind of testing will help someone debug the kernel behaviour on that device.
These were actually added by me intentionally on my kenelci-test branch precisely to test this feature and see how they then get parsed in kernelci.org. The veyron-jaq actually boots fine with mainline kernel, with no warnings.
FYI kernelci boot details showing the warnings (run on a different device with the same kernel branch):
https://staging.kernelci.org/boot/id/5a5629d7cf3a0f142db37ae8/ http://staging-storage.kernelci.org/gtucker/kernelci-test/kernelci-test-052-...
Guillaume
-
Guillaume Tucker -
Neil Williams