On 27 March 2017 at 14:54, Ковалёв Сергей SKovalev@ptsecurity.com wrote:
Thank you Neil for you reply.
Please keep the list in CC:
Compare with: https://staging.validation.linaro.org/scheduler/job/168802
I have tried https://staging.validation.linaro.org/scheduler/job/168802/definition but iPXE stuck on it. I have amd64 machine with UEFI.
"stuck" ? This is a standard amd64 Debian kernel with modules and initramfs. It is already UEFI-aware. Does the machine run Debian natively? Is there a Debian kernel you can use in your LAVA submissions (with modules and ramdisk)?
First step is to replace these files with images which work on the x86 DUT on staging.validation.linaro.org
I perform kernel development with my colleagues so I have to load our kernels.
Yes, however, to debug what is going on, you should switch to known working files so that you have a valid comparison with known working test jobs. Once debugging has produced some results, then switch back to the locally built kernels. Change one thing at a time.
That just isn't going to work. The initrd needs to come via TFTP but this is an absolute path.
'initrd' is come via TFTP. In context I supply additional kernel boot options.
Your original email quoted:
context: extra_kernel_args: initrd=/rootfs.cpio.gz root=/dev/ram0
rootfs.cpio.gz does not exist when the machine boots. The initramfs will have been downloaded by TFTP and loaded directly into memory, it simply does not exist as a cpio.gz any longer. /dev/ram0 shouldn't be needed with modern kernels. At best, it would seem that these options are ignored.
Debian initramfs log:
Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... done. Warning: fsck not present, so skipping unknown file system mount: can't find /root in /etc/fstab done. Begin: Running /scripts/local-bottom ... done. Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev failed: No such file or directory mount: mounting /dev on /root/dev failed: No such file or directory done. mount: mounting /run on /root/run failed: No such file or directory run-init: current directory on the same filesystem as the root: error 0 Target filesystem doesn't have requested /sbin/init. run-init: current directory on the same filesystem as the root: error 0 run-init: current directory on the same filesystem as the root: error 0 run-init: current directory on the same filesystem as the root: error 0 run-init: current directory on the same filesystem as the root: error 0 run-init: current directory on the same filesystem as the root: error 0 No init found. Try passing init= bootarg. BusyBox v1.22.1 (Debian 1:1.22.0-19) built-in shell (ash) Enter 'help' for a list of built-in commands. Matched prompt #5: (initramfs)
This boot option have been detected before effort to automate the process with LAVA. Without it we could see kernel panic. With it we successfully load kernel and rootfs (from Buildroot). May be in Linaro you embed that boot options in compile time?
No, we do not embed anything in V2 (it's one of the key changes from V1, we don't hid magic like that anymore.)
The files were prepared with: https://git.linaro.org/lava-team/refactoring.git/tree/scripts/x86_64-nfs.sh
You can also see the build log for the original Debian kernel package if relevant.
https://tracker.debian.org/pkg/linux-signed
https://buildd.debian.org/status/fetch.php?pkg=linux-signed&arch=amd64&a...
Running x86_64-nfs.sh in an empty directory will provide access to the config of the kernel itself as well as the initramfs and supporting tools.
It's possible these context arguments are hiding some other problem in the kernel but, as described so far, the options seem to make no sense.
The command line used in our tests is simply: Command line: ip=dhcp console=ttyS0,115200n8 lava_mac={LAVA_MAC} (where LAVA_MAC does not need to be defined for these devices.)
Does the machine run Debian natively?
Yes.
Is there a Debian kernel you can use in your LAVA submissions (with modules and ramdisk)?
No.
Yes, however, to debug what is going on, you should switch to known working files so that you have a valid comparison with known working test jobs.
I couldn't find such files. Though I have files that I could load manually (from iPXE console as in first letter or from USB flash drive).
rootfs.cpio.gz does not exist when the machine boots.
Before that I have tried `rootfs.cpio`. In manual runs I have entered " imgargs bzImage console=ttyS0,115200n8 initrd=/rootfs.cpio root=/dev/ram0" into iPXE console and that works.
/dev/ram0 shouldn't be needed with modern kernels.
My kernel is 4.10.4.
At best, it would seem that these options are ignored.
Without that option I couldn't load kernel even manually.
Running x86_64-nfs.sh in an empty directory will provide access to the config of the kernel itself as well as the initramfs and supporting tools.
Thank you very much. I will try.
Neil,
I have tried the script.
With job definition === device_type: x86 job_name: x86-debian-debootstrap
timeouts: job: minutes: 15 action: minutes: 5 actions: bootloader-action: minutes: 5 bootloader-retry: minutes: 5 bootloader-interrupt: minutes: 5 extract-nfsrootfs: seconds: 90 priority: medium visibility: public
metadata: source: https://git.linaro.org/lava-team/refactoring.git path: ipxe.yaml
actions: - deploy: timeout: minutes: 2 modules_compression: xz to: tftp kernel: url: http://192.168.0.1:8000/linaro/debian/vmlinuz ramdisk: url: http://192.168.0.1:8000/linaro/debian/initrd.gz compression: gz modules: url: http://192.168.0.1:8000/linaro/debian/modules.tar.gz compression: gz os: debian
- boot: method: ipxe commands: ramdisk prompts: - 'root@debian:~#' - '/ #'
- test: timeout: minutes: 5 definitions: - repository: git://git.linaro.org/qa/test-definitions.git from: git path: ubuntu/smoke-tests-basic.yaml name: smoke-tests - repository: http://git.linaro.org/lava-team/lava-functional-tests.git from: git path: lava-test-shell/single-node/singlenode03.yaml name: singlenode-advanced ===
I get kernel panic: === [ 2.054605] No filesystem could mount root, tried: [ 2.059523] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) [ 2.067809] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 3.16.0-4-amd64 #1 Debian 3.16.39-1 [ 2.075902] Hardware name: /DQ87PG, BIOS PGQ8710H.86A.0152.2015.0709.1545 07/09/2015 [ 2.085123] 0000000000000000 ffffffff81514c11 ffffffff817054c8 ffff88040e1b7ea0 [ 2.092576] ffffffff8151195e ffff880400000010 ffff88040e1b7eb0 ffff88040e1b7e50 [ 2.100046] ffff88040e1b7ea0 ffff88040e1b7eb8 0000000000000012 0000000000000001 [ 2.107506] Call Trace: [ 2.109953] [<ffffffff81514c11>] ? dump_stack+0x5d/0x78 [ 2.115256] [<ffffffff8151195e>] ? panic+0xc8/0x206 [ 2.120214] [<ffffffff819035a7>] ? mount_block_root+0x2a9/0x2b8 [ 2.126236] [<ffffffff811bae95>] ? SyS_mknod+0x185/0x210 [ 2.131624] [<ffffffff81903739>] ? prepare_namespace+0x133/0x169 [ 2.137709] [<ffffffff81903258>] ? kernel_init_freeable+0x1d7/0x1e1 [ 2.144071] [<ffffffff8190295e>] ? initcall_blacklist+0xb2/0xb2 [ 2.150077] [<ffffffff81507da0>] ? rest_init+0x80/0x80 [ 2.155318] [<ffffffff81507daa>] ? kernel_init+0xa/0xf0 [ 2.160622] [<ffffffff8151ad18>] ? ret_from_fork+0x58/0x90 [ 2.166185] [<ffffffff81507da0>] ? rest_init+0x80/0x80 [ 2.171458] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff) [ 2.181627] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ===
Hi!
On Mon, Mar 27, 2017 at 03:27:18PM +0000, Ковалёв Сергей wrote:
With job definition
<snip>
I get kernel panic:
[ 2.054605] No filesystem could mount root, tried: [ 2.059523] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) [ 2.067809] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 3.16.0-4-amd64 #1 Debian 3.16.39-1 [ 2.075902] Hardware name: /DQ87PG, BIOS PGQ8710H.86A.0152.2015.0709.1545 07/09/2015 [ 2.085123] 0000000000000000 ffffffff81514c11 ffffffff817054c8 ffff88040e1b7ea0 [ 2.092576] ffffffff8151195e ffff880400000010 ffff88040e1b7eb0 ffff88040e1b7e50 [ 2.100046] ffff88040e1b7ea0 ffff88040e1b7eb8 0000000000000012 0000000000000001 [ 2.107506] Call Trace: [ 2.109953] [<ffffffff81514c11>] ? dump_stack+0x5d/0x78 [ 2.115256] [<ffffffff8151195e>] ? panic+0xc8/0x206 [ 2.120214] [<ffffffff819035a7>] ? mount_block_root+0x2a9/0x2b8 [ 2.126236] [<ffffffff811bae95>] ? SyS_mknod+0x185/0x210 [ 2.131624] [<ffffffff81903739>] ? prepare_namespace+0x133/0x169 [ 2.137709] [<ffffffff81903258>] ? kernel_init_freeable+0x1d7/0x1e1 [ 2.144071] [<ffffffff8190295e>] ? initcall_blacklist+0xb2/0xb2 [ 2.150077] [<ffffffff81507da0>] ? rest_init+0x80/0x80 [ 2.155318] [<ffffffff81507daa>] ? kernel_init+0xa/0xf0 [ 2.160622] [<ffffffff8151ad18>] ? ret_from_fork+0x58/0x90 [ 2.166185] [<ffffffff81507da0>] ? rest_init+0x80/0x80 [ 2.171458] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff) [ 2.181627] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
Could you paste the complete log somewhere? It might help a lot to see exactly how things are booting, plus the complete command line the kernel is trying to use.
Cheers,
Hello Steve,
The log file in attachment (I hope this correct way to share it). The kernel and initrd were built via https://git.linaro.org/lava-team/refactoring.git/tree/scripts/x86_64-nfs.sh (as Neil suggested).
With best regards.
On Tue, Mar 28, 2017 at 02:42:17PM +0000, Ковалёв Сергей wrote:
Hello Steve,
The log file in attachment (I hope this correct way to share it). The
kernel and initrd were built via https://git.linaro.org/lava-team/refactoring.git/tree/scripts/x86_64-nfs.sh (as Neil suggested).
OK, thanks for the log. It helps a lot!
I can see that:
* the dispatcher has grabbed all the files needed for the job ok (up to 2017-03-24T19:02:51.915620) * the dispatcher has added the test overlay ok (up to 2017-03-24T19:03:22.487561) * the dispatcher got iPXE to boot the machine (up to 2017-03-24T19:06:52.065708) * the kernel boots fine, but then can't find root (as you've already seen). There's nothing on the kernel command line to tell it where root is, though:
Command line: vmlinuz ip=dhcp console=ttyS0,115200n8 lava_mac={LAVA_MAC}
Neil suggested earlier that "/dev/ram0 shouldn't be needed with modern kernels", but I don't think the 3.16 kernel you're using will work that way. I'd suggest adding "root=/dev/ram0" to your kernel command line and trying again.
Cheers,
Steve,
As I have already wrote that I have tried kernel 4.10.4 (definition and log in attachment). But I get “Failed to load file: rootfs.cpio.gz” from iPXE.
With best regards.
On Tue, Mar 28, 2017 at 03:51:00PM +0000, Ковалёв Сергей wrote:
Steve,
As I have already wrote that I have tried kernel 4.10.4 (definition and log in attachment). But I get “Failed to load file: rootfs.cpio.gz” from iPXE.
You're specifying "initrd=/rootfs.cpio" on the kernel command line too, you don't need to do that.
Cheers,
Without this kernel complains too: === [ 7.567998] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6 [ 7.575484] Please append a correct "root=" boot option; here are the available partitions: ===
With best regards
I have succeed by manually entering commands to iPXE with image build with script (https://git.linaro.org/lava-team/refactoring.git/tree/scripts/x86_64-nfs.sh): === iPXE> kernel http://192.168.0.1:8000/linaro/debian/vmlinuz http://192.168.0.1:8000/linaro/debian/vmlinuz... ok iPXE> initrd http://192.168.0.1:8000/linaro/debian/initramfs.cpio http://192.168.0.1:8000/linaro/debian/initramfs.cpio... ok iPXE> imgargs vmlinuz console=ttyS0,115200n8 console=ttyS0 initrd=/initramfs.cpio root=/dev/ram0 iPXE> imgstat vmlinuz : 3128032 bytes [EFI] [SELECTED] "console=ttyS0,115200n8 console=ttyS0 initrd=/initramfs.cpio root=/dev/ram0" initramfs.cpio : 45616128 bytes iPXE> boot ===
At the same time LAVA could not achieve this. Am I missing something?
On 29 March 2017 at 12:43, Ковалёв Сергей SKovalev@ptsecurity.com wrote:
I have succeed by manually entering commands to iPXE with image build with script (https://git.linaro.org/lava-team/refactoring.git/tree/scripts/x86_64-nfs.sh):
iPXE> kernel http://192.168.0.1:8000/linaro/debian/vmlinuz http://192.168.0.1:8000/linaro/debian/vmlinuz... ok iPXE> initrd http://192.168.0.1:8000/linaro/debian/initramfs.cpio http://192.168.0.1:8000/linaro/debian/initramfs.cpio... ok iPXE> imgargs vmlinuz console=ttyS0,115200n8 console=ttyS0 initrd=/initramfs.cpio root=/dev/ram0
If imgargs is a command available on your device-type, then you will need to change the device type jinja2 template to issue that command. The default x86 jinja2 template calls 'set extraargs' and 'set console'
https://staging.validation.linaro.org/scheduler/job/168865#L195
iPXE> imgstat vmlinuz : 3128032 bytes [EFI] [SELECTED] "console=ttyS0,115200n8 console=ttyS0 initrd=/initramfs.cpio root=/dev/ram0" initramfs.cpio : 45616128 bytes
Similarly with imgstat
iPXE> boot
At the same time LAVA could not achieve this. Am I missing something?
Please always provide the full log. Mailing lists handle attachments quite well. Selected lines from a log are commonly useless.
Hello Neil,
If imgargs is a command available on your device-type
I could do this without 'imgargs' and it succeed: === set console console=ttyS0,115200n8 lava_mac={LAVA_MAC} set extraargs initrd=/initramfs.cpio root=/dev/ram0 ip=dhcp kernel http://192.168.0.1:8000/linaro/debian/vmlinuz ${extraargs} ${console} initrd http://192.168.0.1:8000/linaro/debian/initramfs.cpio ===
Please always provide the full log.
That was answer to previous letter (with logs) so I didn't include logs again. This time I attach job definition and logs.
So the question is previous: what do you do to make it work or what I have missed?
Thank you very much.
With best regards Sergey Kovalev.
On 29 March 2017 at 13:53, Ковалёв Сергей SKovalev@ptsecurity.com wrote:
Hello Neil,
If imgargs is a command available on your device-type
I could do this without 'imgargs' and it succeed:
set console console=ttyS0,115200n8 lava_mac={LAVA_MAC} set extraargs initrd=/initramfs.cpio root=/dev/ram0 ip=dhcp kernel http://192.168.0.1:8000/linaro/debian/vmlinuz ${extraargs} ${console} initrd http://192.168.0.1:8000/linaro/debian/initramfs.cpio ===
This is not the same as how the lava log shows:
iPXE> [?25hkernel tftp://192.168.0.1/72/tftp-deploy-WDRuCl/kernel/vmlinuz ${extraargs} ${console}"
You've changed the protocol from tftp to http.
Your version of iPXE claims to support TFTP: Features: DNS HTTP iSCSI TFTP SRP AoE EFI Menu However, maybe that isn't working fully.
You're also testing with the initramfs.cpio not the compressed initramfs.cpio.gz
Try putting the files into the tftp location manually and use that path. Also try with the compressed initramfs.
Then, leave the device running and see if it has actually booted but just not sent any output to the console.
Please always provide the full log.
That was answer to previous letter (with logs) so I didn't include logs again. This time I attach job definition and logs.
So the question is previous: what do you do to make it work or what I have missed?
This looks like a difference in how your device behaves compared to the x86 machines we use. Using iPXE means that you are bypassing the UEFI, maybe that could be part of the problem and you'll need to change the boot method from ipxe to uefi-menu with a UEFI menu item which loads Grub etc.
If TFTP doesn't work and http does, you could change the jinja2 template commands to use http but you will have to retain the same directory structure as TFTP for the files to be found by your device.
You need to work out which commands can work and how to adapt the jinja2 template to use those commands.
Hello Neil,
Thank you very much - at last I have a success!
You're also testing with the initramfs.cpio not the compressed initramfs.cpio.gz
That was really the issue. I didn't notice that LAVA put down into iPXE exactly `tftp .../ initramfs.cpio.gz`.
But I have to notice that I really need "extra_kernel_args: initrd=/ramdisk.cpio.gz root=/dev/ram0".
I attach my job definition and full log. Image to load have been generated on my Debian machine with script https://git.linaro.org/lava-team/refactoring.git/tree/scripts/x86_64-nfs.sh.
So I very appreciate you help and comments from Steve McIntyre.
With best regards Sergey Kovalev.
-
Neil Williams -
Steve McIntyre -
Ковалёв Сергей