All qemu's mount rootfs failed on Linux next-20230206 tag due to the following kernel crash.
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Crash log: --------- <3>[ 3.257960] /dev/root: Can't open blockdev <4>[ 3.258940] VFS: Cannot open root device "/dev/sda" or unknown-block(8,0): error -16 <4>[ 3.259704] Please append a correct "root=" boot option; here are the available partitions: <4>[ 3.261088] 0800 2500336 sda <4>[ 3.261186] driver: sd <4>[ 3.262260] 0b00 1048575 sr0 <4>[ 3.262409] driver: sr <3>[ 3.263022] List of all bdev filesystems: <3>[ 3.263553] ext3 <3>[ 3.263708] ext2 <3>[ 3.263994] ext4 <3>[ 3.264160] vfat <3>[ 3.264419] msdos <3>[ 3.264589] iso9660 <3>[ 3.264773] <0>[ 3.265665] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,0) <4>[ 3.266991] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.8.0-rc3-next-20240206 #1 <4>[ 3.267593] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 <4>[ 3.268937] Call Trace: <4>[ 3.269316] <TASK> <4>[ 3.270113] dump_stack_lvl+0x71/0xb0 <4>[ 3.271837] dump_stack+0x14/0x20 <4>[ 3.272128] panic+0x12f/0x2f0 <4>[ 3.272812] ? _printk+0x5d/0x80 <4>[ 3.273097] mount_root_generic+0x26e/0x2b0 <4>[ 3.273941] mount_block_root+0x3f/0x50 <4>[ 3.274212] mount_root+0x60/0x80 <4>[ 3.274610] prepare_namespace+0x7a/0xb0 <4>[ 3.276008] kernel_init_freeable+0x137/0x180 <4>[ 3.276285] ? __pfx_kernel_init+0x10/0x10 <4>[ 3.276563] kernel_init+0x1e/0x1a0 <4>[ 3.276837] ret_from_fork+0x45/0x50 <4>[ 3.277319] ? __pfx_kernel_init+0x10/0x10 <4>[ 3.278176] ret_from_fork_asm+0x1a/0x30 <4>[ 3.278560] </TASK> <0>[ 3.280750] Kernel Offset: 0x1a800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) <0>[ 3.281985] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,0) ]---
Links: - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240206/tes... - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240206/tes...
-- Linaro LKFT https://lkft.linaro.org
On Tue 06-02-24 14:41:17, Naresh Kamboju wrote:
All qemu's mount rootfs failed on Linux next-20230206 tag due to the following kernel crash.
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Crash log:
<3>[ 3.257960] /dev/root: Can't open blockdev <4>[ 3.258940] VFS: Cannot open root device "/dev/sda" or unknown-block(8,0): error -16
Uhuh, -16 is EBUSY so it seems Christian's block device opening changes are suspect? Do you have some sample kconfig available somewhere?
Honza
<4>[ 3.259704] Please append a correct "root=" boot option; here are the available partitions: <4>[ 3.261088] 0800 2500336 sda <4>[ 3.261186] driver: sd <4>[ 3.262260] 0b00 1048575 sr0 <4>[ 3.262409] driver: sr <3>[ 3.263022] List of all bdev filesystems: <3>[ 3.263553] ext3 <3>[ 3.263708] ext2 <3>[ 3.263994] ext4 <3>[ 3.264160] vfat <3>[ 3.264419] msdos <3>[ 3.264589] iso9660 <3>[ 3.264773] <0>[ 3.265665] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,0) <4>[ 3.266991] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.8.0-rc3-next-20240206 #1 <4>[ 3.267593] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 <4>[ 3.268937] Call Trace: <4>[ 3.269316] <TASK> <4>[ 3.270113] dump_stack_lvl+0x71/0xb0 <4>[ 3.271837] dump_stack+0x14/0x20 <4>[ 3.272128] panic+0x12f/0x2f0 <4>[ 3.272812] ? _printk+0x5d/0x80 <4>[ 3.273097] mount_root_generic+0x26e/0x2b0 <4>[ 3.273941] mount_block_root+0x3f/0x50 <4>[ 3.274212] mount_root+0x60/0x80 <4>[ 3.274610] prepare_namespace+0x7a/0xb0 <4>[ 3.276008] kernel_init_freeable+0x137/0x180 <4>[ 3.276285] ? __pfx_kernel_init+0x10/0x10 <4>[ 3.276563] kernel_init+0x1e/0x1a0 <4>[ 3.276837] ret_from_fork+0x45/0x50 <4>[ 3.277319] ? __pfx_kernel_init+0x10/0x10 <4>[ 3.278176] ret_from_fork_asm+0x1a/0x30 <4>[ 3.278560] </TASK> <0>[ 3.280750] Kernel Offset: 0x1a800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) <0>[ 3.281985] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,0) ]---
Links:
- https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240206/tes...
- https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240206/tes...
-- Linaro LKFT https://lkft.linaro.org
On Tue, 6 Feb 2024 at 15:45, Jan Kara jack@suse.cz wrote:
On Tue 06-02-24 14:41:17, Naresh Kamboju wrote:
All qemu's mount rootfs failed on Linux next-20230206 tag due to the following kernel crash.
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Crash log:
<3>[ 3.257960] /dev/root: Can't open blockdev <4>[ 3.258940] VFS: Cannot open root device "/dev/sda" or unknown-block(8,0): error -16
Uhuh, -16 is EBUSY so it seems Christian's block device opening changes are suspect? Do you have some sample kconfig available somewhere?
All build information is in this url, https://storage.tuxsuite.com/public/linaro/lkft/builds/2byqguFVp7MYAEjKo6nJG...
- Naresh
On Tue 06-02-24 15:53:34, Naresh Kamboju wrote:
On Tue, 6 Feb 2024 at 15:45, Jan Kara jack@suse.cz wrote:
On Tue 06-02-24 14:41:17, Naresh Kamboju wrote:
All qemu's mount rootfs failed on Linux next-20230206 tag due to the following kernel crash.
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Crash log:
<3>[ 3.257960] /dev/root: Can't open blockdev <4>[ 3.258940] VFS: Cannot open root device "/dev/sda" or unknown-block(8,0): error -16
Uhuh, -16 is EBUSY so it seems Christian's block device opening changes are suspect? Do you have some sample kconfig available somewhere?
All build information is in this url, https://storage.tuxsuite.com/public/linaro/lkft/builds/2byqguFVp7MYAEjKo6nJG...
Thanks! So for record the config has:
CONFIG_BLK_DEV_WRITE_MOUNTED=y
So we are not hitting any weird corner case with blocking writes to mounted filesystems. It must be something else.
Honza
On Tue, 6 Feb 2024 at 17:58, Jan Kara jack@suse.cz wrote:
On Tue 06-02-24 15:53:34, Naresh Kamboju wrote:
On Tue, 6 Feb 2024 at 15:45, Jan Kara jack@suse.cz wrote:
On Tue 06-02-24 14:41:17, Naresh Kamboju wrote:
All qemu's mount rootfs failed on Linux next-20230206 tag due to the following kernel crash.
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Crash log:
<3>[ 3.257960] /dev/root: Can't open blockdev <4>[ 3.258940] VFS: Cannot open root device "/dev/sda" or unknown-block(8,0): error -16
Uhuh, -16 is EBUSY so it seems Christian's block device opening changes are suspect? Do you have some sample kconfig available somewhere?
All build information is in this url, https://storage.tuxsuite.com/public/linaro/lkft/builds/2byqguFVp7MYAEjKo6nJG...
Thanks! So for record the config has:
CONFIG_BLK_DEV_WRITE_MOUNTED=y
So we are not hitting any weird corner case with blocking writes to mounted filesystems. It must be something else.
As per Anders bisection results the first bad commit pointing to ba858e55b205 ("bdev: open block device as files")
- Naresh
Hi Christian,
On 06.02.2024 16:53, Christian Brauner wrote:
On it.
Ok, can you try: git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs.super.debug please?
I've also encountered this issue during my linux-next daily tests and I confirm that the above branch works fine.
I've applied the diff between e3bfad989976^2 and the above branch (bc7cb6c829e2), which looks following:
diff --git a/init/do_mounts.c b/init/do_mounts.c index 279ad28bf4fb..d8ea839463a5 100644 --- a/init/do_mounts.c +++ b/init/do_mounts.c @@ -19,6 +19,7 @@ #include <linux/ramfs.h> #include <linux/shmem_fs.h> #include <linux/ktime.h> +#include <linux/task_work.h>
#include <linux/nfs_fs.h> #include <linux/nfs_fs_sb.h> @@ -208,6 +209,10 @@ void __init mount_root_generic(char *name, char *pretty_name, int flags) goto out; case -EACCES: case -EINVAL: +#ifdef CONFIG_BLOCK + flush_delayed_fput(); + task_work_run(); +#endif continue; } /*
onto next-20240206 and it fixed all boot problems I've observed on my test farm. :)
Tested-by: Marek Szyprowski m.szyprowski@samsung.com
Best regards
On 2/7/2024 4:35 AM, Marek Szyprowski wrote:
Hi Christian,
On 06.02.2024 16:53, Christian Brauner wrote:
On it.
Ok, can you try: git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs.super.debug please?
I've also encountered this issue during my linux-next daily tests and I confirm that the above branch works fine.
I've applied the diff between e3bfad989976^2 and the above branch (bc7cb6c829e2), which looks following:
diff --git a/init/do_mounts.c b/init/do_mounts.c index 279ad28bf4fb..d8ea839463a5 100644 --- a/init/do_mounts.c +++ b/init/do_mounts.c @@ -19,6 +19,7 @@ #include <linux/ramfs.h> #include <linux/shmem_fs.h> #include <linux/ktime.h> +#include <linux/task_work.h>
#include <linux/nfs_fs.h> #include <linux/nfs_fs_sb.h> @@ -208,6 +209,10 @@ void __init mount_root_generic(char *name, char *pretty_name, int flags) goto out; case -EACCES: case -EINVAL: +#ifdef CONFIG_BLOCK + flush_delayed_fput(); + task_work_run(); +#endif continue; } /*
onto next-20240206 and it fixed all boot problems I've observed on my test farm. :)
Tested-by: Marek Szyprowski m.szyprowski@samsung.com
Best regards
We as well encountered this issue during our CI run. The given branch fixes the issues seen in our run.
Tested-by: Srikanth Aithal sraithal@amd.com
-
Aithal, Srikanth -
Christian Brauner -
Jan Kara -
Marek Szyprowski -
Naresh Kamboju