Thanks for the reply. I was able to reproduce it a couple of weeks ago and I have already submitted a fix.
- Russ
On 8/31/22 22:43, Yujie Liu wrote:
Hi Russ,
On 8/2/2022 04:42, Russ Weight wrote:
Oliver,
On 7/29/22 00:08, kernel test robot wrote:
Greeting,
FYI, we noticed the following commit (built with gcc-11):
commit: a37ddddd86037c896c702b4df416bc4e51b2a5a0 ("selftests: firmware: Add firmware upload selftests") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: kernel-selftests version: kernel-selftests-x86_64-4cb0bec3-1_20220724 with following parameters:
group: firmware ucode: 0xec
test-description: The kernel contains a set of "self tests" under the tools/testing/selftests/ directory. These are intended to be small unit tests to exercise individual code paths in the kernel. test-url: https://www.kernel.org/doc/Documentation/kselftest.txt
on test machine: 8 threads Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz with 28G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
If you fix the issue, kindly add following tag Reported-by: kernel test robot oliver.sang@intel.com
[ 103.520572][ T2443] BUG: KASAN: use-after-free in firmware_upload_unregister (drivers/base/firmware_loader/sysfs_upload.c:395) [ 103.520579][ T2443] Read of size 8 at addr ffff8881e186c808 by task fw_upload.sh/2443 [ 103.528481][ T395] [ 103.534696][ T2443] [ 103.534698][ T2443] CPU: 7 PID: 2443 Comm: fw_upload.sh Not tainted 5.18.0-rc2-00036-ga37ddddd8603 #1 [ 103.534701][ T2443] Hardware name: Dell Inc. OptiPlex 7040/0Y7WYT, BIOS 1.2.8 01/26/2016 [ 103.534703][ T2443] Call Trace: [ 103.534705][ T2443] <TASK> [ 103.534707][ T2443] ? firmware_upload_unregister (drivers/base/firmware_loader/sysfs_upload.c:395)
I believe I understand the problem, but I have been unable to reproduce the error to verify the fix:
394 device_unregister(&fw_sysfs->dev); 395 module_put(fw_upload_priv->module);
The device_unregister() call could result in the dev_release function freeing the fw_upload_priv structure before it is dereferenced on line 395. Copying fw_upload_priv->module to a local variable for use when calling device_unregister() should fix the problem.
[ 103.534713][ T2443] dump_stack_lvl (lib/dump_stack.c:107 (discriminator 4)) [ 103.588011][ T2443] print_address_description+0x1f/0x200 [ 103.594406][ T2443] ? firmware_upload_unregister (drivers/base/firmware_loader/sysfs_upload.c:395) [ 103.600112][ T2443] print_report.cold (mm/kasan/report.c:430) [ 103.604782][ T2443] ? do_raw_spin_lock (arch/x86/include/asm/atomic.h:202 include/linux/atomic/atomic-instrumented.h:543 include/asm-generic/qspinlock.h:82 kernel/locking/spinlock_debug.c:115) [ 103.609624][ T2443] kasan_report (mm/kasan/report.c:162 mm/kasan/report.c:493) [ 103.613861][ T2443] ? firmware_upload_unregister (drivers/base/firmware_loader/sysfs_upload.c:395) [ 103.619561][ T2443] firmware_upload_unregister (drivers/base/firmware_loader/sysfs_upload.c:395) [ 103.625091][ T2443] upload_unregister_store (lib/test_firmware.c:1060 lib/test_firmware.c:1321) [ 103.630377][ T2443] ? sysfs_file_ops (fs/sysfs/file.c:129) [ 103.635046][ T2443] kernfs_fop_write_iter (fs/kernfs/file.c:294) [ 103.640145][ T2443] new_sync_write (fs/read_write.c:505 (discriminator 1)) [ 103.644642][ T2443] ? new_sync_read (fs/read_write.c:494) [ 103.649225][ T2443] ? ksys_write (fs/read_write.c:644) [ 103.653463][ T2443] ? rcu_read_unlock (include/linux/rcupdate.h:723 (discriminator 5)) [ 103.658057][ T2443] ? lock_is_held_type (kernel/locking/lockdep.c:5382 kernel/locking/lockdep.c:5684) [ 103.662909][ T2443] vfs_write (fs/read_write.c:591) [ 103.666984][ T2443] ksys_write (fs/read_write.c:644) [ 103.671057][ T2443] ? __ia32_sys_read (fs/read_write.c:634) [ 103.675645][ T2443] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4501) [ 103.682051][ T2443] ? syscall_enter_from_user_mode (arch/x86/include/asm/irqflags.h:45 arch/x86/include/asm/irqflags.h:80 kernel/entry/common.c:109) [ 103.687756][ T2443] do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80) [ 103.692010][ T2443] ? pick_file (fs/file.c:660) [ 103.696165][ T2443] ? do_raw_spin_unlock (arch/x86/include/asm/atomic.h:29 include/linux/atomic/atomic-instrumented.h:28 include/asm-generic/qspinlock.h:28 kernel/locking/spinlock_debug.c:100 kernel/locking/spinlock_debug.c:140) [ 103.701094][ T2443] ? syscall_exit_to_user_mode (kernel/entry/common.c:129 kernel/entry/common.c:296) [ 103.706539][ T2443] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4501) [ 103.712929][ T2443] ? do_syscall_64 (arch/x86/entry/common.c:87) [ 103.717343][ T2443] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4501) [ 103.723747][ T2443] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:115) [ 103.729451][ T2443] RIP: 0033:0x7f1020308f33 [ 103.733709][ T2443] Code: 8b 15 61 ef 0c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 48 83 ec 28 48 89 54 24 18 All code ======== 0: 8b 15 61 ef 0c 00 mov 0xcef61(%rip),%edx # 0xcef67 6: f7 d8 neg %eax 8: 64 89 02 mov %eax,%fs:(%rdx) b: 48 c7 c0 ff ff ff ff mov $0xffffffffffffffff,%rax 12: eb b7 jmp 0xffffffffffffffcb 14: 0f 1f 00 nopl (%rax) 17: 64 8b 04 25 18 00 00 mov %fs:0x18,%eax 1e: 00 1f: 85 c0 test %eax,%eax 21: 75 14 jne 0x37 23: b8 01 00 00 00 mov $0x1,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 55 ja 0x87 32: c3 retq 33: 0f 1f 40 00 nopl 0x0(%rax) 37: 48 83 ec 28 sub $0x28,%rsp 3b: 48 89 54 24 18 mov %rdx,0x18(%rsp)
Code starting with the faulting instruction
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 55 ja 0x5d 8: c3 retq 9: 0f 1f 40 00 nopl 0x0(%rax) d: 48 83 ec 28 sub $0x28,%rsp 11: 48 89 54 24 18 mov %rdx,0x18(%rsp) [ 103.753040][ T2443] RSP: 002b:00007fffe4075988 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 103.761244][ T2443] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f1020308f33 [ 103.769013][ T2443] RDX: 0000000000000003 RSI: 00005582df845b80 RDI: 0000000000000001 [ 103.776791][ T2443] RBP: 00005582df845b80 R08: 00000000ffffffff R09: 0000000000000003 [ 103.784561][ T2443] R10: 00005582df833c80 R11: 0000000000000246 R12: 0000000000000003 [ 103.792328][ T2443] R13: 00007f10203d96a0 R14: 0000000000000003 R15: 00007f10203d98a0 [ 103.800100][ T2443] </TASK> [ 103.802957][ T2443] [ 103.805125][ T2443] Allocated by task 2443: [ 103.809276][ T2443] kasan_save_stack (mm/kasan/common.c:39) [ 103.813781][ T2443] __kasan_kmalloc (mm/kasan/common.c:45 mm/kasan/common.c:436 mm/kasan/common.c:515 mm/kasan/common.c:524) [ 103.818190][ T2443] firmware_upload_register (drivers/base/firmware_loader/sysfs_upload.c:160) [ 103.824150][ T2443] upload_register_store (lib/test_firmware.c:1279) [ 103.829250][ T2443] kernfs_fop_write_iter (fs/kernfs/file.c:294) [ 103.834350][ T2443] new_sync_write (fs/read_write.c:505 (discriminator 1)) [ 103.838846][ T2443] vfs_write (fs/read_write.c:591) [ 103.842910][ T2443] ksys_write (fs/read_write.c:644) [ 103.846975][ T2443] do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80) [ 103.851217][ T2443] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:115) [ 103.856932][ T2443] [ 103.859100][ T2443] Freed by task 2443: [ 103.862907][ T2443] kasan_save_stack (mm/kasan/common.c:39) [ 103.867415][ T2443] kasan_set_track (mm/kasan/common.c:45) [ 103.871822][ T2443] kasan_set_free_info (mm/kasan/generic.c:372) [ 103.876579][ T2443] __kasan_slab_free (mm/kasan/common.c:368 mm/kasan/common.c:328 mm/kasan/common.c:374) [ 103.881331][ T2443] slab_free_freelist_hook (mm/slub.c:1754) [ 103.886517][ T2443] kfree (mm/slub.c:3510 mm/slub.c:4552) [ 103.890156][ T2443] fw_dev_release (drivers/base/firmware_loader/sysfs.c:102) [ 103.894483][ T2443] device_release (drivers/base/core.c:2235) [ 103.898902][ T2443] kobject_cleanup (lib/kobject.c:677) [ 103.903492][ T2443] firmware_upload_unregister (drivers/base/firmware_loader/sysfs_upload.c:395) [ 103.909034][ T2443] upload_unregister_store (lib/test_firmware.c:1060 lib/test_firmware.c:1321) [ 103.914311][ T2443] kernfs_fop_write_iter (fs/kernfs/file.c:294) [ 103.919429][ T2443] new_sync_write (fs/read_write.c:505 (discriminator 1)) [ 103.923927][ T2443] vfs_write (fs/read_write.c:591) [ 103.927990][ T2443] ksys_write (fs/read_write.c:644) [ 103.932054][ T2443] do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80) [ 103.936290][ T2443] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:115) [ 103.941992][ T2443] [ 103.944159][ T2443] Last potentially related work creation: [ 103.949704][ T2443] kasan_save_stack (mm/kasan/common.c:39) [ 103.954216][ T2443] __kasan_record_aux_stack (mm/kasan/generic.c:348) [ 103.959400][ T2443] insert_work (include/linux/instrumented.h:71 include/asm-generic/bitops/instrumented-non-atomic.h:134 kernel/workqueue.c:635 kernel/workqueue.c:642 kernel/workqueue.c:1361) [ 103.963552][ T2443] __queue_work (kernel/workqueue.c:1520) [ 103.967888][ T2443] queue_work_on (kernel/workqueue.c:1546) [ 103.972141][ T2443] fw_upload_start (drivers/base/firmware_loader/sysfs_upload.c:263) [ 103.976723][ T2443] firmware_loading_store (drivers/base/firmware_loader/sysfs.c:213) [ 103.981910][ T2443] kernfs_fop_write_iter (fs/kernfs/file.c:294) [ 103.987022][ T2443] new_sync_write (fs/read_write.c:505 (discriminator 1)) [ 103.991537][ T2443] vfs_write (fs/read_write.c:591) [ 103.995604][ T2443] ksys_write (fs/read_write.c:644) [ 103.999673][ T2443] do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80) [ 104.003930][ T2443] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:115) [ 104.009631][ T2443] [ 104.011800][ T2443] Second to last potentially related work creation: [ 104.018219][ T2443] kasan_save_stack (mm/kasan/common.c:39) [ 104.022727][ T2443] __kasan_record_aux_stack (mm/kasan/generic.c:348) [ 104.027938][ T2443] insert_work (include/linux/instrumented.h:71 include/asm-generic/bitops/instrumented-non-atomic.h:134 kernel/workqueue.c:635 kernel/workqueue.c:642 kernel/workqueue.c:1361) [ 104.032101][ T2443] __queue_work (kernel/workqueue.c:1520) [ 104.036423][ T2443] queue_work_on (kernel/workqueue.c:1546) [ 104.040658][ T2443] fw_upload_start (drivers/base/firmware_loader/sysfs_upload.c:263) [ 104.045240][ T2443] firmware_loading_store (drivers/base/firmware_loader/sysfs.c:213) [ 104.050423][ T2443] kernfs_fop_write_iter (fs/kernfs/file.c:294) [ 104.055522][ T2443] new_sync_write (fs/read_write.c:505 (discriminator 1)) [ 104.060016][ T2443] vfs_write (fs/read_write.c:591) [ 104.064081][ T2443] ksys_write (fs/read_write.c:644) [ 104.068144][ T2443] do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80) [ 104.072381][ T2443] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:115) [ 104.078083][ T2443] [ 104.080249][ T2443] The buggy address belongs to the object at ffff8881e186c800 [ 104.080249][ T2443] which belongs to the cache kmalloc-512 of size 512 [ 104.094049][ T2443] The buggy address is located 8 bytes inside of [ 104.094049][ T2443] 512-byte region [ffff8881e186c800, ffff8881e186ca00) [ 104.106914][ T2443] [ 104.109084][ T2443] The buggy address belongs to the physical page: [ 104.115315][ T2443] page:0000000037a5888d refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x1e1868 [ 104.125336][ T2443] head:0000000037a5888d order:3 compound_mapcount:0 compound_pincount:0 [ 104.133454][ T2443] flags: 0x17ffffc0010200(slab|head|node=0|zone=2|lastcpupid=0x1fffff) [ 104.141498][ T2443] raw: 0017ffffc0010200 ffffea0005491e00 dead000000000002 ffff888100042c80 [ 104.149885][ T2443] raw: 0000000000000000 0000000000200020 00000001ffffffff 0000000000000000 [ 104.158274][ T2443] page dumped because: kasan: bad access detected [ 104.164492][ T2443]
To reproduce:
git clone https://github.com/intel/lkp-tests.git cd lkp-tests sudo bin/lkp install job.yaml # job file is attached in this email
I have tried these steps on Fedora35 and on CentOS Stream. In both cases I have missing packages that I have not yet resolved:
Error: Unable to find a match: arping lib32gcc-dev libc6-dev-i386 libc6-i386 libc6-x32 libhugetlbfs-dev libmnl-dev libmount-dev libpci3 libreadline-dev libx32asan5 libx32atomic1 libx32gcc1 libx32gcc-dev libx32gomp1 libx32itm1 libx32quadmath0 libx32ubsan1 linux-libc-dev-amd64-cross netcat-openbsd openvswitch-common openvswitch-switch sendip libpci-dev
Simply running the fw_upload selftests in a loop is not sufficient to trigger the error. Can you provide additional instructions for running the lkp tests manually? Do I need a specific OS? How can I access the missing packages?
We can reproduce this error on bare metal by simply running fw_upload.sh.
~/linux/tools/testing/selftests/firmware# ./fw_upload.sh ./fw_upload.sh: firmware upload cancellation works ./fw_upload.sh: firmware upload error handling works ./fw_upload.sh: oversized firmware error handling works ./fw_upload.sh: firmware upload for fw1 works ./fw_upload.sh: firmware upload for fw2 works ./fw_upload.sh: firmware upload for fw3 works
Message from syslogd@debian-x8664 at Sep 1 05:06:54 ... kernel:[ 1090.872590][ T1293] Kernel panic - not syncing: Fatal exception
dmesg read from serial:
[ 1089.654274][ T1293] ================================================================== [ 1089.662192][ T1293] BUG: KASAN: use-after-free in firmware_upload_unregister+0x16e/0x1c0 [ 1089.670282][ T1293] Read of size 8 at addr ffff88873a872008 by task fw_upload.sh/1293 [ 1089.678107][ T1293] [ 1089.680291][ T1293] CPU: 4 PID: 1293 Comm: fw_upload.sh Not tainted 5.18.0-rc2-00036-ga37ddddd8603 #1 [ 1089.689527][ T1293] Hardware name: Dell Inc. OptiPlex 7040/0Y7WYT, BIOS 1.2.8 01/26/2016 [ 1089.697612][ T1293] Call Trace: [ 1089.700751][ T1293] <TASK> [ 1089.703557][ T1293] ? firmware_upload_unregister+0x16e/0x1c0 [ 1089.709313][ T1293] dump_stack_lvl+0x45/0x59 [ 1089.713669][ T1293] print_address_description.constprop.0+0x1f/0x200 [ 1089.720107][ T1293] ? firmware_upload_unregister+0x16e/0x1c0 [ 1089.725851][ T1293] print_report.cold+0x55/0x22c [ 1089.730556][ T1293] ? do_raw_spin_lock+0x12e/0x280 [ 1089.735432][ T1293] kasan_report+0xbe/0x1c0
Could you please help check if the .config file used to compile the kernel matches the one we attached in the original report? Here we attach it again for your reference.
About the issue of missing packages during setting up lkp test environment, we wish to support various OS and distributions, but sometimes our package dependencies are not updated in time, sorry for the inconvenience. We use debian OS in our test environment, and it can install the required packages successfully. We will fix the package issue on other OS soon. For this case, we still recommend to run fw_upload.sh to trigger the error, because it's much easier than setting up lkp tests environment.
-- Thanks, Yujie
Thanks,
- Russ
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run sudo bin/lkp run generated-yaml-file
# if come across any failure that blocks the test, # please remove ~/.lkp and /lkp dir to run from a clean state.