Hi, all!
After running tools/testing/selftests/net/tun, there seems to be some kind of hang in test "FAIL tun.reattach_delete_close" or "FAIL tun.reattach_close_delete".
Two tests exit by timeout, but the processes left are unkillable, even with kill -9 PID:
[root@pc-mtodorov linux_torvalds]# ps -ef | grep tun root 1140 1 0 12:16 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned root 1333 1 0 12:16 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P root 3930 2309 0 12:20 pts/1 00:00:00 tools/testing/selftests/net/tun root 3952 2309 0 12:21 pts/1 00:00:00 tools/testing/selftests/net/tun root 4056 3765 0 12:25 pts/1 00:00:00 grep --color=auto tun [root@pc-mtodorov linux_torvalds]# kill -9 3930 3952 [root@pc-mtodorov linux_torvalds]# ps -ef | grep tun root 1140 1 0 12:16 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned root 1333 1 0 12:16 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P root 3930 2309 0 12:20 pts/1 00:00:00 tools/testing/selftests/net/tun root 3952 2309 0 12:21 pts/1 00:00:00 tools/testing/selftests/net/tun root 4060 3765 0 12:25 pts/1 00:00:00 grep --color=auto tun [root@pc-mtodorov linux_torvalds]#
The kernel seems to be stuck in some loop, and filling the log with the following messages until reboot, where it is also waiting very long on the situation to timeout, which apparently never happens.
Mar 14 11:54:09 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 11:54:19 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 11:54:29 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 11:54:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 11:54:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
The platform is kernel 6.3.0-rc2 on AlmaLinux 8.7 and a LENOVO_MT_10TX_BU_Lenovo_FM_V530S-07ICB (lshw output attached).
The .config is here:
https://domac.alu.hr/~mtodorov/linux/selftests/net-tun/config-6.3.0-rc2-mg-a...
Basically, it is a vanilla Torvalds tree kernel with MGLRU, KMEMLEAK, and CONFIG_DEBUG_KOBJECT enabled. And devres patch.
Please find the strace of the net/tun run attached.
I am available for additional diagnostics.
Hope this helps.
Best regards, Mirsad
On 3/14/23 12:45, Mirsad Todorovac wrote:
Hi, all!
After running tools/testing/selftests/net/tun, there seems to be some kind of hang in test "FAIL tun.reattach_delete_close" or "FAIL tun.reattach_close_delete".
Two tests exit by timeout, but the processes left are unkillable, even with kill -9 PID:
[root@pc-mtodorov linux_torvalds]# ps -ef | grep tun root 1140 1 0 12:16 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned root 1333 1 0 12:16 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P root 3930 2309 0 12:20 pts/1 00:00:00 tools/testing/selftests/net/tun root 3952 2309 0 12:21 pts/1 00:00:00 tools/testing/selftests/net/tun root 4056 3765 0 12:25 pts/1 00:00:00 grep --color=auto tun [root@pc-mtodorov linux_torvalds]# kill -9 3930 3952 [root@pc-mtodorov linux_torvalds]# ps -ef | grep tun root 1140 1 0 12:16 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned root 1333 1 0 12:16 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P root 3930 2309 0 12:20 pts/1 00:00:00 tools/testing/selftests/net/tun root 3952 2309 0 12:21 pts/1 00:00:00 tools/testing/selftests/net/tun root 4060 3765 0 12:25 pts/1 00:00:00 grep --color=auto tun [root@pc-mtodorov linux_torvalds]#
The kernel seems to be stuck in some loop, and filling the log with the following messages until reboot, where it is also waiting very long on the situation to timeout, which apparently never happens.
Mar 14 11:54:09 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 11:54:19 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 11:54:29 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 11:54:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 11:54:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
The platform is kernel 6.3.0-rc2 on AlmaLinux 8.7 and a LENOVO_MT_10TX_BU_Lenovo_FM_V530S-07ICB (lshw output attached).
The .config is here:
https://domac.alu.hr/~mtodorov/linux/selftests/net-tun/config-6.3.0-rc2-mg-a...
Basically, it is a vanilla Torvalds tree kernel with MGLRU, KMEMLEAK, and CONFIG_DEBUG_KOBJECT enabled. And devres patch.
Please find the strace of the net/tun run attached.
I am available for additional diagnostics.
Hi, again!
I've been busy while waiting for reply, so I wondered how would a vanilla kernel go through the test, considering that I've been testing a number of patches lately.
I did a fresh git clone from repo and woa.
Surprisingly, the test with CONFIG_DEBUG_KOBJECT turned off passes:
[root@pc-mtodorov linux_torvalds]# tools/testing/selftests/net/tun TAP version 13 1..5 # Starting 5 tests from 1 test cases. # RUN tun.delete_detach_close ... # OK tun.delete_detach_close ok 1 tun.delete_detach_close # RUN tun.detach_delete_close ... # OK tun.detach_delete_close ok 2 tun.detach_delete_close # RUN tun.detach_close_delete ... # OK tun.detach_close_delete ok 3 tun.detach_close_delete # RUN tun.reattach_delete_close ... # OK tun.reattach_delete_close ok 4 tun.reattach_delete_close # RUN tun.reattach_close_delete ... # OK tun.reattach_close_delete ok 5 tun.reattach_close_delete # PASSED: 5 / 5 tests passed. # Totals: pass:5 fail:0 xfail:0 xpass:0 skip:0 error:0 [root@pc-mtodorov linux_torvalds]#
So, no hanging processes that cannot be killed now.
If you think it is worthy to explore the lockup that occurs when turning CONFIG_DEBUG_KOBJECT=y, I will rebuild once again with these turned on, to clear any doubts.
Until later.
Best regards, Mirsad
On 3/14/23 14:52, Mirsad Todorovac wrote:
On 3/14/23 12:45, Mirsad Todorovac wrote:
Hi, all!
After running tools/testing/selftests/net/tun, there seems to be some kind of hang in test "FAIL tun.reattach_delete_close" or "FAIL tun.reattach_close_delete".
Two tests exit by timeout, but the processes left are unkillable, even with kill -9 PID:
[root@pc-mtodorov linux_torvalds]# ps -ef | grep tun root 1140 1 0 12:16 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned root 1333 1 0 12:16 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P root 3930 2309 0 12:20 pts/1 00:00:00 tools/testing/selftests/net/tun root 3952 2309 0 12:21 pts/1 00:00:00 tools/testing/selftests/net/tun root 4056 3765 0 12:25 pts/1 00:00:00 grep --color=auto tun [root@pc-mtodorov linux_torvalds]# kill -9 3930 3952 [root@pc-mtodorov linux_torvalds]# ps -ef | grep tun root 1140 1 0 12:16 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned root 1333 1 0 12:16 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P root 3930 2309 0 12:20 pts/1 00:00:00 tools/testing/selftests/net/tun root 3952 2309 0 12:21 pts/1 00:00:00 tools/testing/selftests/net/tun root 4060 3765 0 12:25 pts/1 00:00:00 grep --color=auto tun [root@pc-mtodorov linux_torvalds]#
The kernel seems to be stuck in some loop, and filling the log with the following messages until reboot, where it is also waiting very long on the situation to timeout, which apparently never happens.
Mar 14 11:54:09 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 11:54:19 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 11:54:29 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 11:54:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 11:54:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
The platform is kernel 6.3.0-rc2 on AlmaLinux 8.7 and a LENOVO_MT_10TX_BU_Lenovo_FM_V530S-07ICB (lshw output attached).
The .config is here:
https://domac.alu.hr/~mtodorov/linux/selftests/net-tun/config-6.3.0-rc2-mg-a...
Basically, it is a vanilla Torvalds tree kernel with MGLRU, KMEMLEAK, and CONFIG_DEBUG_KOBJECT enabled. And devres patch.
Please find the strace of the net/tun run attached.
I am available for additional diagnostics.
Hi, again!
I've been busy while waiting for reply, so I wondered how would a vanilla kernel go through the test, considering that I've been testing a number of patches lately.
I did a fresh git clone from repo and woa.
Surprisingly, the test with CONFIG_DEBUG_KOBJECT turned off passes:
[root@pc-mtodorov linux_torvalds]# tools/testing/selftests/net/tun TAP version 13 1..5 # Starting 5 tests from 1 test cases. # RUN tun.delete_detach_close ... # OK tun.delete_detach_close ok 1 tun.delete_detach_close # RUN tun.detach_delete_close ... # OK tun.detach_delete_close ok 2 tun.detach_delete_close # RUN tun.detach_close_delete ... # OK tun.detach_close_delete ok 3 tun.detach_close_delete # RUN tun.reattach_delete_close ... # OK tun.reattach_delete_close ok 4 tun.reattach_delete_close # RUN tun.reattach_close_delete ... # OK tun.reattach_close_delete ok 5 tun.reattach_close_delete # PASSED: 5 / 5 tests passed. # Totals: pass:5 fail:0 xfail:0 xpass:0 skip:0 error:0 [root@pc-mtodorov linux_torvalds]#
So, no hanging processes that cannot be killed now.
If you think it is worthy to explore the lockup that occurs when turning CONFIG_DEBUG_KOBJECT=y, I will rebuild once again with these turned on, to clear any doubts.
Confirmed.
With the sole difference of:
[marvin@pc-mtodorov linux_torvalds]$ grep KOBJECT /boot/config-6.3.0-rc2-vanilla-00006-gfc89d7fb499b CONFIG_DEBUG_KOBJECT=y CONFIG_DEBUG_KOBJECT_RELEASE=y # CONFIG_SAMPLE_KOBJECT is not set [marvin@pc-mtodorov linux_torvalds]$
we get again unkillable processes:
[root@pc-mtodorov linux_torvalds]# ps -ef | grep tun root 1157 1 0 16:44 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned root 1331 1 0 16:44 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P root 3479 2315 0 16:45 pts/1 00:00:00 tools/testing/selftests/net/tun root 3512 2315 0 16:45 pts/1 00:00:00 tools/testing/selftests/net/tun root 4091 3364 0 16:49 pts/1 00:00:00 grep --color=auto tun [root@pc-mtodorov linux_torvalds]# kill -9 3479 3512 [root@pc-mtodorov linux_torvalds]# ps -ef | grep tun root 1157 1 0 16:44 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned root 1331 1 0 16:44 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P root 3479 2315 0 16:45 pts/1 00:00:00 tools/testing/selftests/net/tun root 3512 2315 0 16:45 pts/1 00:00:00 tools/testing/selftests/net/tun root 4095 3364 0 16:50 pts/1 00:00:00 grep --color=auto tun [root@pc-mtodorov linux_torvalds]#
Possibly the kernel /proc/cmdline is also important:
[root@pc-mtodorov linux_torvalds]# cat /proc/cmdline BOOT_IMAGE=(hd0,gpt5)/vmlinuz-6.3.0-rc2-vanilla-00006-gfc89d7fb499b root=/dev/mapper/almalinux_desktop--mtodorov-root ro crashkernel=auto resume=/dev/mapper/almalinux_desktop--mtodorov-swap rd.lvm.lv=almalinux_desktop-mtodorov/root rd.lvm.lv=almalinux_desktop-mtodorov/swap loglevel=7 i915.alpha_support=1 debug devres.log=1 [root@pc-mtodorov linux_torvalds]#
After a while, kernel message start looping:
kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:15 ... kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:24 ... kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:26 ... kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
This hangs processes until very late stage of shutdown.
I can confirm that CONFIG_DEBUG_{KOBJECT,KOBJECT_RELEASE}=y were the only changes to .config in between builds.
Best regards, Mirsad
On Tue, Mar 14, 2023 at 9:01 AM Mirsad Todorovac mirsad.todorovac@alu.unizg.hr wrote:
After a while, kernel message start looping:
kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:15 ... kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:24 ... kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:26 ... kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
This hangs processes until very late stage of shutdown.
I can confirm that CONFIG_DEBUG_{KOBJECT,KOBJECT_RELEASE}=y were the only changes to .config in between builds.
Best regards, Mirsad
Try adding in your config
CONFIG_NET_DEV_REFCNT_TRACKER=y CONFIG_NET_NS_REFCNT_TRACKER=y
Thanks.
On 14. 03. 2023. 17:02, Eric Dumazet wrote:
On Tue, Mar 14, 2023 at 9:01 AM Mirsad Todorovac mirsad.todorovac@alu.unizg.hr wrote:
After a while, kernel message start looping:
kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:15 ... kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:24 ... kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:26 ... kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
This hangs processes until very late stage of shutdown.
I can confirm that CONFIG_DEBUG_{KOBJECT,KOBJECT_RELEASE}=y were the only changes to .config in between builds.
Best regards, Mirsad
Try adding in your config
CONFIG_NET_DEV_REFCNT_TRACKER=y CONFIG_NET_NS_REFCNT_TRACKER=y
Thanks.
Not at all.
According to the info here: https://cateee.net/lkddb/web-lkddb/NET_DEV_REFCNT_TRACKER.html no kerenel param was needed.
I have got the same hang, and additional debug information appears to be this (in /var/log/messages):
Mar 14 20:58:20 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:20 pc-mtodorov kernel: leaked reference. Mar 14 20:58:20 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:20 pc-mtodorov kernel: leaked reference. Mar 14 20:58:20 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:20 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:20 pc-mtodorov kernel: leaked reference. Mar 14 20:58:20 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:20 pc-mtodorov kernel: leaked reference. Mar 14 20:58:20 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:30 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:30 pc-mtodorov kernel: leaked reference. Mar 14 20:58:30 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:30 pc-mtodorov kernel: leaked reference. Mar 14 20:58:30 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:30 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:30 pc-mtodorov kernel: leaked reference. Mar 14 20:58:30 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:30 pc-mtodorov kernel: leaked reference. Mar 14 20:58:30 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:40 pc-mtodorov kernel: leaked reference. Mar 14 20:58:40 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:40 pc-mtodorov kernel: leaked reference. Mar 14 20:58:40 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:40 pc-mtodorov kernel: leaked reference. Mar 14 20:58:40 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:40 pc-mtodorov kernel: leaked reference. Mar 14 20:58:40 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:50 pc-mtodorov kernel: leaked reference. Mar 14 20:58:50 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:50 pc-mtodorov kernel: leaked reference. Mar 14 20:58:50 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:50 pc-mtodorov kernel: leaked reference. Mar 14 20:58:50 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:50 pc-mtodorov kernel: leaked reference. Mar 14 20:58:50 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:57 pc-mtodorov kernel: kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak) Mar 14 20:59:00 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:59:00 pc-mtodorov kernel: leaked reference. Mar 14 20:59:00 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:59:00 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:59:00 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:59:00 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:00 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:00 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:00 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:00 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:00 pc-mtodorov kernel: leaked reference. Mar 14 20:59:00 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:59:00 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:59:00 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:59:00 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:00 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:00 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:00 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:00 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:01 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:59:01 pc-mtodorov kernel: leaked reference. Mar 14 20:59:01 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:59:01 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:59:01 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:59:01 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:01 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:01 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:01 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:01 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:01 pc-mtodorov kernel: leaked reference. Mar 14 20:59:01 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:59:01 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:59:01 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:59:01 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:01 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:01 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:01 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:01 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:10 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:59:10 pc-mtodorov kernel: leaked reference. Mar 14 20:59:10 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:59:10 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:59:10 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:59:10 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:10 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:10 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:10 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:10 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:10 pc-mtodorov kernel: leaked reference. Mar 14 20:59:10 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:59:10 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:59:10 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:59:10 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:10 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:10 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:10 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:10 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:11 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:59:11 pc-mtodorov kernel: leaked reference. Mar 14 20:59:11 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:59:11 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:59:11 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:59:11 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:11 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:11 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:11 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:11 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:11 pc-mtodorov kernel: leaked reference. Mar 14 20:59:11 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:59:11 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:59:11 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:59:11 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:11 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:11 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:11 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:11 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc [root@pc-mtodorov marvin]#
I see those "leaked reference" lines are being printed here: https://elixir.bootlin.com/linux/v6.3-rc2/source/lib/ref_tracker.c#L55
However, it is beyond the scope of my knowledge to track the actual leak.
Hope this helps.
Best regards, Mirsad
From: Mirsad Goran Todorovac mirsad.todorovac@alu.unizg.hr Date: Tue, 14 Mar 2023 21:10:55 +0100
On 14. 03. 2023. 17:02, Eric Dumazet wrote:
On Tue, Mar 14, 2023 at 9:01 AM Mirsad Todorovac mirsad.todorovac@alu.unizg.hr wrote:
After a while, kernel message start looping:
kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:15 ... kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:24 ... kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:26 ... kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
This hangs processes until very late stage of shutdown.
I can confirm that CONFIG_DEBUG_{KOBJECT,KOBJECT_RELEASE}=y were the only changes to .config in between builds.
KOBJECT_RELEASE delays freeing kobject with delayed_work.
In the failing tests, e.g. reattach_delete_close, we create a tun dev and do
1. detach (tun_detach()) 2. re-attach (tun_attach()) 3. close
. We release kobjects (tun->dev->(_tx|_rx).kobj) in 1. and recycle the same memory region in 2.
But the kobjects are not actually released due to the delay, thus netdev_queue_add_kobject() and rx_queue_add_kobject() fail.
---8<--- # RUN tun.reattach_delete_close ... [ 179.087589] kobject: 'tx-1' (00000000ee182e2f): kobject_release, parent 000000000643eb5f (delayed 3000) [ 179.088105] kobject: 'rx-1' (000000001c44852d): kobject_release, parent 000000000643eb5f (delayed 1000) [ 179.089097] kobject (00000000ee182e2f): tried to init an initialized object, something is seriously wrong. ---8<---
However, we don't assume the delay and also the failure in tun_set_real_num_queues().
In this case, we have to re-initialise the queues without touching kobjects.
Eric, Are you working on this? If not, let me try fixing this :)
Thanks, Kuniyuki
Best regards, Mirsad
Try adding in your config
CONFIG_NET_DEV_REFCNT_TRACKER=y CONFIG_NET_NS_REFCNT_TRACKER=y
Thanks.
Not at all.
According to the info here: https://cateee.net/lkddb/web-lkddb/NET_DEV_REFCNT_TRACKER.html no kerenel param was needed.
I have got the same hang, and additional debug information appears to be this (in /var/log/messages):
Mar 14 20:58:20 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:20 pc-mtodorov kernel: leaked reference. Mar 14 20:58:20 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:20 pc-mtodorov kernel: leaked reference. Mar 14 20:58:20 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:20 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:20 pc-mtodorov kernel: leaked reference. Mar 14 20:58:20 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:20 pc-mtodorov kernel: leaked reference. Mar 14 20:58:20 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:30 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:30 pc-mtodorov kernel: leaked reference. Mar 14 20:58:30 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:30 pc-mtodorov kernel: leaked reference. Mar 14 20:58:30 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:30 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:30 pc-mtodorov kernel: leaked reference. Mar 14 20:58:30 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:30 pc-mtodorov kernel: leaked reference. Mar 14 20:58:30 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:40 pc-mtodorov kernel: leaked reference. Mar 14 20:58:40 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:40 pc-mtodorov kernel: leaked reference. Mar 14 20:58:40 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:40 pc-mtodorov kernel: leaked reference. Mar 14 20:58:40 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:40 pc-mtodorov kernel: leaked reference. Mar 14 20:58:40 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:50 pc-mtodorov kernel: leaked reference. Mar 14 20:58:50 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:50 pc-mtodorov kernel: leaked reference. Mar 14 20:58:50 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:58:50 pc-mtodorov kernel: leaked reference. Mar 14 20:58:50 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:50 pc-mtodorov kernel: leaked reference. Mar 14 20:58:50 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:58:57 pc-mtodorov kernel: kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak) Mar 14 20:59:00 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:59:00 pc-mtodorov kernel: leaked reference. Mar 14 20:59:00 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:59:00 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:59:00 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:59:00 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:00 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:00 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:00 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:00 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:00 pc-mtodorov kernel: leaked reference. Mar 14 20:59:00 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:59:00 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:59:00 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:59:00 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:00 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:00 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:00 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:00 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:01 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:59:01 pc-mtodorov kernel: leaked reference. Mar 14 20:59:01 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:59:01 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:59:01 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:59:01 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:01 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:01 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:01 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:01 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:01 pc-mtodorov kernel: leaked reference. Mar 14 20:59:01 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:59:01 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:59:01 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:59:01 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:01 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:01 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:01 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:01 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:10 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:59:10 pc-mtodorov kernel: leaked reference. Mar 14 20:59:10 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:59:10 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:59:10 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:59:10 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:10 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:10 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:10 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:10 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:10 pc-mtodorov kernel: leaked reference. Mar 14 20:59:10 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:59:10 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:59:10 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:59:10 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:10 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:10 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:10 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:10 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:11 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3 Mar 14 20:59:11 pc-mtodorov kernel: leaked reference. Mar 14 20:59:11 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0 Mar 14 20:59:11 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0 Mar 14 20:59:11 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0 Mar 14 20:59:11 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:11 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:11 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:11 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:11 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc Mar 14 20:59:11 pc-mtodorov kernel: leaked reference. Mar 14 20:59:11 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190 Mar 14 20:59:11 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250 Mar 14 20:59:11 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0 Mar 14 20:59:11 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0 Mar 14 20:59:11 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20 Mar 14 20:59:11 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0 Mar 14 20:59:11 pc-mtodorov kernel: do_syscall_64+0x5c/0x90 Mar 14 20:59:11 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc [root@pc-mtodorov marvin]#
I see those "leaked reference" lines are being printed here: https://elixir.bootlin.com/linux/v6.3-rc2/source/lib/ref_tracker.c#L55
However, it is beyond the scope of my knowledge to track the actual leak.
Hope this helps.
Best regards, Mirsad
-- Mirsad Goran Todorovac Sistem inženjer Grafički fakultet | Akademija likovnih umjetnosti Sveučilište u Zagrebu System engineer Faculty of Graphic Arts | Academy of Fine Arts University of Zagreb, Republic of Croatia The European Union
On Wed, Mar 15, 2023 at 1:57 PM Kuniyuki Iwashima kuniyu@amazon.com wrote:
However, we don't assume the delay and also the failure in tun_set_real_num_queues().
In this case, we have to re-initialise the queues without touching kobjects.
Eric, Are you working on this? If not, let me try fixing this :)
I am not working on this, please go ahead, thanks !
On 15. 03. 2023. 21:59, Eric Dumazet wrote:
On Wed, Mar 15, 2023 at 1:57 PM Kuniyuki Iwashima kuniyu@amazon.com wrote:
However, we don't assume the delay and also the failure in tun_set_real_num_queues().
In this case, we have to re-initialise the queues without touching kobjects.
Eric, Are you working on this? If not, let me try fixing this :)
I am not working on this, please go ahead, thanks !
Hi,
It's me again. I just have new findings.
[root@pc-mtodorov linux_torvalds]# grep -E '(KOBJECT|TRACKER)' /boot/config-6.3.0-rc2-00006-gfc89d7fb499b CONFIG_REF_TRACKER=y CONFIG_NET_DEV_REFCNT_TRACKER=y CONFIG_NET_NS_REFCNT_TRACKER=y CONFIG_DEBUG_KOBJECT=y # CONFIG_DEBUG_KOBJECT_RELEASE is not set # CONFIG_SAMPLE_KOBJECT is not set # CONFIG_TEST_REF_TRACKER is not set [root@pc-mtodorov linux_torvalds]# uname -rms Linux 6.3.0-rc2-00006-gfc89d7fb499b x86_64 [root@pc-mtodorov linux_torvalds]# grep -E '(KOBJECT|TRACKER)' /boot/config-6.3.0-rc2-00006-gfc89d7fb499b CONFIG_REF_TRACKER=y CONFIG_NET_DEV_REFCNT_TRACKER=y CONFIG_NET_NS_REFCNT_TRACKER=y CONFIG_DEBUG_KOBJECT=y # CONFIG_DEBUG_KOBJECT_RELEASE is not set # CONFIG_SAMPLE_KOBJECT is not set # CONFIG_TEST_REF_TRACKER is not set [root@pc-mtodorov linux_torvalds]# tools/testing/selftests/net/tun TAP version 13 1..5 # Starting 5 tests from 1 test cases. # RUN tun.delete_detach_close ... # OK tun.delete_detach_close ok 1 tun.delete_detach_close # RUN tun.detach_delete_close ... # OK tun.detach_delete_close ok 2 tun.detach_delete_close # RUN tun.detach_close_delete ... # OK tun.detach_close_delete ok 3 tun.detach_close_delete # RUN tun.reattach_delete_close ... # OK tun.reattach_delete_close ok 4 tun.reattach_delete_close # RUN tun.reattach_close_delete ... # OK tun.reattach_close_delete ok 5 tun.reattach_close_delete # PASSED: 5 / 5 tests passed. # Totals: pass:5 fail:0 xfail:0 xpass:0 skip:0 error:0 [root@pc-mtodorov linux_torvalds]#
My interpretation if you allow it is that the bug search can be narrowed to the code that depends on CONFIG_DEBUG_KOBJECT_RELEASE=y.
Best regards, Mirsad
CONFIG_DEBUG_KOBJECT=y alone doesn't seem to be sufficient to trigger the reference leak.
Hope this helps narrow down the search.
linux-kselftest-mirror@lists.linaro.org
-
Eric Dumazet -
Kuniyuki Iwashima -
Mirsad Goran Todorovac -
Mirsad Todorovac