On Fri, 2024-10-04 at 09:39 +0200, Sebastian Andrzej Siewior wrote:

On 2024-09-27 15:01:00 [-0400], Joseph Salisbury wrote:

...

Is it needed in all stable release patch sets, including v5.15?

Yes. I would appreciate backporting it all the way where the code is available. The dependencies 1eacdd71b3436 ("netfilter: nft_counter: Disable BH in nft_counter_offload_stats().") a0b39e2dc7017 ("netfilter: nft_counter: Synchronize nft_counter_reset() against reader.")

were already routed via stable. The problem is that the seqcount has no lock associated so a reader could preempt a writer and then lockup spinning.

Hi,

this needs to be backported to all stable RT trees (just checked 4.19 and 6.1. 5.15 already has it). We observed the reader live-lock issue in "nft_counter_fetch" on 6.1.120-rt47 (leading to a system stall) and were also able to find it with lockdep (see stacktrace below).

I'm wondering if this patch could be applied to linux-stable, even if it is just a performance optimization on non-rt kernels (not a fix).

The patch "netfilter: nft_counter: Use u64_stats_t for statistic" cleanly applies on 6.1.y and 6.1.127-rt48.

Stacktrace from lockdep: [ 33.643632] ------------[ cut here ]------------ [ 33.643637] WARNING: CPU: 0 PID: 972 at include/linux/seqlock.h:269 nft_counter_eval+0x6b/0xd0 [nf_tables] [ 33.643657] Modules linked in: br_netfilter bridge stp llc xt_comment xt_recent xt_hl ip6_tables ip6t_rt ipt_REJECT nf_reject_ipv4 xt_LOG nf_log_syslog nft_limit xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables libcrc32c nfnetlink rfkill intel_rapl_msr intel_rapl_common ccp binfmt_misc kvm irqbypass ghash_clmulni_intel sha512_ssse3 sha512_generic sha256_ssse3 sha1_ssse3 ppdev snd_pcm snd_timer aesni_intel snd crypto_simd cryptd soundcore pcspkr parport_pc iTCO_wdt bochs parport drm_vram_helper intel_pmc_bxt drm_ttm_helper iTCO_vendor_support button ttm drm_kms_helper watchdog sg joydev evdev serio_raw drm fuse loop efi_pstore configfs qemu_fw_cfg ip_tables x_tables autofs4 overlay nls_ascii nls_cp437 vfat fat ext4 crc32c_generic crc16 mbcache jbd2 xts ecb squashfs dm_verity dm_bufio reed_solomon dm_mod sd_mod t10_pi crc64_rocksoft crc64 crc_t10dif crct10dif_generic virtio_net net_failover ahci failover libahci crct10dif_pclmul [ 33.643727] crct10dif_common libata virtio_pci i2c_i801 crc32_pclmul scsi_mod crc32c_intel virtio_pci_legacy_dev i2c_smbus psmouse virtio_pci_modern_dev virtio scsi_common virtio_ring lpc_ich [ 33.643739] CPU: 0 PID: 972 Comm: onboardservice Not tainted 6.1.120-rt47 #1 [ 33.643742] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 [ 33.643744] RIP: 0010:nft_counter_eval+0x6b/0xd0 [nf_tables] [ 33.643759] Code: 52 3f 85 d2 74 26 65 8b 05 ba bd 52 3f 85 c0 75 1b 65 8b 05 e7 b3 52 3f a9 ff ff ff 7f 75 0d 65 8b 05 dd ba 52 3f 85 c0 74 02 <0f> 0b ff 74 24 20 4c 8d 6d 08 45 31 c9 31 c9 41 b8 01 00 00 00 31 [ 33.643776] RSP: 0018:ffffa045007736a0 EFLAGS: 00010202 [ 33.643778] RAX: 0000000000000001 RBX: ffffc044ffc2ae80 RCX: 00000000000026af [ 33.643780] RDX: 0000000000000001 RSI: ffff8d29050db388 RDI: ffffffffc0af49a4 [ 33.643781] RBP: ffff8d293f638060 R08: 0000000000000000 R09: 0000000000000000 [ 33.643782] R10: 0000000000000001 R11: 000000009bb77572 R12: ffffa04500773920 [ 33.643783] R13: ffff8d29011db358 R14: ffff8d29011db208 R15: ffff8d29011db240 [ 33.643807] FS: 000000c000047c90(0000) GS:ffff8d293f600000(0000) knlGS:0000000000000000 [ 33.643811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 33.643813] CR2: 000000c0005fe000 CR3: 000000003a212000 CR4: 00000000003506f0 [ 33.643836] Call Trace: [ 33.643840] <TASK> [ 33.643844] ? __warn+0x82/0xe0 [ 33.643852] ? nft_counter_eval+0x6b/0xd0 [nf_tables] [ 33.643877] ? report_bug+0x10e/0x180 [ 33.643889] ? handle_bug+0x41/0x70 [ 33.643895] ? exc_invalid_op+0x13/0x60 [ 33.643899] ? asm_exc_invalid_op+0x16/0x20 [ 33.643912] ? nft_counter_eval+0x24/0xd0 [nf_tables] [ 33.643931] ? nft_counter_eval+0x6b/0xd0 [nf_tables] [ 33.643962] nft_do_chain+0x45b/0x690 [nf_tables] [ 33.644025] nft_do_chain_ipv4+0x78/0xa0 [nf_tables] [ 33.644046] nf_hook_slow+0x41/0xc0 [ 33.644054] __ip_local_out+0x14c/0x300 [ 33.644062] ? ip_output+0xb0/0xb0 [ 33.644074] __ip_queue_xmit+0x1c0/0x7f0 [ 33.644086] __tcp_transmit_skb+0xabe/0xcb0 [ 33.644107] tcp_write_xmit+0x521/0x14a0 [ 33.644117] __tcp_push_pending_frames+0x32/0xf0 [ 33.644120] tcp_sendmsg_locked+0x4cd/0xc20 [ 33.644133] tcp_sendmsg+0x27/0x40 [ 33.644137] __sock_sendmsg+0x58/0x70 [ 33.644142] sock_write_iter+0x9a/0x100 [ 33.644151] vfs_write+0x2c8/0x330 [ 33.644164] ksys_write+0xc3/0xf0 [ 33.644169] do_syscall_64+0x55/0xb0 [ 33.644173] ? lock_acquire+0xc4/0x2d0 [ 33.644178] ? find_held_lock+0x2b/0x80 [ 33.644182] ? finish_task_switch.isra.0+0xca/0x380 [ 33.644186] ? lock_release+0xd0/0x2d0 [ 33.644191] ? lockdep_hardirqs_on_prepare+0xdc/0x190 [ 33.644196] ? finish_task_switch.isra.0+0xcf/0x380 [ 33.644201] ? __schedule+0x3f8/0xd20 [ 33.644206] ? restore_fpregs_from_fpstate+0x38/0x90 [ 33.644211] ? trace_x86_fpu_regs_activated+0x1f/0xb0 [ 33.644213] ? switch_fpu_return+0x58/0x90 [ 33.644218] ? exit_to_user_mode_prepare+0x1af/0x250 [ 33.644223] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 33.644227] RIP: 0033:0x40720e [ 33.644230] Code: 48 83 ec 38 e8 13 00 00 00 48 83 c4 38 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48 [ 33.644232] RSP: 002b:000000c000069980 EFLAGS: 00000216 ORIG_RAX: 0000000000000001 [ 33.644234] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 000000000040720e [ 33.644236] RDX: 000000000000008c RSI: 000000c0001746c0 RDI: 0000000000000009 [ 33.644237] RBP: 000000c0000699c0 R08: 0000000000000000 R09: 0000000000000000 [ 33.644238] R10: 0000000000000000 R11: 0000000000000216 R12: 000000c000069b00 [ 33.644239] R13: 000000000000000e R14: 000000c00016ed00 R15: 0000000000a88360 [ 33.644250] </TASK> [ 33.644250] irq event stamp: 10266 [ 33.644251] hardirqs last enabled at (10268): [<ffffffff96339836>] vprintk_store+0x326/0x550 [ 33.644256] hardirqs last disabled at (10269): [<ffffffff9633987c>] vprintk_store+0x36c/0x550 [ 33.644259] softirqs last enabled at (9900): [<ffffffff962af77e>] __local_bh_enable_ip+0xfe/0x140 [ 33.644264] softirqs last disabled at (9904): [<ffffffffc0af49a4>] nft_counter_eval+0x24/0xd0 [nf_tables] [ 33.644277] ---[ end trace 0000000000000000 ]---

Best regards, Felix

Sebastian