On Mon, Nov 28, 2022 at 05:08:31PM +0000, Luiz Capitulino wrote:
Hi,
[ Marc, can you help reviewing? Esp. the first patch? ]
This series of backports from upstream to stable 5.15 and 5.10 fixes an issue we're seeing on AWS ARM instances where attaching an EBS volume (which is a nvme device) to the instance after offlining CPUs causes the device to take several minutes to show up and eventually nvme kworkers and other threads start getting stuck.
This series fixes the issue for 5.15.79 and 5.10.155. I can't reproduce it on 5.4. Also, I couldn't reproduce this on x86 even w/ affected kernels.
An easy reproducer is:
- Start an ARM instance with 32 CPUs
- Once the instance is booted, offline all CPUs but CPU 0. Eg: # for i in $(seq 1 32); do chcpu -d $i; done
- Once the CPUs are offline, attach an EBS volume
- Watch lsblk and dmesg in the instance
Eventually, you get this stack trace:
[ 71.842974] pci 0000:00:1f.0: [1d0f:8061] type 00 class 0x010802 [ 71.843966] pci 0000:00:1f.0: reg 0x10: [mem 0x00000000-0x00003fff] [ 71.845149] pci 0000:00:1f.0: PME# supported from D0 D1 D2 D3hot D3cold [ 71.846694] pci 0000:00:1f.0: BAR 0: assigned [mem 0x8011c000-0x8011ffff] [ 71.848458] ACPI: _SB_.PCI0.GSI3: Enabled at IRQ 38 [ 71.850852] nvme nvme1: pci function 0000:00:1f.0 [ 71.851611] nvme 0000:00:1f.0: enabling device (0000 -> 0002) [ 135.887787] nvme nvme1: I/O 22 QID 0 timeout, completion polled [ 197.328276] nvme nvme1: I/O 23 QID 0 timeout, completion polled [ 197.329221] nvme nvme1: 1/0/0 default/read/poll queues [ 243.408619] INFO: task kworker/u64:2:275 blocked for more than 122 seconds. [ 243.409674] Not tainted 5.15.79 #1 [ 243.410270] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.411389] task:kworker/u64:2 state:D stack: 0 pid: 275 ppid: 2 flags:0x00000008 [ 243.412602] Workqueue: events_unbound async_run_entry_fn [ 243.413417] Call trace: [ 243.413797] __switch_to+0x15c/0x1a4 [ 243.414335] __schedule+0x2bc/0x990 [ 243.414849] schedule+0x68/0xf8 [ 243.415334] schedule_timeout+0x184/0x340 [ 243.415946] wait_for_completion+0xc8/0x220 [ 243.416543] __flush_work.isra.43+0x240/0x2f0 [ 243.417179] flush_work+0x20/0x2c [ 243.417666] nvme_async_probe+0x20/0x3c [ 243.418228] async_run_entry_fn+0x3c/0x1e0 [ 243.418858] process_one_work+0x1bc/0x460 [ 243.419437] worker_thread+0x164/0x528 [ 243.420030] kthread+0x118/0x124 [ 243.420517] ret_from_fork+0x10/0x20 [ 258.768771] nvme nvme1: I/O 20 QID 0 timeout, completion polled [ 320.209266] nvme nvme1: I/O 21 QID 0 timeout, completion polled
For completion, I tested the same test-case on x86 with this series applied on 5.15.79 and 5.10.155 as well. It works as expected.
All now queued up, thanks.
greg k-h