在 2022/9/28 AM1:47, Luck, Tony 写道:
I follow and agree with everything up until:
In a conclusion, the error will be handled in a kworker with or without this fix.
It isn't handled during the interrupt (it can't be).
Yes, it is not handled during the interrupt and it does not have to.
Who handles the error if the interrupt happens during the execution of a kthread?
As I mentioned, the GHES driver always queues work into workqueue to handle memory failure of a page in memory_failure_queue(), so the **worker will be scheduled and handle memory failure later**.
Can't use the task_work_add() trick to handle it (because this thread never returns to user mode).
Yes, it can not. And this is the key point to fix.
So how is the error handled?
The workflow to handle hardware error is summery as bellow:
----------------------------------------------------------------------------- [ghes_sdei_critical_callback: current swapper/3, CPU 3] ghes_sdei_critical_callback => __ghes_sdei_callback => ghes_in_nmi_queue_one_entry // peak and read estatus => irq_work_queue(&ghes_proc_irq_work) <=> ghes_proc_in_irq // irq_work [ghes_sdei_critical_callback: return] ----------------------------------------------------------------------------- [ghes_proc_in_irq: current swapper/3, CPU 3] => ghes_do_proc => ghes_handle_memory_failure => ghes_do_memory_failure => memory_failure_queue // put work task on current CPU => if (kfifo_put(&mf_cpu->fifo, entry)) schedule_work_on(smp_processor_id(), &mf_cpu->work); => task_work_add(current, &estatus_node->task_work, TWA_RESUME); // fix here, always added to current [ghes_proc_in_irq: return] ----------------------------------------------------------------------------- // kworker preempts swapper/3 on CPU 3 due to RESCHED flag [memory_failure_work_func: current kworker, CPU 3] => memory_failure_work_func(&mf_cpu->work) => while kfifo_get(&mf_cpu->fifo, &entry); // until get no work => soft/hard offline -----------------------------------------------------------------------------
STEP 0: The firmware notifies hardware error to kernel through is SDEI (ACPI_HEST_NOTIFY_SOFTWARE_DELEGATED).
STEP 1: In SDEI callback (or any NMI-like handler), memory from ghes_estatus_pool is used to save estatus, and added to the ghes_estatus_llist. The swapper running on CPU 3 is interrupted. irq_work_queue() causes ghes_proc_in_irq() to run in IRQ context where each estatus in ghes_estatus_llist is processed.
STEP2: In IRQ context, ghes_proc_in_irq() queues memory failure work on current CPU in workqueue and add task work to sync with the workqueue.
STEP3: The kworker preempts the current running thread and get CPU 3. Then memory failure is processed in kworker.
(STEP4 for user thread: ghes_kick_task_work() is called as task_work to ensure any queued workqueue has been done before returning to user-space. The estatus_node is freed.)
If the task work is not added, estatus_node->task_work.func will be NULL, and estatus_node is freed in STEP 2.
Hope it helps to make the problem clearer. You can also check the stack dumped in key function in above flow.
Best Regards, Shuai
--------------------------------------------------------------------------------------- dump_stack() is added in: - __ghes_sdei_callback() - ghes_proc_in_irq() - memory_failure_queue_kick() - memory_failure_work_func() - memory_failure()
[ 485.457761] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G E 6.0.0-rc5+ #33 [ 485.457769] Hardware name: xxxx [ 485.457771] Call trace: [ 485.457772] dump_backtrace+0xe8/0x12c [ 485.457779] show_stack+0x20/0x50 [ 485.457781] dump_stack_lvl+0x68/0x84 [ 485.457785] dump_stack+0x18/0x34 [ 485.457787] __ghes_sdei_callback+0x24/0x64 [ 485.457789] ghes_sdei_critical_callback+0x5c/0x94 [ 485.457792] sdei_event_handler+0x28/0x90 [ 485.457795] do_sdei_event+0x74/0x160 [ 485.457797] __sdei_handler+0x60/0xf0 [ 485.457799] __sdei_asm_handler+0xbc/0x18c [ 485.457801] cpu_do_idle+0x14/0x80 [ 485.457802] default_idle_call+0x50/0x114 [ 485.457804] cpuidle_idle_call+0x16c/0x1c0 [ 485.457806] do_idle+0xb8/0x110 [ 485.457808] cpu_startup_entry+0x2c/0x34 [ 485.457809] secondary_start_kernel+0xf0/0x144 [ 485.457812] __secondary_switched+0xb0/0xb4
[ 485.459513] EDAC MC0: 1 UE multi-symbol chipkill ECC on unknown memory (node:0 card:3 module:0 rank:0 bank_group:0 bank_address:0 device:0 row:624 column:384 chip_id:0 page:0x89c033 offset:0x400 grain:1 - APEI location: node:0 card:3 module:0 rank:0 bank_group:0 bank_address:0 device:0 row:624 column:384 chip_id:0 status(0x0000000000000400): Storage error in DRAM memory) [ 485.459523] {2}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2 [ 485.470607] {2}[Hardware Error]: event severity: recoverable [ 485.476252] {2}[Hardware Error]: precise tstamp: 2022-09-29 09:31:27 [ 485.482678] {2}[Hardware Error]: Error 0, type: recoverable [ 485.488322] {2}[Hardware Error]: section_type: memory error [ 485.494052] {2}[Hardware Error]: error_status: Storage error in DRAM memory (0x0000000000000400) [ 485.503081] {2}[Hardware Error]: physical_address: 0x000000089c033400 [ 485.509680] {2}[Hardware Error]: node:0 card:3 module:0 rank:0 bank_group:0 bank_address:0 device:0 row:624 column:384 chip_id:0 [ 485.521487] {2}[Hardware Error]: error_type: 5, multi-symbol chipkill ECC
[ 485.528439] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G E 6.0.0-rc5+ #33 [ 485.528440] Hardware name: AlibabaCloud AliServer-Xuanwu2.0AM-02-2UC1P-5B/M, BIOS 1.2.M1.AL.E.132.01 08/23/2022 [ 485.528441] Call trace: [ 485.528441] dump_backtrace+0xe8/0x12c [ 485.528443] show_stack+0x20/0x50 [ 485.528444] dump_stack_lvl+0x68/0x84 [ 485.528446] dump_stack+0x18/0x34 [ 485.528448] ghes_proc_in_irq+0x220/0x250 [ 485.528450] irq_work_single+0x30/0x80 [ 485.528453] irq_work_run_list+0x4c/0x70 [ 485.528455] irq_work_run+0x28/0x44 [ 485.528457] do_handle_IPI+0x2b4/0x2f0 [ 485.528459] ipi_handler+0x24/0x34 [ 485.528461] handle_percpu_devid_irq+0x90/0x1c4 [ 485.528463] generic_handle_domain_irq+0x34/0x50 [ 485.528465] __gic_handle_irq_from_irqson.isra.0+0x130/0x230 [ 485.528468] gic_handle_irq+0x2c/0x60 [ 485.528469] call_on_irq_stack+0x2c/0x38 [ 485.528471] do_interrupt_handler+0x88/0x90 [ 485.528472] el1_interrupt+0x48/0xb0 [ 485.528475] el1h_64_irq_handler+0x18/0x24 [ 485.528476] el1h_64_irq+0x74/0x78 [ 485.528477] __do_softirq+0xa4/0x358 [ 485.528478] __irq_exit_rcu+0x110/0x13c [ 485.528479] irq_exit_rcu+0x18/0x24 [ 485.528480] el1_interrupt+0x4c/0xb0 [ 485.528482] el1h_64_irq_handler+0x18/0x24 [ 485.528483] el1h_64_irq+0x74/0x78 [ 485.528484] arch_cpu_idle+0x18/0x40 [ 485.528485] default_idle_call+0x50/0x114 [ 485.528487] cpuidle_idle_call+0x16c/0x1c0 [ 485.528488] do_idle+0xb8/0x110 [ 485.528489] cpu_startup_entry+0x2c/0x34 [ 485.528491] secondary_start_kernel+0xf0/0x144 [ 485.528493] __secondary_switched+0xb0/0xb4
[ 485.528511] CPU: 3 PID: 12696 Comm: kworker/3:0 Tainted: G E 6.0.0-rc5+ #33 [ 485.528513] Hardware name: AlibabaCloud AliServer-Xuanwu2.0AM-02-2UC1P-5B/M, BIOS 1.2.M1.AL.E.132.01 08/23/2022 [ 485.528514] Workqueue: events memory_failure_work_func [ 485.528518] Call trace: [ 485.528519] dump_backtrace+0xe8/0x12c [ 485.528520] show_stack+0x20/0x50 [ 485.528521] dump_stack_lvl+0x68/0x84 [ 485.528523] dump_stack+0x18/0x34 [ 485.528525] memory_failure_work_func+0xec/0x180 [ 485.528527] process_one_work+0x1f4/0x460 [ 485.528528] worker_thread+0x188/0x3e4 [ 485.528530] kthread+0xd0/0xd4 [ 485.528532] ret_from_fork+0x10/0x20
[ 485.528533] CPU: 3 PID: 12696 Comm: kworker/3:0 Tainted: G E 6.0.0-rc5+ #33 [ 485.528534] Hardware name: AlibabaCloud AliServer-Xuanwu2.0AM-02-2UC1P-5B/M, BIOS 1.2.M1.AL.E.132.01 08/23/2022 [ 485.528535] Workqueue: events memory_failure_work_func [ 485.528537] Call trace: [ 485.528538] dump_backtrace+0xe8/0x12c [ 485.528539] show_stack+0x20/0x50 [ 485.528540] dump_stack_lvl+0x68/0x84 [ 485.528541] dump_stack+0x18/0x34 [ 485.528543] memory_failure+0x50/0x438 [ 485.528544] memory_failure_work_func+0x174/0x180 [ 485.528546] process_one_work+0x1f4/0x460 [ 485.528547] worker_thread+0x188/0x3e4 [ 485.528548] kthread+0xd0/0xd4 [ 485.528550] ret_from_fork+0x10/0x20 [ 485.530622] Memory failure: 0x89c033: recovery action for dirty LRU page: Recovered