...

cper_sec_mem_err::error_type identifies the type of error that occurred if CPER_MEM_VALID_ERROR_TYPE is set. So, set memory failure flags as 0 for Scrub Uncorrected Error (type 14). Otherwise, set memory failure flags as MF_ACTION_REQUIRED.

On x86 the "action required" cases are signaled by a synchronous machine check that is delivered before the instruction that is attempting to consume the uncorrected data retires. I.e., it is guaranteed that the uncorrected error has not been propagated because it is not visible in any architectural state.

APEI signaled errors don't fall into that category on x86 ... the uncorrected data could have been consumed and propagated long before the signaling used for APEI can alert the OS.

Does ARM deliver APEI signals synchronously?

If not, then this patch might deliver a false sense of security to applications about the state of uncorrected data in the system.

-Tony

在 2022/11/2 PM7:53, Shuai Xue 写道:

在 2022/10/29 AM1:25, Luck, Tony 写道:

...

...

...

cper_sec_mem_err::error_type identifies the type of error that occurred if CPER_MEM_VALID_ERROR_TYPE is set. So, set memory failure flags as 0 for Scrub Uncorrected Error (type 14). Otherwise, set memory failure flags as MF_ACTION_REQUIRED.

On x86 the "action required" cases are signaled by a synchronous machine check that is delivered before the instruction that is attempting to consume the uncorrected data retires. I.e., it is guaranteed that the uncorrected error has not been propagated because it is not visible in any architectural state.

On arm, if a 2-bit (uncorrectable) error is detected, and the memory access has been architecturally executed, that error is considered “consumed”. The CPU will take a synchronous error exception, signaled as synchronous external abort (SEA), which is analogously to MCE.

...

APEI signaled errors don't fall into that category on x86 ... the uncorrected data could have been consumed and propagated long before the signaling used for APEI can alert the OS.

Does ARM deliver APEI signals synchronously?

If not, then this patch might deliver a false sense of security to applications about the state of uncorrected data in the system.

Well, it does not always. There are many APEI notification, such as SCI, GSIV, GPIO, SDEI, SEA, etc. Not all APEI notifications are synchronously and it depends on hardware signal. As far as I know, if a UE is detected and consumed, synchronous external abort is signaled to firmware and firmware then performs a first-level triage and synchronously notify OS by SDEI or SEA notification. On the other hand, if CE is detected, a asynchronous interrupt will be signaled and firmware could notify OS by GPIO or GSIV.

Best Regards, Shuai

Hi, Tony,

Prefetch data with UE error triggers async interrupt on both X86 and Arm64 platform (CMCI in X86 and SPI in arm64). It does not belongs to scrub UEs. I have to admit that cper_sec_mem_err::error_type is not an appropriate basis to distinguish "action required" cases.

acpi_hest_generic_data::flags (UEFI spec section N.2.2) could be used to indicate Action Optional (Scrub/Prefetch).

Bit 5 – Latent error: If set this flag indicates that action has been taken to ensure error containment (such a poisoning data), but the error has not been fully corrected and the data has not been consumed. System software may choose to take further corrective action before the data is consumed.

Our hardware team has submitted a proposal to UEFI community to add a new bit:

Bit 8 – sync flag; if set this flag indicates that this event record is synchronous(e.g. cpu core consumes poison data, then cause instruction/data abort); if not set, this event record is asynchronous.

With bit 8, we will know it is "Action Required".

I will send a new patch set to rework GHES error handling after the proposal is accept.

Thank you.

Best Regards Shuai

On 2023/9/25 23:00, Jarkko Sakkinen wrote:

On Tue Sep 19, 2023 at 5:21 AM EEST, Shuai Xue wrote:

...

Hardware errors could be signaled by synchronous interrupt, e.g. when an error is detected by a background scrubber, or signaled by synchronous exception, e.g. when an uncorrected error is consumed. Both synchronous and asynchronous error are queued and handled by a dedicated kthread in workqueue.

commit 7f17b4a121d0 ("ACPI: APEI: Kick the memory_failure() queue for synchronous errors") keep track of whether memory_failure() work was queued, and make task_work pending to flush out the workqueue so that the work for synchronous error is processed before returning to user-space. The trick ensures that the corrupted page is unmapped and poisoned. And after returning to user-space, the task starts at current instruction which triggering a page fault in which kernel will send SIGBUS to current process due to VM_FAULT_HWPOISON.

However, the memory failure recovery for hwpoison-aware mechanisms does not work as expected. For example, hwpoison-aware user-space processes like QEMU register their customized SIGBUS handler and enable early kill mode by seting PF_MCE_EARLY at initialization. Then the kernel will directy notify the process by sending a SIGBUS signal in memory failure with wrong si_code: the actual user-space process accessing the corrupt memory location, but its memory failure work is handled in a kthread context, so it will send SIGBUS with BUS_MCEERR_AO si_code to the actual user-space process instead of BUS_MCEERR_AR in kill_proc().

To this end, separate synchronous and asynchronous error handling into different paths like X86 platform does:

• valid synchronous errors: queue a task_work to synchronously send SIGBUS before ret_to_user.
• valid asynchronous errors: queue a work into workqueue to asynchronously handle memory failure.
• abnormal branches such as invalid PA, unexpected severity, no memory failure config support, invalid GUID section, OOM, etc.

Then for valid synchronous errors, the current context in memory failure is exactly belongs to the task consuming poison data and it will send SIBBUS with proper si_code.

Fixes: 7f17b4a121d0 ("ACPI: APEI: Kick the memory_failure() queue for synchronous errors") Signed-off-by: Shuai Xue xueshuai@linux.alibaba.com Tested-by: Ma Wupeng mawupeng1@huawei.com Reviewed-by: Kefeng Wang wangkefeng.wang@huawei.com Reviewed-by: Xiaofei Tan tanxiaofei@huawei.com Reviewed-by: Baolin Wang baolin.wang@linux.alibaba.com

Did 7f17b4a121d0 actually break something that was not broken before?

If not, this is (afaik) not a bug fix.

Hi, Jarkko,

It did not. It keeps track of whether memory_failure() work was queued, and makes task_work pending to flush out the queue. But if no work queued for synchronous error due to abnormal branches, it does not do a force kill to current process resulting a hard lockup due to exception loop.

It is fine to me to remove the bug fix tag if you insist on removing it.

Best Regards, Shuai

Hi, ALL,

Gentle ping.

Best Regards, Shuai

On 2023/10/7 15:28, Shuai Xue wrote:

Hi, ALL,

I have rewritten the cover letter with the hope that the maintainer will truly understand the necessity of this patch. Both Alibaba and Huawei met the same issue in products, and we hope it could be fixed ASAP.

## Changes Log

changes since v8:

• remove the bug fix tag of patch 2 (per Jarkko Sakkinen)
• remove the declaration of memory_failure_queue_kick (per Naoya Horiguchi)
• rewrite the return value comments of memory_failure (per Naoya Horiguchi)

changes since v7:

• rebase to Linux v6.6-rc2 (no code changed)
• rewritten the cover letter to explain the motivation of this patchset

changes since v6:

• add more explicty error message suggested by Xiaofei
• pick up reviewed-by tag from Xiaofei
• pick up internal reviewed-by tag from Baolin

changes since v5 by addressing comments from Kefeng:

• document return value of memory_failure()
• drop redundant comments in call site of memory_failure()
• make ghes_do_proc void and handle abnormal case within it
• pick up reviewed-by tag from Kefeng Wang

changes since v4 by addressing comments from Xiaofei:

• do a force kill only for abnormal sync errors

changes since v3 by addressing comments from Xiaofei:

• do a force kill for abnormal memory failure error such as invalid PA,

unexpected severity, OOM, etc

• pcik up tested-by tag from Ma Wupeng

changes since v2 by addressing comments from Naoya:

• rename mce_task_work to sync_task_work
• drop ACPI_HEST_NOTIFY_MCE case in is_hest_sync_notify()
• add steps to reproduce this problem in cover letter

changes since v1:

• synchronous events by notify type
• Link: https://lore.kernel.org/lkml/20221206153354.92394-3-xueshuai@linux.alibaba.c...

## Cover Letter

There are two major types of uncorrected recoverable (UCR) errors :

• Action Required (AR): The error is detected and the processor already consumes the memory. OS requires to take action (for example, offline failure page/kill failure thread) to recover this error.

• Action Optional (AO): The error is detected out of processor execution context. Some data in the memory are corrupted. But the data have not been consumed. OS is optional to take action to recover this error.

The main difference between AR and AO errors is that AR errors are synchronous events, while AO errors are asynchronous events. Synchronous exceptions, such as Machine Check Exception (MCE) on X86 and Synchronous External Abort (SEA) on Arm64, are signaled by the hardware when an error is detected and the memory access has architecturally been executed.

Currently, both synchronous and asynchronous errors are queued as AO errors and handled by a dedicated kernel thread in a work queue on the ARM64 platform. For synchronous errors, memory_failure() is synced using a cancel_work_sync trick to ensure that the corrupted page is unmapped and poisoned. Upon returning to user-space, the process resumes at the current instruction, triggering a page fault. As a result, the kernel sends a SIGBUS signal to the current process due to VM_FAULT_HWPOISON.

However, this trick is not always be effective, this patch set improves the recovery process in three specific aspects:

1. Handle synchronous exceptions with proper si_code

ghes_handle_memory_failure() queue both synchronous and asynchronous errors with flag=0. Then the kernel will notify the process by sending a SIGBUS signal in memory_failure() with wrong si_code: BUS_MCEERR_AO to the actual user-space process instead of BUS_MCEERR_AR. The user-space processes rely on the si_code to distinguish to handle memory failure.

For example, hwpoison-aware user-space processes use the si_code: BUS_MCEERR_AO for 'action optional' early notifications, and BUS_MCEERR_AR for 'action required' synchronous/late notifications. Specifically, when a signal with SIGBUS_MCEERR_AR is delivered to QEMU, it will inject a vSEA to Guest kernel. In contrast, a signal with SIGBUS_MCEERR_AO will be ignored by QEMU.[1]

Fix it by seting memory failure flags as MF_ACTION_REQUIRED on synchronous events. (PATCH 1)

2. Handle memory_failure() abnormal fails to avoid a unnecessary reboot

If process mapping fault page, but memory_failure() abnormal return before try_to_unmap(), for example, the fault page process mapping is KSM page. In this case, arm64 cannot use the page fault process to terminate the synchronous exception loop.[4]

This loop can potentially exceed the platform firmware threshold or even trigger a kernel hard lockup, leading to a system reboot. However, kernel has the capability to recover from this error.

Fix it by performing a force kill when memory_failure() abnormal fails or when other abnormal synchronous errors occur. These errors can include situations such as invalid PA, unexpected severity, no memory failure config support, invalid GUID section, OOM, etc. (PATCH 2)

3. Handle memory_failure() in current process context which consuming poison

When synchronous errors occur, memory_failure() assume that current process context is exactly that consuming poison synchronous error.

For example, kill_accessing_process() holds mmap locking of current->mm, does pagetable walk to find the error virtual address, and sends SIGBUS to the current process with error info. However, the mm of kworker is not valid, resulting in a null-pointer dereference. I have fixed this in[3].

commit 77677cdbc2aa mm,hwpoison: check mm when killing accessing process

Another example is that collect_procs()/kill_procs() walk the task list, only collect and send sigbus to task which consuming poison. But memory_failure() is queued and handled by a dedicated kernel thread on arm64 platform.

Fix it by queuing memory_failure() as a task work which runs in current execution context to synchronously send SIGBUS before ret_to_user. (PATCH 2)

** In summary, this patch set handles synchronous errors in task work with proper si_code so that hwpoison-aware process can recover from errors, and fixes (potentially) abnormal cases. **

Lv Ying and XiuQi from Huawei also proposed to address similar problem[2][4]. Acknowledge to discussion with them.

## Steps to Reproduce This Problem

To reproduce this problem:

# STEP1: enable early kill mode #sysctl -w vm.memory_failure_early_kill=1 vm.memory_failure_early_kill = 1

# STEP2: inject an UCE error and consume it to trigger a synchronous error #einj_mem_uc single 0: single vaddr = 0xffffb0d75400 paddr = 4092d55b400 injecting ... triggering ... signal 7 code 5 addr 0xffffb0d75000 page not present Test passed

The si_code (code 5) from einj_mem_uc indicates that it is BUS_MCEERR_AO error and it is not fact.

After this patch set:

# STEP1: enable early kill mode #sysctl -w vm.memory_failure_early_kill=1 vm.memory_failure_early_kill = 1

# STEP2: inject an UCE error and consume it to trigger a synchronous error #einj_mem_uc single 0: single vaddr = 0xffffb0d75400 paddr = 4092d55b400 injecting ... triggering ... signal 7 code 4 addr 0xffffb0d75000 page not present Test passed

The si_code (code 4) from einj_mem_uc indicates that it is BUS_MCEERR_AR error as we expected.

[1] Add ARMv8 RAS virtualization support in QEMU https://patchew.org/QEMU/20200512030609.19593-1-gengdongjiu@huawei.com/ [2] https://lore.kernel.org/lkml/20221205115111.131568-3-lvying6@huawei.com/ [3] https://lkml.kernel.org/r/20220914064935.7851-1-xueshuai@linux.alibaba.com [4] https://lore.kernel.org/lkml/20221209095407.383211-1-lvying6@huawei.com/

Shuai Xue (2): ACPI: APEI: set memory failure flags as MF_ACTION_REQUIRED on synchronous events ACPI: APEI: handle synchronous exceptions in task work

arch/x86/kernel/cpu/mce/core.c | 9 +-- drivers/acpi/apei/ghes.c | 113 ++++++++++++++++++++++----------- include/acpi/ghes.h | 3 - include/linux/mm.h | 1 - mm/memory-failure.c | 22 ++----- 5 files changed, 82 insertions(+), 66 deletions(-)

On 2023/11/23 23:07, Borislav Petkov wrote:

Hi, Borislav,

Thank you for your reply and advice.

On Sat, Oct 07, 2023 at 03:28:16PM +0800, Shuai Xue wrote:

...

However, this trick is not always be effective

So far so good.

What's missing here is why "this trick" is not always effective.

Basically to explain what exactly the problem is.

I think the main point is that this trick for AR error is not effective, because:

- an AR error consumed by current process is deferred to handle in a dedicated kernel thread, but memory_failure() assumes that it runs in the current context - another page fault is not unnecessary, we can send sigbus to current process in the first Synchronous External Abort SEA on arm64 (analogy Machine Check Exception on x86)

...

For example, hwpoison-aware user-space processes use the si_code: BUS_MCEERR_AO for 'action optional' early notifications, and BUS_MCEERR_AR for 'action required' synchronous/late notifications. Specifically, when a signal with SIGBUS_MCEERR_AR is delivered to QEMU, it will inject a vSEA to Guest kernel. In contrast, a signal with SIGBUS_MCEERR_AO will be ignored by QEMU.[1]

Fix it by seting memory failure flags as MF_ACTION_REQUIRED on synchronous events. (PATCH 1)

So you're fixing qemu by "fixing" the kernel?

This doesn't make any sense.

I just give an example that the user space process *really* relys on the si_code of signal to handle hardware errors

Make errors which are ACPI_HEST_NOTIFY_SEA type return MF_ACTION_REQUIRED so that it *happens* to fix your use case.

Sounds like a lot of nonsense to me.

What is the issue here you're trying to solve?

The SIGBUS si_codes defined in include/uapi/asm-generic/siginfo.h says:

/* hardware memory error consumed on a machine check: action required */ #define BUS_MCEERR_AR 4 /* hardware memory error detected in process but not consumed: action optional*/ #define BUS_MCEERR_AO 5

When a synchronous error is consumed by Guest, the kernel should send a signal with BUS_MCEERR_AR instead of BUS_MCEERR_AO.

...

2. Handle memory_failure() abnormal fails to avoid a unnecessary reboot

If process mapping fault page, but memory_failure() abnormal return before try_to_unmap(), for example, the fault page process mapping is KSM page. In this case, arm64 cannot use the page fault process to terminate the synchronous exception loop.[4]

This loop can potentially exceed the platform firmware threshold or even trigger a kernel hard lockup, leading to a system reboot. However, kernel has the capability to recover from this error.

Fix it by performing a force kill when memory_failure() abnormal fails or when other abnormal synchronous errors occur.

Just like that?

Without giving the process the opportunity to even save its other data?

Exactly.

So this all is still very confusing, patches definitely need splitting and this whole thing needs restraint.

You go and do this: you split *each* issue you're addressing into a separate patch and explain it like this:

---

1. Prepare the context for the explanation briefly.

2. Explain the problem at hand.

3. "It happens because of <...>"

4. "Fix it by doing X"

5. "(Potentially do Y)."

---

and each patch explains *exactly* *one* issue, what happens, why it happens and just the fix for it and *why* it is needed.

Otherwise, this is unreviewable.

Thank you for your valuable suggestion, I will split the patches and resubmit a new patch set.

Thx.

Best Regards, Shuai

On 2023/11/25 20:10, Borislav Petkov wrote:

Hi, Borislav,

Thank you for your reply, and sorry for the confusion I made. Please see my rely inline.

Best Regards, Shuai

On Sat, Nov 25, 2023 at 02:44:52PM +0800, Shuai Xue wrote:

...

• an AR error consumed by current process is deferred to handle in a dedicated kernel thread, but memory_failure() assumes that it runs in the current context

On x86? ARM?

Pease point to the exact code flow.

An AR error consumed by current process is deferred to handle in a dedicated kernel thread on ARM platform. The AR error is handled in bellow flow:

----------------------------------------------------------------------------- [usr space task einj_mem_uc consumd data poison, CPU 3] STEP 0

----------------------------------------------------------------------------- [ghes_sdei_critical_callback: current einj_mem_uc, CPU 3] STEP 1 ghes_sdei_critical_callback => __ghes_sdei_callback => ghes_in_nmi_queue_one_entry // peak and read estatus => irq_work_queue(&ghes_proc_irq_work) <=> ghes_proc_in_irq // irq_work [ghes_sdei_critical_callback: return] ----------------------------------------------------------------------------- [ghes_proc_in_irq: current einj_mem_uc, CPU 3] STEP 2 => ghes_do_proc => ghes_handle_memory_failure => ghes_do_memory_failure => memory_failure_queue // put work task on current CPU => if (kfifo_put(&mf_cpu->fifo, entry)) schedule_work_on(smp_processor_id(), &mf_cpu->work); => task_work_add(current, &estatus_node->task_work, TWA_RESUME); [ghes_proc_in_irq: return] ----------------------------------------------------------------------------- // kworker preempts einj_mem_uc on CPU 3 due to RESCHED flag STEP 3 [memory_failure_work_func: current kworker, CPU 3] => memory_failure_work_func(&mf_cpu->work) => while kfifo_get(&mf_cpu->fifo, &entry); // until get no work => memory_failure(entry.pfn, entry.flags); ----------------------------------------------------------------------------- [ghes_kick_task_work: current einj_mem_uc, other cpu] STEP 4 => memory_failure_queue_kick => cancel_work_sync - waiting memory_failure_work_func finish => memory_failure_work_func(&mf_cpu->work) => kfifo_get(&mf_cpu->fifo, &entry); // no work ----------------------------------------------------------------------------- [einj_mem_uc resume at the same PC, trigger a page fault STEP 5

STEP 0: A user space task, named einj_mem_uc consume a poison. The firmware notifies hardware error to kernel through is SDEI (ACPI_HEST_NOTIFY_SOFTWARE_DELEGATED).

STEP 1: The swapper running on CPU 3 is interrupted. irq_work_queue() rasie a irq_work to handle hardware errors in IRQ context

STEP2: In IRQ context, ghes_proc_in_irq() queues memory failure work on current CPU in workqueue and add task work to sync with the workqueue.

STEP3: The kworker preempts the current running thread and get CPU 3. Then memory_failure() is processed in kworker.

STEP4: ghes_kick_task_work() is called as task_work to ensure any queued workqueue has been done before returning to user-space.

STEP5: Upon returning to user-space, the task einj_mem_uc resumes at the current instruction, because the poison page is unmapped by memory_failure() in step 3, so a page fault will be triggered.

memory_failure() assumes that it runs in the current context on both x86 and ARM platform.

for example: memory_failure() in mm/memory-failure.c:

if (flags & MF_ACTION_REQUIRED) { folio = page_folio(p); res = kill_accessing_process(current, folio_pfn(folio), flags); }

...

• another page fault is not unnecessary, we can send sigbus to current process in the first Synchronous External Abort SEA on arm64 (analogy Machine Check Exception on x86)

I have no clue what that means. What page fault?

I mean page fault in step 5. We can simplify the above flow by queuing memory_failure() as a task work for AR errors in step 3 directly.

...

I just give an example that the user space process *really* relys on the si_code of signal to handle hardware errors

No, don't give examples.

Explain what the exact problem is you're seeing, in your use case, point to the code and then state how you think it should be fixed and why.

Right now your text is "all over the place" and I have no clue what you even want.

Ok, got it. Thank you.

...

The SIGBUS si_codes defined in include/uapi/asm-generic/siginfo.h says:

/* hardware memory error consumed on a machine check: action required */
#define BUS_MCEERR_AR	4
/* hardware memory error detected in process but not consumed: action optional*/
#define BUS_MCEERR_AO	5

When a synchronous error is consumed by Guest, the kernel should send a signal with BUS_MCEERR_AR instead of BUS_MCEERR_AO.

Can you drop this "synchronous" bla and concentrate on the error *severity*?

I think you want to say that there are some types of errors for which error handling needs to happen immediately and for some reason that doesn't happen.

Which errors are those? Types?

Why do you need them to be handled immediately?

Well, the severity defined on x86 and ARM platform is quite different. I guess you mean taxonomy of producer error types.

- X86: Software recoverable action required (SRAR)

A UCR error that *requires* system software to take a recovery action on this processor *before scheduling another stream of execution on this processor*. (15.6.3 UCR Error Classification in Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3)

- ARM: Recoverable state (UER)

The PE determines that software *must* take action to locate and repair the error to successfully recover execution. This might be because the exception was taken before the error was architecturally consumed by the PE, at the point when the PE was not be able to make correct progress without either consuming the error or *otherwise making the state of the PE unrecoverable*. (2.3.2 PE error state classification in Arm RAS Supplement https://documentation-service.arm.com/static/63185614f72fad1903828eda)

I think above two types of error need to be handled immediately.

...

Exactly.

No, not exactly. Why is it ok to do that? What are the implications of this?

Is immediate killing the right decision?

Is this ok for *every* possible kernel running out there - not only for your use case?

And so on and so on...

I don't have a clear answer here. I guess the poison data only effects the user space task which triggers exception. A panic is not necessary.

On x86 platform, the current error handling of memory_failure() in kill_me_maybe() is just send a sigbus forcely.

kill_me_maybe():

ret = memory_failure(pfn, flags); if (ret == -EHWPOISON || ret == -EOPNOTSUPP) return;

pr_err("Memory error not recovered"); kill_me_now(cb);

Do you have any comments or suggestion about this? I don't change x86 behavior.

For arm64 platform, step 3 in above flow, memory_failure_work_func(), the call site of memory_failure(), does not handle the return code of memory_failure(). I just add the same behavior.