linux-kernel - Re: [PATCH v9 0/2] ACPI: APEI: handle synchronous errors in task work with proper si

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <8cefd789-36da-4208-9511-f826a4508612@linux.alibaba.com>
Date:   Fri, 1 Dec 2023 10:58:42 +0800
From:   Shuai Xue <xueshuai@...ux.alibaba.com>
To:     James Morse <james.morse@....com>, Borislav Petkov <bp@...en8.de>
Cc:     rafael@...nel.org, wangkefeng.wang@...wei.com,
        tanxiaofei@...wei.com, mawupeng1@...wei.com, tony.luck@...el.com,
        linmiaohe@...wei.com, naoya.horiguchi@....com,
        gregkh@...uxfoundation.org, will@...nel.org, jarkko@...nel.org,
        linux-acpi@...r.kernel.org, linux-mm@...ck.org,
        linux-kernel@...r.kernel.org, akpm@...ux-foundation.org,
        linux-edac@...r.kernel.org, acpica-devel@...ts.linuxfoundation.org,
        stable@...r.kernel.org, x86@...nel.org, justin.he@....com,
        ardb@...nel.org, ying.huang@...el.com, ashish.kalra@....com,
        baolin.wang@...ux.alibaba.com, tglx@...utronix.de,
        mingo@...hat.com, dave.hansen@...ux.intel.com, lenb@...nel.org,
        hpa@...or.com, robert.moore@...el.com, lvying6@...wei.com,
        xiexiuqi@...wei.com, zhuo.song@...ux.alibaba.com
Subject: Re: [PATCH v9 0/2] ACPI: APEI: handle synchronous errors in task work
 with proper si_code



On 2023/12/1 01:43, James Morse wrote:
> Hi Boris,
> 
> On 30/11/2023 14:40, Borislav Petkov wrote:
>> FTR, this is starting to make sense, thanks for explaining.
>>
>> Replying only to this one for now:
>>
>> On Thu, Nov 30, 2023 at 10:58:53AM +0800, Shuai Xue wrote:
>>> To reproduce this problem:
>>>
>>> 	# STEP1: enable early kill mode
>>> 	#sysctl -w vm.memory_failure_early_kill=1
>>> 	vm.memory_failure_early_kill = 1
>>>
>>> 	# STEP2: inject an UCE error and consume it to trigger a synchronous error
>>
>> So this is for ARM folks to deal with, BUT:
>>
>> A consumed uncorrectable error on x86 means panic. On some hw like on
>> AMD, that error doesn't even get seen by the OS but the hw does
>> something called syncflood to prevent further error propagation. So
>> there's no any action required - the hw does that.

The "consume" is at the application point of view, e.g. a memory read. If
poison is enable, then a SRAR error will be detected and a MCE raised
at the point of the consumption in the execution flow.

A generic Intel x86 hw behaves like below:

1. UE Error Inject at a known Physical Address. (by einj_mem_uc through EINJ interface)
2. Core Issue a Memory Read to the same Physical Address (by a singe memory read)
3. iMC Detects the error.
4. HA logs UCA error and signals CMCI if enabled
5. HA Forward data with poison indication bit set.
6. CBo detects the Poison data. Does not log any error.
7. MLC detects the Poison data.
8. DCU detects the Poison data, logs SRAR error and trigger MCERR if recoverable
9. OS/VMM takes corresponding recovery action based on affected state.

In our example:
-  step 2 is triggered by a singe memory read.
-  step 8: UCR errors detected on data load, MCACOD 134H, triggering MCERR
-  step 9: the kernel is excepted to send sigbus with si_code BUS_MCEERR_AR (code 4)

I also run the same test  in AMD EPYC platform, e.g. Milan, Genoa, which
behaves the same as Intel Xeon platform, e.g. Icelake, SPR.

The ARMv8.2 RAS extension support similar data poison mechanism, a
Synchronous External Abort on arm64 (analogy Machine Check Exception on
x86) will be trigger in setp 8. See James comments for details.  But the
kernel sends sigbus with si_code BUS_MCEERR_AO (code 5) , tested on
Alibaba Yitian710 and Huawei Kunepng 920.


>>
>> But I'd like to hear from ARM folks whether consuming an uncorrectable
>> error even lets software run. Dunno.
> 
> I think we mean different things by 'consume' here.
> 
> I'd assume Shuai's test is poisoning a cache-line. When the CPU tries to access that
> cache-line it will get an 'external abort' signal back from the memory system. Shuai - is
> this what you mean by 'consume' - the CPU received external abort from the poisoned cache
> line?
> 

Yes, exactly. Thank you for point it out. We are talking about synchronous errors.

> It's then up to the CPU whether it can put the world back in order to take this as
> synchronous-external-abort or asynchronous-external-abort, which for arm64 are two
> different interrupt/exception types.
> The synchronous exceptions can't be masked, but the asynchronous one can.
> If by the time the asynchronous-external-abort interrupt/exception has been unmasked, the
> CPU has used the poisoned value in some calculation (which is what we usually mean by
> consume) which has resulted in a memory access - it will report the error as 'uncontained'
> because the error has been silently propagated. APEI should always report those a 'fatal',
> and there is little point getting the OS involved at this point. Also in this category are
> things like 'tag ram corruption', where you can no longer trust anything about memory.
> 
> Everything in this thread is about synchronous errors where this can't happen. The CPU
> stops and does takes an interrupt/exception instead.
> 
> 

Thank you for explaining.

Best Regards,
Shuai