linux-kernel - Re: [PATCH] EDAC/{i10nm,skx,skx_common}: Support multiple clumps

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <Z1Jieg7ACUMZLsF-@hpe.com>
Date: Thu, 5 Dec 2024 20:33:30 -0600
From: Kyle Meyer <kyle.meyer@....com>
To: "Zhuo, Qiuxu" <qiuxu.zhuo@...el.com>
Cc: "Luck, Tony" <tony.luck@...el.com>, "bp@...en8.de" <bp@...en8.de>,
        "james.morse@....com" <james.morse@....com>,
        "mchehab@...nel.org" <mchehab@...nel.org>,
        "rric@...nel.org" <rric@...nel.org>,
        "linux-edac@...r.kernel.org" <linux-edac@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH] EDAC/{i10nm,skx,skx_common}: Support multiple clumps

On Fri, Dec 06, 2024 at 01:26:12AM +0000, Zhuo, Qiuxu wrote:
> > From: Kyle Meyer <kyle.meyer@....com>
> > Sent: Friday, December 6, 2024 8:57 AM
> > To: Luck, Tony <tony.luck@...el.com>
> > Cc: bp@...en8.de; james.morse@....com; mchehab@...nel.org;
> > rric@...nel.org; linux-edac@...r.kernel.org; linux-kernel@...r.kernel.org
> > Subject: Re: [PATCH] EDAC/{i10nm,skx,skx_common}: Support multiple
> > clumps
> > 
> > On Thu, Dec 05, 2024 at 03:57:55PM -0800, Luck, Tony wrote:
> > > > +int skx_get_src_id(struct skx_dev *d, int off, u8 *id) { #ifdef
> > > > +CONFIG_NUMA
> > > > +	return skx_get_pkg_id(d, id);
> > > > +#else
> > > > +	u32 reg;
> > > > +
> > > > +	if (pci_read_config_dword(d->util_all, off, &reg)) {
> > > > +		skx_printk(KERN_ERR, "Failed to read src id\n");
> > > > +		return -ENODEV;
> > > > +	}
> > > > +
> > > > +	*id = GET_BITFIELD(reg, 12, 14);
> > > > +	return 0;
> > > > +#endif
> > >
> > > Doh ... I alwasy forget about IS_ENABLED(). This can be written:
> > >
> > >
> > > int skx_get_src_id(struct skx_dev *d, int off, u8 *id) {
> > > 	u32 reg;
> > >
> > > 	if (IS_ENABLED(CONFIG_NUMA))
> > > 		return skx_get_pkg_id(d, id);
> > >
> > > 	if (pci_read_config_dword(d->util_all, off, &reg)) {
> > > 		skx_printk(KERN_ERR, "Failed to read src id\n");
> > > 		return -ENODEV;
> > > 	}
> > >
> > > 	*id = GET_BITFIELD(reg, 12, 14);
> > > 	return 0;
> > > }
> > 
> > Looks good.
> > 
> > > 1) Does this work? I tried on a non-clumpy system that is NUMA.
> > 
> > Yes, I just tested this on a Sapphire Rapids system with multiple UPI domains.
> > 
> > > 2) Is it better (assuming #fidef factored off into a .h file)?
> > 
> > IMO, yes, but there's one subtle difference. EDAC will not load on systems that
> > have a single UPI domain when CONFIG_NUMA is enabled but numa=off,
> > because
> > pcibus_to_node() in skx_get_pkg_id() will return NUMA_NO_NODE (-1). Is
> > that a case that we need to worry about?
> 
> I think we need to make the EDAC load/work even in this case. 
> Regardless of CONFIG_NUMA or whether numa=off is set or not, could we do it like this:
> 
> int skx_get_src_id(struct skx_dev *d, int off, u8 *id)
> {
>         u32 reg;
> 
>         if (!skx_get_pkg_id(d, id))
>                 return 0;
> 
>         if (pci_read_config_dword(d->util_all, off, &reg)) {
>                 skx_printk(KERN_ERR, "Failed to read src id\n");
>                 return -ENODEV;
>         }
> 
>         *id = GET_BITFIELD(reg, 12, 14);
>         return 0;
> }

So, we're back to the original issue on systems with multiple UPI/QPI domains
when NUMA is disabled.

Systems with multiple UPI/QPI domains can't use source IDs to map devices
to sockets. skx_get_src_id() will successfully read the source ID from PCI
configuration space registers but it might not map to the correct socket because
each UPI/QPI domain has identical repeating source IDs.

For example, 8 sockets with 2 UPI/QPI domains:

Socket 0 -> Source ID 0
Socket 1 -> Source ID 1
Socket 2 -> Source ID 2
Socket 3 -> Source ID 3
Socket 4 -> Source ID 0
Socket 5 -> Source ID 1
Socket 6 -> Source ID 2
Socket 7 -> Source ID 3

EDAC will successfully load, but it will not find the the corresponding device
for errors on sockets 4 though 7 (for example, see skx_common.c:178).

Looking at my original patch, EDAC will not load when a system has multiple UPI/
QPI domains and NUMA is disabled. We fail early with "Failed to get package ID
from NUMA information" instead of later when an error occurs.

Thanks,
Kyle Meyer