Message-ID: <f9fa8335c52444398e1736854c887fb7@huawei.com>
Date: Mon, 24 Mar 2025 10:37:08 +0000
From: Shiju Jose <shiju.jose@...wei.com>
To: Jonathan Cameron <jonathan.cameron@...wei.com>
CC: "linux-cxl@...r.kernel.org" <linux-cxl@...r.kernel.org>,
	"dan.j.williams@...el.com" <dan.j.williams@...el.com>, "dave@...olabs.net"
	<dave@...olabs.net>, "dave.jiang@...el.com" <dave.jiang@...el.com>,
	"alison.schofield@...el.com" <alison.schofield@...el.com>,
	"vishal.l.verma@...el.com" <vishal.l.verma@...el.com>, "ira.weiny@...el.com"
	<ira.weiny@...el.com>, "david@...hat.com" <david@...hat.com>,
	"Vilas.Sridharan@....com" <Vilas.Sridharan@....com>,
	"linux-edac@...r.kernel.org" <linux-edac@...r.kernel.org>,
	"linux-acpi@...r.kernel.org" <linux-acpi@...r.kernel.org>,
	"linux-mm@...ck.org" <linux-mm@...ck.org>, "linux-kernel@...r.kernel.org"
	<linux-kernel@...r.kernel.org>, "bp@...en8.de" <bp@...en8.de>,
	"tony.luck@...el.com" <tony.luck@...el.com>, "rafael@...nel.org"
	<rafael@...nel.org>, "lenb@...nel.org" <lenb@...nel.org>,
	"mchehab@...nel.org" <mchehab@...nel.org>, "leo.duran@....com"
	<leo.duran@....com>, "Yazen.Ghannam@....com" <Yazen.Ghannam@....com>,
	"rientjes@...gle.com" <rientjes@...gle.com>, "jiaqiyan@...gle.com"
	<jiaqiyan@...gle.com>, "Jon.Grimm@....com" <Jon.Grimm@....com>,
	"dave.hansen@...ux.intel.com" <dave.hansen@...ux.intel.com>,
	"naoya.horiguchi@....com" <naoya.horiguchi@....com>, "james.morse@....com"
	<james.morse@....com>, "jthoughton@...gle.com" <jthoughton@...gle.com>,
	"somasundaram.a@....com" <somasundaram.a@....com>, "erdemaktas@...gle.com"
	<erdemaktas@...gle.com>, "pgonda@...gle.com" <pgonda@...gle.com>,
	"duenwen@...gle.com" <duenwen@...gle.com>, "gthelen@...gle.com"
	<gthelen@...gle.com>, "wschwartz@...erecomputing.com"
	<wschwartz@...erecomputing.com>, "dferguson@...erecomputing.com"
	<dferguson@...erecomputing.com>, "wbs@...amperecomputing.com"
	<wbs@...amperecomputing.com>, "nifan.cxl@...il.com" <nifan.cxl@...il.com>,
	tanxiaofei <tanxiaofei@...wei.com>, "Zengtao (B)" <prime.zeng@...ilicon.com>,
	Roberto Sassu <roberto.sassu@...wei.com>, "kangkang.shen@...urewei.com"
	<kangkang.shen@...urewei.com>, wanghuiqiang <wanghuiqiang@...wei.com>,
	Linuxarm <linuxarm@...wei.com>
Subject: RE: [PATCH v2 2/8] EDAC: Update documentation for the CXL memory
 patrol scrub control feature


>-----Original Message-----
>From: Jonathan Cameron <jonathan.cameron@...wei.com>
>Sent: 21 March 2025 10:03
>To: Shiju Jose <shiju.jose@...wei.com>
>Cc: linux-cxl@...r.kernel.org; dan.j.williams@...el.com; dave@...olabs.net;
>dave.jiang@...el.com; alison.schofield@...el.com; vishal.l.verma@...el.com;
>ira.weiny@...el.com; david@...hat.com; Vilas.Sridharan@....com; linux-
>edac@...r.kernel.org; linux-acpi@...r.kernel.org; linux-mm@...ck.org; linux-
>kernel@...r.kernel.org; bp@...en8.de; tony.luck@...el.com; rafael@...nel.org;
>lenb@...nel.org; mchehab@...nel.org; leo.duran@....com;
>Yazen.Ghannam@....com; rientjes@...gle.com; jiaqiyan@...gle.com;
>Jon.Grimm@....com; dave.hansen@...ux.intel.com;
>naoya.horiguchi@....com; james.morse@....com; jthoughton@...gle.com;
>somasundaram.a@....com; erdemaktas@...gle.com; pgonda@...gle.com;
>duenwen@...gle.com; gthelen@...gle.com;
>wschwartz@...erecomputing.com; dferguson@...erecomputing.com;
>wbs@...amperecomputing.com; nifan.cxl@...il.com; tanxiaofei
><tanxiaofei@...wei.com>; Zengtao (B) <prime.zeng@...ilicon.com>; Roberto
>Sassu <roberto.sassu@...wei.com>; kangkang.shen@...urewei.com;
>wanghuiqiang <wanghuiqiang@...wei.com>; Linuxarm
><linuxarm@...wei.com>
>Subject: Re: [PATCH v2 2/8] EDAC: Update documentation for the CXL memory
>patrol scrub control feature
>
>On Thu, 20 Mar 2025 18:04:39 +0000
><shiju.jose@...wei.com> wrote:
>
>> From: Shiju Jose <shiju.jose@...wei.com>
>>
>> Update the Documentation/edac/scrub.rst to include descriptions and
>> policies for CXL memory device-based and CXL region-based patrol scrub
>> control.
>>
>> Note: This may require inputs from CXL memory experts regarding
>> region-based scrubbing policies.
>
>So I suggested the region interfaces in the first place.  It's all about usecases and
>'why' we might increase the scrub rate.
>Ultimately the hardware is controlled in a device wide way, so we could have
>made it complex userspace problem to deal with it on a perf device.
>The region interfaces are there as a simplification not because they are strictly
>necessary.
>
>Anyhow, the use cases:
>
>1) Scrubbing because a device is showing unexpectedly high errors.  That
>   control needs to be at device granularity.  If one device in an interleave
>   set (backing a region) is dodgy, why make them all do more work?
>
>2) Scrubbing may apply to memory that isn't online at all yet.  Nice to know
>   if we have a problem before we start using it!  Likely this is setting
>   system wide defaults on boot.
>
>3) Scrubbing at higher rate because software has decided that we want
>   more reliability for particular data.  I've been calling this
>   Differentiated Reliability.  That data sits in a region which
>   may cover part of multiple devices. The region interfaces are about
>   supporting this use case.
>
>So now the question is what do we do if both interfaces are poked because
>someone cares simultaneously about 1 and 3?
>
>I'd suggest just laying out a set for rules on how to set the scrub rates for any
>mixture of requirements, rather than making the driver work out the optimum
>combination.
>
>>
>> Signed-off-by: Shiju Jose <shiju.jose@...wei.com>
>> ---
>>  Documentation/edac/scrub.rst | 47
>> ++++++++++++++++++++++++++++++++++++
>>  1 file changed, 47 insertions(+)
>>
>> diff --git a/Documentation/edac/scrub.rst
>> b/Documentation/edac/scrub.rst index daab929cdba1..d1c02bd90090 100644
>> --- a/Documentation/edac/scrub.rst
>> +++ b/Documentation/edac/scrub.rst
>> @@ -264,3 +264,51 @@ Sysfs files are documented in
>> `Documentation/ABI/testing/sysfs-edac-scrub`
>>
>>  `Documentation/ABI/testing/sysfs-edac-ecs`
>> +
>> +Examples
>> +--------
>> +
>> +The usage takes the form shown in these examples:
>> +
>> +1. CXL memory device patrol scrubber
>> +
>> +1.1 Device based scrubbing
>> +
>> +CXL memory is exposed to memory management subsystem and ultimately
>> +userspace via CXL devices.
>> +
>> +For cases where hardware interleave controls do not directly map to
>> +regions of Physical Address space, perhaps due to interleave the
>> +approach described in
>> +1.2 Region based scrubbing section, which is specific to CXL regions
>> +should be followed.
>
>These sentences end up a bit unwieldy. Perhaps simply a forwards reference.
>
>When combining control via the device interfaces and region interfaces see
>1.2 Region bases scrubbing.
>
>
>
>> In those cases settings on the presented interface may interact with
>> +direct control via a device instance specific interface and care must be taken.
>> +
>> +Sysfs files for scrubbing are documented in
>> +`Documentation/ABI/testing/sysfs-edac-scrub`
>> +
>> +1.2. Region based scrubbing
>> +
>> +CXL memory is exposed to memory management subsystem and ultimately
>> +userspace via CXL regions. CXL Regions represent mapped memory
>> +capacity in system physical address space. These can incorporate one
>> +or more parts of multiple CXL memory devices with traffic interleaved
>> +across them. The user may want to control the scrub rate via this
>> +more abstract region instead of having to figure out the constituent
>> +devices and program them separately. The scrub rate for each device
>> +covers the whole device. Thus if multiple regions use parts of that
>> +device then requests for scrubbing of other regions may result in a higher
>scrub rate than requested for this specific region.
>> +
>> +1. When user sets scrub rate for a memory region, the scrub rate for all the
>CXL
>> +   memory devices interleaved under that region is updated with the same
>scrub
>> +   rate.
>
>Note that this may affect multiple regions.
>
>> +
>> +2. When user sets scrub rate for a memory device, only the scrub rate for
>that
>> +   memory devices is updated though device may be part of a memory region
>and
>> +   does not change scrub rate of other memory devices of that memory
>region.
>> +
>> +3. Scrub rate of a CXL memory device may be set via EDAC device or region
>scrub
>> +   interface simultaneously. Care must be taken to prevent a race condition,
>or
>> +   only region-based setting may be allowed.
>
>So is this saying if you want to mix and match, set region first then device next?
>Can we just lay out the rules to set up a weird mixture.  We could add more
>smarts to the driver but do we care as mixing 1 and 3 above is probably
>unusual?
>
>1. Taking each region in turn from lowest desired scrub rate to highest and set
>   their scrub rates.  Later regions may override the scrub rate on individual
>   devices (and hence potentially whole regions).
>
>2. Take each device for which enhanced scrubbing is required (higher rate) and
>   set those scrub rates.  This will override the scrub rates of individual devices
>   leaving any that are not specifically set to scrub at the maximum rate required
>   for any of the regions they are involved in backing.

Thanks. Will incorporate these info and rules in the next version.
>
>
>> +
>> +Sysfs files for scrubbing are documented in
>> +`Documentation/ABI/testing/sysfs-edac-scrub`

Shiju

Powered by blists - more mailing lists