Message-ID: <e185be25-797a-4e98-b2b9-4acc963c321f@open-hieco.net>
Date: Sun, 7 Dec 2025 00:14:12 +0800
From: Xiaochen Shen <shenxiaochen@...n-hieco.net>
To: "Luck, Tony" <tony.luck@...el.com>,
 "Chatre, Reinette" <reinette.chatre@...el.com>, "bp@...en8.de"
 <bp@...en8.de>, "fenghuay@...dia.com" <fenghuay@...dia.com>
Cc: "babu.moger@....com" <babu.moger@....com>,
 "james.morse@....com" <james.morse@....com>,
 "Dave.Martin@....com" <Dave.Martin@....com>, "x86@...nel.org"
 <x86@...nel.org>, "linux-kernel@...r.kernel.org"
 <linux-kernel@...r.kernel.org>, shenxiaochen@...n-hieco.net
Subject: Re: [PATCH 2/2] x86/resctrl: Fix memory bandwidth counter width for
 Hygon

Hi Tony,

On 12/6/2025 4:33 AM, Luck, Tony wrote:
>> We have observed a test case where an incorrect counter width leads to random unexpected memory bandwidth readings:
>> https://github.com/shenxiaochen/my_documents/blob/main/memory_bandwidth_counter_width_and_overflow_issue_steps_to_reproduce.txt
>>
>> The issue was resolved by applying this patch.
> Clearly something is going wrong, and you sometime see enormous values for
> memory bandwidth. But I'm still puzzled about what is going wrong.
> 
> I pasted the resctrl wraparound function into a small user-mode test, and it
> seems to be able to ignore bits above the width used.
> 
> The test below prints "2" when told the width is either 24 or 32.
> 

--The test output:----
# ./counter_width
width = 24 2
width = 32 2
----------------------


> Your patch to use width = 32 is good, but if the problem isn't in the mbm_overflow_count()
> function, then you might just have made the problem 256X harder to hit.
> 

>From my understanding, mbm_overflow_count() works as expected if a correct counter width is passed as a parameter.

In my opinion, the root cause is that:
The incorrect counter width (24-bits) is passed to mbm_overflow_count(), which is much smaller than the hardware counter width (32-bits).
As a result, the data between bit 24 and bit 32 of the counter is *discarded* unexpectedly in bandwidth delta calculation in this scenario:
(1) Kernel firstly reads the hardware counter.
(2) Kernel secondly reads the hardware counter with 1 second interval.
Between (1) and (2), if 32-bits hardware counter really hits overflow, mbm_overflow_count() still handles with 24-bits counter width as a parameter:

static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width)
{
        u64 shift = 64 - width, chunks;

        chunks = (cur_msr << shift) - (prev_msr << shift);
        return chunks >> shift;
}

The calculated bandwidth delta is incorrect at this time, because the data between bit 24 and bit 32 of the counter is *discarded* unexpectedly.

See more debugging information as below.


> Question: What is the value of "hw_res->mon_scale" on a Hygon system?

I have double confirmed with Hygon hardware architect, in the testing Hygon system:
hw_res->mon_scale: 64
hw_res->mbm_width: 32

Here is a rough calculation for the theoretical max bandwidth by the hardware counter:
(1) 32-bits width and mon_scale is 64:
2 ^32 * 64 = 2 ^38 = 256 (B/s)

(2) 24-bits width and mon_scale is 64 (this is highly likely to cause overflow):
2 ^24 * 64 = 2 ^30 = 1G (B/s)


FYI - debugging for the overflow issue:
--------------------------------------------
I run the test case in [1] again, try to capture more useful information:
[1]https://github.com/shenxiaochen/my_documents/blob/main/memory_bandwidth_counter_width_and_overflow_issue_steps_to_reproduce.txt

# ./mbm_total_verbose.sh
...
// Normal data
total b/w (bytes/s): 31192093760 (77687336665701824 - 77687305473608064)
total b/w (bytes/s): 31205362048 (77687367871063872 - 77687336665701824)
77687336665701824 / 64 = 1213864635401591 (0x45000E2660577)
77687367871063872 / 64 = 1213865122985373 (0x45000FF75F59D)

// Unexpected calculated bandwidth value, hardware should hit overflow:
total b/w (bytes/s): 1125656235801792 (78813024106865664 - 77687367871063872)  

// The data analysis:
// Before overflow, low 32-bits of the counter is close to overflow (0xFF75F59D). Will hit hardware overflow immediately!
Read 1: 77687367871063872 / 64 = 1213865122985373 (0x45000FF75F59D)

// 1 second later, after overflow, the low 32-bits of the counter is (0x1C864190).
Read 2: 78813024106865664 / 64 = 1231453501669776 (0x460001C864190)

// The calculated delta is incorrect (the data between bit 24 and bit 32 of the counter is discarded unexpectedly). We see the unexpected bandwidth value:
total b/w (bytes/s): 1125656235801792 (78813024106865664 - 77687367871063872)


Best regards,
Xiaochen Shen

Powered by blists - more mailing lists