linux-kernel - Re: [PATCH v9 19/26] x86/resctrl: Add the interface to unassign a MBM counter

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <6514f004-c83b-422e-a31d-f9e1f399cb5f@amd.com>
Date: Tue, 5 Nov 2024 13:58:15 -0600
From: "Moger, Babu" <babu.moger@....com>
To: Peter Newman <peternewman@...gle.com>
Cc: corbet@....net, reinette.chatre@...el.com, tglx@...utronix.de,
 mingo@...hat.com, bp@...en8.de, dave.hansen@...ux.intel.com,
 fenghua.yu@...el.com, x86@...nel.org, hpa@...or.com, thuth@...hat.com,
 paulmck@...nel.org, rostedt@...dmis.org, akpm@...ux-foundation.org,
 xiongwei.song@...driver.com, pawan.kumar.gupta@...ux.intel.com,
 daniel.sneddon@...ux.intel.com, perry.yuan@....com, sandipan.das@....com,
 kai.huang@...el.com, xiaoyao.li@...el.com, seanjc@...gle.com,
 jithu.joseph@...el.com, brijesh.singh@....com, xin3.li@...el.com,
 ebiggers@...gle.com, andrew.cooper3@...rix.com, mario.limonciello@....com,
 james.morse@....com, tan.shaopeng@...itsu.com, tony.luck@...el.com,
 linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org,
 maciej.wieczor-retman@...el.com, eranian@...gle.com, jpoimboe@...nel.org,
 thomas.lendacky@....com
Subject: Re: [PATCH v9 19/26] x86/resctrl: Add the interface to unassign a MBM
 counter

Hi Peter,

On 11/5/24 04:35, Peter Newman wrote:
> Hi Babu,
> 
> On Mon, Nov 4, 2024 at 7:21 PM Moger, Babu <babu.moger@....com> wrote:
>>
>> Hi Peter,
>>
>> On 11/4/24 08:16, Peter Newman wrote:
>>> Hi Babu,
>>>
>>> On Wed, Oct 30, 2024 at 12:25 AM Babu Moger <babu.moger@....com> wrote:
>>>>
>>>> The mbm_cntr_assign mode provides a limited number of hardware counters
>>>> that can be assigned to an RMID, event pair to monitor bandwidth while
>>>> assigned. If all counters are in use, the kernel will show an error
>>>> message: "Out of MBM assignable counters" when a new assignment is
>>>> requested. To make space for a new assignment, users must unassign an
>>>> already assigned counter.
>>>>
>>>> Introduce an interface that allows for the unassignment of counter IDs
>>>> from both the group and the domain. Additionally, ensure that the global
>>>> counter is released if it is no longer assigned to any domains.
>>>
>>> This seems unnecessarily restrictive. What's wrong with monitoring
>>> different groups in different domains?
>>
>> Yes. User can monitor different groups in different domains. But, they
>> will have to use different global counter for each group.
> 
> What is a global counter anyways? It sounds like an artifact of an
> earlier revision. This concept does not sound intuitive to the user.


# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
32

This is a global counter. We have totally 32 hardware counters.

This is tracked by the bitmap mbm_cntr_free_map.


> 
>>
>> Here is an example.
>>
>> #cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>> groupA//0=t;1=_;
>> groupB//0=_;1=l;
>>
>> Group A - counter 0 (Assigned to total event in Domain 0)
>> Group B - counter 1 (Assigned to local event in Domain 1)
>>
>> We allocate two different counters here.  Now we are left with 30 counters
>> (max 32).
>>
>>
>> This is similar to CLOSID management we follow in resctrl. This is not a
>> new restriction,
> 
> It is a restriction in a new feature that resembles a restriction in
> an existing feature.
> 
> I don't see what function the global allocator serves now that there
> is already a per-domain allocator. My best guess is that it avoids the
> case of an mbm_assign_control write that succeeds in some domains but
> fails in others.
> 
> I admit I said earlier that I was only planning to allocate globally,
> but now that I'm evaluating how to make resctrl's monitoring
> functionality scale on large systems, I'm being forced to reconsider.
> 
> As long as this is only a limitation I can fix later, I don't see it
> as an obstacle. There would just need to be better documentation of
> what sort of internal data structures the user needs to visualize in
> order to use this feature successfully.


We have totally 32 global counters. That means we can assign up to 32 events.

Assigning events requires sending an IPI to write the MSR
(MSR_IA32_L3_QOS_ABMC_CFG) on every domain affected.

So, we wanted another bitmap to track if status of the assignment on each
domain. This is tracked by mbm_cntr_map. This bit is updated when we send
the IPI on that domain.

I dont consider this as a limitation. This helps to avoid sending
unnecessary IPIs to all the domains when user wants to assign an event.
This is kind of improvement I would say.

We still have the option to applying the assignment to all the domains by
setting "*" for the domain.

-- 
Thanks
Babu Moger