Message-ID: <Z7cw4JTp3Hfx/4li@e133380.arm.com>
Date: Thu, 20 Feb 2025 13:40:48 +0000
From: Dave Martin <Dave.Martin@....com>
To: Peter Newman <peternewman@...gle.com>
Cc: Reinette Chatre <reinette.chatre@...el.com>,
	Babu Moger <babu.moger@....com>, corbet@....net, tglx@...utronix.de,
	mingo@...hat.com, bp@...en8.de, dave.hansen@...ux.intel.com,
	tony.luck@...el.com, fenghua.yu@...el.com, x86@...nel.org,
	hpa@...or.com, paulmck@...nel.org, akpm@...ux-foundation.org,
	thuth@...hat.com, rostedt@...dmis.org, xiongwei.song@...driver.com,
	pawan.kumar.gupta@...ux.intel.com, daniel.sneddon@...ux.intel.com,
	jpoimboe@...nel.org, perry.yuan@....com, sandipan.das@....com,
	kai.huang@...el.com, xiaoyao.li@...el.com, seanjc@...gle.com,
	xin3.li@...el.com, andrew.cooper3@...rix.com, ebiggers@...gle.com,
	mario.limonciello@....com, james.morse@....com,
	tan.shaopeng@...itsu.com, linux-doc@...r.kernel.org,
	linux-kernel@...r.kernel.org, maciej.wieczor-retman@...el.com,
	eranian@...gle.com
Subject: Re: [PATCH v11 17/23] x86/resctrl: Auto assign/unassign counters
 when mbm_cntr_assign is enabled

On Thu, Feb 20, 2025 at 11:35:56AM +0100, Peter Newman wrote:
> Hi Reinette,
> 
> On Wed, Feb 19, 2025 at 6:55 PM Reinette Chatre
> <reinette.chatre@...el.com> wrote:
> >
> > Hi Dave and Peter,
> >
> > On 2/19/25 6:09 AM, Peter Newman wrote:
> > > Hi Dave,
> > >
> > > On Wed, Feb 19, 2025 at 2:41 PM Dave Martin <Dave.Martin@....com> wrote:
> > >>
> > >> Hi,
> > >>
> > >> On Wed, Jan 22, 2025 at 02:20:25PM -0600, Babu Moger wrote:
> > >>> Assign/unassign counters on resctrl group creation/deletion. Two counters
> > >>> are required per group, one for MBM total event and one for MBM local
> > >>> event.
> > >>>
> > >>> There are a limited number of counters available for assignment. If these
> > >>> counters are exhausted, the kernel will display the error message: "Out of
> > >>> MBM assignable counters". However, it is not necessary to fail the
> > >>> creation of a group due to assignment failures. Users have the flexibility
> > >>> to modify the assignments at a later time.
> > >>
> > >> If we are doing this, should turning mbm_cntr_assign mode on also
> > >> trigger auto-assingment for all extant monitoring groups?
> > >>
> > >> Either way though, this auto-assignment feels like a potential nuisance
> > >> for userspace.
> >
> > hmmm ... this auto-assignment was created with the goal to help userspace.
> > In mbm_cntr_assign mode the user will only see data when a counter is assigned
> > to an event. mbm_cntr_assign mode is selected as default on a system that
> > supports ABMC. Without auto assignment a user will thus see different
> > behavior when reading the monitoring events when the user switches to a kernel with
> > assignable counter support: Before assignable counter support events will have
> > data, with assignable counter support the events will not have data.
> >
> > I understood that interfaces should not behave differently when user space
> > switches kernels and that is what the auto assignment aims to solve.
> >
> > >>
> > >> If the userspace use-case requires too many monitoring groups for the
> > >> available counters, then the kernel will auto-assign counters to a
> > >> random subset of groups which may or may not be the ones that userspace
> > >> wanted to monitor; then userspace must manually look for the assigned
> > >> counters and unassign some of them before they can be assigned where
> > >> userspace actually wanted them.
> > >>
> > >> This is not impossible for userspace to cope with, but it feels
> > >> awkward.
> > >>
> > >> Is there a way to inhibit auto-assignment?
> > >>
> > >> Or could automatic assignments be considered somehow "weak", so that
> > >> new explicit assignments can steal automatically assigned counters
> > >> without the need to unassign them explicitly?
> > >
> > > We had an incomplete discussion about this early on[1]. I guess I
> > > didn't revisit it because I found it was trivial to add a flag that
> > > inhibits the assignment behavior during mkdir and had moved on to
> > > bigger issues.
> >
> > Could you please remind me how a user will set this flag?
> 
> Quoting my original suggestion[1]:
> 
>  "info/L3_MON/mbm_assign_on_mkdir?
> 
>   boolean (parsed with kstrtobool()), defaulting to true?"
> 
> After mount, any groups that got counters on creation would have to be
> cleaned up, but at least that can be done with forward progress once
> the flag is cleared.
> 
> I was able to live with that as long as there aren't users polling for
> resctrl to be mounted and immediately creating groups. For us, a
> single container manager service manages resctrl.
> 
> >
> > >
> > > If an agent creating directories isn't coordinated with the agent
> > > managing counters, a series of creating and destroying a group could
> > > prevent a monitor assignment from ever succeeding because it's not
> > > possible to atomically discover the name of the new directory that
> > > stole the previously-available counter and reassign it.
> > >
> > > However, if the counter-manager can get all the counters assigned once
> > > and only move them with atomic reassignments, it will become
> > > impossible to snatch them with a mkdir.
> > >
> >
> > You have many points that makes auto-assignment not be ideal but I
> > remain concerned that not doing something like this will break
> > existing users who are not as familiar with resctrl internals.
> 
> I agree auto-assignment should be the default. I just want an official
> way to turn it off.
> 
> Thanks!
> -Peter
> 
> [1] https://lore.kernel.org/lkml/CALPaoCiJ9ELXkij-zsAhxC1hx8UUR+KMPJH6i8c8AT6_mtXs+Q@mail.gmail.com/
> 

+1

That's basically my position -- the auto-assignment feels like a
_potential_ nuisance for ABMC-aware users, but it depends on what they
are trying to do.  Migration of non-ABMC-aware users will be easier for
basic use cases if auto-assignment occurs by default (as in this
series).

Having an explicit way to turn this off seems perfectly reasonable
(and could be added later on, if not provided in this series).


What about the question re whether turning mbm_cntr_assign mode on
should trigger auto-assignment?

Currently turning this mode off and then on again has the effect of
removing all automatic assignments for extant groups.  This feels
surprising and/or unintentional (?)

Cheers
---Dave

Powered by blists - more mailing lists