Message-ID: <b79b74ed-8d52-45a9-80fd-47ed030f362f@arm.com>
Date: Mon, 26 Jan 2026 14:30:14 +0000
From: Ben Horgan <ben.horgan@....com>
To: Catalin Marinas <catalin.marinas@....com>
Cc: amitsinght@...vell.com, baisheng.gao@...soc.com,
 baolin.wang@...ux.alibaba.com, carl@...amperecomputing.com,
 dave.martin@....com, david@...nel.org, dfustini@...libre.com,
 fenghuay@...dia.com, gshan@...hat.com, james.morse@....com,
 jonathan.cameron@...wei.com, kobak@...dia.com, lcherian@...vell.com,
 linux-arm-kernel@...ts.infradead.org, linux-kernel@...r.kernel.org,
 peternewman@...gle.com, punit.agrawal@....qualcomm.com,
 quic_jiles@...cinc.com, reinette.chatre@...el.com, rohit.mathew@....com,
 scott@...amperecomputing.com, sdonthineni@...dia.com,
 tan.shaopeng@...itsu.com, xhao@...ux.alibaba.com, will@...nel.org,
 corbet@....net, maz@...nel.org, oupton@...nel.org, joey.gouly@....com,
 suzuki.poulose@....com, kvmarm@...ts.linux.dev
Subject: Re: [PATCH v3 06/47] arm64: mpam: Context switch the MPAM registers

Hi Catalin,

On 1/23/26 14:29, Catalin Marinas wrote:
> Hi Ben,
> 
> On Mon, Jan 19, 2026 at 12:23:13PM +0000, Ben Horgan wrote:
>> On 1/15/26 17:58, Catalin Marinas wrote:
>>> On Mon, Jan 12, 2026 at 04:58:33PM +0000, Ben Horgan wrote:
>>>> +static inline void mpam_thread_switch(struct task_struct *tsk)
>>>> +{
>>>> +	u64 oldregval;
>>>> +	int cpu = smp_processor_id();
>>>> +	u64 regval = mpam_get_regval(tsk);
>>>> +
>>>> +	if (!static_branch_likely(&mpam_enabled))
>>>> +		return;
>>>> +
>>>> +	if (regval == READ_ONCE(arm64_mpam_global_default))
>>>> +		regval = READ_ONCE(per_cpu(arm64_mpam_default, cpu));
>>>> +
>>>> +	oldregval = READ_ONCE(per_cpu(arm64_mpam_current, cpu));
>>>> +	if (oldregval == regval)
>>>> +		return;
>>>> +
>>>> +	write_sysreg_s(regval, SYS_MPAM1_EL1);
>>>> +	isb();
>>>> +
>>>> +	/* Synchronising the EL0 write is left until the ERET to EL0 */
>>>> +	write_sysreg_s(regval, SYS_MPAM0_EL1);
>>>
>>> Since we have an isb() already, does it make any difference if we write
>>> MPAM0 before the barrier? Similar question for other places where we
>>> write these two registers.
>>
>> The reason for the isb() placement is to document that it's not required
>> for the MPAM0_EL1. All instructions running at EL1take their MPAM
>> configuration from MPAM1_EL1. This includes LDTR and STTR as you asked
>> about in a different thread.
> 
> It's fine to keep it this way if LDTR/STTR are not affected by the MPAM0
> register.
> 
>>>> +
>>>> +	WRITE_ONCE(per_cpu(arm64_mpam_current, cpu), regval);
>>>
>>> Is it too expensive to read the MPAM sysregs and avoid carrying around
>>> another per-CPU state? You use it for pm restoring but we could just
>>> save it in cpu_do_suspend() like other sysregs. Not a big issue, it just
>>> feels like this function got unnecessarily complicated (it took me a bit
>>> to figure out what it all does).
>>
>> It's done this way as it matches what's done in x86. I expect it would
>> be cheap enough to read the register to check whether a write is required.
> 
> Since you use it for CPU suspend/resume, I guess it's fine, it saves us
> from having to preserve it in the low-level asm sleep code. I don't have
> a strong preference either way.
> 
>>> A related question - is resctrl_arch_set_cdp_enabled() always called in
>>> non-preemptible contexts? We potentially have a race between setting
>>> current->mpam_partid_msg and arm64_mpam_global_default, so the check in
>>> mpam_thread_switch() can get confused.
>>
>> The resctrl filesystem can only ever get mounted once and
>> resctrl_arch_set_cdp_enabled() is always called in mount. In
>> rdt_get_tree(), rdt_enable_ctx() calls resctrl_arch_set_cdp_enabled().
>> This is done while holding the rdtgroup_mutex and the user can't change
>> the mpam configuration from the default until the mutex is released and
>> it can claim it.
> 
> What if resctrl_arch_set_cdp_enabled() gets preempted between updating
> current task partid and setting arm64_mpam_global_default? The mutex
> doesn't help.

I see, I had misinterpreted your question.

When looking into this I spotted that the per-cpu default,
arm64_mpam_default, is not updated on the cdp enable and so the default
tasks carry on running with the non-cdp default pmg/partid configuration.

On the race itself, if current->mpam_partid_pmg is set but not
arm64_mpam_global_default then if the current task is preempted at that
point there is a discrepancy. When the task gets context switched back
in then the comparison in mpam_thread_switch() will be false when true
would be expected.

In order to update arm64_mpam_default and make sure the mpam variables
and registers for the online cpus are in the right state by the end of
resctrl_arch_set_cdp_enabled() I'll add a per cpu call to
resctrl_arch_sync_cpu_closid_rmid(). I'll also add something in the
resume path so that arm64_mpam_default gets set correctly for the cpus
that were offline.

> 
>>> And I couldn't figure out where the MPAMx_EL1 registers are written. If
>>> any global/per-cpu/per-task value is changed, does the kernel wait until
>>> the next thread switch to write the sysreg? The only places I can found
>>> touching these sysregs are the thread switch, pm notifiers and KVM.
>>
>> If the task configuration is changed then the MPAM sysreg will only be
>> updated the next time it goes into the kernel.
> 
> Is it updated when it goes into the kernel or when we have a context
> switch? If you have a long-running user thread that is never scheduled
> out, it may not notice the update even if it entered the kernel (but no
> context switch).

Yes, only on context switch.

> 
>> So, just eventually
>> consistent. For cpu configuration, update_closid_rmid() gets called
>> which ipis the cpus and ends up calling mpam_thread_switch() from
>> resctrl_arch_sched_in().
> 
> I see, it should be fine.
> 

Thanks,

Ben

Powered by blists - more mailing lists