[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <d4873add-ac24-47dd-be45-28af3698c334@intel.com>
Date: Fri, 16 Aug 2024 15:33:35 -0700
From: Reinette Chatre <reinette.chatre@...el.com>
To: Babu Moger <babu.moger@....com>, <corbet@....net>, <fenghua.yu@...el.com>,
<tglx@...utronix.de>, <mingo@...hat.com>, <bp@...en8.de>,
<dave.hansen@...ux.intel.com>
CC: <x86@...nel.org>, <hpa@...or.com>, <paulmck@...nel.org>,
<rdunlap@...radead.org>, <tj@...nel.org>, <peterz@...radead.org>,
<yanjiewtw@...il.com>, <kim.phillips@....com>, <lukas.bulwahn@...il.com>,
<seanjc@...gle.com>, <jmattson@...gle.com>, <leitao@...ian.org>,
<jpoimboe@...nel.org>, <rick.p.edgecombe@...el.com>,
<kirill.shutemov@...ux.intel.com>, <jithu.joseph@...el.com>,
<kai.huang@...el.com>, <kan.liang@...ux.intel.com>,
<daniel.sneddon@...ux.intel.com>, <pbonzini@...hat.com>,
<sandipan.das@....com>, <ilpo.jarvinen@...ux.intel.com>,
<peternewman@...gle.com>, <maciej.wieczor-retman@...el.com>,
<linux-doc@...r.kernel.org>, <linux-kernel@...r.kernel.org>,
<eranian@...gle.com>, <james.morse@....com>
Subject: Re: [PATCH v6 22/22] x86/resctrl: Introduce interface to modify
assignment states of the groups
Hi Babu,
On 8/6/24 3:00 PM, Babu Moger wrote:
> Introduce the interface to assign MBM events in ABMC mode.
>
> Events can be enabled or disabled by writing to file
> /sys/fs/resctrl/info/L3_MON/mbm_control
>
> Format is similar to the list format with addition of opcode for the
> assignment operation.
> "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
>
> Format for specific type of groups:
>
> * Default CTRL_MON group:
> "//<domain_id><opcode><flags>"
>
> * Non-default CTRL_MON group:
> "<CTRL_MON group>//<domain_id><opcode><flags>"
>
> * Child MON group of default CTRL_MON group:
> "/<MON group>/<domain_id><opcode><flags>"
>
> * Child MON group of non-default CTRL_MON group:
> "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
>
> Domain_id '*' will apply the flags on all the domains.
>
> Opcode can be one of the following:
>
> = Update the assignment to match the flags
> + assign a MBM event
> - unassign a MBM event
>
> Assignment flags can be one of the following:
> t MBM total event
> l MBM local event
> tl Both total and local MBM events
> _ None of the MBM events. Valid only with '=' opcode.
(please note comments in this area of cover letter)
>
> Signed-off-by: Babu Moger <babu.moger@....com>
> ---
> v6: Added support assign all if domain id is '*'
> Fixed the allocation of counter id if it not assigned already.
>
> v5: Interface name changed from mbm_assign_control to mbm_control.
> Fixed opcode and flags combination.
> '=_" is valid.
> "-_" amd "+_" is not valid.
> Minor message update.
> Renamed the function with prefix - rdtgroup_.
> Corrected few documentation mistakes.
> Rebase related changes after SNC support.
>
> v4: Added domain specific assignments. Fixed the opcode parsing.
>
> v3: New patch.
> Addresses the feedback to provide the global assignment interface.
> https://lore.kernel.org/lkml/c73f444b-83a1-4e9a-95d3-54c5165ee782@intel.com/
> ---
> Documentation/arch/x86/resctrl.rst | 94 +++++++-
> arch/x86/kernel/cpu/resctrl/internal.h | 7 +
> arch/x86/kernel/cpu/resctrl/rdtgroup.c | 315 ++++++++++++++++++++++++-
> 3 files changed, 414 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
> index 113c22ba6db3..ae3b17b7cefe 100644
> --- a/Documentation/arch/x86/resctrl.rst
> +++ b/Documentation/arch/x86/resctrl.rst
> @@ -346,7 +346,7 @@ with the following files:
> t MBM total event is enabled.
> l MBM local event is enabled.
> tl Both total and local MBM events are enabled.
> - _ None of the MBM events are enabled.
> + _ None of the MBM events are enabled. Only works with opcode '=' for write.
>
> Examples:
> ::
> @@ -365,6 +365,98 @@ with the following files:
> enabled on domain 0 and 1.
>
>
> + Assignment state can be updated by writing to the interface.
> +
> + Format is similar to the list format with addition of opcode for the
> + assignment operation.
> +
> + "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
> +
> + Format for each type of groups:
> +
> + * Default CTRL_MON group:
> + "//<domain_id><opcode><flags>"
> +
> + * Non-default CTRL_MON group:
> + "<CTRL_MON group>//<domain_id><opcode><flags>"
> +
> + * Child MON group of default CTRL_MON group:
> + "/<MON group>/<domain_id><opcode><flags>"
> +
> + * Child MON group of non-default CTRL_MON group:
> + "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
> +
> + Domain_id '*' wil apply the flags on all the domains.
> +
> + Opcode can be one of the following:
> + ::
> +
> + = Update the assignment to match the MBM event.
> + + Assign a MBM event.
"Assign a new MBM event without impacting existing assignments."?
> + - Unassign a MBM event.
(similar)
> +
> + Examples:
> + ::
> +
> + Initial group status:
> + # cat /sys/fs/resctrl/info/L3_MON/mbm_control
> + non_default_ctrl_mon_grp//0=tl;1=tl;
> + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl;
> + //0=tl;1=tl;
> + /child_default_mon_grp/0=tl;1=tl;
> +
> + To update the default group to assign only total MBM event on domain 0:
> + # echo "//0=t" > /sys/fs/resctrl/info/L3_MON/mbm_control
> +
> + Assignment status after the update:
> + # cat /sys/fs/resctrl/info/L3_MON/mbm_control
> + non_default_ctrl_mon_grp//0=tl;1=tl;
> + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl;
> + //0=t;1=tl;
> + /child_default_mon_grp/0=tl;1=tl;
> +
> + To update the MON group child_default_mon_grp to remove total MBM event on domain 1:
> + # echo "/child_default_mon_grp/1-t" > /sys/fs/resctrl/info/L3_MON/mbm_control
> +
> + Assignment status after the update:
> + $ cat /sys/fs/resctrl/info/L3_MON/mbm_control
> + non_default_ctrl_mon_grp//0=tl;1=tl;
> + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl;
> + //0=t;1=tl;
> + /child_default_mon_grp/0=tl;1=l;
> +
> + To update the MON group non_default_ctrl_mon_grp/child_non_default_mon_grp to
> + unassign both local and total MBM events on domain 1:
> + # echo "non_default_ctrl_mon_grp/child_non_default_mon_grp/1=_" >
> + /sys/fs/resctrl/info/L3_MON/mbm_control
> +
> + Assignment status after the update:
> + non_default_ctrl_mon_grp//0=tl;1=tl;
> + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_;
> + //0=t;1=tl;
> + /child_default_mon_grp/0=tl;1=l;
> +
> + To update the default group to add a local MBM event domain 0.
> + # echo "//0+l" > /sys/fs/resctrl/info/L3_MON/mbm_control
> +
> + Assignment status after the update:
> + # cat /sys/fs/resctrl/info/L3_MON/mbm_control
> + non_default_ctrl_mon_grp//0=tl;1=tl;
> + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_;
> + //0=tl;1=tl;
> + /child_default_mon_grp/0=tl;1=l;
> +
> + To update the non default CTRL_MON group non_default_ctrl_mon_grp to unassign all
> + the MBM events on all the domains.
> + # echo "non_default_ctrl_mon_grp//*=_" > /sys/fs/resctrl/info/L3_MON/mbm_control
> +
> + Assignment status after the update:
> + #cat /sys/fs/resctrl/info/L3_MON/mbm_control
> + non_default_ctrl_mon_grp//0=_;1=_;
> + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_;
> + //0=tl;1=tl;
> + /child_default_mon_grp/0=tl;1=l;
> +
> "max_threshold_occupancy":
> Read/write file provides the largest value (in
> bytes) at which a previously used LLC_occupancy
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index ba3012f8f940..5af225b4a497 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -46,6 +46,13 @@
>
> #define MON_CNTR_UNSET U32_MAX
>
> +/*
> + * Assignment flags for ABMC feature
(this is resctrl fs code)
> + */
> +#define ASSIGN_NONE 0
> +#define ASSIGN_TOTAL BIT(QOS_L3_MBM_TOTAL_EVENT_ID)
> +#define ASSIGN_LOCAL BIT(QOS_L3_MBM_LOCAL_EVENT_ID)
> +
> /**
> * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that
> * aren't marked nohz_full
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index d7aadca5e4ab..8567fb3a6274 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -1034,6 +1034,318 @@ static int rdtgroup_mbm_control_show(struct kernfs_open_file *of,
> return 0;
> }
>
> +/*
> + * Update the assign states for the domain.
> + *
> + * If this is a new assignment for the group then allocate a counter and update
> + * the assignment else just update the assign state
> + */
> +static int rdtgroup_assign_update(struct rdtgroup *rdtgrp, enum resctrl_event_id evtid,
> + struct rdt_mon_domain *d)
> +{
> + int ret, index;
> +
> + index = mon_event_config_index_get(evtid);
> + if (index == INVALID_CONFIG_INDEX)
> + return -EINVAL;
(wrong spacing ... see checkpatch.pl)
> +
> + if (rdtgrp->mon.cntr_id[index] == MON_CNTR_UNSET) {
> + ret = rdtgroup_alloc_cntr(rdtgrp, index);
> + if (ret < 0)
> + goto out_done;
> + }
> +
> + /* Update the state on all domains if d == NULL */
> + if (d == NULL) {
if (!d) ... (checkpatch)
> + ret = rdtgroup_assign_cntr(rdtgrp, evtid);
> + } else {
> + ret = resctrl_arch_assign_cntr(d, evtid, rdtgrp->mon.rmid,
> + rdtgrp->mon.cntr_id[index],
> + rdtgrp->closid, 1);
> + if (!ret)
> + set_bit(rdtgrp->mon.cntr_id[index], d->mbm_cntr_map);
> + }
> +
> +out_done:
> + return ret;
> +}
Please merge this with almost identical rdtgroup_assign_cntr()
> +
> +/*
> + * Update the unassign state for the domain.
> + *
> + * Free the counter if it is unassigned on all the domains else just
> + * update the unassign state
> + */
> +static int rdtgroup_unassign_update(struct rdtgroup *rdtgrp, enum resctrl_event_id evtid,
> + struct rdt_mon_domain *d)
> +{
> + struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
> + int ret = 0, index;
> +
> + index = mon_event_config_index_get(evtid);
> + if (index == INVALID_CONFIG_INDEX)
> + return -EINVAL;
(wrong spacing ... see checkpatch.pl)
> +
> + if (rdtgrp->mon.cntr_id[index] == MON_CNTR_UNSET)
> + goto out_done;
> +
> + if (d == NULL) {
if (!d)
> + ret = rdtgroup_unassign_cntr(rdtgrp, evtid);
> + } else {
> + ret = resctrl_arch_assign_cntr(d, evtid, rdtgrp->mon.rmid,
> + rdtgrp->mon.cntr_id[index],
> + rdtgrp->closid, 0);
> + if (!ret) {
> + clear_bit(rdtgrp->mon.cntr_id[index], d->mbm_cntr_map);
> + rdtgroup_free_cntr(r, rdtgrp, index);
> + }
> + }
> +
> +out_done:
> + return ret;
> +}
Please merge this with almost identical rdtgroup_unassign_cntr()
> +
> +static int rdtgroup_str_to_mon_state(char *flag)
> +{
> + int i, mon_state = 0;
> +
> + for (i = 0; i < strlen(flag); i++) {
> + switch (*(flag + i)) {
> + case 't':
> + mon_state |= ASSIGN_TOTAL;
> + break;
> + case 'l':
> + mon_state |= ASSIGN_LOCAL;
> + break;
> + case '_':
> + mon_state = ASSIGN_NONE;
> + break;
It looks like this supports flags like "_lt", treating it as assigning
both local and total. I expect this should remove all flags instead?
> + default:
> + break;
> + }
> + }
> +
> + return mon_state;
> +}
hmmm ... so you removed assigning mon_state to ASSIGN_NONE from default,
but that did not change what this function returns since ASSIGN_NONE is 0
and mon_state is initialized to 0. Unknown flags should cause error so
that it is possible to add flags in the future. Above prevents us from
ever adding new flags.
> +
> +static struct rdtgroup *rdtgroup_find_grp(enum rdt_group_type rtype, char *p_grp, char *c_grp)
rdtgroup_find_grp() -> rdtgroup_find_grp_by_name()?
> +{
> + struct rdtgroup *rdtg, *crg;
> +
> + if (rtype == RDTCTRL_GROUP && *p_grp == '\0') {
> + return &rdtgroup_default;
> + } else if (rtype == RDTCTRL_GROUP) {
> + list_for_each_entry(rdtg, &rdt_all_groups, rdtgroup_list)
> + if (!strcmp(p_grp, rdtg->kn->name))
> + return rdtg;
> + } else if (rtype == RDTMON_GROUP) {
> + list_for_each_entry(rdtg, &rdt_all_groups, rdtgroup_list) {
> + if (!strcmp(p_grp, rdtg->kn->name)) {
> + list_for_each_entry(crg, &rdtg->mon.crdtgrp_list,
> + mon.crdtgrp_list) {
> + if (!strcmp(c_grp, crg->kn->name))
> + return crg;
> + }
> + }
> + }
> + }
> +
> + return NULL;
> +}
> +
> +static int rdtgroup_process_flags(struct rdt_resource *r,
> + enum rdt_group_type rtype,
> + char *p_grp, char *c_grp, char *tok)
> +{
> + int op, mon_state, assign_state, unassign_state;
> + char *dom_str, *id_str, *op_str;
> + struct rdt_mon_domain *d;
> + struct rdtgroup *rdtgrp;
> + unsigned long dom_id;
> + int ret, found = 0;
> +
> + rdtgrp = rdtgroup_find_grp(rtype, p_grp, c_grp);
> +
> + if (!rdtgrp) {
> + rdt_last_cmd_puts("Not a valid resctrl group\n");
> + return -EINVAL;
> + }
> +
> +next:
> + if (!tok || tok[0] == '\0')
> + return 0;
> +
> + /* Start processing the strings for each domain */
> + dom_str = strim(strsep(&tok, ";"));
> +
> + op_str = strpbrk(dom_str, "=+-");
> +
> + if (op_str) {
> + op = *op_str;
> + } else {
> + rdt_last_cmd_puts("Missing operation =, +, -, _ character\n");
> + return -EINVAL;
> + }
> +
> + id_str = strsep(&dom_str, "=+-");
> +
> + /* Check for domain id '*' which means all domains */
> + if (id_str && *id_str == '*') {
> + d = NULL;
> + goto check_state;
> + } else if (!id_str || kstrtoul(id_str, 10, &dom_id)) {
> + rdt_last_cmd_puts("Missing domain id\n");
> + return -EINVAL;
> + }
> +
> + /* Verify if the dom_id is valid */
> + list_for_each_entry(d, &r->mon_domains, hdr.list) {
> + if (d->hdr.id == dom_id) {
> + found = 1;
> + break;
> + }
> + }
> +
> + if (!found) {
> + rdt_last_cmd_printf("Invalid domain id %ld\n", dom_id);
> + return -EINVAL;
> + }
> +
> +check_state:
> + mon_state = rdtgroup_str_to_mon_state(dom_str);
Function should return error and exit here.
> +
> + assign_state = 0;
> + unassign_state = 0;
> +
> + switch (op) {
> + case '+':
> + if (mon_state == ASSIGN_NONE) {
> + rdt_last_cmd_puts("Invalid assign opcode\n");
> + goto out_fail;
> + }
> + assign_state = mon_state;
> + break;
> + case '-':
> + if (mon_state == ASSIGN_NONE) {
> + rdt_last_cmd_puts("Invalid assign opcode\n");
> + goto out_fail;
> + }
> + unassign_state = mon_state;
> + break;
> + case '=':
> + assign_state = mon_state;
> + unassign_state = (ASSIGN_TOTAL | ASSIGN_LOCAL) & ~assign_state;
> + break;
> + default:
> + break;
> + }
> +
> + if (assign_state & ASSIGN_TOTAL) {
> + ret = rdtgroup_assign_update(rdtgrp, QOS_L3_MBM_TOTAL_EVENT_ID, d);
> + if (ret)
> + goto out_fail;
> + }
Should unassign occur before assign so that unassign can make counters available for
assign that follows?
> +
> + if (assign_state & ASSIGN_LOCAL) {
> + ret = rdtgroup_assign_update(rdtgrp, QOS_L3_MBM_LOCAL_EVENT_ID, d);
> + if (ret)
> + goto out_fail;
> + }
> +
> + if (unassign_state & ASSIGN_TOTAL) {
> + ret = rdtgroup_unassign_update(rdtgrp, QOS_L3_MBM_TOTAL_EVENT_ID, d);
> + if (ret)
> + goto out_fail;
> + }
> +
> + if (unassign_state & ASSIGN_LOCAL) {
> + ret = rdtgroup_unassign_update(rdtgrp, QOS_L3_MBM_LOCAL_EVENT_ID, d);
> + if (ret)
> + goto out_fail;
> + }
> +
> + goto next;
> +
> +out_fail:
> +
> + return -EINVAL;
> +}
> +
> +static ssize_t rdtgroup_mbm_control_write(struct kernfs_open_file *of,
> + char *buf, size_t nbytes,
> + loff_t off)
> +{
> + struct rdt_resource *r = of->kn->parent->priv;
> + char *token, *cmon_grp, *mon_grp;
> + int ret;
> +
> + if (!resctrl_arch_get_abmc_enabled())
> + return -EINVAL;
This needs to be protected by mutex.
> +
> + /* Valid input requires a trailing newline */
> + if (nbytes == 0 || buf[nbytes - 1] != '\n')
> + return -EINVAL;
> +
> + buf[nbytes - 1] = '\0';
> +
> + cpus_read_lock();
> + mutex_lock(&rdtgroup_mutex);
> + rdt_last_cmd_clear();
> +
> + while ((token = strsep(&buf, "\n")) != NULL) {
> + if (strstr(token, "//")) {
> + /*
> + * The CTRL_MON group processing:
> + * default CTRL_MON group: "//<flags>"
> + * non-default CTRL_MON group: "<CTRL_MON group>//flags"
> + * The CTRL_MON group will be empty string if it is a
> + * default group.
> + */
> + cmon_grp = strsep(&token, "//");
> +
> + /*
> + * strsep returns empty string for contiguous delimiters.
> + * Make sure check for two consecutive delimiters and
> + * advance the token.
> + */
> + mon_grp = strsep(&token, "//");
> + if (*mon_grp != '\0') {
> + rdt_last_cmd_printf("Invalid CTRL_MON group format %s\n", token);
> + ret = -EINVAL;
> + break;
> + }
> +
> + ret = rdtgroup_process_flags(r, RDTCTRL_GROUP, cmon_grp, mon_grp, token);
> + if (ret)
> + break;
> + } else if (strstr(token, "/")) {
> + /*
> + * MON group processing:
> + * MON_GROUP inside default CTRL_MON group: "/<MON group>/<flags>"
> + * MON_GROUP within CTRL_MON group: "<CTRL_MON group>/<MON group>/<flags>"
> + */
> + cmon_grp = strsep(&token, "/");
Isn't strsep(&token, "//") the same as strsep(&token, "/")? It looks like these two big branches
can be merged.
> +
> + /* Extract the MON_GROUP. It cannot be empty string */
> + mon_grp = strsep(&token, "/");
> + if (*mon_grp == '\0') {
> + rdt_last_cmd_printf("Invalid MON_GROUP format %s\n", token);
> + ret = -EINVAL;
> + break;
> + }
> +
> + ret = rdtgroup_process_flags(r, RDTMON_GROUP, cmon_grp, mon_grp, token);
> + if (ret)
> + break;
> + }
> + }
> +
> + mutex_unlock(&rdtgroup_mutex);
> + cpus_read_unlock();
> +
> + return ret ?: nbytes;
> +}
> +
> #ifdef CONFIG_PROC_CPU_RESCTRL
>
> /*
> @@ -2277,9 +2589,10 @@ static struct rftype res_common_files[] = {
> },
> {
> .name = "mbm_control",
> - .mode = 0444,
> + .mode = 0644,
> .kf_ops = &rdtgroup_kf_single_ops,
> .seq_show = rdtgroup_mbm_control_show,
> + .write = rdtgroup_mbm_control_write,
> },
> {
> .name = "cpus_list",
Reinette
Powered by blists - more mailing lists