[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <d5d7706c-0c12-46bf-9a4d-9bd80db2c83f@amd.com>
Date: Mon, 21 Oct 2024 12:04:28 -0500
From: "Moger, Babu" <babu.moger@....com>
To: Reinette Chatre <reinette.chatre@...el.com>, corbet@....net,
fenghua.yu@...el.com, tglx@...utronix.de, mingo@...hat.com, bp@...en8.de,
dave.hansen@...ux.intel.com
Cc: x86@...nel.org, hpa@...or.com, paulmck@...nel.org, rdunlap@...radead.org,
tj@...nel.org, peterz@...radead.org, yanjiewtw@...il.com,
kim.phillips@....com, lukas.bulwahn@...il.com, seanjc@...gle.com,
jmattson@...gle.com, leitao@...ian.org, jpoimboe@...nel.org,
rick.p.edgecombe@...el.com, kirill.shutemov@...ux.intel.com,
jithu.joseph@...el.com, kai.huang@...el.com, kan.liang@...ux.intel.com,
daniel.sneddon@...ux.intel.com, pbonzini@...hat.com, sandipan.das@....com,
ilpo.jarvinen@...ux.intel.com, peternewman@...gle.com,
maciej.wieczor-retman@...el.com, linux-doc@...r.kernel.org,
linux-kernel@...r.kernel.org, eranian@...gle.com, james.morse@....com
Subject: Re: [PATCH v8 25/25] x86/resctrl: Introduce interface to modify
assignment states of the groups
Hi Reinette,
On 10/15/24 22:43, Reinette Chatre wrote:
> Hi Babu,
>
> On 10/9/24 10:39 AM, Babu Moger wrote:
>> Introduce the interface to assign MBM events in mbm_cntr_assign mode.
>>
>> Events can be enabled or disabled by writing to file
>> /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>>
>> Format is similar to the list format with addition of opcode for the
>> assignment operation.
>> "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
>>
>> Format for specific type of groups:
>>
>> * Default CTRL_MON group:
>> "//<domain_id><opcode><flags>"
>>
>> * Non-default CTRL_MON group:
>> "<CTRL_MON group>//<domain_id><opcode><flags>"
>>
>> * Child MON group of default CTRL_MON group:
>> "/<MON group>/<domain_id><opcode><flags>"
>>
>> * Child MON group of non-default CTRL_MON group:
>> "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
>>
>> Domain_id '*' will apply the flags on all the domains.
>>
>> Opcode can be one of the following:
>>
>> = Update the assignment to match the flags
>> + Assign a new MBM event without impacting existing assignments.
>> - Unassign a MBM event from currently assigned events.
>>
>> Assignment flags can be one of the following:
>> t MBM total event
>> l MBM local event
>> tl Both total and local MBM events
>> _ None of the MBM events. Valid only with '=' opcode. This flag cannot
>> be combined with other flags.
>>
>> Signed-off-by: Babu Moger <babu.moger@....com>
>> ---
>> v8: Moved unassign as the first action during the assign modification.
>> Assign none "_" takes priority. Cannot be mixed with other flags.
>> Updated the documentation and .rst file format. htmldoc looks ok.
>>
>> v7: Simplified the parsing (strsep(&token, "//") in rdtgroup_mbm_assign_control_write().
>> Added mutex lock in rdtgroup_mbm_assign_control_write() while processing.
>> Renamed rdtgroup_find_grp to rdtgroup_find_grp_by_name.
>> Fixed rdtgroup_str_to_mon_state to return error for invalid flags.
>> Simplified the calls rdtgroup_assign_cntr by merging few functions earlier.
>> Removed ABMC reference in FS code.
>> Reinette commented about handling the combination of flags like 'lt_' and '_lt'.
>> Not sure if we need to change the behaviour here. Processed them sequencially right now.
>> Users have the liberty to pass the flags. Restricting it might be a problem later.
>>
>> v6: Added support assign all if domain id is '*'
>> Fixed the allocation of counter id if it not assigned already.
>>
>> v5: Interface name changed from mbm_assign_control to mbm_control.
>> Fixed opcode and flags combination.
>> '=_" is valid.
>> "-_" amd "+_" is not valid.
>> Minor message update.
>> Renamed the function with prefix - rdtgroup_.
>> Corrected few documentation mistakes.
>> Rebase related changes after SNC support.
>>
>> v4: Added domain specific assignments. Fixed the opcode parsing.
>>
>> v3: New patch.
>> Addresses the feedback to provide the global assignment interface.
>> https://lore.kernel.org/lkml/c73f444b-83a1-4e9a-95d3-54c5165ee782@intel.com/
>> ---
>> Documentation/arch/x86/resctrl.rst | 115 +++++++++++-
>> arch/x86/kernel/cpu/resctrl/internal.h | 10 ++
>> arch/x86/kernel/cpu/resctrl/rdtgroup.c | 233 ++++++++++++++++++++++++-
>> 3 files changed, 356 insertions(+), 2 deletions(-)
>>
>> diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
>> index b85d3bc3e301..77bb0b095127 100644
>> --- a/Documentation/arch/x86/resctrl.rst
>> +++ b/Documentation/arch/x86/resctrl.rst
>> @@ -336,7 +336,8 @@ with the following files:
>> t MBM total event is assigned.
>> l MBM local event is assigned.
>> tl Both total and local MBM events are assigned.
>> - _ None of the MBM events are assigned.
>> + _ None of the MBM events are assigned. Only works with opcode '=' for write
>> + and cannot be combined with other flags.
>>
>> Examples:
>> ::
>> @@ -354,6 +355,118 @@ with the following files:
>> There are four resctrl groups. All the groups have total and local MBM events
>> assigned on domain 0 and 1.
>>
>> + Assignment state can be updated by writing to the interface.
>> +
>> + Format is similar to the list format with addition of opcode for the
>> + assignment operation.
>> +
>> + "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
>> +
>> + Format for each type of groups:
>> +
>> + * Default CTRL_MON group:
>> + "//<domain_id><opcode><flags>"
>> +
>> + * Non-default CTRL_MON group:
>> + "<CTRL_MON group>//<domain_id><opcode><flags>"
>> +
>> + * Child MON group of default CTRL_MON group:
>> + "/<MON group>/<domain_id><opcode><flags>"
>> +
>> + * Child MON group of non-default CTRL_MON group:
>> + "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
>> +
>> + Domain_id '*' will apply the flags on all the domains.
>> +
>> + Opcode can be one of the following:
>> + ::
>> +
>> + = Update the assignment to match the MBM event.
>> + + Assign a new MBM event without impacting existing assignments.
>> + - Unassign a MBM event from currently assigned events.
>> +
>> + Examples:
>> + Initial group status:
>> + ::
>> +
>> + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>> + non_default_ctrl_mon_grp//0=tl;1=tl;
>> + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl;
>> + //0=tl;1=tl;
>> + /child_default_mon_grp/0=tl;1=tl;
>> +
>> + To update the default group to assign only total MBM event on domain 0:
>> + ::
>> +
>> + # echo "//0=t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>> +
>> + Assignment status after the update:
>> + ::
>> +
>> + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>> + non_default_ctrl_mon_grp//0=tl;1=tl;
>> + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl;
>> + //0=t;1=tl;
>> + /child_default_mon_grp/0=tl;1=tl;
>> +
>> + To update the MON group child_default_mon_grp to remove total MBM event on domain 1:
>> + ::
>> +
>> + # echo "/child_default_mon_grp/1-t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>> +
>> + Assignment status after the update:
>> + ::
>> +
>> + $ cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>
> Please be consistent by always using "# cat", not sometimes "$ cat" as above.
Sure.
>
>> + non_default_ctrl_mon_grp//0=tl;1=tl;
>> + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl;
>> + //0=t;1=tl;
>> + /child_default_mon_grp/0=tl;1=l;
>> +
>> + To update the MON group non_default_ctrl_mon_grp/child_non_default_mon_grp to unassign
>> + both local and total MBM events on domain 1:
>> + ::
>> +
>> + # echo "non_default_ctrl_mon_grp/child_non_default_mon_grp/1=_" >
>> + /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>> +
>> + Assignment status after the update:
>> + ::
>> +
>
> Missing "# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control"
Sure.
>
>> + non_default_ctrl_mon_grp//0=tl;1=tl;
>> + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_;
>> + //0=t;1=tl;
>> + /child_default_mon_grp/0=tl;1=l;
>> +
>> + To update the default group to add a local MBM event domain 0.
>> + ::
>> +
>> + # echo "//0+l" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>> +
>> + Assignment status after the update:
>> + ::
>> +
>> + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>> + non_default_ctrl_mon_grp//0=tl;1=tl;
>> + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_;
>> + //0=tl;1=tl;
>> + /child_default_mon_grp/0=tl;1=l;
>> +
>> + To update the non default CTRL_MON group non_default_ctrl_mon_grp to unassign all the
>> + MBM events on all the domains.
>> + ::
>> +
>> + # echo "non_default_ctrl_mon_grp//*=_" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>> +
>> + Assignment status after the update:
>> + ::
>> +
>> + #cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
>
> Please be consistent with spacing "# cat" vs "#cat". This is very noticeable when
> viewing the formatted docs.
Sure.
>
>> + non_default_ctrl_mon_grp//0=_;1=_;
>> + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_;
>> + //0=tl;1=tl;
>> + /child_default_mon_grp/0=tl;1=l;
>> +
>> "max_threshold_occupancy":
>> Read/write file provides the largest value (in
>> bytes) at which a previously used LLC_occupancy
>> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
>> index a6f40d3115f4..e8d6a430dc4a 100644
>> --- a/arch/x86/kernel/cpu/resctrl/internal.h
>> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
>> @@ -74,6 +74,16 @@
>> */
>> #define MBM_EVENT_ARRAY_INDEX(_event) ((_event) - 2)
>>
>> +/*
>> + * Assignment flags for mbm_cntr_assign feature
>> + */
>> +enum {
>> + ASSIGN_NONE = 0,
>> + ASSIGN_TOTAL = BIT(QOS_L3_MBM_TOTAL_EVENT_ID),
>> + ASSIGN_LOCAL = BIT(QOS_L3_MBM_LOCAL_EVENT_ID),
>> + ASSIGN_INVALID,
>> +};
>> +
>> /**
>> * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that
>> * aren't marked nohz_full
>> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> index cf92ceb0f05e..6095146e3ba4 100644
>> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> @@ -1040,6 +1040,236 @@ static int rdtgroup_mbm_assign_control_show(struct kernfs_open_file *of,
>> return 0;
>> }
>>
>> +static int rdtgroup_str_to_mon_state(char *flag)
>> +{
>> + int i, mon_state = ASSIGN_NONE;
>> +
>> + for (i = 0; i < strlen(flag); i++) {
>> + switch (*(flag + i)) {
>> + case 't':
>> + mon_state |= ASSIGN_TOTAL;
>> + break;
>> + case 'l':
>> + mon_state |= ASSIGN_LOCAL;
>> + break;
>> + case '_':
>> + return ASSIGN_NONE;
>> + default:
>> + return ASSIGN_INVALID;
>> + }
>> + }
>> +
>> + return mon_state;
>> +}
>> +
>> +static struct rdtgroup *rdtgroup_find_grp_by_name(enum rdt_group_type rtype,
>> + char *p_grp, char *c_grp)
>> +{
>> + struct rdtgroup *rdtg, *crg;
>> +
>> + if (rtype == RDTCTRL_GROUP && *p_grp == '\0') {
>> + return &rdtgroup_default;
>> + } else if (rtype == RDTCTRL_GROUP) {
>> + list_for_each_entry(rdtg, &rdt_all_groups, rdtgroup_list)
>> + if (!strcmp(p_grp, rdtg->kn->name))
>> + return rdtg;
>> + } else if (rtype == RDTMON_GROUP) {
>> + list_for_each_entry(rdtg, &rdt_all_groups, rdtgroup_list) {
>> + if (!strcmp(p_grp, rdtg->kn->name)) {
>> + list_for_each_entry(crg, &rdtg->mon.crdtgrp_list,
>> + mon.crdtgrp_list) {
>> + if (!strcmp(c_grp, crg->kn->name))
>> + return crg;
>> + }
>> + }
>> + }
>> + }
>> +
>> + return NULL;
>> +}
>> +
>> +static int rdtgroup_process_flags(struct rdt_resource *r,
>> + enum rdt_group_type rtype,
>> + char *p_grp, char *c_grp, char *tok)
>> +{
>> + int op, mon_state, assign_state, unassign_state;
>> + char *dom_str, *id_str, *op_str;
>> + struct rdt_mon_domain *d;
>> + struct rdtgroup *rdtgrp;
>> + unsigned long dom_id;
>> + int ret, found = 0;
>> +
>> + rdtgrp = rdtgroup_find_grp_by_name(rtype, p_grp, c_grp);
>> +
>> + if (!rdtgrp) {
>> + rdt_last_cmd_puts("Not a valid resctrl group\n");
>> + return -EINVAL;
>> + }
>> +
>> +next:
>> + if (!tok || tok[0] == '\0')
>> + return 0;
>> +
>> + /* Start processing the strings for each domain */
>> + dom_str = strim(strsep(&tok, ";"));
>> +
>> + op_str = strpbrk(dom_str, "=+-");
>> +
>> + if (op_str) {
>> + op = *op_str;
>> + } else {
>> + rdt_last_cmd_puts("Missing operation =, +, - character\n");
>> + return -EINVAL;
>> + }
>> +
>> + id_str = strsep(&dom_str, "=+-");
>> +
>> + /* Check for domain id '*' which means all domains */
>> + if (id_str && *id_str == '*') {
>> + d = NULL;
>> + goto check_state;
>> + } else if (!id_str || kstrtoul(id_str, 10, &dom_id)) {
>> + rdt_last_cmd_puts("Missing domain id\n");
>> + return -EINVAL;
>> + }
>> +
>> + /* Verify if the dom_id is valid */
>> + list_for_each_entry(d, &r->mon_domains, hdr.list) {
>> + if (d->hdr.id == dom_id) {
>> + found = 1;
>> + break;
>> + }
>> + }
>> +
>> + if (!found) {
>> + rdt_last_cmd_printf("Invalid domain id %ld\n", dom_id);
>> + return -EINVAL;
>> + }
>> +
>> +check_state:
>> + mon_state = rdtgroup_str_to_mon_state(dom_str);
>> +
>> + if (mon_state == ASSIGN_INVALID) {
>> + rdt_last_cmd_puts("Invalid assign flag\n");
>> + goto out_fail;
>> + }
>> +
>> + assign_state = 0;
>> + unassign_state = 0;
>> +
>> + switch (op) {
>> + case '+':
>> + if (mon_state == ASSIGN_NONE) {
>> + rdt_last_cmd_puts("Invalid assign opcode\n");
>> + goto out_fail;
>> + }
>> + assign_state = mon_state;
>> + break;
>> + case '-':
>> + if (mon_state == ASSIGN_NONE) {
>> + rdt_last_cmd_puts("Invalid assign opcode\n");
>> + goto out_fail;
>> + }
>> + unassign_state = mon_state;
>> + break;
>> + case '=':
>> + assign_state = mon_state;
>> + unassign_state = (ASSIGN_TOTAL | ASSIGN_LOCAL) & ~assign_state;
>> + break;
>> + default:
>> + break;
>> + }
>> +
>> + if (unassign_state & ASSIGN_TOTAL) {
>> + ret = rdtgroup_unassign_cntr_event(r, rdtgrp, d, QOS_L3_MBM_TOTAL_EVENT_ID);
>> + if (ret)
>> + goto out_fail;
>> + }
>> +
>> + if (unassign_state & ASSIGN_LOCAL) {
>> + ret = rdtgroup_unassign_cntr_event(r, rdtgrp, d, QOS_L3_MBM_LOCAL_EVENT_ID);
>> + if (ret)
>> + goto out_fail;
>> + }
>> +
>> + if (assign_state & ASSIGN_TOTAL) {
>> + ret = rdtgroup_assign_cntr_event(r, rdtgrp, d, QOS_L3_MBM_TOTAL_EVENT_ID);
>> + if (ret)
>> + goto out_fail;
>> + }
>> +
>> + if (assign_state & ASSIGN_LOCAL) {
>> + ret = rdtgroup_assign_cntr_event(r, rdtgrp, d, QOS_L3_MBM_LOCAL_EVENT_ID);
>> + if (ret)
>> + goto out_fail;
>> + }
>> +
>> + goto next;
>> +
>> +out_fail:
>
> Is it possible to print a message to the command status to give some details about which
> request failed? I am wondering about a scenario where a user changes multiple domains of
> multiple groups, since the operation does not undo changes, it will fail without information
> to user space about which setting triggered the failure and which settings succeeded.
> This is similar to what is done when user attempts to move several tasks ... the error will
> indicate which task triggered failure so that user space knows what completed successfully.
Will add something like this on failure.
rdt_last_cmd_printf("Total event assign failed on domain %d\n", dom_id);
>
>> +
>> + return -EINVAL;
>> +}
>> +
>> +static ssize_t rdtgroup_mbm_assign_control_write(struct kernfs_open_file *of,
>> + char *buf, size_t nbytes, loff_t off)
>> +{
>> + struct rdt_resource *r = of->kn->parent->priv;
>> + char *token, *cmon_grp, *mon_grp;
>> + enum rdt_group_type rtype;
>> + int ret;
>> +
>> + /* Valid input requires a trailing newline */
>> + if (nbytes == 0 || buf[nbytes - 1] != '\n')
>> + return -EINVAL;
>> +
>> + buf[nbytes - 1] = '\0';
>> +
>> + cpus_read_lock();
>> + mutex_lock(&rdtgroup_mutex);
>> +
>> + if (!resctrl_arch_mbm_cntr_assign_enabled(r)) {
>> + rdt_last_cmd_puts("mbm_cntr_assign mode is not enabled\n");
>
> Writing to last_cmd_status_buf here ...
Sure.
>
>> + mutex_unlock(&rdtgroup_mutex);
>> + cpus_read_unlock();
>> + return -EINVAL;
>> + }
>> +
>> + rdt_last_cmd_clear();
>
> ... but initializing buffer here.
> Sidenote: This was an issue before. If you receive comments about
> items in patches, please do check if those comments apply to other patches also.
Missed it.
>
>> +
>> + while ((token = strsep(&buf, "\n")) != NULL) {
>> + if (strstr(token, "/")) {
>> + /*
>> + * The write command follows the following format:
>> + * “<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>”
>> + * Extract the CTRL_MON group.
>> + */
>> + cmon_grp = strsep(&token, "/");
>> +
>> + /*
>> + * Extract the MON_GROUP.
>> + * strsep returns empty string for contiguous delimiters.
>> + * Empty mon_grp here means it is a RDTCTRL_GROUP.
>> + */
>> + mon_grp = strsep(&token, "/");
>> +
>> + if (*mon_grp == '\0')
>> + rtype = RDTCTRL_GROUP;
>> + else
>> + rtype = RDTMON_GROUP;
>> +
>> + ret = rdtgroup_process_flags(r, rtype, cmon_grp, mon_grp, token);
>> + if (ret)
>> + break;
>> + }
>> + }
>> +
>> + mutex_unlock(&rdtgroup_mutex);
>> + cpus_read_unlock();
>> +
>> + return ret ?: nbytes;
>> +}
>> +
>> #ifdef CONFIG_PROC_CPU_RESCTRL
>>
>> /*
>> @@ -2328,9 +2558,10 @@ static struct rftype res_common_files[] = {
>> },
>> {
>> .name = "mbm_assign_control",
>> - .mode = 0444,
>> + .mode = 0644,
>> .kf_ops = &rdtgroup_kf_single_ops,
>> .seq_show = rdtgroup_mbm_assign_control_show,
>> + .write = rdtgroup_mbm_assign_control_write,
>> },
>> {
>> .name = "cpus_list",
>
> On a high level this looks ok but this code needs to be more robust. This will parse
> data from user space that may include all kinds of input ... think malicious user or
> a buggy script. I am not able to test this code but I tried to work through what will
> happen under some wrong input and found some issues. For example, if user space provides
> input like '//\n' then rdtgroup_process_flags() will be called with token == NULL. This will
> result in rdtgroup_process_flags() returning "success", but fortunately do nothing, for
> this invalid input. A more severe example is with input like '//0=\n', from what I can tell
> this will result in rdtgroup_str_to_mon_state() called with dom_str==NULL that will treat
> this as ASSIGN_NONE and proceed as if user provided '//0=_'.
> This was just some scenarios with basic input that could be typos, no real stress tests.
> I stopped here though since I believe it is already clear this needs to be more robust.
> Please do test this interface by exercising it with invalid input and corner cases.
Agree.
But, tested the cases you mentioned above. It seems to handle as expected.
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
//0=tl;1=tl;2=tl;3=tl;4=tl;5=tl;6=tl;7=tl;8=tl;9=tl;10=tl;11=tl;
#echo '//\n' > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
bash: echo: write error: Invalid argument
# cat /sys/fs/resctrl/info/last_cmd_status
Missing operation =, +, - character
#echo '//0=\n' > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
bash: echo: write error: Invalid argument
#cat /sys/fs/resctrl/info/last_cmd_status
Invalid assign flag
#echo '/0=\n' > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
bash: echo: write error: Invalid argument
# cat /sys/fs/resctrl/info/last_cmd_status
Not a valid resctrl group
The assign state did not change.
#cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
//0=tl;1=tl;2=tl;3=tl;4=tl;5=tl;6=tl;7=tl;8=tl;9=tl;10=tl;11=tl;
Sure. will test some more combinations to be sure.
--
Thanks
Babu Moger
Powered by blists - more mailing lists