linux-kernel - Re: [PATCH] perf/core: Fix hung issue on perf stat command during cpu hotplug

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20201008131520.GY2628@hirez.programming.kicks-ass.net>
Date:   Thu, 8 Oct 2020 15:15:20 +0200
From:   Peter Zijlstra <peterz@...radead.org>
To:     kajoljain <kjain@...ux.ibm.com>
Cc:     acme@...nel.org, jolsa@...hat.com, linux-kernel@...r.kernel.org,
        linux-perf-users@...r.kernel.org, maddy@...ux.ibm.com,
        mingo@...hat.com, mark.rutland@....com,
        alexander.shishkin@...ux.intel.com, namhyung@...nel.org,
        daniel@...earbox.net, brho@...gle.com, srikar@...ux.vnet.ibm.com
Subject: Re: [PATCH] perf/core: Fix hung issue on perf stat command during
 cpu hotplug

On Thu, Oct 08, 2020 at 05:55:35PM +0530, kajoljain wrote:
> 
> 
> On 8/27/20 12:17 PM, Kajol Jain wrote:
> > Commit 2ed6edd33a21 ("perf: Add cond_resched() to task_function_call()")
> > added assignment of ret value as -EAGAIN in case function
> > call to 'smp_call_function_single' fails.
> > For non-zero ret value, it did
> > 'ret = !ret ? data.ret : -EAGAIN;', which always
> > assign -EAGAIN to ret and make second if condition useless.
> > 
> > In scenarios like when executing a perf stat with --per-thread option, and
> > if any of the monitoring cpu goes offline, the 'smp_call_function_single'
> > function could return -ENXIO, and with the above check,
> > task_function_call hung and increases CPU
> > usage (because of repeated 'smp_call_function_single()')
> > 
> > Recration scenario:
> > 	# perf stat -a --per-thread && (offline a CPU )
> > 
> > Patch here removes the tertiary condition added as part of that
> > commit and added a check for NULL and -EAGAIN.
> 
> Hi Peter,
>      Please let me know if you have any comment on this patch.

Yes, sorry. I've got it now. Thanks!

---
Subject: perf: Fix task_function_call() error handling
From: Kajol Jain <kjain@...ux.ibm.com>
Date: Thu, 27 Aug 2020 12:17:32 +0530

From: Kajol Jain <kjain@...ux.ibm.com>

The error handling introduced by commit:

  2ed6edd33a21 ("perf: Add cond_resched() to task_function_call()")

looses any return value from smp_call_function_single() that is not
{0, -EINVAL}. This is a problem because it will return -EXNIO when the
target CPU is offline. Worse, in that case it'll turn into an infinite
loop.

Fixes: 2ed6edd33a21 ("perf: Add cond_resched() to task_function_call()")
Reported-by: Srikar Dronamraju <srikar@...ux.vnet.ibm.com>
Signed-off-by: Kajol Jain <kjain@...ux.ibm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@...radead.org>
Reviewed-by: Barret Rhoden <brho@...gle.com>
Tested-by: Srikar Dronamraju <srikar@...ux.vnet.ibm.com>
Link: https://lkml.kernel.org/r/20200827064732.20860-1-kjain@linux.ibm.com
---
 kernel/events/core.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -99,7 +99,7 @@ static void remote_function(void *data)
  * retry due to any failures in smp_call_function_single(), such as if the
  * task_cpu() goes offline concurrently.
  *
- * returns @func return value or -ESRCH when the process isn't running
+ * returns @func return value or -ESRCH or -ENXIO when the process isn't running
  */
 static int
 task_function_call(struct task_struct *p, remote_function_f func, void *info)
@@ -115,7 +115,8 @@ task_function_call(struct task_struct *p
 	for (;;) {
 		ret = smp_call_function_single(task_cpu(p), remote_function,
 					       &data, 1);
-		ret = !ret ? data.ret : -EAGAIN;
+		if (!ret)
+			ret = data.ret;
 
 		if (ret != -EAGAIN)
 			break;