| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20180715130827.GA5071@danjae.aot.lge.com>
Date: Sun, 15 Jul 2018 22:08:27 +0900
From: Namhyung Kim <namhyung@...nel.org>
To: Jiri Olsa <jolsa@...nel.org>
Cc: Arnaldo Carvalho de Melo <acme@...nel.org>,
lkml <linux-kernel@...r.kernel.org>,
Ingo Molnar <mingo@...nel.org>,
David Ahern <dsahern@...il.com>,
Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
Peter Zijlstra <a.p.zijlstra@...llo.nl>,
Kan Liang <kan.liang@...el.com>,
Andi Kleen <ak@...ux.intel.com>,
Lukasz Odzioba <lukasz.odzioba@...el.com>,
Wang Nan <wangnan0@...wei.com>, kernel-team@....com
Subject: Re: [PATCH 1/4] perf tools: Fix struct comm_str removal crash
Hi Jiri,
On Thu, Jul 12, 2018 at 04:20:20PM +0200, Jiri Olsa wrote:
> We occasionaly hit following assert failure in perf top,
> when processing the /proc info in multiple threads.
>
> perf: ...include/linux/refcount.h:109: refcount_inc:
> Assertion `!(!refcount_inc_not_zero(r))' failed.
>
> The gdb backtrace looks like this:
>
> [Switching to Thread 0x7ffff11ba700 (LWP 13749)]
> 0x00007ffff50839fb in raise () from /lib64/libc.so.6
> (gdb)
> #0 0x00007ffff50839fb in raise () from /lib64/libc.so.6
> #1 0x00007ffff5085800 in abort () from /lib64/libc.so.6
> #2 0x00007ffff507c0da in __assert_fail_base () from /lib64/libc.so.6
> #3 0x00007ffff507c152 in __assert_fail () from /lib64/libc.so.6
> #4 0x0000000000535373 in refcount_inc (r=0x7fffdc009be0)
> at ...include/linux/refcount.h:109
> #5 0x00000000005354f1 in comm_str__get (cs=0x7fffdc009bc0)
> at util/comm.c:24
> #6 0x00000000005356bd in __comm_str__findnew (str=0x7fffd000b260 ":2",
> root=0xbed5c0 <comm_str_root>) at util/comm.c:72
> #7 0x000000000053579e in comm_str__findnew (str=0x7fffd000b260 ":2",
> root=0xbed5c0 <comm_str_root>) at util/comm.c:95
> #8 0x000000000053582e in comm__new (str=0x7fffd000b260 ":2",
> timestamp=0, exec=false) at util/comm.c:111
> #9 0x00000000005363bc in thread__new (pid=2, tid=2) at util/thread.c:57
> #10 0x0000000000523da0 in ____machine__findnew_thread (machine=0xbfde38,
> threads=0xbfdf28, pid=2, tid=2, create=true) at util/machine.c:457
> #11 0x0000000000523eb4 in __machine__findnew_thread (machine=0xbfde38,
> ...
>
> The failing assertion is this one:
>
> REFCOUNT_WARN(!refcount_inc_not_zero(r), ...
>
> The problem is that we keep global comm_str_root list, which
> is accessed by multiple threads during the perf top startup
> and following 2 paths can race:
>
> thread 1:
> ...
> thread__new
> comm__new
> comm_str__findnew
> down_write(&comm_str_lock);
> __comm_str__findnew
> comm_str__get
>
> thread 2:
> ...
> comm__override or comm__free
> comm_str__put
> refcount_dec_and_test
> down_write(&comm_str_lock);
> rb_erase(&cs->rb_node, &comm_str_root);
>
> Because thread 2 first decrements the refcnt and only after then it
> removes the struct comm_str from the list, the thread 1 can find this
> object on the list with refcnt equls to 0 and hit the assert.
>
> This patch fixes the thread 2 path, by removing the struct comm_str
> FIRST from the list and only AFTER calling comm_str__put on it. This
> way the thread 1 finds only valid objects on the list.
I'm not sure we can unconditionally remove the comm_str from the tree.
It should be removed only if refcount is going to zero IMHO.
Otherwise it could end up having multiple comm_str entry for a same
name.
Thanks,
Namhyung
>
> We also need to ensure now, that only one caller removes the struct
> comm_str, from the list. Adding 'removed' bool to track that.
>
> Link: http://lkml.kernel.org/n/tip-vrizt6sw1lu1ybsrl9l0wwln@git.kernel.org
> Signed-off-by: Jiri Olsa <jolsa@...nel.org>
> ---
> tools/perf/util/comm.c | 30 +++++++++++++++++++++++++-----
> 1 file changed, 25 insertions(+), 5 deletions(-)
>
> diff --git a/tools/perf/util/comm.c b/tools/perf/util/comm.c
> index 7798a2cc8a86..7f1c6e63e3e6 100644
> --- a/tools/perf/util/comm.c
> +++ b/tools/perf/util/comm.c
> @@ -12,6 +12,7 @@ struct comm_str {
> char *str;
> struct rb_node rb_node;
> refcount_t refcnt;
> + bool removed;
> };
>
> /* Should perhaps be moved to struct machine */
> @@ -28,9 +29,6 @@ static struct comm_str *comm_str__get(struct comm_str *cs)
> static void comm_str__put(struct comm_str *cs)
> {
> if (cs && refcount_dec_and_test(&cs->refcnt)) {
> - down_write(&comm_str_lock);
> - rb_erase(&cs->rb_node, &comm_str_root);
> - up_write(&comm_str_lock);
> zfree(&cs->str);
> free(cs);
> }
> @@ -117,6 +115,28 @@ struct comm *comm__new(const char *str, u64 timestamp, bool exec)
> return comm;
> }
>
> +static void __comm_str__remove(struct comm_str *cs)
> +{
> + down_write(&comm_str_lock);
> + if (!cs->removed) {
> + rb_erase(&cs->rb_node, &comm_str_root);
> + cs->removed = true;
> + }
> + up_write(&comm_str_lock);
> +}
> +
> +static void comm_str__remove(struct comm_str *cs)
> +{
> + if (!cs->removed)
> + __comm_str__remove(cs);
> +}
> +
> +static void comm_str__exit(struct comm_str *cs)
> +{
> + comm_str__remove(cs);
> + comm_str__put(cs);
> +}
> +
> int comm__override(struct comm *comm, const char *str, u64 timestamp, bool exec)
> {
> struct comm_str *new, *old = comm->comm_str;
> @@ -125,7 +145,7 @@ int comm__override(struct comm *comm, const char *str, u64 timestamp, bool exec)
> if (!new)
> return -ENOMEM;
>
> - comm_str__put(old);
> + comm_str__exit(old);
> comm->comm_str = new;
> comm->start = timestamp;
> if (exec)
> @@ -136,7 +156,7 @@ int comm__override(struct comm *comm, const char *str, u64 timestamp, bool exec)
>
> void comm__free(struct comm *comm)
> {
> - comm_str__put(comm->comm_str);
> + comm_str__exit(comm->comm_str);
> free(comm);
> }
>
> --
> 2.17.1
>
Powered by blists - more mailing lists