| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAJpZYjUvaXfmvMH5FjpAUSj4EiEgbMd4CyOobPXKoDhpzknvJQ@mail.gmail.com> Date: Thu, 27 Feb 2025 11:20:18 -0800 From: Chun-Tse Shao <ctshao@...gle.com> To: Athira Rajeev <atrajeev@...ux.ibm.com> Cc: "open list:PERFORMANCE EVENTS SUBSYSTEM" <linux-kernel@...r.kernel.org>, Peter Zijlstra <peterz@...radead.org>, Ingo Molnar <mingo@...hat.com>, Arnaldo Carvalho de Melo <acme@...nel.org>, Namhyung Kim <namhyung@...nel.org>, Mark Rutland <mark.rutland@....com>, Alexander Shishkin <alexander.shishkin@...ux.intel.com>, Jiri Olsa <jolsa@...nel.org>, Ian Rogers <irogers@...gle.com>, Adrian Hunter <adrian.hunter@...el.com>, Kan Liang <kan.liang@...ux.intel.com>, nick.forrington@....com, "open list:PERFORMANCE EVENTS SUBSYSTEM" <linux-perf-users@...r.kernel.org>, bpf@...r.kernel.org Subject: Re: [PATCH v8 0/4] Tracing contention lock owner call stack Thank you Athira! On Thu, Feb 27, 2025 at 3:11 AM Athira Rajeev <atrajeev@...ux.ibm.com> wrote: > > > > > On 27 Feb 2025, at 5:58 AM, Chun-Tse Shao <ctshao@...gle.com> wrote: > > > > For perf lock contention, the current owner tracking (-o option) only > > works with per-thread mode (-t option). Enabling call stack mode for > > owner can be useful for diagnosing why a system running slow in > > lock contention. > > > > Example output: > > $ sudo ~/linux/tools/perf/perf lock con -abvo -Y mutex -E16 perf bench sched pipe > > ... > > contended total wait max wait avg wait type caller > > > > 171 1.55 ms 20.26 us 9.06 us mutex pipe_read+0x57 > > 0xffffffffac6318e7 pipe_read+0x57 > > 0xffffffffac623862 vfs_read+0x332 > > 0xffffffffac62434b ksys_read+0xbb > > 0xfffffffface604b2 do_syscall_64+0x82 > > 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 > > 36 193.71 us 15.27 us 5.38 us mutex pipe_write+0x50 > > 0xffffffffac631ee0 pipe_write+0x50 > > 0xffffffffac6241db vfs_write+0x3bb > > 0xffffffffac6244ab ksys_write+0xbb > > 0xfffffffface604b2 do_syscall_64+0x82 > > 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 > > 4 51.22 us 16.47 us 12.80 us mutex do_epoll_wait+0x24d > > 0xffffffffac691f0d do_epoll_wait+0x24d > > 0xffffffffac69249b do_epoll_pwait.part.0+0xb > > 0xffffffffac693ba5 __x64_sys_epoll_pwait+0x95 > > 0xfffffffface604b2 do_syscall_64+0x82 > > 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 > > 2 20.88 us 11.95 us 10.44 us mutex do_epoll_wait+0x24d > > 0xffffffffac691f0d do_epoll_wait+0x24d > > 0xffffffffac693943 __x64_sys_epoll_wait+0x73 > > 0xfffffffface604b2 do_syscall_64+0x82 > > 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 > > 1 7.33 us 7.33 us 7.33 us mutex do_epoll_ctl+0x6c1 > > 0xffffffffac692e01 do_epoll_ctl+0x6c1 > > 0xffffffffac6937e0 __x64_sys_epoll_ctl+0x70 > > 0xfffffffface604b2 do_syscall_64+0x82 > > 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 > > 1 6.64 us 6.64 us 6.64 us mutex do_epoll_ctl+0x3d4 > > 0xffffffffac692b14 do_epoll_ctl+0x3d4 > > 0xffffffffac6937e0 __x64_sys_epoll_ctl+0x70 > > 0xfffffffface604b2 do_syscall_64+0x82 > > 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 > > > > === owner stack trace === > > > > 3 31.24 us 15.27 us 10.41 us mutex pipe_read+0x348 > > 0xffffffffac631bd8 pipe_read+0x348 > > 0xffffffffac623862 vfs_read+0x332 > > 0xffffffffac62434b ksys_read+0xbb > > 0xfffffffface604b2 do_syscall_64+0x82 > > 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 > > ... > > > > v8: > > Fix compilation error found by Athira Rajeev and Namhyung Kim. > > Tested with v8 and compiles fine, thanks for addressing the issue. > > Tested-by: Athira Rajeev <atrajeev@...ux.ibm.com> > > > > > v7: lore.kernel.org/20250224184742.4144931-1-ctshao@...gle.com > > Remove duplicate contention records. > > > > v6: lore.kernel.org/20250219214400.3317548-1-ctshao@...gle.com > > Free allocated memory in error patch. > > Add description in man page. > > > > v5: lore.kernel.org/20250212222859.2086080-1-ctshao@...gle.com > > Move duplicated code into function. > > Remove code to retrieve undesired callstack at the end of `contention_end()`. > > Other minor fix based on Namhyung's review. > > > > v4: lore.kernel.org/20250130052510.860318-1-ctshao@...gle.com > > Use `__sync_fetch_and_add()` to generate owner stackid automatically. > > Use `__sync_fetch_and_add(..., -1)` to workaround compiler error from > > `__sync_fetch_and_sub()` > > Remove unnecessary include headers. > > Dedicate on C-style comment. > > Other minor fix based on Namhyung's review. > > > > v3: lore.kernel.org/20250129001905.619859-1-ctshao@...gle.com > > Rename/shorten multiple variables. > > Implement owner stackid. > > Add description for lock function return code in `contention_end()`. > > Other minor fix based on Namhyung's review. > > > > v2: lore.kernel.org/20250113052220.2105645-1-ctshao@...gle.com > > Fix logic deficit in v1 patch 2/4. > > > > v1: lore.kernel.org/20250110051346.1507178-1-ctshao@...gle.com > > > > Chun-Tse Shao (4): > > perf lock: Add bpf maps for owner stack tracing > > perf lock: Retrieve owner callstack in bpf program > > perf lock: Make rb_tree helper functions generic > > perf lock: Report owner stack in usermode > > > > tools/perf/Documentation/perf-lock.txt | 5 +- > > tools/perf/builtin-lock.c | 56 +++- > > tools/perf/util/bpf_lock_contention.c | 85 +++++- > > .../perf/util/bpf_skel/lock_contention.bpf.c | 245 +++++++++++++++++- > > tools/perf/util/bpf_skel/lock_data.h | 7 + > > tools/perf/util/lock-contention.h | 7 + > > 6 files changed, 372 insertions(+), 33 deletions(-) > > > > -- > > 2.48.1.658.g4767266eb4-goog > > > > >
Powered by blists - more mailing lists