| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Fri, 24 Feb 2017 13:13:23 +0530
From: Ravi Bangoria <ravi.bangoria@...ux.vnet.ibm.com>
To: mingo@...hat.com, acme@...nel.org, masami.hiramatsu.pt@...achi.com
Cc: brendan.d.gregg@...il.com, peterz@...radead.org,
alexander.shishkin@...ux.intel.com, wangnan0@...wei.com,
jolsa@...nel.org, ak@...ux.intel.com, treeze.taeung@...il.com,
mathieu.poirier@...aro.org, hekuang@...wei.com,
sukadev@...ux.vnet.ibm.com, ananth@...ibm.com,
naveen.n.rao@...ux.vnet.ibm.com, adrian.hunter@...el.com,
linux-kernel@...r.kernel.org, hemant@...ux.vnet.ibm.com,
Ravi Bangoria <ravi.bangoria@...ux.vnet.ibm.com>
Subject: [PATCH v3 0/2] perf/sdt: Directly record SDT events with 'perf record'
All events from 'perf list', except SDT events, can be directly recorded
with 'perf record'. But, the flow is little different for SDT events.
Probe point for SDT event needs to be created using 'perf probe' before
recording it using 'perf record'.
As suggested by Ingo[1], it's better to make this process simple by
creating probe point automatically with 'perf record' for SDT events.
Same effort was done by Hemant some time ago[2]. This patch series
is based on work he has done.
Changes in v3:
- Rebased to current acme/perf/core
- v2 was having build failure if LIB_ELF support is missing. Solved
it in this series.
- Event starting with 'sdt_' will be considered as SDT event. v2
was forcing to prepend SDT event name with '%' for 'perf record'
but not for 'perf probe'. This change was suggested by Brendan/
Masami in one of the review comment in v1.[3]
- There was one more problem with v2. If internal 'perf probe' fails
to lookup SDT event in cache because of too big list (-E2BIG),
'perf record' starts recording cycles event instead of showing
warning/error. This can totally confuse user. I've resolved it. ex:
With v2:
$ perf record -a -e sdt_qemu:*
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.246 MB perf.data (1523 samples) ]
$ perf evlist
cpu-clock
With this patch:
$ perf record -a -e sdt_qemu:*
event syntax error: 'sdt_qemu:*'
\___ Cache lookup failed
- This patch series still allows user to place a probe manually with
'perf probe' but also shows a hint:
$ perf probe sdt_libpthread:mutex_entry
...
Hint: SDT event can be directly recorded with 'perf record'. No need to create probe manually.
- v2 was always trying to look for sdt event in probe cache by ignoring
entries of uprobe_events. Hence, it creates new probe points for event
even if it already exists. This was confusing from user's point of
view.
With v2:
$ perf probe sdt_libpthread:mutex_entry
Added new events:
sdt_libpthread:mutex_entry (on %mutex_entry in /usr/lib64/libpthread-2.24.so)
sdt_libpthread:mutex_entry_1 (on %mutex_entry in /usr/lib64/libpthread-2.24.so)
$ cat /sys/kernel/debug/tracing/uprobe_events
p:sdt_libpthread/mutex_entry /usr/lib64/libpthread-2.24.so:0x0000000000009ddb
p:sdt_libpthread/mutex_entry_1 /usr/lib64/libpthread-2.24.so:0x000000000000bcbb
$ perf record -a -e %sdt_libpthread:mutex_entry
Warning : Recording on 2 occurrences of sdt_libpthread:mutex_entry
$ perf evlist
sdt_libpthread:mutex_entry_3
sdt_libpthread:mutex_entry_2
There are two issues:
1st, It confuses user about new names.
2nd, (and more important) Perf won't allow you to record
'sdt_libpthread:mutex_entry_1' with 'perf record' even if it
exists in uprobe_events, because it won't find event with that
name in probe cache.
I've solved these issues. This patch gives first priority to existing
entries of uprobe_events file. After that, it looks into probe cache.
For ex,
With this patch:
$ perf probe sdt_libpthread:mutex_entry
Added new events:
sdt_libpthread:mutex_entry (on %mutex_entry in /usr/lib64/libpthread-2.24.so)
sdt_libpthread:mutex_entry_1 (on %mutex_entry in /usr/lib64/libpthread-2.24.so)
$ perf record -a -e sdt_libpthread:mutex_entry_1
Matching event(s) from uprobe_events:
sdt_libpthread:mutex_entry_1 0xbcbb@...r/lib64/libpthread-2.24.so
Use 'perf probe -d <event>' to delete event(s).
It also lists matching entries as 'name addr@...ename' followed by the
hint about how to delete them with 'perf probe -d'.
If there is no matching entry found in uprobe_events with that *name*,
perf will find all events with that name from probe cache. Once again
it checks whether these events exists in uprobe_events, but this time
it uses *address* and *filename* instead of eventname. If entry found,
it reuse that entry instead of creating new one. In continuation of
above example,
With this patch:
$ perf probe -d sdt_libpthread:mutex_entry
Removed event: sdt_libpthread:mutex_entry
$ cat /sys/kernel/debug/tracing/uprobe_events
p:sdt_libpthread/mutex_entry_1 /usr/lib64/libpthread-2.24.so:0x000000000000bcbb
$ perf record -a -e sdt_libpthread:*
Matching event(s) from uprobe_events:
sdt_libpthread:mutex_entry_1 0xbcbb@...r/lib64/libpthread-2.24.so
Use 'perf probe -d <event>' to delete event(s).
Warning: Recording on 35 occurrences of sdt_libpthread:*
$ perf evlist
sdt_libpthread:mutex_entry
sdt_libpthread:pthread_create
sdt_libpthread:mutex_entry_1
...
Here, Perf has reused entry for event with 0xbcbb address, but also it
has created new entry for event with 0x9ddb address. It also maintains
list of entries created for particular record session, and uses that
list to remove entries at the end of session.
Finally, If somehow tool fails to clean events from uprobe_events at
the end of session, user has to clean events manually with
'perf probe -d'. But perf will give Warning in such case. For ex,
$ perf record -a -e sdt_libpthread:mutex_entry
Warning: Recording on 2 occurrences of sdt_libpthread:mutex_entry
/** Fails with segfault **/
$ cat /sys/kernel/debug/tracing/uprobe_events
p:sdt_libpthread/mutex_entry /usr/lib64/libpthread-2.24.so:0x0000000000009ddb
p:sdt_libpthread/mutex_entry_1 /usr/lib64/libpthread-2.24.so:0x000000000000bcbb
When next time user tries to record, it will show a warning:
$ perf record -a -e sdt_libpthread:mutex_entry
Matching event(s) from uprobe_events:
sdt_libpthread:mutex_entry 0x9ddb@...r/lib64/libpthread-2.24.so
Use 'perf probe -d <event>' to delete event(s).
Warning: Found 2 events from probe-cache with name 'sdt_libpthread:mutex_entry'.
Since probe point already exists with this name, recording only 1 event.
Hint: Please use 'perf probe -d sdt_libpthread:mutex_entry*' to allow record on all events.
But no such warning for 'sdt_libpthread:mutex_entry_1'.
$ perf record -a -e sdt_libpthread:mutex_entry_1
Matching event(s) from uprobe_events:
sdt_libpthread:mutex_entry_1 0xbcbb@...r/lib64/libpthread-2.24.so
Use 'perf probe -d <event>' to delete event(s).
[1] https://lkml.org/lkml/2017/2/7/59
[2] https://lkml.org/lkml/2016/5/3/810
[3] https://lkml.org/lkml/2016/5/2/689
Hemant Kumar (1):
perf/sdt: Directly record SDT events with 'perf record'
Ravi Bangoria (1):
perf/sdt: Introduce util func is_sdt_event()
tools/lib/api/fs/tracing_path.c | 17 +-
tools/perf/builtin-probe.c | 21 ++-
tools/perf/builtin-record.c | 23 +++
tools/perf/perf.h | 2 +
tools/perf/util/parse-events.c | 56 +++++-
tools/perf/util/parse-events.h | 2 +
tools/perf/util/probe-event.c | 53 +++++-
tools/perf/util/probe-event.h | 4 +
tools/perf/util/probe-file.c | 376 ++++++++++++++++++++++++++++++++++++++++
tools/perf/util/probe-file.h | 27 +++
tools/perf/util/util.c | 12 ++
tools/perf/util/util.h | 2 +
12 files changed, 567 insertions(+), 28 deletions(-)
--
2.9.3
Powered by blists - more mailing lists