| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAP-5=fUETVGR+BVsHPyEAPFHcAQNRE0BJjXFyaqFtmwt8PY2HA@mail.gmail.com> Date: Mon, 29 Apr 2024 12:50:34 -0700 From: Ian Rogers <irogers@...gle.com> To: Arnaldo Carvalho de Melo <acme@...nel.org> Cc: Peter Zijlstra <peterz@...radead.org>, Ingo Molnar <mingo@...hat.com>, Namhyung Kim <namhyung@...nel.org>, Mark Rutland <mark.rutland@....com>, Alexander Shishkin <alexander.shishkin@...ux.intel.com>, Jiri Olsa <jolsa@...nel.org>, Adrian Hunter <adrian.hunter@...el.com>, Kan Liang <kan.liang@...ux.intel.com>, James Clark <james.clark@....com>, Athira Rajeev <atrajeev@...ux.vnet.ibm.com>, Colin Ian King <colin.i.king@...il.com>, nabijaczleweli@...ijaczleweli.xyz, Leo Yan <leo.yan@...ux.dev>, Song Liu <song@...nel.org>, Ilkka Koskinen <ilkka@...amperecomputing.com>, Ben Gainey <ben.gainey@....com>, K Prateek Nayak <kprateek.nayak@....com>, Yanteng Si <siyanteng@...ngson.cn>, Sun Haiyong <sunhaiyong@...ngson.cn>, Changbin Du <changbin.du@...wei.com>, Andi Kleen <ak@...ux.intel.com>, Thomas Richter <tmricht@...ux.ibm.com>, Masami Hiramatsu <mhiramat@...nel.org>, Dima Kogan <dima@...retsauce.net>, zhaimingbing <zhaimingbing@...s.chinamobile.com>, Paran Lee <p4ranlee@...il.com>, Li Dong <lidong@...o.com>, Tiezhu Yang <yangtiezhu@...ngson.cn>, Yang Jihong <yangjihong1@...wei.com>, Chengen Du <chengen.du@...onical.com>, linux-perf-users@...r.kernel.org, linux-kernel@...r.kernel.org Subject: Re: [PATCH v5 0/7] dso/dsos memory savings and clean up On Mon, Apr 29, 2024 at 12:17 PM Arnaldo Carvalho de Melo <acme@...nel.org> wrote: > > On Mon, Apr 29, 2024 at 11:46:07AM -0700, Ian Rogers wrote: > > 7 more patches from: > > https://lore.kernel.org/lkml/20240202061532.1939474-1-irogers@google.com/ > > a near half year old adventure in trying to lower perf's dynamic > > memory use. Bits like the memory overhead of opendir are on the > > sidelines for now, too much fighting over how > > distributions/C-libraries present getdents. These changes are more > > good old fashioned replace an rb-tree with a sorted array and add > > reference count tracking. > > > > The changes migrate dsos code, the collection of dso structs, more > > into the dsos.c/dsos.h files. As with maps and threads, this is done > > so the internals can be changed - replacing a linked list (for fast > > iteration) and an rb-tree (for fast finds) with a lazily sorted > > array. The complexity of operations remain roughly the same, although > > iterating an array is likely faster than iterating a linked list, the > Th> memory usage is at least reduced by half. > > > > As fixing the memory usage necessitates changing operations like find, > > modify these operations so that they increment the reference count to > > avoid races like a find in dsos and a remove. Similarly tighten up > > lock usage so that operations working on dsos state hold the > > appropriate lock. Note, since this series is partially applied in the > > perf-tools-next tree currently some memory leaks have been introduced. > > > > v5. Rebase, adding use of accessors to dso as necessary. Previous > > versions were all rebases or dropping merged patches. > > So, on an Intel machine: > > ⬢[acme@...lbox perf-tools-next]$ git log --oneline -8 > fb401385575211e6 (HEAD -> perf-tools-next) perf dso: Use container_of to avoid a pointer in 'struct dso_data' > 0fe118d129ab1c77 perf dso: Reference counting related fixes > 35e44adf6103a407 perf dso: Add reference count checking and accessor functions > 35673675ebbbac5d perf dsos: Switch hand code to bsearch() > 654d60f2f5c737cd perf dsos: Remove __dsos__findnew_link_by_longname_id() > 94b0ba802e090b66 perf dsos: Remove __dsos__addnew() > 47692286dd856469 perf dsos: Switch backing storage to array from rbtree/list > 8c618b58c89ce4c2 (perf-tools-next.korg/perf-tools-next, acme.korg/perf-tools-next) perf test: Reintroduce -p/--parallel and make -S/--sequential the default > ⬢[acme@...lbox perf-tools-next]$ > > When I'm at: > > 8c618b58c89ce4c2 (perf-tools-next.korg/perf-tools-next, acme.korg/perf-tools-next) perf test: Reintroduce -p/--parallel and make -S/--sequential the default > > root@x1:~# perf -v > perf version 6.9.rc5.g8c618b58c89c > root@x1:~# perf test "lock contention" > 87: kernel lock contention analysis test : Ok > root@x1:~# perf test "lock contention" > 87: kernel lock contention analysis test : Ok > root@x1:~# perf test "lock contention" > 87: kernel lock contention analysis test : Ok > root@x1:~# time perf test "lock contention" > 87: kernel lock contention analysis test : Ok > > real 0m9.143s > user 0m5.201s > sys 0m4.812s > root@x1:~# > > Moving to the first patch in this series: > > ⬢[acme@...lbox perf-tools-next]$ git log --oneline -2 > 47692286dd856469 (HEAD) perf dsos: Switch backing storage to array from rbtree/list > 8c618b58c89ce4c2 (perf-tools-next.korg/perf-tools-next, acme.korg/perf-tools-next) perf test: Reintroduce -p/--parallel and make -S/--sequential the default > ⬢[acme@...lbox perf-tools-next]$ alias m > alias m='rm -rf ~/libexec/perf-core/ ; make -k CORESIGHT=1 O=/tmp/build/$(basename $PWD)/ -C tools/perf install-bin && perf test pythond' > ⬢[acme@...lbox perf-tools-next]$ rm -rf /tmp/build/$(basename $PWD)/ ; mkdir -p /tmp/build/$(basename $PWD)/ ; m > <SNIP> > > root@x1:~# perf -v > perf version 6.9.rc5.g47692286dd85 > root@x1:~# perf test "lock contention" > 87: kernel lock contention analysis test : FAILED! > root@x1:~# perf test "lock contention" > 87: kernel lock contention analysis test : FAILED! > root@x1:~# perf test "lock contention" > 87: kernel lock contention analysis test : FAILED! > root@x1:~# perf test "lock contention" > 87: kernel lock contention analysis test : FAILED! > root@x1:~# perf test -vvv "lock contention" > 87: kernel lock contention analysis test: > --- start --- > test child forked, pid 2279518 > Testing perf lock record and perf lock contention > [Fail] Recorded result count is not 1: 2 > ---- end(-1) ---- > 87: kernel lock contention analysis test : FAILED! > root@x1:~# > > This breaks bisectability, but then lets see if at the end of the series > it works: > > ⬢[acme@...lbox perf-tools-next]$ git log --oneline -8 > fb401385575211e6 (HEAD -> perf-tools-next) perf dso: Use container_of to avoid a pointer in 'struct dso_data' > 0fe118d129ab1c77 perf dso: Reference counting related fixes > 35e44adf6103a407 perf dso: Add reference count checking and accessor functions > 35673675ebbbac5d perf dsos: Switch hand code to bsearch() > 654d60f2f5c737cd perf dsos: Remove __dsos__findnew_link_by_longname_id() > 94b0ba802e090b66 perf dsos: Remove __dsos__addnew() > 47692286dd856469 perf dsos: Switch backing storage to array from rbtree/list > 8c618b58c89ce4c2 (perf-tools-next.korg/perf-tools-next, acme.korg/perf-tools-next) perf test: Reintroduce -p/--parallel and make -S/--sequential the default > ⬢[acme@...lbox perf-tools-next]$ rm -rf /tmp/build/$(basename $PWD)/ ; mkdir -p /tmp/build/$(basename $PWD)/ ; m > > root@x1:~# perf -v > perf version 6.9.rc5.gfb4013855752 > root@x1:~# perf test "lock contention" > 87: kernel lock contention analysis test : FAILED! > root@x1:~# perf test "lock contention" > 87: kernel lock contention analysis test : FAILED! > root@x1:~# perf test "lock contention" > 87: kernel lock contention analysis test : FAILED! > root@x1:~# perf test "lock contention" > 87: kernel lock contention analysis test : FAILED! > root@x1:~# perf test -vvv "lock contention" > 87: kernel lock contention analysis test: > --- start --- > test child forked, pid 2289060 > Testing perf lock record and perf lock contention > Testing perf lock contention --use-bpf > Testing perf lock record and perf lock contention at the same time > [Fail] Recorded result count is not 1: 2 > ---- end(-1) ---- > 87: kernel lock contention analysis test : FAILED! > root@x1:~# > > ⬢[acme@...lbox perf-tools-next]$ gcc --version | head -1 > gcc (GCC) 13.2.1 20240316 (Red Hat 13.2.1-7) > ⬢[acme@...lbox perf-tools-next]$ rpm -q gcc > gcc-13.2.1-7.fc39.x86_64 > ⬢[acme@...lbox perf-tools-next]$ > > root@x1:~# grep -m1 'model name' /proc/cpuinfo > model name : 13th Gen Intel(R) Core(TM) i7-1365U > root@x1:~# free > total used free shared buff/cache available > Mem: 32507912 9081644 3531432 1987616 22554868 23426268 > Swap: 8388604 314112 8074492 > root@x1:~# Thanks, will check. Will likely need a v6 to fix. v5 was just addressing the rebase issues. Ian > - Arnaldo
Powered by blists - more mailing lists