lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <Zi_yKzGMoDU0-L3K@x1>
Date: Mon, 29 Apr 2024 16:16:59 -0300
From: Arnaldo Carvalho de Melo <acme@...nel.org>
To: Ian Rogers <irogers@...gle.com>
Cc: Peter Zijlstra <peterz@...radead.org>, Ingo Molnar <mingo@...hat.com>,
	Namhyung Kim <namhyung@...nel.org>,
	Mark Rutland <mark.rutland@....com>,
	Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
	Jiri Olsa <jolsa@...nel.org>,
	Adrian Hunter <adrian.hunter@...el.com>,
	Kan Liang <kan.liang@...ux.intel.com>,
	James Clark <james.clark@....com>,
	Athira Rajeev <atrajeev@...ux.vnet.ibm.com>,
	Colin Ian King <colin.i.king@...il.com>,
	nabijaczleweli@...ijaczleweli.xyz, Leo Yan <leo.yan@...ux.dev>,
	Song Liu <song@...nel.org>,
	Ilkka Koskinen <ilkka@...amperecomputing.com>,
	Ben Gainey <ben.gainey@....com>,
	K Prateek Nayak <kprateek.nayak@....com>,
	Yanteng Si <siyanteng@...ngson.cn>,
	Sun Haiyong <sunhaiyong@...ngson.cn>,
	Changbin Du <changbin.du@...wei.com>,
	Andi Kleen <ak@...ux.intel.com>,
	Thomas Richter <tmricht@...ux.ibm.com>,
	Masami Hiramatsu <mhiramat@...nel.org>,
	Dima Kogan <dima@...retsauce.net>,
	zhaimingbing <zhaimingbing@...s.chinamobile.com>,
	Paran Lee <p4ranlee@...il.com>, Li Dong <lidong@...o.com>,
	Tiezhu Yang <yangtiezhu@...ngson.cn>,
	Yang Jihong <yangjihong1@...wei.com>,
	Chengen Du <chengen.du@...onical.com>,
	linux-perf-users@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH v5 0/7] dso/dsos memory savings and clean up

On Mon, Apr 29, 2024 at 11:46:07AM -0700, Ian Rogers wrote:
> 7 more patches from:
> https://lore.kernel.org/lkml/20240202061532.1939474-1-irogers@google.com/
> a near half year old adventure in trying to lower perf's dynamic
> memory use. Bits like the memory overhead of opendir are on the
> sidelines for now, too much fighting over how
> distributions/C-libraries present getdents. These changes are more
> good old fashioned replace an rb-tree with a sorted array and add
> reference count tracking.
> 
> The changes migrate dsos code, the collection of dso structs, more
> into the dsos.c/dsos.h files. As with maps and threads, this is done
> so the internals can be changed - replacing a linked list (for fast
> iteration) and an rb-tree (for fast finds) with a lazily sorted
> array. The complexity of operations remain roughly the same, although
> iterating an array is likely faster than iterating a linked list, the
Th> memory usage is at least reduced by half.
> 
> As fixing the memory usage necessitates changing operations like find,
> modify these operations so that they increment the reference count to
> avoid races like a find in dsos and a remove. Similarly tighten up
> lock usage so that operations working on dsos state hold the
> appropriate lock. Note, since this series is partially applied in the
> perf-tools-next tree currently some memory leaks have been introduced.
> 
> v5. Rebase, adding use of accessors to dso as necessary. Previous
>     versions were all rebases or dropping merged patches.

So, on an Intel machine:

⬢[acme@...lbox perf-tools-next]$ git log --oneline -8
fb401385575211e6 (HEAD -> perf-tools-next) perf dso: Use container_of to avoid a pointer in 'struct dso_data'
0fe118d129ab1c77 perf dso: Reference counting related fixes
35e44adf6103a407 perf dso: Add reference count checking and accessor functions
35673675ebbbac5d perf dsos: Switch hand code to bsearch()
654d60f2f5c737cd perf dsos: Remove __dsos__findnew_link_by_longname_id()
94b0ba802e090b66 perf dsos: Remove __dsos__addnew()
47692286dd856469 perf dsos: Switch backing storage to array from rbtree/list
8c618b58c89ce4c2 (perf-tools-next.korg/perf-tools-next, acme.korg/perf-tools-next) perf test: Reintroduce -p/--parallel and make -S/--sequential the default
⬢[acme@...lbox perf-tools-next]$

When I'm at:

8c618b58c89ce4c2 (perf-tools-next.korg/perf-tools-next, acme.korg/perf-tools-next) perf test: Reintroduce -p/--parallel and make -S/--sequential the default

root@x1:~# perf -v
perf version 6.9.rc5.g8c618b58c89c
root@x1:~# perf test "lock contention"
 87: kernel lock contention analysis test                            : Ok
root@x1:~# perf test "lock contention"
 87: kernel lock contention analysis test                            : Ok
root@x1:~# perf test "lock contention"
 87: kernel lock contention analysis test                            : Ok
root@x1:~# time perf test "lock contention"
 87: kernel lock contention analysis test                            : Ok

real	0m9.143s
user	0m5.201s
sys	0m4.812s
root@x1:~#

Moving to the first patch in this series:

⬢[acme@...lbox perf-tools-next]$ git log --oneline -2
47692286dd856469 (HEAD) perf dsos: Switch backing storage to array from rbtree/list
8c618b58c89ce4c2 (perf-tools-next.korg/perf-tools-next, acme.korg/perf-tools-next) perf test: Reintroduce -p/--parallel and make -S/--sequential the default
⬢[acme@...lbox perf-tools-next]$ alias m
alias m='rm -rf ~/libexec/perf-core/ ; make -k CORESIGHT=1 O=/tmp/build/$(basename $PWD)/ -C tools/perf install-bin && perf test pythond'
⬢[acme@...lbox perf-tools-next]$ rm -rf /tmp/build/$(basename $PWD)/ ; mkdir -p /tmp/build/$(basename $PWD)/ ; m
<SNIP>

root@x1:~# perf -v
perf version 6.9.rc5.g47692286dd85
root@x1:~# perf test "lock contention"
 87: kernel lock contention analysis test                            : FAILED!
root@x1:~# perf test "lock contention"
 87: kernel lock contention analysis test                            : FAILED!
root@x1:~# perf test "lock contention"
 87: kernel lock contention analysis test                            : FAILED!
root@x1:~# perf test "lock contention"
 87: kernel lock contention analysis test                            : FAILED!
root@x1:~# perf test -vvv "lock contention"
 87: kernel lock contention analysis test:
--- start ---
test child forked, pid 2279518
Testing perf lock record and perf lock contention
[Fail] Recorded result count is not 1: 2
---- end(-1) ----
 87: kernel lock contention analysis test                            : FAILED!
root@x1:~#

This breaks bisectability, but then lets see if at the end of the series
it works:

⬢[acme@...lbox perf-tools-next]$ git log --oneline -8
fb401385575211e6 (HEAD -> perf-tools-next) perf dso: Use container_of to avoid a pointer in 'struct dso_data'
0fe118d129ab1c77 perf dso: Reference counting related fixes
35e44adf6103a407 perf dso: Add reference count checking and accessor functions
35673675ebbbac5d perf dsos: Switch hand code to bsearch()
654d60f2f5c737cd perf dsos: Remove __dsos__findnew_link_by_longname_id()
94b0ba802e090b66 perf dsos: Remove __dsos__addnew()
47692286dd856469 perf dsos: Switch backing storage to array from rbtree/list
8c618b58c89ce4c2 (perf-tools-next.korg/perf-tools-next, acme.korg/perf-tools-next) perf test: Reintroduce -p/--parallel and make -S/--sequential the default
⬢[acme@...lbox perf-tools-next]$ rm -rf /tmp/build/$(basename $PWD)/ ; mkdir -p /tmp/build/$(basename $PWD)/ ; m

root@x1:~# perf -v
perf version 6.9.rc5.gfb4013855752
root@x1:~# perf test "lock contention"
 87: kernel lock contention analysis test                            : FAILED!
root@x1:~# perf test "lock contention"
 87: kernel lock contention analysis test                            : FAILED!
root@x1:~# perf test "lock contention"
 87: kernel lock contention analysis test                            : FAILED!
root@x1:~# perf test "lock contention"
 87: kernel lock contention analysis test                            : FAILED!
root@x1:~# perf test -vvv "lock contention"
 87: kernel lock contention analysis test:
--- start ---
test child forked, pid 2289060
Testing perf lock record and perf lock contention
Testing perf lock contention --use-bpf
Testing perf lock record and perf lock contention at the same time
[Fail] Recorded result count is not 1: 2
---- end(-1) ----
 87: kernel lock contention analysis test                            : FAILED!
root@x1:~#

⬢[acme@...lbox perf-tools-next]$ gcc --version | head -1
gcc (GCC) 13.2.1 20240316 (Red Hat 13.2.1-7)
⬢[acme@...lbox perf-tools-next]$ rpm -q gcc
gcc-13.2.1-7.fc39.x86_64
⬢[acme@...lbox perf-tools-next]$

root@x1:~# grep -m1 'model name' /proc/cpuinfo 
model name	: 13th Gen Intel(R) Core(TM) i7-1365U
root@x1:~# free
               total        used        free      shared  buff/cache   available
Mem:        32507912     9081644     3531432     1987616    22554868    23426268
Swap:        8388604      314112     8074492
root@x1:~#

- Arnaldo

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ