linux-kernel - Re: [PATCH v2 00/17] Support dynamic opening of capstone/llvm remove BUILD

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAP-5=fW5xmir_26CrQN50TWkzab3GueUvne4VWhWhqc82p6LvA@mail.gmail.com>
Date: Thu, 23 Jan 2025 13:24:44 -0800
From: Ian Rogers <irogers@...gle.com>
To: Andi Kleen <ak@...ux.intel.com>
Cc: Peter Zijlstra <peterz@...radead.org>, Ingo Molnar <mingo@...hat.com>, 
	Arnaldo Carvalho de Melo <acme@...nel.org>, Namhyung Kim <namhyung@...nel.org>, 
	Mark Rutland <mark.rutland@....com>, 
	Alexander Shishkin <alexander.shishkin@...ux.intel.com>, Jiri Olsa <jolsa@...nel.org>, 
	Adrian Hunter <adrian.hunter@...el.com>, Kan Liang <kan.liang@...ux.intel.com>, 
	Nathan Chancellor <nathan@...nel.org>, Nick Desaulniers <ndesaulniers@...gle.com>, 
	Bill Wendling <morbo@...gle.com>, Justin Stitt <justinstitt@...gle.com>, 
	Aditya Gupta <adityag@...ux.ibm.com>, "Steinar H. Gunderson" <sesse@...gle.com>, 
	Charlie Jenkins <charlie@...osinc.com>, Changbin Du <changbin.du@...wei.com>, 
	"Masami Hiramatsu (Google)" <mhiramat@...nel.org>, James Clark <james.clark@...aro.org>, 
	Kajol Jain <kjain@...ux.ibm.com>, Athira Rajeev <atrajeev@...ux.vnet.ibm.com>, 
	Li Huafei <lihuafei1@...wei.com>, Dmitry Vyukov <dvyukov@...gle.com>, 
	Chaitanya S Prakash <chaitanyas.prakash@....com>, linux-kernel@...r.kernel.org, 
	linux-perf-users@...r.kernel.org, llvm@...ts.linux.dev, 
	Song Liu <song@...nel.org>, bpf@...r.kernel.org
Subject: Re: [PATCH v2 00/17] Support dynamic opening of capstone/llvm remove BUILD_NONDISTRO

On Thu, Jan 23, 2025 at 10:19 AM Andi Kleen <ak@...ux.intel.com> wrote:
>
> > In certain scenarios, like data centers, it can be useful to
> > statically link all your dependencies to avoid dll hell.
>
> Yes but it won't be loaded into memory if not used. Executable
> loading is all lazy. Maybe look a page fault trace for loading
> perf if you don't believe me.
>
> So you're trying to optimize disk space here?
>
> I didn't see that in the cover letter.

For me yes, for distributions it is dependencies. This is already in
the v3 message:
https://lore.kernel.org/lkml/20250122174308.350350-1-irogers@google.com/

> It doesn't seem like a very good reason for such an intrusive patch kit.

The capstone and LLVM code is preexisting. Moving the capstone/llvm
code to their own files isn't dependent on dlopen, it does make it
nicer to have a single place we're doing dlopen. The change to shim
the capstone/LLVM calls looks like this:
https://github.com/googleprodkernel/linux-perf/blob/google_tools_master/tools/perf/util/llvm.c#L160-L182
That is a shim is introduced that either calls through to the function
if we're linking against libcapstone/llvm or does the dlsym. There are
7 such functions in the LLVM code. I don't think shimming 7 functions
is at the scale of hugely intrusive.

> If it's a serious concern maybe investigate an executable compressor?

Perhaps just have a squashfs partition.

Fwiw, excluding dependencies I think compression on the events is a
good solution. Convert json events/metrics to a sysfs file with the
cpuid in the path, add the compressed file to the binary as data, find
"json" events by iterating the directories in the compressed file,
etc. A single filesystem approach to event lookup can mean we do some
kind of unionfs style lookup of events, which could support users
adding their own events/metrics in a directory. Zip doesn't support
compressing across files, which is something of a requirement here,
other formats do but it's a case of optimizing for some kind of
libarchive sweet spot. The opportunity here is that about 70% of the
binary is event encodings, a compressed file is about 30% of the
current binary size, so we could reduce the binary size by about 40%.

> > The X86
> > disassembler alone in libllvm is of a size comparable to the perf tool
>
> I agree that LLVM is a serious bloat and DLL hell concern, but I don't think
> dlopen is the answer here.

Agreed, but it's where the code is at. addr2line command or use LLVM
for some performance. I think having an inbuilt solution would be best
longer term - we spend energy trying to parse and understand text
output from tools/libraries when the information is just sitting there
in the instruction encoding. Such a solution would be brittle for
things like new dwarf information, so we may want to have fallbacks
like LLVM but having a loosely coupled dependency using dlopen feels
preferable there, to aid package maintainers.

Thanks,
Ian