| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <YdigKX55J1y5xUiZ@kernel.org>
Date: Fri, 7 Jan 2022 17:18:49 -0300
From: Arnaldo Carvalho de Melo <acme@...nel.org>
To: Ian Rogers <irogers@...gle.com>
Cc: Andrii Nakryiko <andrii.nakryiko@...il.com>,
Song Liu <songliubraving@...com>, Jiri Olsa <jolsa@...nel.org>,
Steven Rostedt <rostedt@...dmis.org>,
Jiri Slaby <jirislaby@...nel.org>,
Namhyung Kim <namhyung@...nel.org>, bpf <bpf@...r.kernel.org>,
Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
Quentin Monnet <quentin@...valent.com>
Subject: Re: perf build broken seemingly due to libbpf changes, checking...
Em Fri, Jan 07, 2022 at 11:26:50AM -0800, Ian Rogers escreveu:
> On Fri, Jan 7, 2022 at 10:32 AM Arnaldo Carvalho de Melo <acme@...nel.org> wrote:
> > Em Thu, Jan 06, 2022 at 07:30:34PM -0800, Ian Rogers escreveu:
> > > On Thu, Jan 6, 2022 at 2:04 PM Ian Rogers <irogers@...gle.com> wrote:
> > > > On Thu, Jan 6, 2022 at 1:44 PM Andrii Nakryiko <andrii.nakryiko@...il.com> wrote:
> > > > > On Thu, Jan 6, 2022 at 1:42 PM Ian Rogers <irogers@...gle.com> wrote:
> > > So tools/lib also provides subcmd, symbol and api. These will need
> > > Makefiles to allow an install and likely the header file structure
> > > altering. This seems like too big a fix for the next 5.16rc, wdyt?
> > Right, I think the best thing is to revert the patch Jiri pointed out,
> > right?
> Your call. There is a latent bug that with LIBTRACEEVENT_DYNAMIC we
> are using tools/lib/traceevent header files. Reverting the change
> means we don't break because of this, but it means that people
> building with LIBTRACEEVENT_DYNAMIC and newer libtraceevent (at least
> my employer :-) ) lose logging. I can carry the change locally, so not
> a big loss :-)
Just for a while, we should get this fixed for v5.17, for v5.16 a revert
is required :-\
> There are a few issues stemming from this:
> 1) we've identified the current build is wrong for xxx_DYNAMIC options
> as tools/lib versions headers always override
yeap
> 2) to address this we should make the tools/lib things proper
> libraries like libbpf, libtraceevent, etc.
yeap
> 3) once we have proper libraries, we need to update the perf build to
> build non-dynamic libraries then depend on the built/installed header
> files
yeap
> I expect at least some of this is going to break when testing on many
> distributions as that just seems to be what always happens, and
> changing the build in this significant way is going to have
> implications. Doing this means that the code base is in better shape
> and logging works.
Right, its nice that I have a mechanism to test build in 80+ distro
versions/cross-build environments. :-)
> To counter some of the many distribution pain, do
> you have a way to reproduce your testing? My OpenSuSE recipe is:
So I'm adding this:
[perfbuilder@...e tumbleweed]$ git diff rx_and_build.sh
diff --git a/opensuse/tumbleweed/rx_and_build.sh b/opensuse/tumbleweed/rx_and_build.sh
index fbc8845..0510ef1 100755
--- a/opensuse/tumbleweed/rx_and_build.sh
+++ b/opensuse/tumbleweed/rx_and_build.sh
@@ -11,11 +11,19 @@ build_perf_gcc() {
make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" NO_LIBELF=1 -C tools/perf O=/tmp/build/perf || exit 1
rm -rf /tmp/build/perf ; mkdir /tmp/build/perf
make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" NO_LIBBPF=1 -C tools/perf O=/tmp/build/perf || exit 1
+ [ -d /usr/include/traceevent/ ] && \
+ make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" LIBTRACEEVENT_DYNAMIC=1 -C tools/perf O=/tmp/build/perf || exit 1
set +o xtrace
}
build_perf_clang() {
set -o xtrace
+
+ if [ ! $NO_BUILD_BPF_SKEL ] ; then
+ rm -rf /tmp/build/perf ; mkdir /tmp/build/perf
+ make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" BUILD_BPF_SKEL=1 -C tools/perf O=/tmp/build/perf || exit 1
+ fi
+
rm -rf /tmp/build/perf ; mkdir /tmp/build/perf
make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" -C tools/perf O=/tmp/build/perf CC=clang || exit 1
rm -rf /tmp/build/perf ; mkdir /tmp/build/perf
@@ -26,6 +34,8 @@ build_perf_clang() {
make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" LIBCLANGLLVM=1 -C tools/perf O=/tmp/build/perf CC=clang || exit 1
rm -rf /tmp/build/perf ; mkdir /tmp/build/perf
make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" LIBCLANGLLVM=1 -C tools/perf O=/tmp/build/perf || exit 1
+ [ -d /usr/include/traceevent/ ] && \
+ make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" LIBTRACEEVENT_DYNAMIC=1 -C tools/perf O=/tmp/build/perf || exit 1
set +o xtrace
}
@@ -48,6 +58,7 @@ TARBALL=`basename $TARBALL_URL`
xzcat $TARBALL | tar xvf -
SRCDIR=`echo $TARBALL | sed -r 's/(.*).tar\..*/\1/g'`
cd /git/$SRCDIR
+echo -n BUILD_TARBALL_HEAD=
cat HEAD
# print the version for dm to harvest and put in the status line
[perfbuilder@...e tumbleweed]$
And I'll add this to 'make -C tools/perf build-test' too, what I have in
rx_and_build.sh is a limited set of 'build-test', building with it in
all containers take more time, it is possible if one does:
[perfbuilder@...e ~]$ export BUILD_TARBALL=http://192.168.100.2/perf/perf-5.16.0-rc8.tar.xz
[perfbuilder@...e ~]$ export BUILD_CMD='make -C tools/perf build-cmd'
[perfbuilder@...e ~]$ dm
Then it will do it for gcc and for clang, takes a while...
My Dockerfile for tumbleweed, btw, is:
[perfbuilder@...e tumbleweed]$ cat Dockerfile
# acmel/linux-perf-tools-build-opensuse:tumbleweed
FROM docker.io/opensuse/tumbleweed
MAINTAINER Arnaldo Carvalho de Melo <acme@...nel.org>
# shadow for groupadd
RUN zypper -n update && \
zypper -n install shadow \
make gcc gcc-c++ flex bison cpio \
bc file findutils clang llvm curl tar xz \
libunwind-devel openssl-devel slang-devel python-devel \
systemtap-sdt-devel gtk2-devel xz-devel binutils-devel \
libelf-devel libdw-devel audit-devel libzstd-devel \
java-1_8_0-openjdk-headless libcap-devel \
clang-devel llvm-devel perl libnuma-devel \
libbpf-devel libtraceevent-devel \
babeltrace-devel OpenCSD-devel cmake xmlto asciidoc && \
zypper clean --all && \
mkdir -m 777 -p /git /tmp/build/perf /tmp/build/objtool /tmp/build/linux && \
groupadd -r perfbuilder && \
useradd -m -r -g perfbuilder perfbuilder && \
chown -R perfbuilder.perfbuilder /tmp/build/ /git/
USER perfbuilder
COPY rx_and_build.sh /
COPY .bash_profile /home/perfbuilder/
ENTRYPOINT ["/rx_and_build.sh"]
[perfbuilder@...e tumbleweed]$
> ```
> # Get the image
> docker pull opensuse/tumbleweed
> # Start it with an interactive bash shell and mounting the current
> directory as /kernel-src
> sudo docker run --privileged -it --net=host --env="DISPLAY" --mount
> type=bind,source="$(pwd)",target=/kernel-src opensuse/tumbleweed
> /bin/bash
> # Install missing rpms
> zypper install make gcc diffutils flex bison kernel-devel findutils
> libelf-devel python3 kernel-kvmsmall-devel glibc-devel
> # Go to /kernel-src and build into /tmp, etc.
> ```
> But finding every distribution, every rpm, etc. is quite laborious.
Yeah, but I did it already for quite a few distros :-)
[perfbuilder@...e linux-perf-tools-build]$ find . -name Dockerfile | nl
1 ./alpine/3.10/Dockerfile
2 ./alpine/3.11/Dockerfile
3 ./alpine/3.12/Dockerfile
4 ./alpine/3.13/Dockerfile
5 ./alpine/3.5/Dockerfile
6 ./alpine/3.6/Dockerfile
7 ./alpine/3.7/Dockerfile
8 ./alpine/3.8/Dockerfile
9 ./alpine/3.9/Dockerfile
10 ./alpine/edge/Dockerfile
11 ./alpine/3.14/Dockerfile
12 ./alpine/3.4/Dockerfile
13 ./alpine/3.15/Dockerfile
14 ./alt/p8/Dockerfile
15 ./alt/p9/Dockerfile
16 ./alt/sisyphus/Dockerfile
17 ./alt/p10/Dockerfile
18 ./amazonlinux/1/Dockerfile
19 ./amazonlinux/2/Dockerfile
20 ./android/end-of-life/r22b/arm/Dockerfile
21 ./android/end-of-life/r12b/arm/Dockerfile
22 ./android/end-of-life/r15c/arm/Dockerfile
23 ./archlinux/base/Dockerfile
24 ./centos/8/Dockerfile
25 ./centos/end-of-life/5/Dockerfile
26 ./centos/end-of-life/6/Dockerfile
27 ./centos/end-of-life/7/Dockerfile
28 ./centos/stream/Dockerfile
29 ./clearlinux/latest/Dockerfile
30 ./debian/10/Dockerfile
31 ./debian/9/Dockerfile
32 ./debian/end-of-life/7/Dockerfile
33 ./debian/end-of-life/8/Dockerfile
34 ./debian/experimental/x-arm64/Dockerfile
35 ./debian/experimental/x-mips/Dockerfile
36 ./debian/experimental/x-mips64/Dockerfile
37 ./debian/experimental/x-mipsel/Dockerfile
38 ./debian/experimental/Dockerfile
39 ./debian/11/Dockerfile
40 ./fedora/22/Dockerfile
41 ./fedora/23/Dockerfile
42 ./fedora/24/x-ARC-uClibc/Dockerfile
43 ./fedora/24/Dockerfile
44 ./fedora/25/Dockerfile
45 ./fedora/26/Dockerfile
46 ./fedora/27/Dockerfile
47 ./fedora/28/Dockerfile
48 ./fedora/29/Dockerfile
49 ./fedora/30/Dockerfile
50 ./fedora/31/Dockerfile
51 ./fedora/32/Dockerfile
52 ./fedora/33/Dockerfile
53 ./fedora/34/x-ARC-glibc/Dockerfile
54 ./fedora/34/x-ARC-uClibc/Dockerfile
55 ./fedora/34/Dockerfile
56 ./fedora/end-of-life/21/Dockerfile
57 ./fedora/end-of-life/20/Dockerfile
58 ./fedora/rawhide/Dockerfile
59 ./fedora/35/Dockerfile
60 ./gentoo/end-of-life/stage3-amd64/Dockerfile
61 ./gentoo/stage3/Dockerfile
62 ./mageia/5/Dockerfile
63 ./mageia/6/Dockerfile
64 ./mageia/7/Dockerfile
65 ./mageia/8/Dockerfile
66 ./manjaro/base/Dockerfile
67 ./openmandriva/cooker/Dockerfile
68 ./opensuse/15.0/Dockerfile
69 ./opensuse/15.1/Dockerfile
70 ./opensuse/15.2/Dockerfile
71 ./opensuse/15.3/Dockerfile
72 ./opensuse/end-of-life/13.2/Dockerfile
73 ./opensuse/end-of-life/42.1/Dockerfile
74 ./opensuse/end-of-life/42.2/Dockerfile
75 ./opensuse/end-of-life/42.3/Dockerfile
76 ./opensuse/tumbleweed/Dockerfile
77 ./opensuse/15.4/Dockerfile
78 ./oraclelinux/8/Dockerfile
79 ./oraclelinux/end-of-life/6/Dockerfile
80 ./oraclelinux/end-of-life/7/Dockerfile
81 ./rhel7/Dockerfile
82 ./ubuntu/16.04/x-arm/Dockerfile
83 ./ubuntu/16.04/x-arm64/Dockerfile
84 ./ubuntu/16.04/x-powerpc/Dockerfile
85 ./ubuntu/16.04/x-powerpc64/Dockerfile
86 ./ubuntu/16.04/x-powerpc64el/Dockerfile
87 ./ubuntu/16.04/x-s390/Dockerfile
88 ./ubuntu/16.04/Dockerfile
89 ./ubuntu/18.04/x-arm/Dockerfile
90 ./ubuntu/18.04/x-arm64/Dockerfile
91 ./ubuntu/18.04/x-m68k/Dockerfile
92 ./ubuntu/18.04/x-powerpc/Dockerfile
93 ./ubuntu/18.04/x-powerpc64/Dockerfile
94 ./ubuntu/18.04/x-powerpc64el/Dockerfile
95 ./ubuntu/18.04/x-riscv64/Dockerfile
96 ./ubuntu/18.04/x-s390/Dockerfile
97 ./ubuntu/18.04/x-sh4/Dockerfile
98 ./ubuntu/18.04/x-sparc64/Dockerfile
99 ./ubuntu/18.04/Dockerfile
100 ./ubuntu/20.04/x-powerpc64el/Dockerfile
101 ./ubuntu/20.04/x-s390/Dockerfile
102 ./ubuntu/20.04/Dockerfile
103 ./ubuntu/20.04/x-arm/Dockerfile
104 ./ubuntu/20.10/Dockerfile
105 ./ubuntu/21.04/Dockerfile
106 ./ubuntu/21.04/end-of-life/x-mips/Dockerfile
107 ./ubuntu/21.04/end-of-life/x-mips64/Dockerfile
108 ./ubuntu/end-of-life/15.04/Dockerfile
109 ./ubuntu/end-of-life/15.10/Dockerfile
110 ./ubuntu/end-of-life/16.10/Dockerfile
111 ./ubuntu/end-of-life/17.04/Dockerfile
112 ./ubuntu/end-of-life/17.10/Dockerfile
113 ./ubuntu/end-of-life/18.10/Dockerfile
114 ./ubuntu/end-of-life/19.04/Dockerfile
115 ./ubuntu/end-of-life/19.10/Dockerfile
116 ./ubuntu/end-of-life/19.10/x-alpha/Dockerfile
117 ./ubuntu/end-of-life/19.10/x-arm64/Dockerfile
118 ./ubuntu/end-of-life/19.10/x-hppa/Dockerfile
119 ./ubuntu/end-of-life/12.04/Dockerfile
120 ./ubuntu/end-of-life/14.04/end-of-life/x-linaro-arm64/Dockerfile
121 ./ubuntu/end-of-life/14.04/Dockerfile
122 ./ubuntu/21.10/Dockerfile
123 ./ubuntu/22.04/Dockerfile
124 ./almalinux/8/Dockerfile
125 ./rockylinux/8/Dockerfile
126 ./ubi8/Dockerfile
[perfbuilder@...e linux-perf-tools-build]$
Powered by blists - more mailing lists