| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Sat, 8 Jan 2022 14:38:56 -0800
From: Ian Rogers <irogers@...gle.com>
To: Arnaldo Carvalho de Melo <acme@...nel.org>
Cc: Andrii Nakryiko <andrii.nakryiko@...il.com>,
Song Liu <songliubraving@...com>, Jiri Olsa <jolsa@...nel.org>,
Steven Rostedt <rostedt@...dmis.org>,
Jiri Slaby <jirislaby@...nel.org>,
Namhyung Kim <namhyung@...nel.org>, bpf <bpf@...r.kernel.org>,
Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
Quentin Monnet <quentin@...valent.com>
Subject: Re: perf build broken seemingly due to libbpf changes, checking...
On Fri, Jan 7, 2022 at 12:18 PM Arnaldo Carvalho de Melo
<acme@...nel.org> wrote:
>
> Em Fri, Jan 07, 2022 at 11:26:50AM -0800, Ian Rogers escreveu:
> > On Fri, Jan 7, 2022 at 10:32 AM Arnaldo Carvalho de Melo <acme@...nel.org> wrote:
> > > Em Thu, Jan 06, 2022 at 07:30:34PM -0800, Ian Rogers escreveu:
> > > > On Thu, Jan 6, 2022 at 2:04 PM Ian Rogers <irogers@...gle.com> wrote:
> > > > > On Thu, Jan 6, 2022 at 1:44 PM Andrii Nakryiko <andrii.nakryiko@...il.com> wrote:
> > > > > > On Thu, Jan 6, 2022 at 1:42 PM Ian Rogers <irogers@...gle.com> wrote:
> > > > So tools/lib also provides subcmd, symbol and api. These will need
> > > > Makefiles to allow an install and likely the header file structure
> > > > altering. This seems like too big a fix for the next 5.16rc, wdyt?
>
> > > Right, I think the best thing is to revert the patch Jiri pointed out,
> > > right?
>
> > Your call. There is a latent bug that with LIBTRACEEVENT_DYNAMIC we
> > are using tools/lib/traceevent header files. Reverting the change
> > means we don't break because of this, but it means that people
> > building with LIBTRACEEVENT_DYNAMIC and newer libtraceevent (at least
> > my employer :-) ) lose logging. I can carry the change locally, so not
> > a big loss :-)
>
> Just for a while, we should get this fixed for v5.17, for v5.16 a revert
> is required :-\
>
> > There are a few issues stemming from this:
> > 1) we've identified the current build is wrong for xxx_DYNAMIC options
> > as tools/lib versions headers always override
>
> yeap
>
> > 2) to address this we should make the tools/lib things proper
> > libraries like libbpf, libtraceevent, etc.
>
> yeap
>
> > 3) once we have proper libraries, we need to update the perf build to
> > build non-dynamic libraries then depend on the built/installed header
> > files
>
> yeap
>
> > I expect at least some of this is going to break when testing on many
> > distributions as that just seems to be what always happens, and
> > changing the build in this significant way is going to have
> > implications. Doing this means that the code base is in better shape
> > and logging works.
>
> Right, its nice that I have a mechanism to test build in 80+ distro
> versions/cross-build environments. :-)
>
> > To counter some of the many distribution pain, do
> > you have a way to reproduce your testing? My OpenSuSE recipe is:
>
> So I'm adding this:
>
> [perfbuilder@...e tumbleweed]$ git diff rx_and_build.sh
> diff --git a/opensuse/tumbleweed/rx_and_build.sh b/opensuse/tumbleweed/rx_and_build.sh
> index fbc8845..0510ef1 100755
> --- a/opensuse/tumbleweed/rx_and_build.sh
> +++ b/opensuse/tumbleweed/rx_and_build.sh
> @@ -11,11 +11,19 @@ build_perf_gcc() {
> make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" NO_LIBELF=1 -C tools/perf O=/tmp/build/perf || exit 1
> rm -rf /tmp/build/perf ; mkdir /tmp/build/perf
> make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" NO_LIBBPF=1 -C tools/perf O=/tmp/build/perf || exit 1
> + [ -d /usr/include/traceevent/ ] && \
> + make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" LIBTRACEEVENT_DYNAMIC=1 -C tools/perf O=/tmp/build/perf || exit 1
> set +o xtrace
> }
>
> build_perf_clang() {
> set -o xtrace
> +
> + if [ ! $NO_BUILD_BPF_SKEL ] ; then
> + rm -rf /tmp/build/perf ; mkdir /tmp/build/perf
> + make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" BUILD_BPF_SKEL=1 -C tools/perf O=/tmp/build/perf || exit 1
> + fi
> +
> rm -rf /tmp/build/perf ; mkdir /tmp/build/perf
> make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" -C tools/perf O=/tmp/build/perf CC=clang || exit 1
> rm -rf /tmp/build/perf ; mkdir /tmp/build/perf
> @@ -26,6 +34,8 @@ build_perf_clang() {
> make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" LIBCLANGLLVM=1 -C tools/perf O=/tmp/build/perf CC=clang || exit 1
> rm -rf /tmp/build/perf ; mkdir /tmp/build/perf
> make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" LIBCLANGLLVM=1 -C tools/perf O=/tmp/build/perf || exit 1
> + [ -d /usr/include/traceevent/ ] && \
> + make $EXTRA_MAKE_ARGS ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE EXTRA_CFLAGS="$EXTRA_CFLAGS" LIBTRACEEVENT_DYNAMIC=1 -C tools/perf O=/tmp/build/perf || exit 1
> set +o xtrace
> }
>
> @@ -48,6 +58,7 @@ TARBALL=`basename $TARBALL_URL`
> xzcat $TARBALL | tar xvf -
> SRCDIR=`echo $TARBALL | sed -r 's/(.*).tar\..*/\1/g'`
> cd /git/$SRCDIR
> +echo -n BUILD_TARBALL_HEAD=
> cat HEAD
>
> # print the version for dm to harvest and put in the status line
> [perfbuilder@...e tumbleweed]$
>
> And I'll add this to 'make -C tools/perf build-test' too, what I have in
> rx_and_build.sh is a limited set of 'build-test', building with it in
> all containers take more time, it is possible if one does:
>
> [perfbuilder@...e ~]$ export BUILD_TARBALL=http://192.168.100.2/perf/perf-5.16.0-rc8.tar.xz
> [perfbuilder@...e ~]$ export BUILD_CMD='make -C tools/perf build-cmd'
> [perfbuilder@...e ~]$ dm
>
> Then it will do it for gcc and for clang, takes a while...
>
> My Dockerfile for tumbleweed, btw, is:
>
> [perfbuilder@...e tumbleweed]$ cat Dockerfile
> # acmel/linux-perf-tools-build-opensuse:tumbleweed
> FROM docker.io/opensuse/tumbleweed
> MAINTAINER Arnaldo Carvalho de Melo <acme@...nel.org>
> # shadow for groupadd
> RUN zypper -n update && \
> zypper -n install shadow \
> make gcc gcc-c++ flex bison cpio \
> bc file findutils clang llvm curl tar xz \
> libunwind-devel openssl-devel slang-devel python-devel \
> systemtap-sdt-devel gtk2-devel xz-devel binutils-devel \
> libelf-devel libdw-devel audit-devel libzstd-devel \
> java-1_8_0-openjdk-headless libcap-devel \
> clang-devel llvm-devel perl libnuma-devel \
> libbpf-devel libtraceevent-devel \
> babeltrace-devel OpenCSD-devel cmake xmlto asciidoc && \
> zypper clean --all && \
> mkdir -m 777 -p /git /tmp/build/perf /tmp/build/objtool /tmp/build/linux && \
> groupadd -r perfbuilder && \
> useradd -m -r -g perfbuilder perfbuilder && \
> chown -R perfbuilder.perfbuilder /tmp/build/ /git/
> USER perfbuilder
> COPY rx_and_build.sh /
> COPY .bash_profile /home/perfbuilder/
> ENTRYPOINT ["/rx_and_build.sh"]
> [perfbuilder@...e tumbleweed]$
>
> > ```
> > # Get the image
> > docker pull opensuse/tumbleweed
> > # Start it with an interactive bash shell and mounting the current
> > directory as /kernel-src
> > sudo docker run --privileged -it --net=host --env="DISPLAY" --mount
> > type=bind,source="$(pwd)",target=/kernel-src opensuse/tumbleweed
> > /bin/bash
> > # Install missing rpms
> > zypper install make gcc diffutils flex bison kernel-devel findutils
> > libelf-devel python3 kernel-kvmsmall-devel glibc-devel
> > # Go to /kernel-src and build into /tmp, etc.
> > ```
> > But finding every distribution, every rpm, etc. is quite laborious.
>
> Yeah, but I did it already for quite a few distros :-)
>
> [perfbuilder@...e linux-perf-tools-build]$ find . -name Dockerfile | nl
> 1 ./alpine/3.10/Dockerfile
> 2 ./alpine/3.11/Dockerfile
> 3 ./alpine/3.12/Dockerfile
> 4 ./alpine/3.13/Dockerfile
> 5 ./alpine/3.5/Dockerfile
> 6 ./alpine/3.6/Dockerfile
> 7 ./alpine/3.7/Dockerfile
> 8 ./alpine/3.8/Dockerfile
> 9 ./alpine/3.9/Dockerfile
> 10 ./alpine/edge/Dockerfile
> 11 ./alpine/3.14/Dockerfile
> 12 ./alpine/3.4/Dockerfile
> 13 ./alpine/3.15/Dockerfile
> 14 ./alt/p8/Dockerfile
> 15 ./alt/p9/Dockerfile
> 16 ./alt/sisyphus/Dockerfile
> 17 ./alt/p10/Dockerfile
> 18 ./amazonlinux/1/Dockerfile
> 19 ./amazonlinux/2/Dockerfile
> 20 ./android/end-of-life/r22b/arm/Dockerfile
> 21 ./android/end-of-life/r12b/arm/Dockerfile
> 22 ./android/end-of-life/r15c/arm/Dockerfile
> 23 ./archlinux/base/Dockerfile
> 24 ./centos/8/Dockerfile
> 25 ./centos/end-of-life/5/Dockerfile
> 26 ./centos/end-of-life/6/Dockerfile
> 27 ./centos/end-of-life/7/Dockerfile
> 28 ./centos/stream/Dockerfile
> 29 ./clearlinux/latest/Dockerfile
> 30 ./debian/10/Dockerfile
> 31 ./debian/9/Dockerfile
> 32 ./debian/end-of-life/7/Dockerfile
> 33 ./debian/end-of-life/8/Dockerfile
> 34 ./debian/experimental/x-arm64/Dockerfile
> 35 ./debian/experimental/x-mips/Dockerfile
> 36 ./debian/experimental/x-mips64/Dockerfile
> 37 ./debian/experimental/x-mipsel/Dockerfile
> 38 ./debian/experimental/Dockerfile
> 39 ./debian/11/Dockerfile
> 40 ./fedora/22/Dockerfile
> 41 ./fedora/23/Dockerfile
> 42 ./fedora/24/x-ARC-uClibc/Dockerfile
> 43 ./fedora/24/Dockerfile
> 44 ./fedora/25/Dockerfile
> 45 ./fedora/26/Dockerfile
> 46 ./fedora/27/Dockerfile
> 47 ./fedora/28/Dockerfile
> 48 ./fedora/29/Dockerfile
> 49 ./fedora/30/Dockerfile
> 50 ./fedora/31/Dockerfile
> 51 ./fedora/32/Dockerfile
> 52 ./fedora/33/Dockerfile
> 53 ./fedora/34/x-ARC-glibc/Dockerfile
> 54 ./fedora/34/x-ARC-uClibc/Dockerfile
> 55 ./fedora/34/Dockerfile
> 56 ./fedora/end-of-life/21/Dockerfile
> 57 ./fedora/end-of-life/20/Dockerfile
> 58 ./fedora/rawhide/Dockerfile
> 59 ./fedora/35/Dockerfile
> 60 ./gentoo/end-of-life/stage3-amd64/Dockerfile
> 61 ./gentoo/stage3/Dockerfile
> 62 ./mageia/5/Dockerfile
> 63 ./mageia/6/Dockerfile
> 64 ./mageia/7/Dockerfile
> 65 ./mageia/8/Dockerfile
> 66 ./manjaro/base/Dockerfile
> 67 ./openmandriva/cooker/Dockerfile
> 68 ./opensuse/15.0/Dockerfile
> 69 ./opensuse/15.1/Dockerfile
> 70 ./opensuse/15.2/Dockerfile
> 71 ./opensuse/15.3/Dockerfile
> 72 ./opensuse/end-of-life/13.2/Dockerfile
> 73 ./opensuse/end-of-life/42.1/Dockerfile
> 74 ./opensuse/end-of-life/42.2/Dockerfile
> 75 ./opensuse/end-of-life/42.3/Dockerfile
> 76 ./opensuse/tumbleweed/Dockerfile
> 77 ./opensuse/15.4/Dockerfile
> 78 ./oraclelinux/8/Dockerfile
> 79 ./oraclelinux/end-of-life/6/Dockerfile
> 80 ./oraclelinux/end-of-life/7/Dockerfile
> 81 ./rhel7/Dockerfile
> 82 ./ubuntu/16.04/x-arm/Dockerfile
> 83 ./ubuntu/16.04/x-arm64/Dockerfile
> 84 ./ubuntu/16.04/x-powerpc/Dockerfile
> 85 ./ubuntu/16.04/x-powerpc64/Dockerfile
> 86 ./ubuntu/16.04/x-powerpc64el/Dockerfile
> 87 ./ubuntu/16.04/x-s390/Dockerfile
> 88 ./ubuntu/16.04/Dockerfile
> 89 ./ubuntu/18.04/x-arm/Dockerfile
> 90 ./ubuntu/18.04/x-arm64/Dockerfile
> 91 ./ubuntu/18.04/x-m68k/Dockerfile
> 92 ./ubuntu/18.04/x-powerpc/Dockerfile
> 93 ./ubuntu/18.04/x-powerpc64/Dockerfile
> 94 ./ubuntu/18.04/x-powerpc64el/Dockerfile
> 95 ./ubuntu/18.04/x-riscv64/Dockerfile
> 96 ./ubuntu/18.04/x-s390/Dockerfile
> 97 ./ubuntu/18.04/x-sh4/Dockerfile
> 98 ./ubuntu/18.04/x-sparc64/Dockerfile
> 99 ./ubuntu/18.04/Dockerfile
> 100 ./ubuntu/20.04/x-powerpc64el/Dockerfile
> 101 ./ubuntu/20.04/x-s390/Dockerfile
> 102 ./ubuntu/20.04/Dockerfile
> 103 ./ubuntu/20.04/x-arm/Dockerfile
> 104 ./ubuntu/20.10/Dockerfile
> 105 ./ubuntu/21.04/Dockerfile
> 106 ./ubuntu/21.04/end-of-life/x-mips/Dockerfile
> 107 ./ubuntu/21.04/end-of-life/x-mips64/Dockerfile
> 108 ./ubuntu/end-of-life/15.04/Dockerfile
> 109 ./ubuntu/end-of-life/15.10/Dockerfile
> 110 ./ubuntu/end-of-life/16.10/Dockerfile
> 111 ./ubuntu/end-of-life/17.04/Dockerfile
> 112 ./ubuntu/end-of-life/17.10/Dockerfile
> 113 ./ubuntu/end-of-life/18.10/Dockerfile
> 114 ./ubuntu/end-of-life/19.04/Dockerfile
> 115 ./ubuntu/end-of-life/19.10/Dockerfile
> 116 ./ubuntu/end-of-life/19.10/x-alpha/Dockerfile
> 117 ./ubuntu/end-of-life/19.10/x-arm64/Dockerfile
> 118 ./ubuntu/end-of-life/19.10/x-hppa/Dockerfile
> 119 ./ubuntu/end-of-life/12.04/Dockerfile
> 120 ./ubuntu/end-of-life/14.04/end-of-life/x-linaro-arm64/Dockerfile
> 121 ./ubuntu/end-of-life/14.04/Dockerfile
> 122 ./ubuntu/21.10/Dockerfile
> 123 ./ubuntu/22.04/Dockerfile
> 124 ./almalinux/8/Dockerfile
> 125 ./rockylinux/8/Dockerfile
> 126 ./ubi8/Dockerfile
> [perfbuilder@...e linux-perf-tools-build]$
Cool. Do you have this available externally?
Thanks,
Ian
Powered by blists - more mailing lists