netdev - Re: bpftool cross-compilation for arm64

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <20181219173226.GK20955@mini-arch.hsd1.ca.comcast.net>
Date:   Wed, 19 Dec 2018 09:32:26 -0800
From:   Stanislav Fomichev <sdf@...ichev.me>
To:     Ivan Babrou <ivan@...udflare.com>
Cc:     netdev@...r.kernel.org, jakub.kicinski@...ronome.com,
        Ignat Korchagin <ignat@...udflare.com>
Subject: Re: bpftool cross-compilation for arm64

On 12/18, Ivan Babrou wrote:
> On Tue, Dec 18, 2018 at 7:02 PM Stanislav Fomichev <sdf@...ichev.me> wrote:
> >
> > On 12/18, Ivan Babrou wrote:
> > > On Tue, Dec 18, 2018 at 6:06 PM Stanislav Fomichev <sdf@...ichev.me> wrote:
> > > >
> > > > On 12/18, Ivan Babrou wrote:
> > > > > On Tue, Dec 18, 2018 at 4:30 PM Stanislav Fomichev <sdf@...ichev.me> wrote:
> > > > > >
> > > > > > On 12/18, Ivan Babrou wrote:
> > > > > > > Hey,
> > > > > > >
> > > > > > > Is it possible to cross-compile bpftool for arm64? I'm trying to do
> > > > > > > the same thing we do for perf, but bpftool Makefile doesn't seem to
> > > > > > > support CROSS_COMPILE variable, so one needs to pass
> > > > > > > CC=aarch64-linux-gnu-gcc as well. Problems do not end there as it
> > > > > > What error are you getting? Can you try setting the ARCH:
> > > > > > make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
> > > > >
> > > > > It just fails with the following:
> > > > >
> > > > > $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
> > > > >
> > > > > Auto-detecting system features:
> > > > > ...                        libbfd: [ on  ]
> > > > > ...        disassembler-four-args: [ OFF ]
> > > > >
> > > > > make[1]: Entering directory '/tmp/build/linux-4.19.10/tools/lib/bpf'
> > > > > make[1]: Leaving directory '/tmp/build/linux-4.19.10/tools/lib/bpf'
> > > > >   LINK     bpftool
> > > > > /usr/bin/ld: /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a(libbpf-in.o):
> > > > > Relocations in generic ELF (EM: 183)
> > > > > /usr/bin/ld: /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a(libbpf-in.o):
> > > > > Relocations in generic ELF (EM: 183)
> > > > > /usr/bin/ld: /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a(libbpf-in.o):
> > > > > Relocations in generic ELF (EM: 183)
> > > > > /usr/bin/ld: /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a(libbpf-in.o):
> > > > > Relocations in generic ELF (EM: 183)
> > > > > /usr/bin/ld: /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a(libbpf-in.o):
> > > > > Relocations in generic ELF (EM: 183)
> > > > > /usr/bin/ld: /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a(libbpf-in.o):
> > > > > Relocations in generic ELF (EM: 183)
> > > > > /usr/bin/ld: /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a(libbpf-in.o):
> > > > > Relocations in generic ELF (EM: 183)
> > > > > /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a: error adding symbols:
> > > > > File in wrong format
> > > > > collect2: error: ld returned 1 exit status
> > > > > Makefile:96: recipe for target 'bpftool' failed
> > > > > make: *** [bpftool] Error 1
> > > > >
> > > > > The reason is that CROSS_COMPILE doesn't do anything:
> > > > >
> > > > > $ grep ^CC Makefile
> > > > > CC = gcc
> > > > Ah, we do "include ../../scripts/Makefile.include" on the first line of the
> > > > Makefile, it should define proper CC, but this can override it.
> > > >
> > > > Did you try removing it? Or changing to CC ?= gcc
> > >
> > > Removing definitely helps.
> > Do you want to send a patch for that? Otherwise, I can write it
> > down somewhere and send the fix out once that perf patch that I'm
> > waiting for lands in.
> 
> Sure, I'll try to send a patch tomorrow.
> 
> > >
> > > > >
> > > > > > [..]
> > > > > > > requires EXTRA_LDFLAGS="-lz" too. With that I'm getting a long sad
> > > > > > > error about missing stuff from libbfd:
> > > > > > For this I've been waiting for the following patch to be merged
> > > > > > first: https://lkml.org/lkml/2018/11/16/243
> > > > > > Basically, on some platforms, -lbfd requires -liberty and/or -lz.
> > > > > > We need to test each case individually for bpftool, but the feature
> > > > > > tests are semi-broken without the patch above.
> > > > >
> > > > > Adding -liberty doesn't help:
> > > > >
> > > > > $ make ARCH=arm64 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
> > > > > CC=aarch64-linux-gnu-gcc EXTRA_LDFLAGS="-lz -liberty"^C
> > > > > (reverse-i-search)`QUI': make -C
> > > > > /tmp/build/linux-4.19.10/tools/bpf/bpftool KBUILD_BUILD_TIMESTAMP="Mon
> > > > > Sep 27 00:00:00 UTC 2010" ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
> > > > > CC=aarch64-linux-gnu-gcc EXTRA_LDFLAGS="-lz -liberty"
> > > > > O=/tmp/build/arm64/bpftool
> > > > > DESTDIR=/tmp/build/arm64/debian/bpftooltmp/usr install
> > > > > LOCALVERSION=-cloudflare-2018.12.6-dev QUIET_LINK= ^CIET_CC=
> > > > > ivan@...338:/tmp/build/linux-4.19.10/tools/bpf/bpftool$ make
> > > > > ARCH=arm64 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
> > > > > CC=aarch64-linux-gnu-gcc EXTRA_LDFLAGS="-lz -liberty" QUIET_LINK=
> > > > > QUIET_CC=
> > > > >
> > > > > Auto-detecting system features:
> > > > > ...                        libbfd: [ on  ]
> > > > > ...        disassembler-four-args: [ OFF ]
> > > > >
> > > > > make[1]: Entering directory '/tmp/build/linux-4.19.10/tools/lib/bpf'
> > > > > make[1]: Leaving directory '/tmp/build/linux-4.19.10/tools/lib/bpf'
> > > > > aarch64-linux-gnu-gcc -O2 -W -Wall -Wextra -Wno-unused-parameter
> > > > > -Wshadow -Wno-missing-field-initializers -DPACKAGE='"bpftool"'
> > > > > -D__EXPORTED_HEADERS__ -I/tmp/build/linux-4.19.10/kernel/bpf/
> > > > > -I/tmp/build/linux-4.19.10/tools/include
> > > > > -I/tmp/build/linux-4.19.10/tools/include/uapi
> > > > > -I/tmp/build/linux-4.19.10/tools/lib/bpf
> > > > > -I/tmp/build/linux-4.19.10/tools/perf -DBPFTOOL_VERSION='"4.19.10"'
> > > > > -DCOMPAT_NEED_REALLOCARRAY -lz -liberty -o bpftool map_perf_ring.o
> > > > > xlated_dumper.o perf.o prog.o btf_dumper.o common.o cgroup.o main.o
> > > > > json_writer.o cfg.o map.o jit_disasm.o disasm.o
> > > > > /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a -lelf -lbfd -lopcodes
> > > > > /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a
> > > >
> > > > EXTRA_LDFLAGS are added too early :-( Looking at your command:
> > > > gcc ... -lz -liberty -o bpftool map_perf_ring.o ... .../libbpf.a -lelf -lbfd -lopcodes.
> > > >
> > > > Try changing LIBS in the tools/bpf/bpftool/Makefile to:
> > > >
> > > > LIBS = -lelf -lbfd -liberty -lz -lopcodes $(LIBBPF)
> > >
> > > I also needed -ldl:
> > >
> > > LIBS = -lelf -lbfd -lz -liberty -ldl -lopcodes $(LIBBPF)
> > >
> > > Still it fails:
> > > LIBS = -lelf -lbfd -lz -liberty -ldl -lopcodes $(LIBBPF)
> > It looks like at least floatformat_ieee_single_little comes from
> > libiberty, so doing something like the following should help (moving
> > -liberty to the right):
> >
> > LIBS = -lelf -lbfd -lz -ldl -lopcodes -liberty $(LIBBPF)
> 
> Yes, that did it:
> 
> LIBS = -lelf -lbfd -lz -ldl -lopcodes -liberty $(LIBBPF)
Awesome, I can try to fix it once my perf patch lands, until then you'd
have to hack this LIBS in the Makefile :-(
> 
> > Btw, why do you need -dl? I didn't see it complain about any dlxxx symbols.
> 
> It does complain if I remove it:
> 
> $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- QUIET_LINK=
> 
> Auto-detecting system features:
> ...                        libbfd: [ on  ]
> ...        disassembler-four-args: [ OFF ]
> 
> make[1]: Entering directory '/tmp/build/linux-4.19.10/tools/lib/bpf'
> make[1]: Leaving directory '/tmp/build/linux-4.19.10/tools/lib/bpf'
> aarch64-linux-gnu-gcc -O2 -W -Wall -Wextra -Wno-unused-parameter
> -Wshadow -Wno-missing-field-initializers -DPACKAGE='"bpftool"'
> -D__EXPORTED_HEADERS__ -I/tmp/build/linux-4.19.10/kernel/bpf/
> -I/tmp/build/linux-4.19.10/tools/include
> -I/tmp/build/linux-4.19.10/tools/include/uapi
> -I/tmp/build/linux-4.19.10/tools/lib/bpf
> -I/tmp/build/linux-4.19.10/tools/perf -DBPFTOOL_VERSION='"4.19.10"'
> -DCOMPAT_NEED_REALLOCARRAY  -o bpftool map_perf_ring.o xlated_dumper.o
> perf.o prog.o btf_dumper.o common.o cgroup.o main.o json_writer.o
> cfg.o map.o jit_disasm.o disasm.o
> /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a -lelf -lbfd -lz
> -lopcodes -liberty /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a
> /lib/aarch64-linux-gnu/libbfd.a(plugin.o): In function `try_load_plugin':
> /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/plugin.c:218:
> undefined reference to `dlopen'
> /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/plugin.c:225:
> undefined reference to `dlsym'
> /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/plugin.c:221:
> undefined reference to `dlerror'
> collect2: error: ld returned 1 exit status
> Makefile:96: recipe for target 'bpftool' failed
> make: *** [bpftool] Error 1
> 
> Thanks for helping out wit this!
> 
> > >
> > > $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- QUIET_LINK=
> > >
> > > Auto-detecting system features:
> > > ...                        libbfd: [ on  ]
> > > ...        disassembler-four-args: [ OFF ]
> > >
> > > make[1]: Entering directory '/tmp/build/linux-4.19.10/tools/lib/bpf'
> > > make[1]: Leaving directory '/tmp/build/linux-4.19.10/tools/lib/bpf'
> > > aarch64-linux-gnu-gcc -O2 -W -Wall -Wextra -Wno-unused-parameter
> > > -Wshadow -Wno-missing-field-initializers -DPACKAGE='"bpftool"'
> > > -D__EXPORTED_HEADERS__ -I/tmp/build/linux-4.19.10/kernel/bpf/
> > > -I/tmp/build/linux-4.19.10/tools/include
> > > -I/tmp/build/linux-4.19.10/tools/include/uapi
> > > -I/tmp/build/linux-4.19.10/tools/lib/bpf
> > > -I/tmp/build/linux-4.19.10/tools/perf -DBPFTOOL_VERSION='"4.19.10"'
> > > -DCOMPAT_NEED_REALLOCARRAY  -o bpftool map_perf_ring.o xlated_dumper.o
> > > perf.o prog.o btf_dumper.o common.o cgroup.o main.o json_writer.o
> > > cfg.o map.o jit_disasm.o disasm.o
> > > /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a -lelf -lbfd -lz
> > > -liberty -ldl -lopcodes
> > > /tmp/build/linux-4.19.10/tools/lib/bpf/libbpf.a
> > > /lib/aarch64-linux-gnu/libopcodes.a(arm-dis.o): In function `print_insn_neon':
> > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/opcodes/../../opcodes/arm-dis.c:4452:
> > > undefined reference to `floatformat_ieee_single_little'
> > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/opcodes/../../opcodes/arm-dis.c:4452:
> > > undefined reference to `floatformat_ieee_single_little'
> > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/opcodes/../../opcodes/arm-dis.c:4452:
> > > undefined reference to `floatformat_to_double'
> > > collect2: error: ld returned 1 exit status
> > > Makefile:96: recipe for target 'bpftool' failed
> > > make: *** [bpftool] Error 1
> > >
> > > > This is, again, a semi-known issue, I need that perf patch to get into
> > > > the tree first. It's probably time to ping that original thread, it's
> > > > been quit for about a month...
> > > >
> > > > > /lib/aarch64-linux-gnu/libbfd.a(elflink.o): In function
> > > > > `elf_link_add_object_symbols':
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/elflink.c:4938:
> > > > > undefined reference to `objalloc_free_block'
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/elflink.c:5185:
> > > > > undefined reference to `_sch_istable'
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/elflink.c:5185:
> > > > > undefined reference to `_sch_istable'
> > > > > /lib/aarch64-linux-gnu/libbfd.a(elflink.o): In function
> > > > > `bfd_elf_size_dynamic_sections':
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/elflink.c:6373:
> > > > > undefined reference to `lbasename'
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/elflink.c:6604:
> > > > > undefined reference to `lbasename'
> > > > > /lib/aarch64-linux-gnu/libbfd.a(elflink.o): In function `elf_output_implib':
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/elflink.c:11172:
> > > > > undefined reference to `xmalloc'
> > > > > /lib/aarch64-linux-gnu/libbfd.a(archures.o): In function `bfd_default_scan':
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/archures.c:1171:
> > > > > undefined reference to `_sch_istable'
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/archures.c:1171:
> > > > > undefined reference to `_sch_istable'
> > > > > ...
> > > > > /lib/aarch64-linux-gnu/libopcodes.a(arm-dis.o): In function `print_insn_neon':
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/opcodes/../../opcodes/arm-dis.c:4452:
> > > > > undefined reference to `floatformat_ieee_single_little'
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/opcodes/../../opcodes/arm-dis.c:4452:
> > > > > undefined reference to `floatformat_ieee_single_little'
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/opcodes/../../opcodes/arm-dis.c:4452:
> > > > > undefined reference to `floatformat_to_double'
> > > > > /lib/aarch64-linux-gnu/libopcodes.a(arm-dis.o): In function
> > > > > `parse_disassembler_options':
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/opcodes/../../opcodes/arm-dis.c:6177:
> > > > > undefined reference to `_sch_istable'
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/opcodes/../../opcodes/arm-dis.c:6177:
> > > > > undefined reference to `_sch_istable'
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/opcodes/../../opcodes/arm-dis.c:6180:
> > > > > undefined reference to `_sch_istable'
> > > > > /lib/aarch64-linux-gnu/libopcodes.a(arm-dis.o): In function `print_insn':
> > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/opcodes/../../opcodes/arm-dis.c:6754:
> > > > > undefined reference to `_sch_istable'
> > > > > collect2: error: ld returned 1 exit status
> > > > > Makefile:96: recipe for target 'bpftool' failed
> > > > > make: *** [bpftool] Error 1
> > > > >
> > > > > > >
> > > > > > > /lib/aarch64-linux-gnu/libbfd.a(elflink.o): In function
> > > > > > > `elf_link_add_object_symbols':
> > > > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/elflink.c:4938:
> > > > > > > undefined reference to `objalloc_free_block'
> > > > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/elflink.c:5185:
> > > > > > > undefined reference to `_sch_istable'
> > > > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/elflink.c:5185:
> > > > > > > undefined reference to `_sch_istable'
> > > > > > > /lib/aarch64-linux-gnu/libbfd.a(elflink.o): In function
> > > > > > > `bfd_elf_size_dynamic_sections':
> > > > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/elflink.c:6373:
> > > > > > > undefined reference to `lbasename'
> > > > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/elflink.c:6604:
> > > > > > > undefined reference to `lbasename'
> > > > > > > /lib/aarch64-linux-gnu/libbfd.a(elflink.o): In function `elf_output_implib':
> > > > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/elflink.c:11172:
> > > > > > > undefined reference to `xmalloc'
> > > > > > > /lib/aarch64-linux-gnu/libbfd.a(archures.o): In function `bfd_default_scan':
> > > > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/archures.c:1171:
> > > > > > > undefined reference to `_sch_istable'
> > > > > > > /build/binutils-YyZpUv/binutils-2.28/builddir-single/bfd/../../bfd/archures.c:1171:
> > > > > > > undefined reference to `_sch_istable'
> > > > > > >
> > > > > > > Binutils for arm64 are installed:
> > > > > > >
> > > > > > > $ dpkg -l | fgrep binutils
> > > > > > > ii  binutils                           2.28-5
> > > > > > > amd64        GNU assembler, linker and binary utilities
> > > > > > > ii  binutils-aarch64-linux-gnu         2.28-5
> > > > > > > amd64        GNU binary utilities, for aarch64-linux-gnu target
> > > > > > > ii  binutils-dev                       2.28-5
> > > > > > > amd64        GNU binary utilities (BFD development files)
> > > > > > >
> > > > > > > I'd love for it to work out of the box just like perf.