Message-ID: <54c7f0bf-7f6b-464e-0cce-b48ab1b05201@fb.com>
Date:   Mon, 5 Oct 2020 20:05:35 -0700
From:   Yonghong Song <yhs@...com>
To:     Andrii Nakryiko <andrii.nakryiko@...il.com>
CC:     bpf <bpf@...r.kernel.org>, Networking <netdev@...r.kernel.org>,
        Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Kernel Team <kernel-team@...com>
Subject: Re: [PATCH bpf-next 1/2] samples/bpf: change Makefile to cope with
 latest llvm



On 10/5/20 12:27 PM, Andrii Nakryiko wrote:
> On Fri, Oct 2, 2020 at 10:16 PM Yonghong Song <yhs@...com> wrote:
>>
>>
>>
>> On 10/2/20 9:22 PM, Andrii Nakryiko wrote:
>>> On Fri, Oct 2, 2020 at 7:19 PM Yonghong Song <yhs@...com> wrote:
>>>>
>>>> With latest llvm trunk, bpf programs under samples/bpf
>>>> directory, if using CORE, may experience the following
>>>> errors:
>>>>
>>>> LLVM ERROR: Cannot select: intrinsic %llvm.preserve.struct.access.index
>>>> PLEASE submit a bug report to https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.llvm.org_&d=DwIBaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=DA8e1B5r073vIqRrFz7MRA&m=_D9tvuWQ6EbYqcMBdVB0qqRMVdV6Etws5ITtx8Pa1ZM&s=BwTAvhipPl-Az_WaiJDbqU8yl__NvG8W4HmCqWqHdqg&e=  and include the crash backtrace.
>>>> Stack dump:
>>>> 0.      Program arguments: llc -march=bpf -filetype=obj -o samples/bpf/test_probe_write_user_kern.o
>>>> 1.      Running pass 'Function Pass Manager' on module '<stdin>'.
>>>> 2.      Running pass 'BPF DAG->DAG Pattern Instruction Selection' on function '@..._prog1'
>>>>    #0 0x000000000183c26c llvm::sys::PrintStackTrace(llvm::raw_ostream&, int)
>>>>       (/data/users/yhs/work/llvm-project/llvm/build.cur/install/bin/llc+0x183c26c)
>>>> ...
>>>>    #7 0x00000000017c375e (/data/users/yhs/work/llvm-project/llvm/build.cur/install/bin/llc+0x17c375e)
>>>>    #8 0x00000000016a75c5 llvm::SelectionDAGISel::CannotYetSelect(llvm::SDNode*)
>>>>       (/data/users/yhs/work/llvm-project/llvm/build.cur/install/bin/llc+0x16a75c5)
>>>>    #9 0x00000000016ab4f8 llvm::SelectionDAGISel::SelectCodeCommon(llvm::SDNode*, unsigned char const*,
>>>>       unsigned int) (/data/users/yhs/work/llvm-project/llvm/build.cur/install/bin/llc+0x16ab4f8)
>>>> ...
>>>> Aborted (core dumped) | llc -march=bpf -filetype=obj -o samples/bpf/test_probe_write_user_kern.o
>>>>
>>>> The reason is due to llvm change https://urldefense.proofpoint.com/v2/url?u=https-3A__reviews.llvm.org_D87153&d=DwIBaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=DA8e1B5r073vIqRrFz7MRA&m=_D9tvuWQ6EbYqcMBdVB0qqRMVdV6Etws5ITtx8Pa1ZM&s=fo_LvXqHJx_m0m0pJJiDdOUcOzVXm2_iYoxPhpqpzng&e=
>>>> where the CORE relocation global generation is moved from the beginning
>>>> of target dependent optimization (llc) to the beginning
>>>> of target independent optimization (opt).
>>>>
>>>> Since samples/bpf programs did not use vmlinux.h and its clang compilation
>>>> uses native architecture, we need to adjust arch triple at opt level
>>>> to do CORE relocation global generation properly. Otherwise, the above
>>>> error will appear.
>>>>
>>>> This patch fixed the issue by introduce opt and llvm-dis to compilation chain,
>>>> which will do proper CORE relocation global generation as well as O2 level
>>>> optimization. Tested with llvm10, llvm11 and trunk/llvm12.
>>>>
>>>> Signed-off-by: Yonghong Song <yhs@...com>
>>>> ---
>>>>    samples/bpf/Makefile | 6 +++++-
>>>>    1 file changed, 5 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile
>>>> index 4f1ed0e3cf9f..79c5fdea63d2 100644
>>>> --- a/samples/bpf/Makefile
>>>> +++ b/samples/bpf/Makefile
>>>> @@ -211,6 +211,8 @@ TPROGLDLIBS_xsk_fwd         += -pthread
>>>>    #  make M=samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang
>>>>    LLC ?= llc
>>>>    CLANG ?= clang
>>>> +OPT ?= opt
>>>> +LLVM_DIS ?= llvm-dis
>>>>    LLVM_OBJCOPY ?= llvm-objcopy
>>>>    BTF_PAHOLE ?= pahole
>>>>
>>>> @@ -314,7 +316,9 @@ $(obj)/%.o: $(src)/%.c
>>>>                   -Wno-address-of-packed-member -Wno-tautological-compare \
>>>>                   -Wno-unknown-warning-option $(CLANG_ARCH_ARGS) \
>>>>                   -I$(srctree)/samples/bpf/ -include asm_goto_workaround.h \
>>>> -               -O2 -emit-llvm -c $< -o -| $(LLC) -march=bpf $(LLC_FLAGS) -filetype=obj -o $@
>>>> +               -O2 -emit-llvm -Xclang -disable-llvm-passes -c $< -o - | \
>>>> +               $(OPT) -O2 -mtriple=bpf-pc-linux | $(LLVM_DIS) | \
>>>> +               $(LLC) -march=bpf $(LLC_FLAGS) -filetype=obj -o $@
> 
> This is an extremely unusual set of steps, and might be worthwhile to
> leave a comment explaining what's going on, so that I or someone else
> doesn't ask the same question few months later :)

Sure. Will add some comments and send v2.

> 
> At any rate, this fixes the issue, so:
> 
> Acked-by: Andrii Nakryiko <andrii@...nel.org>
> 
>>>
>>> I keep forgetting exact details of why we do this native clang + llc
>>> pipeline instead of just doing `clang -target bpf`? Is it still
>>
>> samples/bpf programs did not use vmlinux.h. they directly use
>> kernel-devel headers, hence they need to first compile with native arch
>> for clang but later change target arch to bpf to generate final byte code.
>> They cannot just do 'clang -target bpf' without vmlinux.h.
> 
> Ok, right, thanks for explanation. I wonder if this "native" clang +
> llc pass will also help with vmlinux.h on 32-bit architectures (though
> with my recent patches that shouldn't be necessary, so this is more of
> a curiosity, rather than the need).

Yes, I think this should help 32-bit as all the 32-bit long/ptr size is 
captured in IR. I probably tried before with 32bit x86 uprobe, but I 
forgot details...

> 
>>
>> But changing to use vmlinux.h is a much bigger project and I merely
>> want to make it just work so people won't make/compile samples/bpf
>> and get compilation errors.
> 
> Right, of course.
> 
>>
>>> relevant and necessary, or we can just simplify it now?
>>>
>>>>    ifeq ($(DWARF2BTF),y)
>>>>           $(BTF_PAHOLE) -J $@
>>>>    endif
>>>> --
>>>> 2.24.1
>>>>

Powered by blists - more mailing lists