Date:   Sat, 27 Jul 2019 08:43:24 +0200
From:   Sedat Dilek <sedat.dilek@...il.com>
To:     Alexei Starovoitov <alexei.starovoitov@...il.com>
Cc:     Yonghong Song <yhs@...com>, Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Martin Lau <kafai@...com>, Song Liu <songliubraving@...com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "bpf@...r.kernel.org" <bpf@...r.kernel.org>,
        Clang-Built-Linux ML <clang-built-linux@...glegroups.com>,
        Kees Cook <keescook@...omium.org>,
        Nick Desaulniers <ndesaulniers@...gle.com>,
        Nathan Chancellor <natechancellor@...il.com>
Subject: Re: next-20190723: bpf/seccomp - systemd/journald issue?

On Sat, Jul 27, 2019 at 4:24 AM Alexei Starovoitov
<alexei.starovoitov@...il.com> wrote:
>
> On Fri, Jul 26, 2019 at 2:19 PM Sedat Dilek <sedat.dilek@...il.com> wrote:
> >
> > On Fri, Jul 26, 2019 at 11:10 PM Yonghong Song <yhs@...com> wrote:
> > >
> > >
> > >
> > > On 7/26/19 2:02 PM, Sedat Dilek wrote:
> > > > On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@...il.com> wrote:
> > > >>
> > > >> Hi Yonghong Song,
> > > >>
> > > >> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@...com> wrote:
> > > >>>
> > > >>>
> > > >>>
> > > >>> On 7/26/19 1:26 AM, Sedat Dilek wrote:
> > > >>>> Hi,
> > > >>>>
> > > >>>> I have opened a new issue in the ClangBuiltLinux issue tracker.
> > > >>>
> > > >>> Glad to know clang 9 has asm goto support and now It can compile
> > > >>> kernel again.
> > > >>>
> > > >>
> > > >> Yupp.
> > > >>
> > > >>>>
> > > >>>> I am seeing a problem in the area bpf/seccomp causing
> > > >>>> systemd/journald/udevd services to fail.
> > > >>>>
> > > >>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
> > > >>>> to connect stdout to the journal socket, ignoring: Connection refused
> > > >>>>
> > > >>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
> > > >>>> BFD linker ld.bfd on Debian/buster AMD64.
> > > >>>> In both cases I use clang-9 (prerelease).
> > > >>>
> > > >>> Looks like it is a lld bug.
> > > >>>
> > > >>> I see the stack trace has __bpf_prog_run32() which is used by
> > > >>> kernel bpf interpreter. Could you try to enable bpf jit
> > > >>>     sysctl net.core.bpf_jit_enable = 1
> > > >>> If this passed, it will prove it is interpreter related.
> > > >>>
> > > >>
> > > >> After...
> > > >>
> > > >> sysctl -w net.core.bpf_jit_enable=1
> > > >>
> > > >> I can start all failed systemd services.
> > > >>
> > > >> systemd-journald.service
> > > >> systemd-udevd.service
> > > >> haveged.service
> > > >>
> > > >> This is in maintenance mode.
> > > >>
> > > >> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
> > > >>
> > > >
> > > > This is what I did:
> > >
> > > I probably won't have cycles to debug this potential lld issue.
> > > Maybe you already did, I suggest you put enough reproducible
> > > details in the bug you filed against lld so they can take a look.
> > >
> >
> > I understand and will put the journalctl-log into the CBL issue
> > tracker and update informations.
> >
> > Thanks for your help understanding the BPF correlations.
> >
> > Is setting 'net.core.bpf_jit_enable = 2' helpful here?
>
> jit_enable=1 is enough.
> Or use CONFIG_BPF_JIT_ALWAYS_ON to workaround.
>
> It sounds like clang miscompiles interpreter.
> modprobe test_bpf
> should be able to point out which part of interpreter is broken.

BROKEN: test_bpf: #294 BPF_MAXINSNS: Jump, gap, jump, ... jited:0

- Sedat -

Steps to reproduce:

# sysctl -n net.core.bpf_jit_enable
1

# modprobe -v test_bpf

[ Full dmesg-log attached ]

+[Sat Jul 27 07:08:54 2019] test_bpf: #294 BPF_MAXINSNS: Jump, gap,
jump, ... jited:0
+[Sat Jul 27 07:08:54 2019] BUG: unable to handle page fault for
address: ffffffffbea03370
+[Sat Jul 27 07:08:54 2019] #PF: supervisor read access in kernel mode
+[Sat Jul 27 07:08:54 2019] #PF: error_code(0x0000) - not-present page
+[Sat Jul 27 07:08:54 2019] PGD 53a0e067 P4D 53a0e067 PUD 53a0f063 PMD
450369063 PTE 800fffffacbfc062
+[Sat Jul 27 07:08:54 2019] Oops: 0000 [#43] SMP PTI
+[Sat Jul 27 07:08:54 2019] CPU: 1 PID: 591 Comm: modprobe Tainted: G
    D           5.3.0-rc1-7-amd64-cbl-asmgoto #7~buster+dileks1
+[Sat Jul 27 07:08:54 2019] Hardware name: LENOVO
20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019
+[Sat Jul 27 07:08:54 2019] RIP: 0010:___bpf_prog_run+0x40/0x14f0
+[Sat Jul 27 07:08:54 2019] Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c 00
00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c c3
48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e a0 be 48 83 f8 3b 7f 62 48 83
f8 1e 0f 8f c8 00
+[Sat Jul 27 07:08:54 2019] RSP: 0018:ffffb3140067ba58 EFLAGS: 00010246
+[Sat Jul 27 07:08:54 2019] RAX: ffffb3140067bb00 RBX:
ffffb31400255038 RCX: 0000000000000018
+[Sat Jul 27 07:08:54 2019] RDX: ffffb3140067bae0 RSI:
00000000000000ac RDI: ffffb3140067ba80
+[Sat Jul 27 07:08:54 2019] RBP: ffffb3140067ba70 R08:
ffffffffbf575562 R09: 0000000000000008
+[Sat Jul 27 07:08:54 2019] R10: 0000000000000000 R11:
ffffffffbdfb8210 R12: 0000000000000000
+[Sat Jul 27 07:08:54 2019] R13: ffffb31400255000 R14:
0000000000000000 R15: ffffb3140067ba80
+[Sat Jul 27 07:08:54 2019] FS:  00007fe10c790200(0000)
GS:ffff90f7d2480000(0000) knlGS:0000000000000000
+[Sat Jul 27 07:08:54 2019] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
+[Sat Jul 27 07:08:54 2019] CR2: ffffffffbea03370 CR3:
000000044bb78004 CR4: 00000000003606e0
+[Sat Jul 27 07:08:54 2019] Call Trace:
+[Sat Jul 27 07:08:54 2019]  __bpf_prog_run32+0x44/0x70
+[Sat Jul 27 07:08:54 2019]  ? vprintk_func+0x1cc/0x230
+[Sat Jul 27 07:08:54 2019]  ? __set_cyc2ns_scale+0x130/0x130
+[Sat Jul 27 07:08:54 2019]  ? ktime_get+0x53/0xb0
+[Sat Jul 27 07:08:54 2019]  __run_one+0x3f/0xe2 [test_bpf]
+[Sat Jul 27 07:08:54 2019]  test_bpf+0x3d6/0x5ac [test_bpf]
+[Sat Jul 27 07:08:54 2019]  ? 0xffffffffc0be9000
+[Sat Jul 27 07:08:54 2019]  init_module+0x15/0x26 [test_bpf]
+[Sat Jul 27 07:08:54 2019]  do_one_initcall+0xf9/0x280
+[Sat Jul 27 07:08:54 2019]  ? free_pcppages_bulk+0x28f/0x380
+[Sat Jul 27 07:08:54 2019]  ? free_unref_page_commit+0x93/0x170
+[Sat Jul 27 07:08:54 2019]  ? _cond_resched+0x1a/0x50
+[Sat Jul 27 07:08:54 2019]  ? kmem_cache_alloc_trace+0x1e5/0x230
+[Sat Jul 27 07:08:54 2019]  do_init_module+0x60/0x230
+[Sat Jul 27 07:08:54 2019]  load_module+0x30c0/0x33f0
+[Sat Jul 27 07:08:54 2019]  ? kernel_read_file_from_fd+0x46/0x80
+[Sat Jul 27 07:08:54 2019]  __se_sys_finit_module+0x102/0x110
+[Sat Jul 27 07:08:54 2019]  do_syscall_64+0x59/0x90
+[Sat Jul 27 07:08:54 2019]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
+[Sat Jul 27 07:08:54 2019] RIP: 0033:0x7fe10c8aaf59
+[Sat Jul 27 07:08:54 2019] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00
0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8
4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 07 6f 0c 00
f7 d8 64 89 01 48
+[Sat Jul 27 07:08:54 2019] RSP: 002b:00007ffed6130f78 EFLAGS:
00000246 ORIG_RAX: 0000000000000139
+[Sat Jul 27 07:08:54 2019] RAX: ffffffffffffffda RBX:
0000564035bfbce0 RCX: 00007fe10c8aaf59
+[Sat Jul 27 07:08:54 2019] RDX: 0000000000000000 RSI:
00005640347d13f0 RDI: 0000000000000003
+[Sat Jul 27 07:08:54 2019] RBP: 00005640347d13f0 R08:
0000000000000000 R09: 0000564035bfd8b0
+[Sat Jul 27 07:08:54 2019] R10: 0000000000000003 R11:
0000000000000246 R12: 0000000000000000
+[Sat Jul 27 07:08:54 2019] R13: 0000564035bfbe50 R14:
0000000000040000 R15: 0000564035bfbce0
+[Sat Jul 27 07:08:54 2019] Modules linked in: test_bpf(+) binfmt_misc
nfsd auth_rpcgss nfs_acl lockd grace i2c_dev parport_pc ppdev lp
parport sunrpc efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic
mbcache crc16 jbd2 btrfs zstd_decompress zstd_compress algif_skcipher
af_alg sd_mod dm_crypt dm_mod raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor uas usb_storage scsi_mod
hid_generic usbhid hid raid6_pq libcrc32c raid1 raid0 multipath linear
md_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
nvme aesni_intel xhci_pci xhci_hcd i2c_i801 nvme_core i915
i2c_algo_bit aes_x86_64 glue_helper crypto_simd e1000e cryptd
drm_kms_helper psmouse usbcore intel_lpss_pci drm intel_lpss thermal
wmi video button
+[Sat Jul 27 07:08:54 2019] CR2: ffffffffbea03370
+[Sat Jul 27 07:08:54 2019] ---[ end trace e8c8702f8ca94ac9 ]---
+[Sat Jul 27 07:08:54 2019] RIP: 0010:___bpf_prog_run+0x40/0x14f0
+[Sat Jul 27 07:08:54 2019] Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c 00
00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c c3
48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e a0 be 48 83 f8 3b 7f 62 48 83
f8 1e 0f 8f c8 00
+[Sat Jul 27 07:08:54 2019] RSP: 0018:ffffb31400327cb8 EFLAGS: 00010246
+[Sat Jul 27 07:08:54 2019] RAX: ffffb31400327d60 RBX:
ffffb314000e9038 RCX: 0000000000000002
+[Sat Jul 27 07:08:54 2019] RDX: ffffb31400327d40 RSI:
00000000000000ac RDI: ffffb31400327ce0
+[Sat Jul 27 07:08:54 2019] RBP: ffffb31400327cd0 R08:
0000000000000000 R09: ffffb31400327f58
+[Sat Jul 27 07:08:54 2019] R10: 0000000000000000 R11:
ffffffffbdfb8210 R12: 000000007fff0000
+[Sat Jul 27 07:08:54 2019] R13: ffffb31400327eb8 R14:
0000000000000000 R15: ffffb31400327ce0
+[Sat Jul 27 07:08:54 2019] FS:  00007fe10c790200(0000)
GS:ffff90f7d2480000(0000) knlGS:0000000000000000
+[Sat Jul 27 07:08:54 2019] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
+[Sat Jul 27 07:08:54 2019] CR2: ffffffffbea03370 CR3:
000000044bb78004 CR4: 00000000003606e0

View attachment "dmesg_5.3.0-rc1-7-amd64-cbl-asmgoto_moprobe-test_bpf.txt" of type "text/plain" (146400 bytes)

Powered by blists - more mailing lists