netdev - Re: next-20190723: bpf/seccomp

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CA+icZUXsPRWmH3i-9=TK-=2HviubRqpAeDJGriWHgK1fkFhgUg@mail.gmail.com>
Date:   Fri, 26 Jul 2019 23:02:35 +0200
From:   Sedat Dilek <sedat.dilek@...il.com>
To:     Yonghong Song <yhs@...com>
Cc:     Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Martin Lau <kafai@...com>, Song Liu <songliubraving@...com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "bpf@...r.kernel.org" <bpf@...r.kernel.org>,
        Clang-Built-Linux ML <clang-built-linux@...glegroups.com>,
        Kees Cook <keescook@...omium.org>,
        Nick Desaulniers <ndesaulniers@...gle.com>,
        Nathan Chancellor <natechancellor@...il.com>
Subject: Re: next-20190723: bpf/seccomp - systemd/journald issue?

On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@...il.com> wrote:
>
> Hi Yonghong Song,
>
> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@...com> wrote:
> >
> >
> >
> > On 7/26/19 1:26 AM, Sedat Dilek wrote:
> > > Hi,
> > >
> > > I have opened a new issue in the ClangBuiltLinux issue tracker.
> >
> > Glad to know clang 9 has asm goto support and now It can compile
> > kernel again.
> >
>
> Yupp.
>
> > >
> > > I am seeing a problem in the area bpf/seccomp causing
> > > systemd/journald/udevd services to fail.
> > >
> > > [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
> > > to connect stdout to the journal socket, ignoring: Connection refused
> > >
> > > This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
> > > BFD linker ld.bfd on Debian/buster AMD64.
> > > In both cases I use clang-9 (prerelease).
> >
> > Looks like it is a lld bug.
> >
> > I see the stack trace has __bpf_prog_run32() which is used by
> > kernel bpf interpreter. Could you try to enable bpf jit
> >    sysctl net.core.bpf_jit_enable = 1
> > If this passed, it will prove it is interpreter related.
> >
>
> After...
>
> sysctl -w net.core.bpf_jit_enable=1
>
> I can start all failed systemd services.
>
> systemd-journald.service
> systemd-udevd.service
> haveged.service
>
> This is in maintenance mode.
>
> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
>

This is what I did:

Jul 26 22:43:06 iniza kernel: BUG: unable to handle page fault for
address: ffffffffa8203370
Jul 26 22:43:06 iniza kernel: #PF: supervisor read access in kernel mode
Jul 26 22:43:06 iniza kernel: #PF: error_code(0x0000) - not-present page
Jul 26 22:43:06 iniza kernel: PGD 2cfa0e067 P4D 2cfa0e067 PUD
2cfa0f063 PMD 450829063 PTE 800ffffd30bfc062
Jul 26 22:43:06 iniza kernel: Oops: 0000 [#3] SMP PTI
Jul 26 22:43:06 iniza kernel: CPU: 3 PID: 436 Comm: systemd-udevd
Tainted: G      D           5.3.0-rc1-7-amd64-cbl-asmgoto
#7~buster+dileks1
Jul 26 22:43:06 iniza kernel: Hardware name: LENOVO
20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019
Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0
Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c
00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c
c3 48 83 c3 08 0f b6
 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48 83 f8 1e 0f 8f c8 00
Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec0327a88 EFLAGS: 00010246
Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec0327b30 RBX:
ffffb3cec00d1038 RCX: 0000000000000000
Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec0327b10 RSI:
00000000000000ac RDI: ffffb3cec0327ab0
Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec0327aa0 R08:
ffff9b33c94c0a00 R09: 0000000000000000
Jul 26 22:43:06 iniza kernel: R10: ffff9b33cfe14e00 R11:
ffffffffa77b8210 R12: 0000000000000000
Jul 26 22:43:06 iniza kernel: R13: ffffb3cec00d1000 R14:
0000000000000000 R15: ffffb3cec0327ab0
Jul 26 22:43:06 iniza kernel: FS:  00007f7ac2d28d40(0000)
GS:ffff9b33d2580000(0000) knlGS:0000000000000000
Jul 26 22:43:06 iniza kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3:
000000044f3ea006 CR4: 00000000003606e0
Jul 26 22:43:06 iniza kernel: Call Trace:
Jul 26 22:43:06 iniza kernel:  __bpf_prog_run32+0x44/0x70
Jul 26 22:43:06 iniza kernel:  ? security_sock_rcv_skb+0x3f/0x60
Jul 26 22:43:06 iniza kernel:  sk_filter_trim_cap+0xe4/0x220
Jul 26 22:43:06 iniza kernel:  ? __skb_clone+0x2e/0x100
Jul 26 22:43:06 iniza kernel:  netlink_broadcast_filtered+0x2df/0x4f0
Jul 26 22:43:06 iniza kernel:  netlink_sendmsg+0x34f/0x3c0
Jul 26 22:43:06 iniza kernel:  ___sys_sendmsg+0x315/0x330
Jul 26 22:43:06 iniza kernel:  ? seccomp_run_filters+0x54/0x110
Jul 26 22:43:06 iniza kernel:  ? filename_parentat+0x210/0x490
Jul 26 22:43:06 iniza kernel:  ? __seccomp_filter+0xf7/0x6e0
Jul 26 22:43:06 iniza kernel:  ? __d_alloc+0x159/0x1c0
Jul 26 22:43:06 iniza kernel:  ? kmem_cache_free+0x1e/0x5c0
Jul 26 22:43:06 iniza kernel:  ? fast_dput+0x73/0xb0
Jul 26 22:43:06 iniza kernel:  __x64_sys_sendmsg+0x97/0xe0
Jul 26 22:43:06 iniza kernel:  do_syscall_64+0x59/0x90
Jul 26 22:43:06 iniza kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 26 22:43:06 iniza kernel: RIP: 0033:0x7f7ac3519914
Jul 26 22:43:06 iniza kernel: Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff
ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 e9 5d 0c 00 8b 00 85 c0 75
13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 41
89 d4 55 48 89 f5 53
Jul 26 22:43:06 iniza kernel: RSP: 002b:00007ffcfb66a478 EFLAGS:
00000246 ORIG_RAX: 000000000000002e
Jul 26 22:43:06 iniza kernel: RAX: ffffffffffffffda RBX:
0000561e28ac9390 RCX: 00007f7ac3519914
Jul 26 22:43:06 iniza kernel: RDX: 0000000000000000 RSI:
00007ffcfb66a4a0 RDI: 000000000000000d
Jul 26 22:43:06 iniza kernel: RBP: 0000561e28acd210 R08:
0000561e28990140 R09: 0000000000000002
Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11:
0000000000000246 R12: 0000000000000000
Jul 26 22:43:06 iniza kernel: R13: 0000000000000000 R14:
000000000000005e R15: 00007ffcfb66a490
Jul 26 22:43:06 iniza kernel: Modules linked in: nfsd auth_rpcgss
nfs_acl lockd grace i2c_dev parport_pc ppdev lp parport sunrpc
efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic mbcache crc16
jbd2 btrfs zstd_decompress zstd_compress algif_skcipher af_alg sd_mod
uas usb_storage scsi_mod hid_generic usbhid hid dm_crypt dm_mod raid10
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
raid6_pq libcrc32c raid1 raid0 multipath linear md_mod
crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
aesni_intel i915 intel_lpss_pci nvme aes_x86_64 glue_helper
i2c_algo_bit crypto_simd cryptd xhci_pci psmouse e1000e drm_kms_helper
xhci_hcd i2c_i801 nvme_core intel_lpss drm usbcore thermal wmi video
button
Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370
Jul 26 22:43:06 iniza kernel: ---[ end trace 312670b063bd0391 ]---
Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0
Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c
00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c
c3 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48
83 f8 1e 0f 8f c8 00
Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec0253cb8 EFLAGS: 00010246
Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec0253d60 RBX:
ffffb3cec00e9038 RCX: 0000000000000002
Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec0253d40 RSI:
00000000000000ac RDI: ffffb3cec0253ce0
Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec0253cd0 R08:
0000000000000000 R09: ffffb3cec0253f58
Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11:
ffffffffa77b8210 R12: 000000007fff0000
Jul 26 22:43:06 iniza kernel: R13: ffffb3cec0253eb8 R14:
0000000000000000 R15: ffffb3cec0253ce0
Jul 26 22:43:06 iniza kernel: FS:  00007f7ac2d28d40(0000)
GS:ffff9b33d2580000(0000) knlGS:0000000000000000
Jul 26 22:43:06 iniza kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3:
000000044f3ea006 CR4: 00000000003606e0
Jul 26 22:43:06 iniza kernel: BUG: unable to handle page fault for
address: ffffffffa8203370
Jul 26 22:43:06 iniza kernel: #PF: supervisor read access in kernel mode
Jul 26 22:43:06 iniza kernel: #PF: error_code(0x0000) - not-present page
Jul 26 22:43:06 iniza kernel: PGD 2cfa0e067 P4D 2cfa0e067 PUD
2cfa0f063 PMD 450829063 PTE 800ffffd30bfc062
Jul 26 22:43:06 iniza kernel: Oops: 0000 [#4] SMP PTI
Jul 26 22:43:06 iniza kernel: CPU: 0 PID: 437 Comm: systemd-udevd
Tainted: G      D           5.3.0-rc1-7-amd64-cbl-asmgoto
#7~buster+dileks1
Jul 26 22:43:06 iniza kernel: Hardware name: LENOVO
20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019
Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0
Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c
00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c
c3 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48
83 f8 1e 0f 8f c8 00
Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec032fa88 EFLAGS: 00010246
Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec032fb30 RBX:
ffffb3cec00d1038 RCX: 0000000000000000
Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec032fb10 RSI:
00000000000000ac RDI: ffffb3cec032fab0
Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec032faa0 R08:
ffff9b33cf34b000 R09: 0000000000000000
Jul 26 22:43:06 iniza kernel: R10: ffff9b33cf3a3400 R11:
ffffffffa77b8210 R12: 0000000000000000
Jul 26 22:43:06 iniza kernel: R13: ffffb3cec00d1000 R14:
0000000000000000 R15: ffffb3cec032fab0
Jul 26 22:43:06 iniza kernel: FS:  00007f7ac2d28d40(0000)
GS:ffff9b33d2400000(0000) knlGS:0000000000000000
Jul 26 22:43:06 iniza kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3:
000000044724a001 CR4: 00000000003606f0
Jul 26 22:43:06 iniza kernel: Call Trace:
Jul 26 22:43:06 iniza kernel:  __bpf_prog_run32+0x44/0x70
Jul 26 22:43:06 iniza kernel:  ? prep_new_page+0x47/0x1a0
Jul 26 22:43:06 iniza kernel:  ? security_sock_rcv_skb+0x3f/0x60
Jul 26 22:43:06 iniza kernel:  sk_filter_trim_cap+0xe4/0x220
Jul 26 22:43:06 iniza kernel:  ? __skb_clone+0x2e/0x100
Jul 26 22:43:06 iniza kernel:  netlink_broadcast_filtered+0x2df/0x4f0
Jul 26 22:43:06 iniza kernel:  netlink_sendmsg+0x34f/0x3c0
Jul 26 22:43:06 iniza kernel:  ___sys_sendmsg+0x315/0x330
Jul 26 22:43:06 iniza kernel:  ? seccomp_run_filters+0x54/0x110
Jul 26 22:43:06 iniza kernel:  ? filename_parentat+0x210/0x490
Jul 26 22:43:06 iniza kernel:  ? __seccomp_filter+0xf7/0x6e0
Jul 26 22:43:06 iniza kernel:  ? __d_alloc+0x159/0x1c0
Jul 26 22:43:06 iniza kernel:  ? kmem_cache_free+0x1e/0x5c0
Jul 26 22:43:06 iniza kernel:  ? fast_dput+0x73/0xb0
Jul 26 22:43:06 iniza kernel:  __x64_sys_sendmsg+0x97/0xe0
Jul 26 22:43:06 iniza kernel:  do_syscall_64+0x59/0x90
Jul 26 22:43:06 iniza kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 26 22:43:06 iniza kernel: RIP: 0033:0x7f7ac3519914
Jul 26 22:43:06 iniza kernel: Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff
ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 e9 5d 0c 00 8b 00 85 c0 75
13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 41
89 d4 55 48 89 f5 53
Jul 26 22:43:06 iniza kernel: RSP: 002b:00007ffcfb66a478 EFLAGS:
00000246 ORIG_RAX: 000000000000002e
Jul 26 22:43:06 iniza kernel: RAX: ffffffffffffffda RBX:
0000561e28aaa600 RCX: 00007f7ac3519914
Jul 26 22:43:06 iniza kernel: RDX: 0000000000000000 RSI:
00007ffcfb66a4a0 RDI: 000000000000000e
Jul 26 22:43:06 iniza kernel: RBP: 0000561e28aaaac0 R08:
0000561e28990140 R09: 0000000000000002
Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11:
0000000000000246 R12: 0000000000000000
Jul 26 22:43:06 iniza kernel: R13: 0000000000000000 R14:
000000000000005d R15: 00007ffcfb66a490
Jul 26 22:43:06 iniza kernel: Modules linked in: nfsd auth_rpcgss
nfs_acl lockd grace i2c_dev parport_pc ppdev lp parport sunrpc
efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic mbcache crc16
jbd2 btrfs zstd_decompress zstd_compress algif_skcipher af_alg sd_mod
uas usb_storage scsi_mod hid_generic usbhid hid dm_crypt dm_mod raid10
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
raid6_pq libcrc32c raid1 raid0 multipath linear md_mod
crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
aesni_intel i915 intel_lpss_pci nvme aes_x86_64 glue_helper
i2c_algo_bit crypto_simd cryptd xhci_pci psmouse e1000e drm_kms_helper
xhci_hcd i2c_i801 nvme_core intel_lpss drm usbcore thermal wmi video
button
Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370
Jul 26 22:43:06 iniza kernel: ---[ end trace 312670b063bd0392 ]---

Full `journalctl -xb` attached.

- Sedat -

Download attachment "journalctl-xb.txt.gz" of type "application/gzip" (26676 bytes)