netdev - Possible deadlock in bpf queue map

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <CABcoxUbYwuZUL-xm1+5juO42nJMgpQX7cNyQELYz+g2XkZi9TQ@mail.gmail.com>
Date: Wed, 6 Sep 2023 22:02:05 -0700
From: Hsin-Wei Hung <hsinweih@....edu>
To: Alexei Starovoitov <ast@...nel.org>, Daniel Borkmann <daniel@...earbox.net>, 
	Andrii Nakryiko <andrii@...nel.org>, Martin KaFai Lau <kafai@...com>, Song Liu <songliubraving@...com>, 
	Yonghong Song <yhs@...com>, John Fastabend <john.fastabend@...il.com>, KP Singh <kpsingh@...nel.org>, 
	netdev@...r.kernel.org, bpf@...r.kernel.org
Subject: Possible deadlock in bpf queue map

Hi,

Our bpf fuzzer, a customized Syzkaller, triggered a lockdep warning in
the bpf queue map in v5.15. Since queue_stack_maps.c has no major changes
since v5.15, we think this should still exist in the latest kernel.
The bug can be occasionally triggered, and we suspect one of the
eBPF programs involved to be the following one. We also attached the lockdep
warning at the end.

#define DEFINE_BPF_MAP_NO_KEY(the_map, TypeOfMap, MapFlags,
TypeOfValue, MaxEntries) \
        struct {                                                        \
            __uint(type, TypeOfMap);                                    \
            __uint(map_flags, (MapFlags));                              \
            __uint(max_entries, (MaxEntries));                          \
            __type(value, TypeOfValue);                                 \
        } the_map SEC(".maps");

DEFINE_BPF_MAP_NO_KEY(map_0, BPF_MAP_TYPE_QUEUE, 0 | BPF_F_WRONLY,
struct_0, 162);
SEC("perf_event")
int func(struct bpf_perf_event_data *ctx) {
        char v0[96] = {};
        uint64_t v1 = 0;
        v1 = bpf_map_pop_elem(&map_0, v0);
        return 163819661;
}


The program is attached to the following perf event.

struct perf_event_attr attr_type_hw = {
        .type = PERF_TYPE_HARDWARE,
        .config = PERF_COUNT_HW_CPU_CYCLES,
        .sample_freq = 50,
        .inherit = 1,
        .freq = 1,
};

================================WARNING: inconsistent lock state
5.15.26+ #2 Not tainted
--------------------------------
inconsistent {INITIAL USE} -> {IN-NMI} usage.
syz-executor.5/19749 [HC1[1]:SC0[0]:HE0:SE1] takes:
ffff88804c9fc198 (&qs->lock){..-.}-{2:2}, at: __queue_map_get+0x31/0x250
{INITIAL USE} state was registered at:
  lock_acquire+0x1a3/0x4b0
  _raw_spin_lock_irqsave+0x48/0x60
  __queue_map_get+0x31/0x250
  bpf_prog_577904e86c81dead_func+0x12/0x4b4
  trace_call_bpf+0x262/0x5d0
  perf_trace_run_bpf_submit+0x91/0x1c0
  perf_trace_sched_switch+0x46c/0x700
  __schedule+0x11b5/0x24a0
  schedule+0xd4/0x270
  futex_wait_queue_me+0x25f/0x520
  futex_wait+0x1e0/0x5f0
  do_futex+0x395/0x1890
  __x64_sys_futex+0x1cb/0x480
  do_syscall_64+0x3b/0xc0
  entry_SYSCALL_64_after_hwframe+0x44/0xae
irq event stamp: 13640
hardirqs last  enabled at (13639): [<ffffffff95eb2bf4>]
_raw_spin_unlock_irq+0x24/0x40
hardirqs last disabled at (13640): [<ffffffff95eb2d4d>]
_raw_spin_lock_irqsave+0x5d/0x60
softirqs last  enabled at (13464): [<ffffffff93e26de5>] __sys_bpf+0x3e15/0x4e80
softirqs last disabled at (13462): [<ffffffff93e26da3>] __sys_bpf+0x3dd3/0x4e80

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&qs->lock);
  <Interrupt>
    lock(&qs->lock);

 *** DEADLOCK ***

3 locks held by syz-executor.5/19749:
 #0: ffff88801904dd80 (&sb->s_type->i_mutex_key#17){++++}-{3:3}, at:
path_openat+0x1585/0x2760
 #1: ffff88800924c020 (&chan->lock){-.-.}-{2:2}, at:
p9_virtio_request+0x18c/0x640
 #2: ffffffff977d32e0 (rcu_read_lock){....}-{1:2}, at:
is_bpf_text_address+0x5/0x180

stack backtrace:
CPU: 0 PID: 19749 Comm: syz-executor.5 Not tainted 5.15.26+ #2
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.13.0-1ubuntu1.1 04/01/2014
Call Trace:
 <NMI>
 dump_stack_lvl+0x9d/0xc9
 lock_acquire.cold+0x3b/0x40
 ? lock_release+0x6c0/0x6c0
 ? __queue_map_get+0x31/0x250
 ? __perf_event_header__init_id+0x2bd/0x610
 ? look_up_lock_class+0x76/0xd0
 ? lock_acquire+0x282/0x4b0
 _raw_spin_lock_irqsave+0x48/0x60
 ? __queue_map_get+0x31/0x250
 __queue_map_get+0x31/0x250
 bpf_prog_9e60c62721f22dfa_func+0x5a/0xc18
 bpf_overflow_handler+0x17b/0x460
 ? perf_output_read+0x12b0/0x12b0
 ? __perf_event_account_interrupt+0xe9/0x3a0
 __perf_event_overflow+0x13f/0x3c0
 handle_pmi_common+0x59b/0x980
 ? intel_pmu_save_and_restart+0x100/0x100
 ? intel_bts_interrupt+0x10d/0x3d0
 intel_pmu_handle_irq+0x28a/0x8b0
 perf_event_nmi_handler+0x4d/0x70
 nmi_handle+0x147/0x390
 default_do_nmi+0x40/0x100
 exc_nmi+0x152/0x170
 end_repeat_nmi+0x16/0x55
RIP: 0010:lock_acquire+0x1b4/0x4b0
Code: f7 6a 00 45 0f b6 c9 6a 00 ff b4 24 f8 00 00 00 ff 74 24 18 e8
cd 98 ff ff b8 ff ff ff ff 48 83 c4 20 65 0f c1 05 fc 19 50 6c <83> f8
01 0f 85 7f 02 00 00 48 83 7c 24 08 00 0f 85 50 02 00 00 48
RSP: 0018:ffff88802825ec30 EFLAGS: 00000057
RAX: 0000000000000001 RBX: 1ffff1100504bd88 RCX: 00000000dea5863f
RDX: 1ffff110053cd1b6 RSI: 1ffff1100504bd6f RDI: 00000000fe94d1a3
RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff99172ba7
R10: fffffbfff322e574 R11: 0000000000000001 R12: 0000000000000002
R13: 0000000000000000 R14: ffffffff977d32e0 R15: 0000000000000000
 ? lock_acquire+0x1b4/0x4b0
 ? lock_acquire+0x1b4/0x4b0
 </NMI>