netdev - BUG: using __this_cpu_add() in preemptible in tcp_make

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <2d2ad1e5-8b03-0c59-4cf1-6a5cc85bbd94@cloudflare.com>
Date:   Wed, 18 Jan 2023 10:56:51 -0600
From:   Frederick Lawler <fred@...udflare.com>
To:     netdev@...r.kernel.org
Cc:     kernel-team@...udflare.com, edumazet@...gle.com,
        davem@...emloft.net, yoshfuji@...ux-ipv6.org, dsahern@...nel.org,
        kuba@...nel.org, pabeni@...hat.com
Subject: BUG: using __this_cpu_add() in preemptible in tcp_make_synack()

Hello,

We've been testing Linux 6.1.4, and came across an intermittent "BUG: 
using __this_cpu_add() in preemptible" [1] in our services leveraging 
TCP_FASTOPEN and our kernel configured with:

CONFIG_PREEMPT_BUILD=y
CONFIG_PREEMPT_VOLUNTARY=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_DYNAMIC=y
CONFIG_PREEMPT_RCU=y
CONFIG_HAVE_PREEMPT_DYNAMIC=y
CONFIG_HAVE_PREEMPT_DYNAMIC_CALL=y
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_DEBUG_PREEMPT=y

I'm not sure how related this is to commit 0a375c822497 ("tcp: 
tcp_rtx_synack() can be called from process context
"), as I haven't found a reliable reproducer yet.

The stack trace below has audit_*(), but we have other traces with 
do_tcp_setsockopt() mixed in the backtrace. Once we get to 
tcp_rx_synack() in the __release_sock() path, we get the same problem.

[1]:
BUG: using __this_cpu_add() in preemptible [00000000] code: nginx-ssl/209282
caller is tcp_make_synack+0x38d/0x5a0
CPU: 3 PID: 209282 Comm: nginx-ssl Kdump: loaded Tainted: G           O 
      6.1.4-cloudflare-2023.1.2 #1
Hardware name: Quanta Cloud Technology Inc. QuantaPlex T42S-2U/T42S-2U 
MB (Lewisburg-1G) CLX, BIOS 3B16.Q102 02/19/2020
Call Trace:
<TASK>
  dump_stack_lvl+0x34/0x48
  check_preemption_disabled+0xdd/0xe0
  tcp_make_synack+0x38d/0x5a0
  tcp_v4_send_synack+0x50/0x1f0
  tcp_rtx_synack+0x55/0x140
  ? load_balance+0xa91/0xd40
  ? _copy_to_iter+0x1d6/0x560
  inet_rtx_syn_ack+0x16/0x30
  tcp_check_req+0x39f/0x660
  tcp_rcv_state_process+0xa3/0x1020
  ? tcp_mstamp_refresh+0xe/0x40
  ? tcp_update_recv_tstamps+0x61/0x90
  ? tcp_recvmsg_locked+0x1eb/0x960
  tcp_v4_do_rcv+0xc6/0x280
  __release_sock+0xb4/0xc0
  release_sock+0x2b/0x90
  tcp_recvmsg+0x7c/0x200
  inet_recvmsg+0x52/0x130
  __sys_recvfrom+0xa8/0x120
  ? audit_filter_inodes.part.0+0x2e/0x110
  ? auditd_test_task+0x3c/0x50
  ? __audit_syscall_entry+0xd5/0x120
  __x64_sys_recvfrom+0x20/0x30
  do_syscall_64+0x38/0x90
  entry_SYSCALL_64_after_hwframe+0x4b/0xb5
RIP: 0033:0x7fafc19adc74
Code: 89 4c 24 1c e8 2d 41 f8 ff 44 8b 54 24 1c 8b 3c 24 45 31 c9 89 c5 
48 8b 54 24 10 48 8b 74 24 08 45 31 c0 b8 2d 00 00 00 0f 05 <48> 3d 00 
f0 ff ff 77 34 89 ef 48 89 04 24 e8 59 41 f8 ff 48 8b 04
RSP: 002b:00007ffc42e18a20 EFLAGS: 00000246 ORIG_RAX: 000000000000002d
RAX: ffffffffffffffda RBX: 00007faf26457870 RCX: 00007fafc19adc74
RDX: 0000000000000001 RSI: 00007ffc42e18a60 RDI: 000000000000076a
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000002 R11: 0000000000000246 R12: 00007faf2261fd40
R13: 00007faf26457870 R14: 00007faf1fe962c0 R15: 00007ffc42e18a60
</TASK>

Thanks,
Fred