netdev - TPROXY + Attempt to release TCP socket in state 1

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Tue, 5 Jul 2022 08:52:23 +0000
From:   Michelle Bies <mimbies@...look.com>
To:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: TPROXY + Attempt to release TCP socket in state 1

Hi

I'm trying to run squid with TPROXY enabled but the system reboots after logging these messages:

  IPv4: Attempt to release TCP socket in state 1 00000000dfe7f997

and after some seconds:

  rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
  rcu:5-....: (6 GPs behind) idle=f9e/0/0x1 softirq=184030/184030 fqs=1901
  (detected by 1, t=6302 jiffies, g=894549, q=17504)
  Sending NMI from CPU 1 to CPUs 5:
  NMI backtrace for cpu 5
  CPU: 5 PID: 0 Comm: swapper/5 Tainted: GO 5.4.181+ #9
  Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.12.1 12/04/2020
  RIP: 0010:__inet_lookup_established+0x4c/0xf7
  Code: 48 89 f5 53 48 c1 e5 20 44 89 cb 48 09 c5 44 09 c3 e8 7b fe ff ff 41 89 c6 41 89 c5 49 8b 07 45 23 77 10 4e 8d 3c f0 49 8b 17 <f6> c2 01 0f 85 89 00 00 00 44 39 6a a0 75 7b 39 5a a4 75 76 48 39
  RSP: 0018:ffffc90000208c48 EFLAGS: 00000a12
  RAX: ffff88844ad00000 RBX: 0000000001bb18e1 RCX: 00000000baf65c60
  RDX: 000000000002b145 RSI: 00000000e5cc7389 RDI: 000000009da543b1
  RBP: 490e1a683c63a8c0 R08: 00000000000018e1 R09: 0000000001bb0000
  R10: ffff8883c96da400 R11: 00000000000018e1 R12: ffffffff822b3740
  R13: 00000000593d39c6 R14: 00000000000139c6 R15: ffff88844ad9ce30
  FS:0000000000000000(0000) GS:ffff88844d940000(0000) knlGS:0000000000000000
  CS:0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007fd0297e8eb0 CR3: 00000003d37d4005 CR4: 00000000003606e0
  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  Call Trace:
 <IRQ>
 nf_sk_lookup_slow_v4+0x224/0xfbe [nf_socket_ipv4]
 ? do_raw_spin_lock+0x2b/0x52
 socket_match.isra.0+0x2f/0xf9 [xt_socket]
 ipt_do_table+0x26f/0x5c1 [ip_tables]
 ? nf_ct_key_equal+0x38/0x5d [nf_conntrack]
 ? nf_conntrack_in+0x2bd/0x46b [nf_conntrack]
 nf_hook_slow+0x3c/0xb4
 nf_hook.constprop.0+0xa5/0xc8
 ? l3mdev_l3_rcv.constprop.0+0x50/0x50
 ip_rcv+0x41/0x61
 __netif_receive_skb_one_core+0x74/0x95
 process_backlog+0x97/0x122
 net_rx_action+0xf5/0x2a3
 __do_softirq+0xc2/0x1c6
 irq_exit+0x41/0x80
 call_function_single_interrupt+0xf/0x20
 </IRQ>
  RIP: 0010:mwait_idle+0x5f/0x75
  Code: f0 31 d2 48 89 d1 65 48 8b 04 25 40 ac 01 00 0f 01 c8 48 8b 08 48 c1 e9 03 83 e1 01 75 0e e8 2b c3 6c ff 48 89 c8 fb 0f 01 c9 <eb> 01 fb bf 15 00 00 00     65 48 8b 34 25 40 ac 01 00 e9 26 c3 6c ff
  RSP: 0018:ffffc900000f3ee0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
  RAX: 0000000000000000 RBX: ffff88844beee3c0 RCX: 0000000000000000
  RDX: 0000000000000000 RSI: ffff88844beee3c0 RDI: 0000000000000015  
  RBP: 0000000000000000 R08: 0000000000840188 R09: 0000000000000000
  R10: ffff88844d964b80 R11: ffff88844d964bc0 R12: 0000000000000000
  R13: 0000000000000005 R14: 0000000000000000 R15: 0000000000000000
 do_idle+0xcf/0x1da
 ? do_idle+0x2/0x1da 
 cpu_startup_entry+0x1a/0x1c 
 start_secondary+0x14b/0x169 
 secondary_startup_64+0xa4/0xb0
  ------------[ cut here ]------------ 
  NETDEV WATCHDOG: eth2 (igb): transmit queue 4 timed out 
  WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:480 dev_watchdog+0xcf/0x128
  Modules linked in: xt_CLASSIFY xt_nfacct sch_sfq xt_IMQ xt_NFLOG xt_limit xt_pkttype xt_nat xt_MASQUERADE xt_REDIRECT xt_connlimit nf_conncount xt_time xt_geoip(O) xt_iprange xt_NFQUEUE xt_TPROXY nf_tproxy_ipv6 nf_tproxy_ipv4 xt_mac xt_mark 8021q garp mrp xt_multiport xt_socket nf_socket_ipv4 nf_socket_ipv6 ebtable_filter ebtable_nat ebtables xt_state xt_conntrack iptable_filter iptable_nat xt_set xt_connlabel xt_connmark iptable_mangle xt_recent iptable_raw sch_htb ip_set_hash_ipportip ip_set_hash_ip nfnetlink_acct nf_nat_pptp nf_conntrack_pptp nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat nfnetlink_log ip_set nfnetlink_queue nf_conntrack_netlink tun nfnetlink intel_lpss_pci intel_lpss imq igb sch_fq_codel nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bonding llc e1000e e1000 ip_tables
  CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O5.4.181+ #9
  Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.12.1 12/04/2020
  RIP: 0010:dev_watchdog+0xcf/0x128
  Code: 4b a1 00 00 75 38 48 89 ef c6 05 39 4b a1 00 01 e8 2e 7b fd ff 44 89 e1 48 89 ee 48 c7 c7 fb e6 0d 82 48 89 c2 e8 a0 98 0d 00 <0f> 0b eb 10 41 ff c4 48 05 40 01 00 00 41 39 f4 75 9d eb 13 48 8b
  RSP: 0018:ffffc90000158ec0 EFLAGS: 00010282
  RAX: 0000000000000000 RBX: ffff88844b234440 RCX: 0000000000000007
  RDX: 00000000000003f2 RSI: ffffc90000158db4 RDI: ffff88844d85b5b0
  RBP: ffff88844b234000 R08: 0000000000000001 R09: 0000000000014600
  R10: 0000000000000000 R11: 000000000000005c R12: 0000000000000004
  R13: ffffc90000158ef8 R14: ffffffff822050c0 R15: 0000000000000002
  FS:0000000000000000(0000) GS:ffff88844d840000(0000) knlGS:0000000000000000 
  CS:0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007faad50f2000 CR3: 0000000437d0e005 CR4: 00000000003606e0
  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  Call Trace:
 <IRQ> 
 call_timer_fn.isra.0+0x18/0x6f
 ? netif_tx_lock+0x7a/0x7a 
 __run_timers.part.0+0x12d/0x163 
 ? hrtimer_forward+0x73/0x7b 
 ? tick_sched_timer+0x57/0x62
 ? timerqueue_add+0x62/0x68
 run_timer_softirq+0x21/0x43 
 __do_softirq+0xc2/0x1c6 
 irq_exit+0x41/0x80
 smp_apic_timer_interrupt+0x6f/0x7a
 apic_timer_interrupt+0xf/0x20
 </IRQ>
  RIP: 0010:mwait_idle+0x5f/0x75
  Code: f0 31 d2 48 89 d1 65 48 8b 04 25 40 ac 01 00 0f 01 c8 48 8b 08 48 c1 e9 03 83 e1 01 75 0e e8 2b c3 6c ff 48 89 c8 fb 0f 01 c9 <eb> 01 fb bf 15 00 00 00 65 48 8b 34 25 40 ac 01 00 e9 26 c3 6c ff
  RSP: 0018:ffffc900000d3ee0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
  RAX: 0000000000000000 RBX: ffff88844beeaac0 RCX: 0000000000000000
  RDX: 0000000000000000 RSI: ffff88844beeaac0 RDI: 0000000000000015
  RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
  R10: ffff88844d864b80 R11: ffff88844d864bc0 R12: 0000000000000000
  R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
 do_idle+0xcf/0x1da
 cpu_startup_entry+0x1a/0x1c
 start_secondary+0x14b/0x169
 secondary_startup_64+0xa4/0xb0
  ---[ end trace 9e50b2e05e0ee06d ]---

My current kernel is 5.4 and these are my iptables config:

 iptables -t mangle -A PREROUTING -p tcp -m multiport --sport 80 -m socket -m conntrack --ctdir REPLY -j DIVERT
 iptables -t mangle -A DIVERT -j MARK --set-mark 1
 iptables -t mangle -A DIVERT -j ACCEPT
 iptables -t mangle -A PREROUTING -p tcp -m multiport --dports 80 -j TPROXY --tproxy-mark 1 --on-port 3129