netdev - Re: Urgent Bug Report Kernel crash 6.5.2

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
Date: Tue, 19 Sep 2023 23:09:29 +0300
From: Martin Zaharinov <micron10@...il.com>
To: Paolo Abeni <pabeni@...hat.com>
Cc: netdev <netdev@...r.kernel.org>,
 Eric Dumazet <edumazet@...gle.com>,
 patchwork-bot+netdevbpf@...nel.org,
 Jakub Kicinski <kuba@...nel.org>,
 Stephen Hemminger <stephen@...workplumber.org>,
 kuba+netdrv@...nel.org,
 dsahern@...il.com,
 Florian Westphal <fw@...len.de>,
 Pablo Neira Ayuso <pablo@...filter.org>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2

Hi Eric

Yes this patch is not come in 6.5 kernel and queue for 6.6 i test but not ok for now.

One more i find same error have in old kernel 6.4.8  , update to kernel 6.5.4 and same error is come .

Like this is hard to catch bug

see logs :


[1462610.861373] ------------[ cut here ]------------
[1462610.861480] rcuref - imbalanced put()
[1462610.861491] WARNING: CPU: 22 PID: 0 at lib/rcuref.c:267 rcuref_put_slowpath+0x5f/0x70
[1462610.861718] Modules linked in: nft_limit nf_conntrack_netlink  pppoe pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp bonding ixgbe mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
[1462610.862004] CPU: 22 PID: 0 Comm: swapper/22 Tainted: G           O       6.4.8 #1
[1462610.863244] Hardware name: Supermicro Super Server/X10SRW-F, BIOS 3.4 06/05/2021
[1462610.863368] RIP: 0010:rcuref_put_slowpath+0x5f/0x70
[1462610.863469] Code: 31 c0 eb e2 80 3d 02 cd e6 00 00 74 0a c7 03 00 00 00 e0 31 c0 eb cf 48 c7 c7 7f 68 e5 a4 c6 05 e8 cc e6 00 01 e8 e1 ab c7 ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 e2
[1462610.863637] RSP: 0018:ffffaee60070cc38 EFLAGS: 00010292
[1462610.863736] RAX: 0000000000000019 RBX: ffffa1cdc35e5780 RCX: 00000000fff7ffff
[1462610.863857] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: 00000000ffffffea
[1462610.864129] RBP: ffffa1cf6aeb8de8 R08: 0000000000000000 R09: 00000000fff7ffff
[1462610.864250] R10: ffffa1d51b000000 R11: 0000000000000003 R12: ffffa1cdc35e5740
[1462610.864370] R13: ffffa1cdc35e57a8 R14: ffffa1d51fda9008 R15: 00000000ade2eb6e
[1462610.864489] FS:  0000000000000000(0000) GS:ffffa1d51fd80000(0000) knlGS:0000000000000000
[1462610.864615] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1462610.864713] CR2: 00007f057b8ad000 CR3: 0000000141881003 CR4: 00000000001706e0
[1462610.864833] Call Trace:
[1462610.864928]  <IRQ>
[1462610.865021]  ? __warn+0x6c/0x130
[1462610.865124]  ? report_bug+0x1e4/0x260
[1462610.865223]  ? handle_bug+0x36/0x70
[1462610.865318]  ? exc_invalid_op+0x17/0x1a0
[1462610.865414]  ? asm_exc_invalid_op+0x16/0x20
[1462610.865517]  ? rcuref_put_slowpath+0x5f/0x70
[1462610.865618]  ? rcuref_put_slowpath+0x5f/0x70
[1462610.865719]  dst_release+0x2c/0x60
[1462610.865817]  rt_cache_route+0xbd/0xf0
[1462610.865913]  rt_set_nexthop.isra.0+0x1b6/0x440
[1462610.866008]  ip_route_input_slow+0x90e/0xc60
[1462610.866116]  ? nf_conntrack_udp_packet+0x16c/0x230 [nf_conntrack]
[1462610.866229]  ip_route_input_noref+0xed/0x100
[1462610.866328]  ip_rcv_finish_core.isra.0+0xb1/0x410
[1462610.866425]  ip_rcv+0xed/0x130
[1462610.866522]  ? ip_rcv_core.constprop.0+0x350/0x350
[1462610.866621]  process_backlog+0x10c/0x230
[1462610.866719]  __napi_poll+0x20/0x180
[1462610.866818]  net_rx_action+0x2a4/0x390
[1462610.866921]  __do_softirq+0xd0/0x202
[1462610.867020]  do_softirq+0x58/0x80
[1462610.867116]  </IRQ>
[1462610.867206]  <TASK>
[1462610.867298]  flush_smp_call_function_queue+0x3f/0x60
[1462610.867403]  do_idle+0x14d/0x210
[1462610.867500]  cpu_startup_entry+0x14/0x20
[1462610.867602]  start_secondary+0xec/0xf0
[1462610.867701]  secondary_startup_64_no_verify+0xf9/0xfb
[1462610.867799]  </TASK>
[1462610.867891] ---[ end trace 0000000000000000 ]—


And this si 6.5.4 : 

[39651.441371] ------------[ cut here ]------------
[39651.441455] rcuref - imbalanced put()
[39651.441470] WARNING: CPU: 12 PID: 0 at lib/rcuref.c:267 rcuref_put_slowpath+0x5f/0x70
[39651.441633] Modules linked in: nft_limit pppoe pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp igb i2c_algo_bit i40e ixgbe mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
[39651.441805] CPU: 12 PID: 0 Comm: swapper/12 Tainted: G           O       6.5.3 #1
[39651.441911] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./EP2C612D8, BIOS P2.30 04/30/2018
[39651.442035] RIP: 0010:rcuref_put_slowpath+0x5f/0x70
[39651.442131] Code: 31 c0 eb e2 80 3d 86 ae e6 00 00 74 0a c7 03 00 00 00 e0 31 c0 eb cf 48 c7 c7 68 f6 e2 9a c6 05 6c ae e6 00 01 e8 11 71 c7 ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 e2
[39651.442294] RSP: 0018:ffffbb9a404b4de8 EFLAGS: 00010296
[39651.442390] RAX: 0000000000000019 RBX: ffffa13ac9a32640 RCX: 00000000fff7ffff
[39651.442513] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: 00000000ffffffea
[39651.442630] RBP: ffffa13a44a04000 R08: 0000000000000000 R09: 00000000fff7ffff
[39651.442748] R10: ffffa1419ae00000 R11: 0000000000000003 R12: ffffa13ab640bec0
[39651.442866] R13: 0000000000000000 R14: 0000000000000010 R15: ffffbb9a404b4f60
[39651.442985] FS:  0000000000000000(0000) GS:ffffa1419f900000(0000) knlGS:0000000000000000
[39651.443106] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[39651.443201] CR2: 0000564f9e23f6e0 CR3: 000000010bcea002 CR4: 00000000003706e0
[39651.443319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[39651.443438] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[39651.443558] Call Trace:
[39651.443647]  <IRQ>
[39651.443736]  ? __warn+0x6c/0x130
[39651.443829]  ? report_bug+0x1e4/0x260
[39651.443924]  ? handle_bug+0x36/0x70
[39651.444016]  ? exc_invalid_op+0x17/0x1a0
[39651.444109]  ? asm_exc_invalid_op+0x16/0x20
[39651.444202]  ? rcuref_put_slowpath+0x5f/0x70
[39651.444297]  ? rcuref_put_slowpath+0x5f/0x70
[39651.444391]  dst_release+0x2c/0x60
[39651.444487]  __dev_queue_xmit+0x56c/0xbd0
[39651.444582]  ? nf_hook_slow+0x36/0xa0
[39651.444675]  ip_finish_output2+0x27b/0x520
[39651.444770]  process_backlog+0x10c/0x230
[39651.444866]  __napi_poll+0x20/0x180
[39651.444961]  net_rx_action+0x2a4/0x390
[39651.445055]  __do_softirq+0xd0/0x202
[39651.445148]  do_softirq+0x3a/0x50
[39651.445241]  </IRQ>
[39651.445329]  <TASK>
[39651.445416]  flush_smp_call_function_queue+0x3f/0x50
[39651.445516]  do_idle+0x14d/0x210
[39651.445609]  cpu_startup_entry+0x14/0x20
[39651.445702]  start_secondary+0xe1/0xf0
[39651.445797]  secondary_startup_64_no_verify+0x167/0x16b
[39651.445893]  </TASK>
[39651.445982] ---[ end trace 0000000000000000 ]—


best regards,
Martin

> On 17 Sep 2023, at 14:55, Martin Zaharinov <micron10@...il.com> wrote:
> 
> Hi Eric
> is it possible bug to come from this patch : https://patchwork.kernel.org/project/netdevbpf/cover/20230911170531.828100-1-edumazet@google.com/ 
> 
> 
> m.
> 
>> On 17 Sep 2023, at 14:40, Martin Zaharinov <micron10@...il.com> wrote:
>> 
>> One more in changelog for kernel 6.5 : https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.5
>> 
>> I see have many bug reports with : 
>> 
>> Sep 17 11:43:11  [127675.395289][    C2]  ? process_backlog+0x10c/0x230
>> Sep 17 11:43:11  [127675.395386][    C2]  ? __napi_poll+0x20/0x180
>> Sep 17 11:43:11  [127675.395478][    C2]  ? net_rx_action+0x2a4/0x390
>> 
>> 
>> In all server have simple nftables rulls , ethernet card is intel xl710 or 82599. its a very simple config.
>> 
>> m. 
>> 
>> 
>> 
>> 
>>> On 16 Sep 2023, at 12:04, Martin Zaharinov <micron10@...il.com> wrote:
>>> 
>>> Hi Paolo
>>> 
>>> in first report machine dont have out of tree module
>>> 
>>> this bug is come after move from kernel 6.2 to 6.3
>>> 
>>> m.
>>> 
>>> On Sat, Sep 16, 2023, 11:27 Paolo Abeni <pabeni@...hat.com> wrote:
>>> On Sat, 2023-09-16 at 02:11 +0300, Martin Zaharinov wrote:
>>>> one more log:
>>>> 
>>>> Sep 12 07:37:29  [151563.298466][    C5] ------------[ cut here ]------------
>>>> Sep 12 07:37:29  [151563.298550][    C5] rcuref - imbalanced put()
>>>> Sep 12 07:37:29  [151563.298564][ C5] WARNING: CPU: 5 PID: 0 at lib/rcuref.c:267 rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
>>>> Sep 12 07:37:29  [151563.298724][    C5] Modules linked in: nft_limit nf_conntrack_netlink vlan_mon(O) pppoe pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_xnatlog(O) ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos [last unloaded: BNGBOOT(O)]
>>>> Sep 12 07:37:29  [151563.298894][    C5] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G           O       6.5.2 #1
>>> 
>>> 
>>> You have out-of-tree modules taint in all the report you shared. Please
>>> try to reproduce the issue with such taint, thanks!
>>> 
>>> Paolo
>>> 
>> 
>