| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <BB129799-E196-428C-909D-721670DD5E21@gmail.com> Date: Wed, 20 Sep 2023 09:05:10 +0300 From: Martin Zaharinov <micron10@...il.com> To: Eric Dumazet <edumazet@...gle.com> Cc: Paolo Abeni <pabeni@...hat.com>, netdev <netdev@...r.kernel.org>, patchwork-bot+netdevbpf@...nel.org, Jakub Kicinski <kuba@...nel.org>, Stephen Hemminger <stephen@...workplumber.org>, kuba+netdrv@...nel.org, dsahern@...il.com, Florian Westphal <fw@...len.de>, Pablo Neira Ayuso <pablo@...filter.org> Subject: Re: Urgent Bug Report Kernel crash 6.5.2 Hi Eric > On 20 Sep 2023, at 6:59, Eric Dumazet <edumazet@...gle.com> wrote: > > On Tue, Sep 19, 2023 at 10:09 PM Martin Zaharinov <micron10@...il.com> wrote: >> >> Hi Eric >> >> Yes this patch is not come in 6.5 kernel and queue for 6.6 i test but not ok for now. > > "not ok for now" ? What does this mean? > Pointing out patches that are not related to your issue is a waste of time. > If this was to bring my attention, this is a bad strategy, because I > will probably not read your future emails. > I'm sorry, I didn't speak correctly. patch is very good but for kernel 6.6. I enjoy your kernel improvements. And thanks for that !! >> >> One more i find same error have in old kernel 6.4.8 , update to kernel 6.5.4 and same error is come . >> >> Like this is hard to catch bug >> >> see logs : >> >> >> [1462610.861373] ------------[ cut here ]------------ >> [1462610.861480] rcuref - imbalanced put() >> [1462610.861491] WARNING: CPU: 22 PID: 0 at lib/rcuref.c:267 rcuref_put_slowpath+0x5f/0x70 >> [1462610.861718] Modules linked in: nft_limit nf_conntrack_netlink pppoe pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp bonding ixgbe mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos >> [1462610.862004] CPU: 22 PID: 0 Comm: swapper/22 Tainted: G O 6.4.8 #1 >> [1462610.863244] Hardware name: Supermicro Super Server/X10SRW-F, BIOS 3.4 06/05/2021 >> [1462610.863368] RIP: 0010:rcuref_put_slowpath+0x5f/0x70 >> [1462610.863469] Code: 31 c0 eb e2 80 3d 02 cd e6 00 00 74 0a c7 03 00 00 00 e0 31 c0 eb cf 48 c7 c7 7f 68 e5 a4 c6 05 e8 cc e6 00 01 e8 e1 ab c7 ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 e2 >> [1462610.863637] RSP: 0018:ffffaee60070cc38 EFLAGS: 00010292 >> [1462610.863736] RAX: 0000000000000019 RBX: ffffa1cdc35e5780 RCX: 00000000fff7ffff >> [1462610.863857] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: 00000000ffffffea >> [1462610.864129] RBP: ffffa1cf6aeb8de8 R08: 0000000000000000 R09: 00000000fff7ffff >> [1462610.864250] R10: ffffa1d51b000000 R11: 0000000000000003 R12: ffffa1cdc35e5740 >> [1462610.864370] R13: ffffa1cdc35e57a8 R14: ffffa1d51fda9008 R15: 00000000ade2eb6e >> [1462610.864489] FS: 0000000000000000(0000) GS:ffffa1d51fd80000(0000) knlGS:0000000000000000 >> [1462610.864615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [1462610.864713] CR2: 00007f057b8ad000 CR3: 0000000141881003 CR4: 00000000001706e0 >> [1462610.864833] Call Trace: >> [1462610.864928] <IRQ> >> [1462610.865021] ? __warn+0x6c/0x130 >> [1462610.865124] ? report_bug+0x1e4/0x260 >> [1462610.865223] ? handle_bug+0x36/0x70 >> [1462610.865318] ? exc_invalid_op+0x17/0x1a0 >> [1462610.865414] ? asm_exc_invalid_op+0x16/0x20 >> [1462610.865517] ? rcuref_put_slowpath+0x5f/0x70 >> [1462610.865618] ? rcuref_put_slowpath+0x5f/0x70 >> [1462610.865719] dst_release+0x2c/0x60 >> [1462610.865817] rt_cache_route+0xbd/0xf0 >> [1462610.865913] rt_set_nexthop.isra.0+0x1b6/0x440 >> [1462610.866008] ip_route_input_slow+0x90e/0xc60 >> [1462610.866116] ? nf_conntrack_udp_packet+0x16c/0x230 [nf_conntrack] >> [1462610.866229] ip_route_input_noref+0xed/0x100 >> [1462610.866328] ip_rcv_finish_core.isra.0+0xb1/0x410 >> [1462610.866425] ip_rcv+0xed/0x130 >> [1462610.866522] ? ip_rcv_core.constprop.0+0x350/0x350 >> [1462610.866621] process_backlog+0x10c/0x230 >> [1462610.866719] __napi_poll+0x20/0x180 >> [1462610.866818] net_rx_action+0x2a4/0x390 >> [1462610.866921] __do_softirq+0xd0/0x202 >> [1462610.867020] do_softirq+0x58/0x80 >> [1462610.867116] </IRQ> >> [1462610.867206] <TASK> >> [1462610.867298] flush_smp_call_function_queue+0x3f/0x60 >> [1462610.867403] do_idle+0x14d/0x210 >> [1462610.867500] cpu_startup_entry+0x14/0x20 >> [1462610.867602] start_secondary+0xec/0xf0 >> [1462610.867701] secondary_startup_64_no_verify+0xf9/0xfb >> [1462610.867799] </TASK> >> [1462610.867891] ---[ end trace 0000000000000000 ]— >> >> >> And this si 6.5.4 : >> >> [39651.441371] ------------[ cut here ]------------ >> [39651.441455] rcuref - imbalanced put() >> [39651.441470] WARNING: CPU: 12 PID: 0 at lib/rcuref.c:267 rcuref_put_slowpath+0x5f/0x70 >> [39651.441633] Modules linked in: nft_limit pppoe pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp igb i2c_algo_bit i40e ixgbe mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos >> [39651.441805] CPU: 12 PID: 0 Comm: swapper/12 Tainted: G O 6.5.3 #1 >> [39651.441911] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./EP2C612D8, BIOS P2.30 04/30/2018 >> [39651.442035] RIP: 0010:rcuref_put_slowpath+0x5f/0x70 >> [39651.442131] Code: 31 c0 eb e2 80 3d 86 ae e6 00 00 74 0a c7 03 00 00 00 e0 31 c0 eb cf 48 c7 c7 68 f6 e2 9a c6 05 6c ae e6 00 01 e8 11 71 c7 ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 e2 >> [39651.442294] RSP: 0018:ffffbb9a404b4de8 EFLAGS: 00010296 >> [39651.442390] RAX: 0000000000000019 RBX: ffffa13ac9a32640 RCX: 00000000fff7ffff >> [39651.442513] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: 00000000ffffffea >> [39651.442630] RBP: ffffa13a44a04000 R08: 0000000000000000 R09: 00000000fff7ffff >> [39651.442748] R10: ffffa1419ae00000 R11: 0000000000000003 R12: ffffa13ab640bec0 >> [39651.442866] R13: 0000000000000000 R14: 0000000000000010 R15: ffffbb9a404b4f60 >> [39651.442985] FS: 0000000000000000(0000) GS:ffffa1419f900000(0000) knlGS:0000000000000000 >> [39651.443106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [39651.443201] CR2: 0000564f9e23f6e0 CR3: 000000010bcea002 CR4: 00000000003706e0 >> [39651.443319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> [39651.443438] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >> [39651.443558] Call Trace: >> [39651.443647] <IRQ> >> [39651.443736] ? __warn+0x6c/0x130 >> [39651.443829] ? report_bug+0x1e4/0x260 >> [39651.443924] ? handle_bug+0x36/0x70 >> [39651.444016] ? exc_invalid_op+0x17/0x1a0 >> [39651.444109] ? asm_exc_invalid_op+0x16/0x20 >> [39651.444202] ? rcuref_put_slowpath+0x5f/0x70 >> [39651.444297] ? rcuref_put_slowpath+0x5f/0x70 >> [39651.444391] dst_release+0x2c/0x60 >> [39651.444487] __dev_queue_xmit+0x56c/0xbd0 >> [39651.444582] ? nf_hook_slow+0x36/0xa0 >> [39651.444675] ip_finish_output2+0x27b/0x520 >> [39651.444770] process_backlog+0x10c/0x230 >> [39651.444866] __napi_poll+0x20/0x180 >> [39651.444961] net_rx_action+0x2a4/0x390 >> [39651.445055] __do_softirq+0xd0/0x202 >> [39651.445148] do_softirq+0x3a/0x50 >> [39651.445241] </IRQ> >> [39651.445329] <TASK> >> [39651.445416] flush_smp_call_function_queue+0x3f/0x50 >> [39651.445516] do_idle+0x14d/0x210 >> [39651.445609] cpu_startup_entry+0x14/0x20 >> [39651.445702] start_secondary+0xe1/0xf0 >> [39651.445797] secondary_startup_64_no_verify+0x167/0x16b >> [39651.445893] </TASK> >> [39651.445982] ---[ end trace 0000000000000000 ]— >> >> >> best regards, >> Martin > > You keep sending traces without symbols, nobody here will even look at them. > Here is trace with symbols : [39651.441371] ------------[ cut here ]------------ [39651.441455] rcuref - imbalanced put() [39651.441470] WARNING: CPU: 12 PID: 0 at lib/rcuref.c:267 rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1)) [39651.441633] Modules linked in: nft_limit pppoe pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp igb i2c_algo_bit i40e ixgbe mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos [39651.441805] CPU: 12 PID: 0 Comm: swapper/12 Tainted: G O 6.5.3 #1 [39651.441911] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./EP2C612D8, BIOS P2.30 04/30/2018 [39651.442035] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1)) [39651.442131] Code: 31 c0 eb e2 80 3d 86 ae e6 00 00 74 0a c7 03 00 00 00 e0 31 c0 eb cf 48 c7 c7 68 f6 e2 9a c6 05 6c ae e6 00 01 e8 11 71 c7 ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 e2 All code ======== 0: 31 c0 xor %eax,%eax 2: eb e2 jmp 0xffffffffffffffe6 4: 80 3d 86 ae e6 00 00 cmpb $0x0,0xe6ae86(%rip) # 0xe6ae91 b: 74 0a je 0x17 d: c7 03 00 00 00 e0 movl $0xe0000000,(%rbx) 13: 31 c0 xor %eax,%eax 15: eb cf jmp 0xffffffffffffffe6 17: 48 c7 c7 68 f6 e2 9a mov $0xffffffff9ae2f668,%rdi 1e: c6 05 6c ae e6 00 01 movb $0x1,0xe6ae6c(%rip) # 0xe6ae91 25: e8 11 71 c7 ff call 0xffffffffffc7713b 2a:* 0f 0b ud2 <-- trapping instruction 2c: eb df jmp 0xd 2e: cc int3 2f: cc int3 30: cc int3 31: cc int3 32: cc int3 33: cc int3 34: cc int3 35: cc int3 36: cc int3 37: cc int3 38: cc int3 39: cc int3 3a: cc int3 3b: 48 89 fa mov %rdi,%rdx 3e: 83 .byte 0x83 3f: e2 .byte 0xe2 Code starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: eb df jmp 0xffffffffffffffe3 4: cc int3 5: cc int3 6: cc int3 7: cc int3 8: cc int3 9: cc int3 a: cc int3 b: cc int3 c: cc int3 d: cc int3 e: cc int3 f: cc int3 10: cc int3 11: 48 89 fa mov %rdi,%rdx 14: 83 .byte 0x83 15: e2 .byte 0xe2 [39651.442294] RSP: 0018:ffffbb9a404b4de8 EFLAGS: 00010296 [39651.442390] RAX: 0000000000000019 RBX: ffffa13ac9a32640 RCX: 00000000fff7ffff [39651.442513] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: 00000000ffffffea [39651.442630] RBP: ffffa13a44a04000 R08: 0000000000000000 R09: 00000000fff7ffff [39651.442748] R10: ffffa1419ae00000 R11: 0000000000000003 R12: ffffa13ab640bec0 [39651.442866] R13: 0000000000000000 R14: 0000000000000010 R15: ffffbb9a404b4f60 [39651.442985] FS: 0000000000000000(0000) GS:ffffa1419f900000(0000) knlGS:0000000000000000 [39651.443106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39651.443201] CR2: 0000564f9e23f6e0 CR3: 000000010bcea002 CR4: 00000000003706e0 [39651.443319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39651.443438] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39651.443558] Call Trace: [39651.443647] <IRQ> [39651.443736] ? __warn (kernel/panic.c:235 kernel/panic.c:673) [39651.443829] ? report_bug (lib/bug.c:180 lib/bug.c:219) [39651.443924] ? handle_bug (arch/x86/kernel/traps.c:324) [39651.444016] ? exc_invalid_op (arch/x86/kernel/traps.c:345 (discriminator 1)) [39651.444109] ? asm_exc_invalid_op (./arch/x86/include/asm/idtentry.h:568) [39651.444202] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1)) [39651.444297] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1)) [39651.444391] dst_release (./arch/x86/include/asm/preempt.h:95 ./include/linux/rcuref.h:151 net/core/dst.c:166) [39651.444487] __dev_queue_xmit (./include/net/dst.h:283 net/core/dev.c:4158) [39651.444582] ? nf_hook_slow (./include/linux/netfilter.h:143 net/netfilter/core.c:626) [39651.444675] ip_finish_output2 (./include/linux/netdevice.h:3088 ./include/net/neighbour.h:528 ./include/net/neighbour.h:542 net/ipv4/ip_output.c:230) [39651.444770] process_backlog (./include/linux/rcupdate.h:781 net/core/dev.c:5896) [39651.444866] __napi_poll (net/core/dev.c:6461) [39651.444961] net_rx_action (net/core/dev.c:6530 net/core/dev.c:6661) [39651.445055] __do_softirq (./arch/x86/include/asm/preempt.h:27 kernel/softirq.c:564) [39651.445148] do_softirq (kernel/softirq.c:463 (discriminator 32) kernel/softirq.c:450 (discriminator 32)) [39651.445241] </IRQ> [39651.445329] <TASK> [39651.445416] flush_smp_call_function_queue (./arch/x86/include/asm/irqflags.h:134 (discriminator 1) kernel/smp.c:570 (discriminator 1)) [39651.445516] do_idle (kernel/sched/idle.c:314) [39651.445609] cpu_startup_entry (kernel/sched/idle.c:378) [39651.445702] start_secondary (arch/x86/kernel/smpboot.c:326) [39651.445797] secondary_startup_64_no_verify (arch/x86/kernel/head_64.S:441) [39651.445893] </TASK> [39651.445982] ---[ end trace 0000000000000000 ]--- > Again, your best route is a bisection. For now its not possible to make bisection , its hard to change kernel on running machine … is there another way to catch from where is come this bug message. Best regards, Martin
Powered by blists - more mailing lists