Message-ID: <c3a5d08e-9c7c-4888-916a-ba4bd22a3319@collabora.com>
Date:   Tue, 29 Aug 2023 18:07:04 +0500
From:   Muhammad Usama Anjum <usama.anjum@...labora.com>
To:     Bagas Sanjaya <bagasdotme@...il.com>,
        "David S. Miller" <davem@...emloft.net>,
        David Ahern <dsahern@...nel.org>,
        Eric Dumazet <edumazet@...gle.com>,
        Jakub Kicinski <kuba@...nel.org>,
        Paolo Abeni <pabeni@...hat.com>
Cc:     Muhammad Usama Anjum <usama.anjum@...labora.com>,
        Linux Networking <netdev@...r.kernel.org>,
        Linux BPF <bpf@...r.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Linux Regressions <regressions@...ts.linux.dev>
Subject: Re: Fwd: inet6_sock_destruct->inet_sock_destruct trigger Call Trace

On 6/16/23 5:43 PM, Bagas Sanjaya wrote:
> Hi,
> 
> I notice a regression report on Bugzilla [1]. Quoting from it:
> 
>> When the IPv6 address or NIC configuration changes, the following kernel warnings may be triggered:
>>
>> Thu Jun 15 09:02:31 2023 daemon.info : 09[KNL] interface utun deleted
>> Thu Jun 15 09:02:31 2023 daemon.info : 13[KNL] interface utun deleted
>> Thu Jun 15 09:02:32 2023 daemon.notice procd: /etc/rc.d/S99zerotier: disabled in config
>> Thu Jun 15 09:02:33 2023 daemon.info procd: - init complete -
>> Thu Jun 15 09:02:45 2023 daemon.info : 09[KNL] interface utun deleted
>> Thu Jun 15 09:02:45 2023 daemon.info : 15[KNL] interface utun deleted
>> Thu Jun 15 09:02:48 2023 user.notice firewall: Reloading firewall due to ifup of lan6 (br-switch)
>> Thu Jun 15 09:02:51 2023 daemon.err uhttpd[2929]: cat: can't open '/tmp/cpu.usage': No such file or directory
>> Thu Jun 15 09:03:03 2023 user.notice firewall: Reloading firewall due to ifup of wg (wg)
>> Thu Jun 15 09:03:03 2023 daemon.info : 13[KNL] interface tunh activated
>> Thu Jun 15 09:03:03 2023 daemon.info : 16[KNL] fe80::5efe:c0a8:7df9 appeared on tunh
>> Thu Jun 15 09:03:03 2023 daemon.info : 08[KNL] 10.10.13.1 appeared on tunh
>> Thu Jun 15 09:03:03 2023 daemon.info : 13[KNL] interface utun deleted
>> Thu Jun 15 09:03:03 2023 daemon.info : 05[KNL] interface utun deleted
>> Thu Jun 15 09:03:04 2023 auth.err passwd: password for root changed by root
>> Thu Jun 15 09:03:17 2023 daemon.err uhttpd[2929]: sh: /etc/init.d/tasks: line 7: extra_command: not found
>> Thu Jun 15 09:03:17 2023 daemon.err uhttpd[2929]: sh: /etc/init.d/tasks: line 8: extra_command: not found
>> Thu Jun 15 09:03:17 2023 daemon.err uhttpd[2929]: sh: /etc/init.d/tasks: line 9: extra_command: not found
>> Thu Jun 15 09:03:17 2023 daemon.err uhttpd[2929]: sh: /etc/init.d/tasks: line 10: extra_command: not found
>> Thu Jun 15 09:03:17 2023 daemon.err uhttpd[2929]: sh: /etc/init.d/tasks: line 11: extra_command: not found
>> Thu Jun 15 09:03:23 2023 daemon.err uhttpd[2929]: sh: /etc/init.d/tasks: line 7: extra_command: not found
>> Thu Jun 15 09:03:23 2023 daemon.err uhttpd[2929]: sh: /etc/init.d/tasks: line 8: extra_command: not found
>> Thu Jun 15 09:03:23 2023 daemon.err uhttpd[2929]: sh: /etc/init.d/tasks: line 9: extra_command: not found
>> Thu Jun 15 09:03:23 2023 daemon.err uhttpd[2929]: sh: /etc/init.d/tasks: line 10: extra_command: not found
>> Thu Jun 15 09:03:23 2023 daemon.err uhttpd[2929]: sh: /etc/init.d/tasks: line 11: extra_command: not found
>> Thu Jun 15 09:03:35 2023 daemon.err uhttpd[2929]: Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used.
>> Thu Jun 15 09:03:35 2023 daemon.err uhttpd[2929]: Warning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 6111 blocks) or continue with the current setting?
>> Thu Jun 15 09:03:59 2023 daemon.info acpid: starting up with netlink and the input layer
>> Thu Jun 15 09:03:59 2023 daemon.info acpid: 1 rule loaded
>> Thu Jun 15 09:03:59 2023 daemon.info acpid: waiting for events: event logging is off
>> Thu Jun 15 09:11:16 2023 daemon.err uhttpd[2929]: getopt: unrecognized option: no-validate
>> Thu Jun 15 10:06:07 2023 daemon.err uhttpd[2929]: getopt: unrecognized option: no-validate
>> Thu Jun 15 10:09:27 2023 daemon.info : 09[KNL] interface utun deleted
>> Thu Jun 15 10:09:27 2023 daemon.info : 13[KNL] interface utun deleted
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.437330] ------------[ cut here ]------------
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.439948] WARNING: CPU: 1 PID: 19 at inet_sock_destruct+0x190/0x1c0
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.440967] Modules linked in: pppoe ppp_async l2tp_ppp i915 wireguard video sch_fq_pie pppox ppp_mppe ppp_generic mt7921u mt7921s mt7921e mt7921_common mt7915e mt76x2u mt76x2e mt76x2_common mt76x02_usb mt76x02_lib mt76_usb mt76_sdio mt76_connac_lib mt76 mac80211 libchacha20poly1305 ipt_REJECT curve25519_x86_64 chacha_x86_64 cfg80211 ax88179_178a zstd xt_time xt_tcpudp xt_tcpmss xt_string xt_statistic xt_state xt_socket xt_recent xt_quota xt_policy xt_pkttype xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_hl xt_helper xt_hashlimit xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_cgroup xt_bpf xt_addrtype xt_TPROXY xt_TCPMSS xt_REDIRECT xt_MASQUERADE xt_LOG xt_IPMARK xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY wmi via_velocity usbnet ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda tulip ts_fsm ts_bm tcp_bbr slhc sch_pie sch_cake rtl8150 r8168 r8152 r8125
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.441064]  poly1305_x86_64 pcnet32 nf_tproxy_ipv6 nf_tproxy_ipv4 nf_socket_ipv6 nf_socket_ipv4 nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_log_syslog nf_flow_table nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conncount mlx5_core mlx4_en mlx4_core mdev macvlan lzo_rle lzo libcurve25519_generic libchacha kvm_intel kvm ipvlan iptable_raw iptable_nat iptable_mangle iptable_filter ipt_ah ipt_ECN ip_tables iommu_v2 igc iavf i40e forcedeth e1000e drm_display_helper drm_buddy crc_ccitt compat_xtables compat cls_flower br_netfilter bnx2x bnx2 alx act_vlan 8139too 8139cp ntfs3 cls_bpf act_bpf sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact configs sg evdev i2c_dev cryptodev xt_set
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.453861]  ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set st ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_NPT ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsd nfs bonding ip_gre gre ixgbe igbvf e1000 amd_xgbe mdio_devres dummy sit mdio l2tp_netlink l2tp_core udp_tunnel ip6_udp_tunnel ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4 ah4 ipip tunnel6 tunnel4 ip_tunnel udp_diag tcp_diag raw_diag inet_diag rpcsec_gss_krb5 auth_rpcgss veth tun nbd xfrm_user xfrm_ipcomp af_key xfrm_algo virtiofs fuse lockd sunrpc grace hfs cifs oid_registry cifs_md4 cifs_arc4 asn1_decoder dns_resolver md_mod nls_utf8 nls_cp950 nls_cp936 ena shortcut_fe_ipv6 shortcut_fe crypto_user
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.462923]  algif_skcipher algif_rng algif_hash algif_aead af_alg sha512_ssse3 sha512_generic sha1_ssse3 sha1_generic seqiv jitterentropy_rng drbg md5 hmac echainiv des_generic libdes deflate cts cmac authencesn authenc arc4 crypto_acompress nls_iso8859_1 nls_cp437 uas sdhci_pltfm xhci_plat_hcd fsl_mph_dr_of ehci_platform ehci_fsl igb vfat fat exfat btrfs zstd_decompress zstd_compress zstd_common xxhash xor raid6_pq lzo_decompress lzo_compress dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax button_hotplug mii libphy tpm cbc sha256_ssse3 sha256_generic libsha256 encrypted_keys trusted
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.472554] CPU: 1 PID: 19 Comm: ksoftirqd/1 Not tainted 6.1.34 #0
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.473025] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.473731] RIP: 0010:inet_sock_destruct+0x190/0x1c0
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.474204] Code: bc 24 40 01 00 00 e8 af 0d f0 ff 49 8b bc 24 88 00 00 00 e8 a2 0d f0 ff 5b 41 5c 5d c3 4c 89 e7 e8 e5 7e ed ff e9 70 ff ff ff <0f> 0b eb c3 0f 0b 41 8b 84 24 54 01 00 00 85 c0 74 9d 0f 0b 41 8b
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.475522] RSP: 0018:ffffc900000afda8 EFLAGS: 00010206
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.475947] RAX: 0000000000000e00 RBX: ffff888015c9b040 RCX: 0000000000000007
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.476450] RDX: 0000000000000000 RSI: 0000000000000e00 RDI: ffff888015c9b040
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.476966] RBP: ffffc900000afdb8 R08: ffff88800aba5900 R09: 000000008020001a
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.477588] R10: 0000000040000000 R11: 0000000000000000 R12: ffff888015c9af80
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.478326] R13: ffff888002931540 R14: ffffc900000afe28 R15: ffff88807dd253f8
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.478872] FS:  0000000000000000(0000) GS:ffff88807dd00000(0000) knlGS:0000000000000000
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.479434] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.479886] CR2: 00007f11fc4cd0a0 CR3: 0000000021cbe004 CR4: 0000000000370ee0
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.480533] Call Trace:
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.480851]  <TASK>
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.481169]  ? show_regs.part.0+0x1e/0x20
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.481631]  ? show_regs.cold+0x8/0xd
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.482248]  ? __warn+0x6e/0xc0
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.483300]  ? inet_sock_destruct+0x190/0x1c0
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.484240]  ? report_bug+0xed/0x140
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.484937]  ? handle_bug+0x46/0x80
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.485448]  ? exc_invalid_op+0x19/0x70
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.485963]  ? asm_exc_invalid_op+0x1b/0x20
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.486360]  ? inet_sock_destruct+0x190/0x1c0
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.486888]  inet6_sock_destruct+0x16/0x20
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.487289]  __sk_destruct+0x23/0x180
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.487638]  rcu_core+0x28f/0x690
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.487964]  rcu_core_si+0x9/0x10
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.488285]  __do_softirq+0xbd/0x1e8
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.488973]  run_ksoftirqd+0x24/0x40
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.489370]  smpboot_thread_fn+0xdb/0x1d0
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.489826]  kthread+0xde/0x110
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.490572]  ? sort_range+0x20/0x20
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.491356]  ? kthread_complete_and_exit+0x20/0x20
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.491839]  ret_from_fork+0x1f/0x30
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.492195]  </TASK>
>> Thu Jun 15 10:09:28 2023 kern.warn kernel: [ 4071.492448] ---[ end trace 0000000000000000 ]---
> 
> Later, the reporter revealed his setup:
> 
>> This is an openwrt gateway device on x86_64 platform. I'm not sure the exact version number that came up, it seems like 6.1.27 was not encountered before. I have encountered it since kernel 6.1.32, but it is also from this version that I have relatively large IPv6 udp traffic, conntrack -L|grep -c udp shows that the number is between 600 - 2000.
> 
> See Bugzilla for the full thread and attached log.
> 
> Anyway, I'm adding it to regzbot:
> 
> #regzbot introduced: v6.1.27..v6.1.32 https://bugzilla.kernel.org/show_bug.cgi?id=217555
> #regzbot title: kernel warning (oops) at inet_sock_destruct
The same warning has been seen on 4.14, 5.15, 6.1 and latest mainline. This
WARN_ON is present from 2008.

#regzbot monitor:
https://lore.kernel.org/all/6144228a-799f-4de3-8483-b7add903df0c@collabora.com

> 
> Thanks.
> 
> [1]: https://bugzilla.kernel.org/show_bug.cgi?id=217555

-- 
BR,
Muhammad Usama Anjum

Powered by blists - more mailing lists