| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <10f49258-ae5d-58f4-be1b-8358dd00af8a@tessares.net> Date: Fri, 28 Jul 2023 18:05:58 +0200 From: Matthieu Baerts <matthieu.baerts@...sares.net> To: netdev <netdev@...r.kernel.org> Cc: MPTCP Upstream <mptcp@...ts.linux.dev> Subject: KMemLeak when creating netns / tun dev / sending NL msg Hello, Our CI validating MPTCP tree recently reported leaks according to KMemLeak. Here are some examples of backtraces decoded with decode_stacktrace.sh: ------------ 8< ------------ unreferenced object 0xffff88800ce2b000 (size 2048): comm "ip", pid 18556, jiffies 4296578479 (age 84.890s) hex dump (first 32 bytes): 08 60 1d 18 80 88 ff ff 00 00 00 00 00 00 00 00 .`.............. 00 00 00 00 00 00 00 00 ea ff ff ff ff ff ff ff ................ backtrace: __kmalloc (include/linux/kasan.h:196) __register_sysctl_table (include/linux/slab.h:586) __devinet_sysctl_register (net/ipv4/devinet.c:2590) devinet_init_net (net/ipv4/devinet.c:2713) ops_init (net/core/net_namespace.c:136) setup_net (net/core/net_namespace.c:339) copy_net_ns (net/core/net_namespace.c:493) create_new_namespaces (kernel/nsproxy.c:110) unshare_nsproxy_namespaces (kernel/nsproxy.c:228 (discriminator 4)) ksys_unshare (kernel/fork.c:3438) __x64_sys_unshare (kernel/fork.c:3507) do_syscall_64 (arch/x86/entry/common.c:50) entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) ------------ 8< ------------ ------------ 8< ------------ unreferenced object 0xffff888010866900 (size 192): comm "ip", pid 18556, jiffies 4296578480 (age 84.901s) hex dump (first 32 bytes): 00 3d 39 07 80 88 ff ff 22 01 00 00 00 00 ad de .=9....."....... 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ backtrace: __kmalloc (include/linux/kasan.h:196) fib_default_rule_add (include/linux/slab.h:586) fib4_rules_init (net/ipv4/fib_rules.c:399) fib_net_init (net/ipv4/fib_frontend.c:1554) ops_init (net/core/net_namespace.c:136) setup_net (net/core/net_namespace.c:339) copy_net_ns (net/core/net_namespace.c:493) create_new_namespaces (kernel/nsproxy.c:110) unshare_nsproxy_namespaces (kernel/nsproxy.c:228 (discriminator 4)) ksys_unshare (kernel/fork.c:3438) __x64_sys_unshare (kernel/fork.c:3507) do_syscall_64 (arch/x86/entry/common.c:50) entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) ------------ 8< ------------ ------------ 8< ------------ unreferenced object 0xffff88800fb5b000 (size 2048): comm "ip", pid 18556, jiffies 4296578502 (age 84.879s) hex dump (first 32 bytes): 00 60 0f 0e 80 88 ff ff 00 00 00 00 00 00 00 00 .`.............. 00 00 00 00 00 00 00 00 ea ff ff ff ff ff ff ff ................ backtrace: __kmalloc (include/linux/kasan.h:196) __register_sysctl_table (include/linux/slab.h:586) nf_conntrack_pernet_init (net/netfilter/nf_conntrack_standalone.c:1109) ops_init (net/core/net_namespace.c:136) setup_net (net/core/net_namespace.c:339) copy_net_ns (net/core/net_namespace.c:493) create_new_namespaces (kernel/nsproxy.c:110) unshare_nsproxy_namespaces (kernel/nsproxy.c:228 (discriminator 4)) ksys_unshare (kernel/fork.c:3438) __x64_sys_unshare (kernel/fork.c:3507) do_syscall_64 (arch/x86/entry/common.c:50) entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) ------------ 8< ------------ The majority comes from "ip" when creating a new netns but a few also come from "packetdrill" when creating a tun device: ------------ 8< ------------ unreferenced object 0xffff8880198da000 (size 2048): comm "packetdrill", pid 20084, jiffies 4296651533 (age 12.029s) hex dump (first 32 bytes): 08 c0 f5 05 80 88 ff ff 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 00 00 ea ff ff ff ff ff ff ff ................ backtrace: __kmalloc (include/linux/kasan.h:196) __register_sysctl_table (include/linux/slab.h:586) __devinet_sysctl_register (net/ipv4/devinet.c:2590) devinet_sysctl_register (net/ipv4/devinet.c:2630) inetdev_init (net/ipv4/devinet.c:286) inetdev_event (net/ipv4/devinet.c:1538) notifier_call_chain (kernel/notifier.c:95) register_netdevice (include/linux/notifier.h:218) tun_set_iff.constprop.0 (drivers/net/tun.c:2846) __tun_chr_ioctl (drivers/net/tun.c:3113) __x64_sys_ioctl (fs/ioctl.c:52) do_syscall_64 (arch/x86/entry/common.c:50) entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) ------------ 8< ------------ But I can also see backtraces not mentioning sysctl, caused by "ip" when communicating with the kernel via netlink: ------------ 8< ------------ unreferenced object 0xffff88801041ee00 (size 256): comm "ip", pid 18567, jiffies 4296579097 (age 84.399s) hex dump (first 32 bytes): 00 40 c8 18 80 88 ff ff e0 00 00 01 00 00 00 00 .@.............. 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ backtrace: kmalloc_trace (mm/slab_common.c:1079) ____ip_mc_inc_group (include/linux/slab.h:582) ip_mc_up (net/ipv4/igmp.c:1791 (discriminator 11)) inetdev_event (net/ipv4/devinet.c:1584) notifier_call_chain (kernel/notifier.c:95) __dev_notify_flags (net/core/dev.c:8574) dev_change_flags (net/core/dev.c:8609) do_setlink (net/core/rtnetlink.c:2867) __rtnl_newlink (net/core/rtnetlink.c:3655) rtnl_newlink (net/core/rtnetlink.c:3703) rtnetlink_rcv_msg (net/core/rtnetlink.c:6424) netlink_rcv_skb (net/netlink/af_netlink.c:2549) netlink_unicast (net/netlink/af_netlink.c:1340) netlink_sendmsg (net/netlink/af_netlink.c:1914) sock_sendmsg (net/socket.c:728) ____sys_sendmsg (net/socket.c:2494) ------------ 8< ------------ ------------ 8< ------------ unreferenced object 0xffff88800fee4c00 (size 512): comm "ip", pid 18567, jiffies 4296579099 (age 84.409s) hex dump (first 32 bytes): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 ................ 80 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff ................ backtrace: kmalloc_trace (mm/slab_common.c:1079) ipv6_add_addr (include/linux/slab.h:582) add_addr (net/ipv6/addrconf.c:3108) addrconf_notify (include/linux/err.h:72) notifier_call_chain (kernel/notifier.c:95) __dev_notify_flags (net/core/dev.c:8574) dev_change_flags (net/core/dev.c:8609) do_setlink (net/core/rtnetlink.c:2867) __rtnl_newlink (net/core/rtnetlink.c:3655) rtnl_newlink (net/core/rtnetlink.c:3703) rtnetlink_rcv_msg (net/core/rtnetlink.c:6424) netlink_rcv_skb (net/netlink/af_netlink.c:2549) netlink_unicast (net/netlink/af_netlink.c:1340) netlink_sendmsg (net/netlink/af_netlink.c:1914) sock_sendmsg (net/socket.c:728) ____sys_sendmsg (net/socket.c:2494) ------------ 8< ------------ For more backtraces, please see: https://github.com/multipath-tcp/mptcp_net-next/issues/424 I don't have much info to share on how to reproduce the issue because we only see this error occasionally and only at the end of the execution of many different tests, when forcing a kmemleak scan. The CI creates and destroys hundreds of netns during these tests (MPTCP selftests + packetdrill). We don't frequently see this issue so it might be hard to reproduce (and could also be a false positive from kmemleak). The last time we had this issue, all tests have been executed with success and have been stopped when the scan had been initiated. So in theory, all netns and tun devices have been properly destroyed (or asked to be destroyed) before the scan. I don't think we had this issue before this month, maybe due to something new coming with v6.5-rc1? We had this issue on top of -net and net-next. Anybody else saw such issue with a reproducer or has a pointer on what could cause that? :) Cheers, Matt -- Tessares | Belgium | Hybrid Access Solutions www.tessares.net
Powered by blists - more mailing lists