Message-ID: <090b720234bc4471bc7f169d3b560f40@SVR-IES-MBX-03.mgc.mentorg.com>
Date:   Wed, 14 Apr 2021 09:18:38 +0000
From:   "Schmid, Carsten" <Carsten_Schmid@...tor.com>
To:     Wei Wang <weiwan@...gle.com>, David Ahern <dsahern@...il.com>
CC:     "davem@...emloft.net" <davem@...emloft.net>,
        "kuznet@....inr.ac.ru" <kuznet@....inr.ac.ru>,
        "yoshfuji@...ux-ipv6.org" <yoshfuji@...ux-ipv6.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: AW: Possible race in ipv4 routing

Hi again,

> -----Ursprüngliche Nachricht-----
> Von: Wei Wang [mailto:weiwan@...gle.com]
> Gesendet: Montag, 1. Februar 2021 19:53
> An: David Ahern <dsahern@...il.com>
> Cc: Schmid, Carsten <Carsten_Schmid@...tor.com>;
> davem@...emloft.net; kuznet@....inr.ac.ru; yoshfuji@...ux-ipv6.org;
> netdev@...r.kernel.org
> Betreff: Re: Possible race in ipv4 routing
>
> On Mon, Feb 1, 2021 at 7:35 AM David Ahern <dsahern@...il.com> wrote:
> >
> > On 2/1/21 2:20 AM, Schmid, Carsten wrote:
> > > Hi,
> > >
> > > on kernel 4.14(.147) i have seen something weird. The stack trace:
> > >
> > > [65064.457920] BUG: unable to handle kernel NULL pointer dereference
> at 0000000000000604
> > > [65064.466677] IP: ip_route_output_key_hash_rcu+0x755/0x850
-------8<-------------
...
-------8<-------------

> > > Fortunately i have a core dump, and analyzed that.
i again have a core dump and ...
... the pointer is not NULL this time:
(see RCX content, which has the pointer's content and is accessed)

[44087.587354] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[44087.594172] Modules linked in: bcmdhd(O) ebt_ip6 ebt_ip ebtable_filter ebtables squashfs zlib_inflate xz_dec veth lzo lzo_compress lzo_decompress esp4 ah4 xfrm4_mode_transport xfrm_user xfrm_algo cls_u32 sch_htb cdc_acm intel_tfm_governor intel_ipu4_psys intel_ipu4_psys_csslib ecryptfs snd_soc_apl_mgu_hu intel_xhci_usb_role_switch roles dwc3 adv728x udc_core intel_ipu4_isys videobuf2_dma_contig videobuf2_memops ipu4_acpi intel_ipu4_isys_csslib coretemp snd_soc_skl videobuf2_v4l2 videobuf2_core sdw_cnl snd_soc_acpi_intel_match snd_soc_acpi snd_soc_core sbi_apl snd_compress i2c_i801 snd_soc_skl_ipc sdw_bus crc8 intel_ipu4_mmu snd_soc_sst_ipc snd_soc_sst_dsp ahci snd_hda_ext_core libahci xhci_pci snd_hda_core mei_me xhci_hcd snd_pcm libata cfg80211 mei snd_timer usbcore dwc3_pci snd scsi_mod usb_common
[44087.607643] skl_ipc_process_reply: 65 callbacks suppressed
[44087.679547]  soundcore rfkill intel_ipu4 iova nfsd auth_rpcgss lockd grace sunrpc zram zsmalloc loop fuse 8021q bridge stp llc inap560t(O) i915 video backlight intel_gtt i2c_algo_bit drm_kms_helper drm firmware_class igb_avb(O) ptp hwmon spi_pxa2xx_platform pps_core [last unloaded: bcmdhd]
[44087.708253] CPU: 1 PID: 248 Comm: 6310_io03 Tainted: G     U     O    4.14.198-apl #1
[44087.717004] task: ffffa0d273b75780 task.stack: ffffa28cc0714000
[44087.723623] RIP: 0010:ip_route_output_key_hash_rcu+0x763/0x860
[44087.730138] RSP: 0018:ffffa28cc0717940 EFLAGS: 00010206
[44087.735976] RAX: ffffa0d0d1f40b00 RBX: ffffa0d26e6d6038 RCX: 000499f24000000a
[44087.743961] RDX: 0000000000000001 RSI: 0000000060c730a0 RDI: 0000000000000000
[44087.751942] RBP: ffffa28cc0717990 R08: 0000000000000000 R09: ffffa0d272e29200
[44087.759919] R10: 0000000000000000 R11: ffffa0d0c60ec300 R12: ffffa0d0d1f40b00
[44087.767897] R13: ffffa28cc07179a0 R14: ffffa0d272e31000 R15: 0000000000000000
[44087.775870] FS:  00007f85affff700(0000) GS:ffffa0d27fc80000(0000) knlGS:0000000000000000
[44087.784917] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[44087.791338] CR2: 00007f3837ec1000 CR3: 00000001f70d4000 CR4: 00000000003406a0
[44087.799314] Call Trace:
[44087.802048]  ip_route_output_key_hash+0x82/0xb0
[44087.807119]  ip_route_output_flow+0x19/0x50
[44087.811798]  ip_queue_xmit+0x389/0x3c0
[44087.815986]  __tcp_transmit_skb+0x598/0x9f0
[44087.820658]  tcp_write_xmit+0x1b7/0xf50
[44087.824942]  __tcp_push_pending_frames+0x30/0xd0
[44087.830109]  tcp_push+0xe7/0x110
[44087.833712]  tcp_sendmsg_locked+0x9ac/0xe50
[44087.838386]  ? __switch_to_asm+0x41/0x70
[44087.842770]  tcp_sendmsg+0x27/0x40
[44087.846574]  inet_sendmsg+0x2f/0xf0
[44087.850468]  sock_sendmsg+0x31/0x40
[44087.854363]  ___sys_sendmsg+0x28d/0x2a0
[44087.858646]  ? wake_up_q+0x54/0x80
[44087.862444]  ? futex_wake+0x8a/0x180
[44087.866437]  ? do_futex+0xc0/0xc60
[44087.870234]  ? tick_program_event+0x3f/0x70
[44087.874912]  ? __fget+0x71/0xa0
[44087.878422]  __sys_sendmsg+0x4f/0x90
[44087.882412]  ? __sys_sendmsg+0x4f/0x90
[44087.886591]  SyS_sendmsg+0x9/0x10
[44087.890290]  do_syscall_64+0x79/0x350
[44087.894380]  ? schedule+0x2e/0x90
[44087.898079]  ? exit_to_usermode_loop+0x5a/0x90
[44087.903040]  entry_SYSCALL_64_after_hwframe+0x41/0xa6
[44087.908680] RIP: 0033:0x7f85bb79e807
[44087.912667] RSP: 002b:00007f85afffceb0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e
[44087.921131] RAX: ffffffffffffffda RBX: 0000000000000132 RCX: 00007f85bb79e807
[44087.929116] RDX: 0000000000004000 RSI: 00007f85afffcf20 RDI: 0000000000000132
[44087.937083] RBP: 00007f85afffcf20 R08: 0000000000000000 R09: 0000000000000000
[44087.945057] R10: 000000000536591c R11: 0000000000000293 R12: 0000000000004000
[44087.953022] R13: 0000560a8a4d5208 R14: 00007f85affff5c0 R15: 0000560a8a8a7040
[44087.960994] Code: 4d c8 4c 89 45 d0 e8 dd d2 bd ff 4c 8b 45 d0 4c 8b 4d c8 e9 83 fa ff ff 48 8b 08 48 8b 89 98 04 00 00 48 85 c9 0f 84 a5 fe ff ff <8b> 89 04 06 00 00 39 88 a0 00 00 00 0f 85 93 fe ff ff 8b 80 80
[44087.982134] RIP: ip_route_output_key_hash_rcu+0x763/0x860 RSP: ffffa28cc0717940

> > > https://www.spinics.net/lists/stable-commits/msg133055.html
> > > But this patch didn't make it into 4.14.
> > >
> > > Can someone check this race condition?
> > >
> >
> > dst->dev is NULL. Adding author of the patch for thoughts.
>
> It definitely looks like the race described in
> https://www.spinics.net/lists/stable-commits/msg133055.html, from all
> the evidence above.
> However, I am not very sure, how rt_is_expired() could crash with NULL
> ptr. I don't think dst->dev could be NULL even after calling
> dst_dev_put(), cause we assign loopback_dev to dst->dev there.
> Also, Carsten mentioned the memdump shows 'dst.obsolete=0x0002'. In
> rt_is_expired(), if 'dst.obsolete=0x0002', we should not call
> rt_is_expired(). So there might be some memory barrier that is
> required in dst_dev_put()?

Do we have this part locked, between checking dst.obsolete=0x0002 and
calling rt_is_expired?

> But anyway, since the fix above replaced dst_dev_put() with
> rt_add_uncached_list(), I believe this crash should also be fixed.

The crash that is seen here has the fix applied, it was a trivial
"move the code a bit" for adaption into 4.14. (and it is meanwhile merged
into 4.14 stable).
Additionally, in my kernel i added a NULL pointer check which obviously doesn't
really help here (this is not upstream):
 static inline bool rt_is_expired(const struct rtable *rth)
 {
-return rth->rt_genid != rt_genid_ipv4(dev_net(rth->dst.dev));
+return (dev_net(rth->dst.dev) == NULL) ||
+   (rth->rt_genid != rt_genid_ipv4(dev_net(rth->dst.dev)));
 }

Any thoughts what could cause this?
For me it looks like a UAF and/or a race.
What i have also found is:
https://syzkaller.appspot.com/bug?id=1157f1b5ae3cf62f60d725868ae9fc53e9050aa0


Best regards
Carsten
-----------------
Mentor Graphics (Deutschland) GmbH, Arnulfstrasse 201, 80634 München Registergericht München HRB 106955, Geschäftsführer: Thomas Heurung, Frank Thürauf

Powered by blists - more mailing lists