Date:   Fri, 16 Apr 2021 12:29:30 +0000
From:   "Schmid, Carsten" <Carsten_Schmid@...tor.com>
To:     Wei Wang <weiwan@...gle.com>, David Ahern <dsahern@...il.com>
CC:     "davem@...emloft.net" <davem@...emloft.net>,
        "kuznet@....inr.ac.ru" <kuznet@....inr.ac.ru>,
        "yoshfuji@...ux-ipv6.org" <yoshfuji@...ux-ipv6.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: AW: Possible race in ipv4 routing

Hi again,

> > > > on kernel 4.14(.147) i have seen something weird. The stack trace:
> > > >
> > > > [65064.457920] BUG: unable to handle kernel NULL pointer dereference
> > at 0000000000000604
> > > > [65064.466677] IP: ip_route_output_key_hash_rcu+0x755/0x850

I have another crash with a very different stack, but also ending up in
rt_is_expired but this time from an interrupt context:
[14330.764616] BUG: unable to handle kernel NULL pointer dereference at 0000000000000608
[14330.773382] IP: ipv4_dst_check+0x1c/0x40
[14330.777761] PGD 0 P4D 0
[14330.780586] Oops: 0000 [#1] PREEMPT SMP NOPTI
[14330.785451] Modules linked in: bcmdhd(O) squashfs zlib_inflate xz_dec ebt_ip6 ebt_ip ebtable_filter ebtables veth lzo lzo_compress lzo_decompress ah4 xfrm4_mode_transport nls_iso8859_1 nls_cp850 vfat fat cfq_iosched sd_mod xfrm_user xfrm_algo tntfs(PO) texfat(PO) usb_storage cls_u32 sch_htb intel_tfm_governor ecryptfs snd_soc_apl_mgu_hu intel_ipu4_psys intel_xhci_usb_role_switch dwc3 roles intel_ipu4_psys_csslib udc_core adv728x coretemp intel_ipu4_isys videobuf2_dma_contig i2c_i801 videobuf2_memops snd_soc_skl ipu4_acpi intel_ipu4_isys_csslib sdw_cnl videobuf2_v4l2 snd_soc_acpi_intel_match videobuf2_core snd_soc_acpi snd_soc_core sbi_apl snd_compress snd_soc_skl_ipc sdw_bus crc8 snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_ext_core intel_ipu4_mmu snd_hda_core ahci snd_pcm xhci_pci xhci_hcd libahci snd_timer
[14330.865056]  cfg80211 snd libata soundcore intel_ipu4 usbcore mei_me rfkill dwc3_pci usb_common scsi_mod iova mei nfsd auth_rpcgss lockd grace sunrpc zram zsmalloc loop fuse 8021q bridge stp llc inap560t(O) i915 video backlight intel_gtt i2c_algo_bit igb_avb(O) drm_kms_helper drm ptp pps_core hwmon spi_pxa2xx_platform firmware_class [last unloaded: bcmdhd]
[14330.900358] CPU: 0 PID: 251 Comm: 6310_io03 Tainted: P     U     O    4.14.198-apl #1
[14330.909105] task: ffff93fd77bd4b00 task.stack: ffff9a0ec0538000
[14330.915712] RIP: 0010:ipv4_dst_check+0x1c/0x40
[14330.920671] RSP: 0018:ffff93fd7fc03ca0 EFLAGS: 00010202
[14330.926504] RAX: 0000000000000000 RBX: ffff93fd6b0def00 RCX: 0000000000000001
[14330.934474] RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffff93fba73b1e00
[14330.942444] RBP: ffff93fd7fc03ca0 R08: 0000000079044900 R09: 0000000000005ba0
[14330.950414] R10: 63c730a037c730a0 R11: 0000000000000001 R12: ffff93fd2b634d80
[14330.958381] R13: ffffffffa26ac800 R14: ffff93fd76c08000 R15: 0000000000000008
[14330.966352] FS:  00007f1b37fff700(0000) GS:ffff93fd7fc00000(0000) knlGS:0000000000000000
[14330.975391] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[14330.981807] CR2: 0000000000000608 CR3: 00000001f6884000 CR4: 00000000003406b0
[14330.989777] Call Trace:
[14330.992505]  <IRQ>
[14330.994750]  tcp_v4_early_demux+0xf1/0x150
[14330.999326]  ip_rcv_finish+0x152/0x370
[14331.003509]  ip_rcv+0x25d/0x370
[14331.007004]  ? ip_local_deliver_finish+0x210/0x210
[14331.012356]  __netif_receive_skb_core+0x202/0x8d0
[14331.017609]  __netif_receive_skb+0x13/0x60
[14331.022181]  netif_receive_skb_internal+0x3b/0x110
[14331.027521]  napi_gro_receive+0xe9/0x110
[14331.031910]  igb_poll+0x6a4/0x1300 [igb_avb]
[14331.036682]  net_rx_action+0xdb/0x320
[14331.040771]  __do_softirq+0xd0/0x308
[14331.044765]  irq_exit+0xba/0xc0
[14331.048268]  do_IRQ+0x81/0xd0
[14331.051572]  common_interrupt+0x8d/0x8d
[14331.055853]  </IRQ>
[14331.058186] RIP: 0010:unix_write_space+0x1/0xa0
[14331.063243] RSP: 0018:ffff9a0ec053bab0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff4d
[14331.071702] RAX: ffffffffa1f5ed80 RBX: ffff93fb87832800 RCX: ffff93fb878320c8
[14331.079673] RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffff93fb87832800
[14331.087644] RBP: ffff9a0ec053bac8 R08: ffff9a0ec053c000 R09: ffff93fd2787ea31
[14331.095615] R10: ffff9a0ec053bcf8 R11: 0000000000000000 R12: ffff93fd4a4aa300
[14331.103587] R13: ffff93fb878320c8 R14: 0000000000000041 R15: 0000000000000000
[14331.111562]  ? unix_seqpacket_sendmsg+0x50/0x50
[14331.116624]  ? sock_wfree+0x39/0x60
[14331.120516]  unix_destruct_scm+0x7e/0xa0
[14331.124897]  skb_release_head_state+0x5b/0xb0
[14331.129760]  skb_release_all+0xd/0x30
[14331.133847]  consume_skb+0x2b/0xb0
[14331.137643]  unix_stream_read_generic+0x7e0/0x8d0
[14331.142900]  ? import_iovec+0x3a/0xe0
[14331.146987]  unix_stream_recvmsg+0x4c/0x70
[14331.151562]  ? unix_state_double_unlock+0x30/0x30
[14331.156816]  sock_recvmsg+0x3b/0x50
[14331.160710]  ___sys_recvmsg+0xd4/0x180
[14331.164899]  ? ktime_get+0x3e/0xa0
[14331.168694]  ? clockevents_program_min_delta+0x53/0x100
[14331.174530]  ? clockevents_program_event+0xee/0x110
[14331.179978]  ? tick_program_event+0x3f/0x70
[14331.184650]  ? __fget+0x71/0xa0
[14331.188157]  __sys_recvmsg+0x4c/0x90
[14331.192147]  ? __sys_recvmsg+0x4c/0x90
[14331.196334]  SyS_recvmsg+0x9/0x10
[14331.200033]  do_syscall_64+0x79/0x350
[14331.204120]  ? schedule+0x2e/0x90
[14331.207818]  ? exit_to_usermode_loop+0x5a/0x90
[14331.212772]  entry_SYSCALL_64_after_hwframe+0x41/0xa6
[14331.218413] RIP: 0033:0x7f1b439a8767
[14331.222401] RSP: 002b:00007f1b37ffdb80 EFLAGS: 00000293 ORIG_RAX: 000000000000002f
[14331.230857] RAX: ffffffffffffffda RBX: 00000000000000be RCX: 00007f1b439a8767
[14331.238827] RDX: 0000000000000000 RSI: 00007f1b37ffdbf0 RDI: 00000000000000be
[14331.246796] RBP: 00007f1b37ffdbf0 R08: 0000000000000000 R09: 0000000000000000
[14331.254766] R10: 0000000000000076 R11: 0000000000000293 R12: 0000000000000000
[14331.262735] R13: 00007f1b37fff5c0 R14: 00007f1b3815d470 R15: 0000000200000001
[14331.270705] Code: 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 66 83 7f 64 ff 55 48 89 e5 75 23 48 8b 07 48 8b 90 98 04 00 00 31 c0 48 85 d2 74 10 <8b> 92 04 06 00 00 39 97 a0 00 00 00 48 0f 44 c7 5d c3 31 c0 5d
[14331.291831] RIP: ipv4_dst_check+0x1c/0x40 RSP: ffff93fd7fc03ca0
[14331.298444] CR2: 0000000000000608

INDIRECT_CALLABLE_SCOPE struct dst_entry *ipv4_dst_check(struct dst_entry *dst,
 u32 cookie)
{
struct rtable *rt = (struct rtable *) dst;

/* All IPV4 dsts are created with ->obsolete set to the value
 * DST_OBSOLETE_FORCE_CHK which forces validation calls down
 * into this function always.
 *
 * When a PMTU/redirect information update invalidates a route,
 * this is indicated by setting obsolete to DST_OBSOLETE_KILL or
 * DST_OBSOLETE_DEAD.
 */
if (dst->obsolete != DST_OBSOLETE_FORCE_CHK || rt_is_expired(rt))
return NULL;
return dst;
}
EXPORT_INDIRECT_CALLABLE(ipv4_dst_check);

static inline bool rt_is_expired(const struct rtable *rth)
{
return (dev_net(rth->dst.dev) == NULL) ||
   (rth->rt_genid != rt_genid_ipv4(dev_net(rth->dst.dev))); <--- crashes here

Any ideas?
This damn smells like an UAF due to a race.

Unfortunately i don't have the coredump. Only the stack.
But no reproducer yet.

Best regards
Carsten
-----------------
Mentor Graphics (Deutschland) GmbH, Arnulfstrasse 201, 80634 München Registergericht München HRB 106955, Geschäftsführer: Thomas Heurung, Frank Thürauf

Powered by blists - more mailing lists