Message-ID: <CAG48ez2Rk5R=tNOa0vViah850ZdvVHn30knDX3pzM4pm67ZM=w@mail.gmail.com>
Date:   Thu, 19 May 2022 20:53:33 +0200
From:   Jann Horn <jannh@...gle.com>
To:     USB list <linux-usb@...r.kernel.org>,
        Network Development <netdev@...r.kernel.org>
Cc:     "David S. Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        Oliver Neukum <oneukum@...e.com>,
        Oleksij Rempel <o.rempel@...gutronix.de>,
        kernel list <linux-kernel@...r.kernel.org>
Subject: usbnet tells minidrivers to unbind while netdev is still up, causing UAFs

[resubmitting to public list - this was previously submitted to
security@...nel.org, but ended up not being resolved within 90 days.
see also https://crbug.com/project-zero/2262.]

I've been digging more into the usbnet code, and it's all really brittle.
I keep hitting random KASAN splats while I'm just trying to normally bring
up drivers (mainly when my fake USB device can't handle some request yet
and exits, which results in a USB disconnect, and apparently usbnet
tends to blow up a lot in various ways if you disconnect before the
device is fully up, at least on a kernel with
CONFIG_RCU_STRICT_GRACE_PERIOD=y).

One particularly easy-to-trigger bug was introduced by
commit 2c9d6c2b871d ("usbnet: run unbind() before unregister_netdev()"),
first in v5.14.

Before that commit, the driver_info->unbind() callback was the last call
to the minidriver during USB disconnect, and so one of the things some
minidrivers do there is to free memory associated with the device.

But after that commit, the semantics of driver_info->unbind() are
completely different: It is called at a point where the networking
subsystem **has no idea** yet that the device is going down.
The netdev might still be up, or in the middle of going up, or going
down, or whatever else netdevs do; and so it is still possible that
e.g. userspace sends some netlink message that results in a call to
the minidriver's ->reset method, and then e.g. aqc111_reset() will
try to access its freed dev->driver_priv, and you get UAF.


I looked at this more, and it turns out that with another
minidriver, you don't even need to race to cause a UAF:
Simply disconnecting a USB device when it is currently up will
reliably cause a UAF.

This is the case with the driver in drivers/net/usb/ax88172a.c
(described as "ASIX AX88172A USB 2.0 Ethernet"), which is only
used for the USB device ID of some demo board, nothing else:

/* ASIX 88172a demo board */
USB_DEVICE(0x0b95, 0x172a),
.driver_info = (unsigned long) &ax88172a_info,

(Even though this driver is only used for talking to some demo
board, it is enabled in kconfig together with all the other ASIX
devices using kconfig flag CONFIG_USB_NET_AX8817X, which is
enabled on many kernels, including Debian, some Android kernels
and Chrome OS. Android and Chrome OS are probably not affected
by this one though, since they run sufficiently old kernels...)

The call graph of how the UAF happens:

usbnet_disconnect
  ax88172a_unbind (as driver_info->unbind)
    kfree(dev->driver_priv)
  unregister_netdev
    unregister_netdevice
      unregister_netdevice_queue
        unregister_netdevice_many
          dev_close_many
            __dev_close_many
              usbnet_stop (as ops->ndo_stop)
                ax88172a_stop (as driver_info->stop)
                  [UAF access to dev->driver_priv]

The driver_info->stop() handler tries to access data that was freed
in driver_info->unbind(). This makes it pretty clear that the
reordering in commit 2c9d6c2b871d broke stuff.


I have no clue how to fix all this though. From what I can tell,
there are two points during usbnet_disconnect() where the
minidriver might want to get a callback:

 - When ->ndo_close() is invoked by the netdev code; at that point,
   the netdev is definitely down but hasn't been completely torn
   down yet. usbnet doesn't currently use ->ndo_open/->ndo_close
   at all.
 - In the spot where the driver_info->unbind callback used to
   happen before commit 2c9d6c2b871d.

But I have no clue whether we need one or both of these, and
which of the things the current ->unbind callbacks do have to
happen at what time.
commit 2c9d6c2b871d claims that the current ->unbind callback
is too late for the minidriver to disconnect PHY, so I guess
that should probably happen in ->ndo_close()? Maybe?


==== kernel splats and reproduction instructions ====

On a system running a normal Debian experimental kernel
(version 5.17.0-rc3-amd64 #1  Debian 5.17~rc3-1~exp1), this also
happens. With slub_debug=PF (to make SLUB poison freed memory
and do some extra consistency checks to make UAFs easier to see),
I get this in dmesg when I attach a fake USB device (over real
USB, using a NET2380 USB device-side controller on another
machine), wait for a few seconds so that it can be brought up
completely, and then disconnect it:

[  138.697877] usb 1-2: new high-speed USB device number 3 using xhci_hcd
[  138.852315] usb 1-2: New USB device found, idVendor=0b95,
idProduct=172a, bcdDevice= 0.00
[  138.856972] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  138.861557] usb 1-2: Product: DUMMY
[  138.866016] usb 1-2: Manufacturer: DUMMY
[  138.870312] usb 1-2: SerialNumber: DUMMY
[  139.403344] asix 1-2:1.0 (unnamed net_device) (uninitialized):
registered mdio bus usb-001:003
[  139.404897] asix 1-2:1.0 eth1: register 'asix' at
usb-0000:00:14.0-2, ASIX AX88172A USB 2.0 Ethernet, 00:12:34:56:78:90
[  139.406292] usbcore: registered new interface driver asix
[  139.408747] usbcore: registered new interface driver cdc_ether
[  139.481399] asix 1-2:1.0 enx001234567890: renamed from eth1
[  140.150427] asix 1-2:1.0 enx001234567890: Connected to phy usb-001:003:00
[  149.299153] usb 1-2: USB disconnect, device number 3
[  149.303077] asix 1-2:1.0 enx001234567890: unregister 'asix'
usb-0000:00:14.0-2, ASIX AX88172A USB 2.0 Ethernet
[  149.306775] asix 1-2:1.0 enx001234567890: deregistering mdio bus usb-001:003
[  149.314206] asix 1-2:1.0 enx001234567890: Disconnecting from phy
kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk\xa5%!LhH\xf2/\xa1ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ\x80\xe5W\xe1\xf9\x99\xff\xff\xe0#ա\xff\xff\xff\xff\x80t\xf0\xc0\xff\xff\xff\xff\x98XW\x82\xf9\x99\xff\xff\x80\xb9[\xe1\xf9\x99\xff\xff
[  149.315731] general protection fault, probably for non-canonical
address 0x6b6b6b6b6b6b6f43: 0000 [#1] PREEMPT SMP PTI
[  149.317234] CPU: 4 PID: 105 Comm: kworker/4:1 Tainted: G
E     5.17.0-rc3-amd64 #1  Debian 5.17~rc3-1~exp1
[  149.318815] Hardware name: [...]
[  149.320214] Workqueue: usb_hub_wq hub_event [usbcore]
[  149.321030] RIP: 0010:phy_stop+0x9/0xf0 [libphy]
[  149.321835] Code: 02 e0 eb d4 48 8b 0c dd 20 ed b9 c0 e9 33 ff ff
ff 4c 89 f7 e8 68 a3 fa df eb c6 e8 11 ff 1a e0 90 0f 1f 44 00 00 41
54 55 53 <8b> 87 d8 03 00 00 4c 8b a7 28 05 00 00 8d 50 ff 83 fa 01 0f
86 af
[  149.322714] RSP: 0018:ffffb68a807efa58 EFLAGS: 00010246
[  149.323560] RAX: 0000000000000000 RBX: ffff99f9e168e980 RCX: 0000000000000000
[  149.324405] RDX: ffffb68a807efa08 RSI: ffffffffa15526f6 RDI: 6b6b6b6b6b6b6b6b
[  149.325251] RBP: ffff99f9e15bbb80 R08: 0000000000000000 R09: ffffb68a807ef758
[  149.326128] R10: ffffb68a807ef750 R11: ffffffffa1cd1568 R12: 0000000000000000
[  149.326965] R13: ffff99f9e168e980 R14: ffffb68a807efad0 R15: ffffb68a807efba0
[  149.327800] FS:  0000000000000000(0000) GS:ffff99fc8f700000(0000)
knlGS:0000000000000000
[  149.328641] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  149.329482] CR2: 00007fc2351ba6f4 CR3: 00000002fb410001 CR4: 00000000001706e0
[  149.330352] Call Trace:
[  149.331198]  <TASK>
[  149.332030]  ax88172a_stop.cold+0x20/0x2e [asix]
[  149.332857]  usbnet_stop+0x64/0x140 [usbnet]
[  149.333676]  __dev_close_many+0x9e/0x110
[  149.334512]  dev_close_many+0x8b/0x140
[  149.335295]  ? __slab_free+0xa0/0x330
[  149.336059]  unregister_netdevice_many+0x158/0x740
[  149.336816]  ? kfree+0x218/0x250
[  149.337573]  unregister_netdevice_queue+0xcb/0x110
[  149.338361]  unregister_netdev+0x18/0x20
[  149.339112]  usbnet_disconnect+0x59/0xb0 [usbnet]
[  149.339864]  usb_unbind_interface+0x8a/0x270 [usbcore]
[  149.340616]  __device_release_driver+0x22d/0x240
[  149.341358]  device_release_driver+0x24/0x30
[  149.342112]  bus_remove_device+0xd8/0x140
[  149.342840]  device_del+0x18b/0x3f0
[  149.343573]  ? kobject_put+0x91/0x1d0
[  149.344307]  usb_disable_device+0xc6/0x1e0 [usbcore]
[  149.345056]  usb_disconnect.cold+0x7b/0x24d [usbcore]
[  149.345802]  hub_event+0xc4c/0x1880 [usbcore]
[  149.346571]  ? preempt_count_sub+0x81/0x90
[  149.347313]  process_one_work+0x1e5/0x3b0
[  149.348055]  ? rescuer_thread+0x370/0x370
[  149.348795]  worker_thread+0x50/0x3a0
[  149.349531]  ? rescuer_thread+0x370/0x370
[  149.350292]  kthread+0xe7/0x110
[  149.351030]  ? kthread_complete_and_exit+0x20/0x20
[  149.351771]  ret_from_fork+0x22/0x30
[  149.352504]  </TASK>
[  149.353234] Modules linked in: cdc_ether(E) asix(E) selftests(E)
usbnet(E) mii(E) nfnetlink(E) rfkill(E) zstd(E) zstd_compress(E)
zram(E) zsmalloc(E) intel_rapl_msr(E) intel_rapl_common(E)
x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) nls_ascii(E)
nls_cp437(E) vfat(E) snd_hda_codec_realtek(E) kvm_intel(E) fat(E)
snd_hda_codec_generic(E) snd_hda_codec_hdmi(E) ledtrig_audio(E) kvm(E)
irqbypass(E) snd_hda_intel(E) crc32_pclmul(E) snd_intel_dspcfg(E)
snd_intel_sdw_acpi(E) snd_hda_codec(E) iTCO_wdt(E) intel_pmc_bxt(E)
iTCO_vendor_support(E) snd_hda_core(E) at24(E) mei_hdcp(E) watchdog(E)
ghash_clmulni_intel(E) snd_hwdep(E) snd_pcm_oss(E) snd_mixer_oss(E)
rapl(E) r8169(E) intel_cstate(E) intel_uncore(E) efi_pstore(E)
pcspkr(E) realtek(E) snd_pcm(E) mdio_devres(E) i2c_i801(E)
snd_timer(E) mei_me(E) i2c_smbus(E) snd(E) ehci_pci(E) sg(E) libphy(E)
soundcore(E) ehci_hcd(E) mei(E) lpc_ich(E) button(E) msr(E)
parport_pc(E) ppdev(E) parport(E) fuse(E) configfs(E) efivarfs(E)
ip_tables(E)
[  149.353267]  x_tables(E) autofs4(E) ext4(E) crc16(E) mbcache(E)
jbd2(E) dm_crypt(E) dm_mod(E) raid10(E) raid456(E) libcrc32c(E)
crc32c_generic(E) async_raid6_recov(E) async_memcpy(E) async_pq(E)
async_xor(E) xor(E) async_tx(E) raid6_pq(E) raid1(E) raid0(E)
multipath(E) linear(E) md_mod(E) hid_generic(E) usbhid(E) hid(E)
sd_mod(E) t10_pi(E) crc_t10dif(E) crct10dif_generic(E)
crct10dif_pclmul(E) crct10dif_common(E) evdev(E) crc32c_intel(E)
i915(E) i2c_algo_bit(E) ahci(E) xhci_pci(E) libahci(E)
drm_kms_helper(E) xhci_hcd(E) cec(E) rc_core(E) libata(E) ttm(E)
aesni_intel(E) crypto_simd(E) usbcore(E) scsi_mod(E) cryptd(E)
scsi_common(E) drm(E) usb_common(E) video(E)
[  149.360796] ---[ end trace 0000000000000000 ]---
[  149.362505] ------------[ cut here ]------------


In my test VM with a bunch of kernel debugging enabled, I get this
KASAN splat (shown here without guess frames) when I attach the same
fake USB device through an emulated HCD:

BUG: KASAN: use-after-free in ax88172a_stop+0xab/0xc0
Read of size 8 at addr ffff88800c684e48 by task kworker/0:2/33

CPU: 0 PID: 33 Comm: kworker/0:2 Not tainted 5.17.0-rc4-00054-gf71077a4d84b #949
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
Workqueue: usb_hub_wq hub_event
Call Trace:
 <TASK>
 dump_stack_lvl+0x45/0x59
 print_address_description.constprop.0+0x1f/0x150
 kasan_report.cold+0x7f/0x11b
 ax88172a_stop+0xab/0xc0
 usbnet_stop+0x13d/0x390
 __dev_close_many+0x18c/0x290
 dev_close_many+0x18a/0x3f0
 unregister_netdevice_many+0x2f8/0x1420
 unregister_netdevice_queue+0x1dc/0x280
 unregister_netdev+0x18/0x20
 usbnet_disconnect+0x118/0x260
 usb_unbind_interface+0x182/0x7e0
 __device_release_driver+0x531/0x670
 device_release_driver+0x26/0x40
 bus_remove_device+0x2ae/0x570
 device_del+0x490/0xb50
 usb_disable_device+0x294/0x600
 usb_disconnect.cold+0x1fb/0x68b
 hub_event+0x1472/0x39d0
 process_one_work+0x91d/0x15d0
 worker_thread+0x57b/0x1240
 kthread+0x2a5/0x350
 ret_from_fork+0x22/0x30
 </TASK>

Allocated by task 33:
 kasan_save_stack+0x1e/0x40
 __kasan_kmalloc+0x81/0xa0
 ax88172a_bind+0x95/0x7b0
 usbnet_probe+0xa62/0x2370
 usb_probe_interface+0x27d/0x760
 really_probe+0x475/0xbd0
 __driver_probe_device+0x18f/0x470
 driver_probe_device+0x49/0x120
 __device_attach_driver+0x199/0x250
 bus_for_each_drv+0x125/0x1b0
 __device_attach+0x1e0/0x3d0
 bus_probe_device+0x1a5/0x260
 device_add+0x971/0x1a70
 usb_set_configuration+0x92b/0x1600
 usb_generic_driver_probe+0x79/0xa0
 usb_probe_device+0xab/0x250
 really_probe+0x475/0xbd0
 __driver_probe_device+0x18f/0x470
 driver_probe_device+0x49/0x120
 __device_attach_driver+0x199/0x250
 bus_for_each_drv+0x125/0x1b0
 __device_attach+0x1e0/0x3d0
 bus_probe_device+0x1a5/0x260
 device_add+0x971/0x1a70
 usb_new_device.cold+0x47d/0xb88
 hub_event+0x20c7/0x39d0
 process_one_work+0x91d/0x15d0
 worker_thread+0x57b/0x1240
 kthread+0x2a5/0x350
 ret_from_fork+0x22/0x30

Freed by task 33:
 kasan_save_stack+0x1e/0x40
 kasan_set_track+0x21/0x30
 kasan_set_free_info+0x20/0x30
 __kasan_slab_free+0xe0/0x110
 kfree+0xa5/0x2b0
 usbnet_disconnect+0xe7/0x260
 usb_unbind_interface+0x182/0x7e0
 __device_release_driver+0x531/0x670
 device_release_driver+0x26/0x40
 bus_remove_device+0x2ae/0x570
 device_del+0x490/0xb50
 usb_disable_device+0x294/0x600
 usb_disconnect.cold+0x1fb/0x68b
 hub_event+0x1472/0x39d0
 process_one_work+0x91d/0x15d0
 worker_thread+0x57b/0x1240
 kthread+0x2a5/0x350
 ret_from_fork+0x22/0x30



If you want to test this yourself, you can use the USB raw gadget
(https://www.kernel.org/doc/html/latest/usb/raw-gadget.html)
with dummy_hcd. Compile the attached testcase, then run it as
"./usb-ax88172a dummy_udc dummy_udc.0", wait a few seconds for
the device to come up, and press CTRL+C to trigger USB disconnect.
This requires CONFIG_USB_DUMMY_HCD=y and CONFIG_USB_RAW_GADGET=y.

View attachment "usb-ax88172a.c" of type "text/x-c-code" (25031 bytes)

Powered by blists - more mailing lists