Message-ID: <ecd964430ff124469ba48e289cf2e7404fcdc068.camel@gmail.com>
Date:   Thu, 07 Jan 2021 00:15:24 +0100
From:   Igor Raits <igor.raits@...il.com>
To:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Kernel panic on shutdown (qede+bond+bridge) - KASAN: use-after-free
 in netif_skb_features+0x90a/0x9b0

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hello,

I've been trying out the latest CentOS 8 Stream kernel and found that I
get kernel panic (https://bugzilla.redhat.com/show_bug.cgi?id=1913481)
when trying to reboot the server. With debug kernel I've got following:

[  531.818434]
==================================================================
[  531.818435] BUG: KASAN: use-after-free in
netif_skb_features+0x90a/0x9b0
[  531.818436] Read of size 8 at addr ffff893c74d54b50 by task systemd-
shutdow/1
[  531.818436]                            
[  531.818437] CPU: 20 PID: 1 Comm: systemd-shutdow Tainted: G        W
I      --------- -  - 4.18.0-259.el8.x86_64+debug #1
[  531.818438] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380
Gen10, BIOS U30 07/16/2020
[  531.818438] Call Trace:                                            
[  531.818438]  dump_stack+0x8e/0xd0                                  
[  531.818439]  print_address_description.constprop.3+0x1f/0x300
[  531.818439]  __kasan_report.cold.7+0x76/0xbf                    
[  531.818440]  ? netif_skb_features+0x90a/0x9b0    
[  531.818440]  kasan_report+0xe/0x20                                 
[  531.818441]  netif_skb_features+0x90a/0x9b0        
[  531.818441]  ? unwind_get_return_address+0x56/0xa0                 
[  531.818442]  ? __skb_gso_segment+0x620/0x620                       
[  531.818442]  ? do_raw_spin_unlock+0x54/0x230              
[  531.818442]  netpoll_start_xmit+0x28/0x680
[  531.818443]  ? netdev_pick_tx+0x13b/0x310        
[  531.818443]  netpoll_send_skb+0x50f/0x760                          
[  531.818444]  ? save_stack+0x19/0x80                                
[  531.818444]  netpoll_start_xmit+0x335/0x680                        
[  531.818444]  netpoll_send_skb+0x50f/0x760                    
[  531.818445]  ? rcu_read_lock_sched_held+0xe0/0xe0                  
[  531.818445]  __br_forward+0x680/0x730 [bridge]                     
[  531.818446]  ? br_forward_finish+0x330/0x330 [bridge]              
[  531.818446]  ? br_flood+0x2ce/0x560 [bridge]  
[  531.818447]  br_dev_xmit+0x9db/0x1150 [bridge]
[  531.818447]  ? br_poll_controller+0x10/0x10 [bridge]
[  531.818448]  ? dev_loopback_xmit+0x120/0x550
[  531.818448]  ? __skb_gso_segment+0x620/0x620
[  531.818448]  ? memset+0x1f/0x40
[  531.818449]  netpoll_start_xmit+0x335/0x680
[  531.818449]  netpoll_send_skb+0x50f/0x760
[  531.818450]  write_msg+0x1e2/0x240 [netconsole]
[  531.818450]  console_unlock+0x602/0x9b0
[  531.818450]  vprintk_emit+0x158/0x490
[  531.818451]  printk+0x9f/0xc5
[  531.818451]  ? kmsg_dump_rewind_nolock+0xd9/0xd9
[  531.818452]  ? qede_free_fp_array+0x24c/0x3b0 [qede]
[  531.818452]  qede_unload+0x133e/0x1ae0 [qede]
[  531.818453]  ? lock_downgrade+0x740/0x740
[  531.818453]  ? qede_free_mem_fp+0xa00/0xa00 [qede]
[  531.818454]  ? lock_contended+0xcd0/0xcd0
[  531.818454]  ? __local_bh_enable_ip+0xb3/0x100
[  531.818455]  ? dev_reset_queue.constprop.16+0xa5/0xd0
[  531.818455]  ? trace_hardirqs_on+0x20/0x195
[  531.818455]  ? __local_bh_enable_ip+0xb3/0x100
[  531.818456]  ? dev_deactivate_many+0x6d1/0x9a0
[  531.818456]  ? __local_bh_enable_ip+0xb3/0x100
[  531.818457]  ? dev_deactivate_many+0x6d1/0x9a0
[  531.818457]  qede_close+0x1f/0xe0 [qede]
[  531.818457]  __dev_close_many+0x18e/0x2b0
[  531.818458]  ? netdev_lower_state_changed+0x110/0x110
[  531.818458]  dev_close_many+0x1e2/0x5b0
[  531.818459]  ? unregister_netdevice_notifier_dev_net+0x190/0x190
[  531.818459]  ? unregister_netdev+0xe/0x20
[  531.818460]  rollback_registered_many+0x365/0xfe0
[  531.818460]  ? lock_contended+0xcd0/0xcd0
[  531.818461]  ? netif_set_real_num_tx_queues+0x700/0x700
[  531.818461]  ? __mutex_lock+0xe03/0x13d0
[  531.818462]  ? lock_acquire+0x1a3/0x970
[  531.818462]  ? quarantine_put+0xa0/0x290
[  531.818462]  ? unregister_netdev+0xe/0x20
[  531.818463]  ? mutex_lock_io_nested+0x12a0/0x12a0
[  531.818463]  rollback_registered+0xd1/0x180
[  531.818464]  ? rollback_registered_many+0xfe0/0xfe0
[  531.818464]  ? rpm_callback+0x210/0x210
[  531.818464]  ? _raw_spin_unlock_irq+0x24/0x40
[  531.818465]  unregister_netdevice_queue+0x18b/0x250
[  531.818465]  unregister_netdev+0x18/0x20
[  531.818466]  __qede_remove+0x1e2/0x4c0 [qede]
[  531.818466]  pci_device_shutdown+0x76/0x120
[  531.818467]  device_shutdown+0x35f/0x690
[  531.818467]  ? __blocking_notifier_call_chain+0x80/0xc0
[  531.818467]  kernel_restart+0xe/0x30
[  531.818468]  __do_sys_reboot+0x298/0x2e0
[  531.818468]  ? kernel_power_off+0xa0/0xa0
[  531.818469]  ? security_file_free+0x3f/0x70
[  531.818469]  ? kmem_cache_free+0x2ac/0x360
[  531.818470]  ? __fget_light+0x55/0x1f0
[  531.818470]  ? do_writev+0xc1/0x220
[  531.818470]  ? vfs_writev+0x2d0/0x2d0
[  531.818471]  ? trace_hardirqs_on_thunk+0x1a/0x20
[  531.818471]  ? trace_hardirqs_on_caller+0x22/0x1a0
[  531.818472]  ? do_syscall_64+0x22/0x420
[  531.818472]  do_syscall_64+0xa5/0x420
[  531.818473]  entry_SYSCALL_64_after_hwframe+0x6a/0xdf
[  531.818473] RIP: 0033:0x7f108ffbb097
[  531.818474] Code: 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00
00 90 f3 0f 1e fa 89 fa be 69 19 12 28 bf ad de e1 fe b8 a9 00 00 00 0f
05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 c1 7d 2c 00 f7 d8 64 89 02 b8
[  531.818475] RSP: 002b:00007ffedba60838 EFLAGS: 00000202 ORIG_RAX:
00000000000000a9
[  531.818476] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
00007f108ffbb097
[  531.818476] RDX: 0000000001234567 RSI: 0000000028121969 RDI:
00000000fee1dead
[  531.818477] RBP: 00007ffedba608e0 R08: 0000000000000000 R09:
00007ffedba5fa57
[  531.818478] R10: 00007ffedba5fe00 R11: 0000000000000202 R12:
0000000000000000
[  531.818478] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000006
[  531.818478]
[  531.818479] Allocated by task 1:
[  531.818479]  save_stack+0x19/0x80
[  531.818480]  __kasan_kmalloc.constprop.10+0xc1/0xd0
[  531.818480]  kmem_cache_alloc_node+0xe5/0x380
[  531.818480]  __alloc_skb+0x97/0x530
[  531.818481]  netpoll_send_udp+0x19b/0x17c0
[  531.818481]  write_msg+0x1e2/0x240 [netconsole]
[  531.818482]  console_unlock+0x602/0x9b0
[  531.818482]  vprintk_emit+0x158/0x490
[  531.818482]  printk+0x9f/0xc5
[  531.818483]  qede_unload+0x133e/0x1ae0 [qede]
[  531.818483]  qede_close+0x1f/0xe0 [qede]
[  531.818484]  __dev_close_many+0x18e/0x2b0
[  531.818484]  dev_close_many+0x1e2/0x5b0
[  531.818485]  rollback_registered_many+0x365/0xfe0
[  531.818485]  rollback_registered+0xd1/0x180
[  531.818485]  unregister_netdevice_queue+0x18b/0x250
[  531.818486]  unregister_netdev+0x18/0x20
[  531.818486]  __qede_remove+0x1e2/0x4c0 [qede]
[  531.818487]  pci_device_shutdown+0x76/0x120
[  531.818487]  device_shutdown+0x35f/0x690
[  531.818488]  kernel_restart+0xe/0x30
[  531.818488]  __do_sys_reboot+0x298/0x2e0
[  531.818488]  do_syscall_64+0xa5/0x420
[  531.818489]  entry_SYSCALL_64_after_hwframe+0x6a/0xdf
[  531.818489]
[  531.818490] Freed by task 1:
[  531.818490]  save_stack+0x19/0x80
[  531.818490]  __kasan_slab_free+0x125/0x170
[  531.818491]  kmem_cache_free+0xcd/0x360
[  531.818491]  zap_completion_queue+0x22a/0x300
[  531.818492]  bond_poll_controller+0x214/0x3b0 [bonding]
[  531.818492]  netpoll_poll_dev+0x110/0x5b0
[  531.818492]  netpoll_send_skb+0x2f9/0x760
[  531.818493]  netpoll_start_xmit+0x335/0x680
[  531.818493]  netpoll_send_skb+0x50f/0x760
[  531.818494]  __br_forward+0x680/0x730 [bridge]
[  531.818494]  br_dev_xmit+0x9db/0x1150 [bridge]
[  531.818495]  netpoll_start_xmit+0x335/0x680
[  531.818495]  netpoll_send_skb+0x50f/0x760
[  531.818495]  write_msg+0x1e2/0x240 [netconsole]
[  531.818496]  console_unlock+0x602/0x9b0
[  531.818496]  vprintk_emit+0x158/0x490
[  531.818497]  printk+0x9f/0xc5
[  531.818497]  qede_unload+0x133e/0x1ae0 [qede]
[  531.818497]  qede_close+0x1f/0xe0 [qede]
[  531.818498]  __dev_close_many+0x18e/0x2b0
[  531.818498]  dev_close_many+0x1e2/0x5b0
[  531.818499]  rollback_registered_many+0x365/0xfe0
[  531.818499]  rollback_registered+0xd1/0x180
[  531.818499]  unregister_netdevice_queue+0x18b/0x250
[  531.818500]  unregister_netdev+0x18/0x20
[  531.818500]  __qede_remove+0x1e2/0x4c0 [qede]
[  531.818501]  pci_device_shutdown+0x76/0x120
[  531.818501]  device_shutdown+0x35f/0x690
[  531.818502]  kernel_restart+0xe/0x30
[  531.818502]  __do_sys_reboot+0x298/0x2e0
[  531.818502]  do_syscall_64+0xa5/0x420
[  531.818503]  entry_SYSCALL_64_after_hwframe+0x6a/0xdf
[  531.818503]
[  531.818504] The buggy address belongs to the object at
ffff893c74d54b40
[  531.818504]  which belongs to the cache skbuff_head_cache of size
256
[  531.818505] The buggy address is located 16 bytes inside of
[  531.818505]  256-byte region [ffff893c74d54b40, ffff893c74d54c40)
[  531.818506] The buggy address belongs to the page:
[  531.818506] page:ffffea02f1d35500 refcount:1 mapcount:0
mapping:ffff88c6abf6d100 index:0x0 compound_mapcount: 0
[  531.818507] flags: 0x57ffffc0008100(slab|head)
[  531.818508] raw: 0057ffffc0008100 dead000000000100 dead000000000200
ffff88c6abf6d100
[  531.818508] raw: 0000000000000000 0000000080330033 00000001ffffffff
0000000000000000
[  531.818509] page dumped because: kasan: bad access detected
[  531.818509]
[  531.818509] Memory state around the buggy address:
[  531.818510]  ffff893c74d54a00: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[  531.818511]  ffff893c74d54a80: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[  531.818511] >ffff893c74d54b00: fc fc fc fc fc fc fc fc fb fb fb fb
fb fb fb fb
[  531.818512]                                                  ^
[  531.818512]  ffff893c74d54b80: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[  531.818513]  ffff893c74d54c00: fb fb fb fb fb fb fb fb fc fc fc fc
fc fc fc fc
[  531.818513]
==================================================================
[  532.636252] kasan: CONFIG_KASAN_INLINE enabled

I quickly checked the upstream kernel tree and could not find anything
related to this. Do you have any ideas other than trying the latest
kernel? Any other additional information I could provide?

Thanks!
- -- 
Igor Raits <igor.raits@...il.com>
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl/2RIwACgkQEV1auJxc
Hh5DDBAAryTjDd5rre0nD1f1CMccK+AdFYu1piSwRM8Mp/SMf4rJUdIr0fS3KiKq
TffflZ3IRM+W57wOLuCjpe6OxC2j+6CNR2zJW4Tfb2bOjzKSfeiGwyW5jc6J2qnz
WN1CnZX3Cb3CnUA6LXnouA1yciy/tYo4G63oc3DT0mwk5Ti9Lp8+9jKOvF4NEu4C
IHRD0VcmCPbhyPkA3ka//NoV3h+foeSQdJth+feOxpTlBLMD88i/qIVsHWRl+V+l
jD6+sv9vkS15oJRDYoiOZnwHDiW36ssJcq0wOwE7ZZ1W9Q377QXR+bYa4tlIEUpb
YRRfpNMREUeUgD3VAAYWtHKPnvb64k5KijL1M+Up5sNB7SiCbZhnXIJ4LFOLWSaE
JizbrqgaTflCSi3b/ZQr4jrcQiGJu0lWjuMOaC3srtQTyg3tNPB6Iy9taF992ySS
m3FdR7d8A3snPW8wXSTmzng+qpd+r1zcDUSWS8zcDrk7M3piQNy6lVAV+rDf9ycL
0fHzkaCYvfN+j2pnZFqpDN83IrqeA3bt4jCniGOS75af+JXJpL+PRCVPJLM14Dpq
j3r3QRkASrxhWNXkIZEwMeTV8ldtdgUV86QK3Z31SPCEY+J62lzeM7SwCZ3jaVZ3
u4tgNsPQkezVYw2HTGAlmu+Whsy73KBy5oFAmoZBjSwgSMg7lXo=
=uvWj
-----END PGP SIGNATURE-----

Powered by blists - more mailing lists