linux-kernel - Re: [RFC PATCH] blk-mq: Clean up references when freeing rqs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <76190c94-c5c1-9553-5509-9969fc323544@huawei.com>
Date:   Thu, 10 Dec 2020 12:22:35 +0000
From:   John Garry <john.garry@...wei.com>
To:     Ming Lei <ming.lei@...hat.com>
CC:     <axboe@...nel.dk>, <linux-block@...r.kernel.org>,
        <linux-kernel@...r.kernel.org>, <hch@....de>, <hare@...e.de>,
        <ppvk@...eaurora.org>, <bvanassche@....org>,
        <kashyap.desai@...adcom.com>
Subject: Re: [RFC PATCH] blk-mq: Clean up references when freeing rqs

Hi Ming,

> 
> So I'll try an experiment today to prove your concern about 
> blk_mq_queue_tag_busy_iter(). Then look at possible solution which 
> builds on patch in $subject, and compare.

JFYI, I was able to trigger this:

[ 1651.324150] 
==================================================================
[ 1651.331400] BUG: KASAN: use-after-free in bt_iter+0xa0/0x120
[ 1651.337054] Read of size 8 at addr ffff00108d589300 by task fio/3052

[ 1651.344891] CPU: 32 PID: 3052 Comm: fio Tainted: GW 
5.10.0-rc4-64839-g2dcf1ee5054f #693
[ 1651.354281] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon 
D05 IT21 Nemo 2.0 RC0 04/18/2018
[ 1651.363407] Call trace:
[ 1651.365861]  dump_backtrace+0x0/0x2d0
[ 1651.369519]  show_stack+0x18/0x68
[ 1651.372833]  dump_stack+0x100/0x16c
[ 1651.376316]  print_address_description.constprop.12+0x6c/0x4e8
[ 1651.382146]  kasan_report+0x130/0x200
[ 1651.385801]  __asan_load8+0x9c/0xd8
[ 1651.389284]  bt_iter+0xa0/0x120
[ 1651.392419]  blk_mq_queue_tag_busy_iter+0x2d8/0x540
[ 1651.397293]  blk_mq_in_flight+0x80/0xb8
[ 1651.401121]  part_stat_show+0xd8/0x238
[ 1651.404867]  dev_attr_show+0x44/0x90
[ 1651.408439]  sysfs_kf_seq_show+0x128/0x1c8
[ 1651.412530]  kernfs_seq_show+0xa0/0xb8
[ 1651.416274]  seq_read_iter+0x1ec/0x6a0
[ 1651.420019]  seq_read+0x1d0/0x250
[ 1651.423331]  kernfs_fop_read+0x70/0x330
[ 1651.427161]  vfs_read+0xe4/0x250
[ 1651.430382]  ksys_read+0xc8/0x178
[ 1651.433690]  __arm64_sys_read+0x44/0x58
[ 1651.437524]  el0_svc_common.constprop.2+0xc4/0x1e8
[ 1651.442309]  do_el0_svc+0x90/0xa0
[ 1651.445619]  el0_sync_handler+0x128/0x178
[ 1651.449623]  el0_sync+0x158/0x180

[ 1651.454418] The buggy address belongs to the page:
[ 1651.459208] page:00000000140c0813 refcount:0 mapcount:0 
mapping:0000000000000000 index:0x0 pfn:0x108d589
[ 1651.468683] flags: 0xbfffc0000000000()
[ 1651.472432] raw: 0bfffc0000000000 0000000000000000 ffffffff42150201 
0000000000000000
[ 1651.480173] raw: 0000000000000000 0000000000000000 00000000ffffffff 
0000000000000000
[ 1651.487909] page dumped because: kasan: bad access detected

[ 1651.494956] Memory state around the buggy address:
[ 1651.499744]  ffff00108d589200: ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff
[ 1651.506960]  ffff00108d589280: ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff
[ 1651.514176] >ffff00108d589300: ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff
[ 1651.521389]       ^
[ 1651.524611]  ffff00108d589380: ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff
[ 1651.531827]  ffff00108d589400: ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff
[ 1651.539044] 
==================================================================
[ 1651.546258] Disabling lock debugging due to kernel taint
john@...ntu:~$

So I run fio on disk providing root partition and another disk, and 
constantly change IO scheduler on other disk.

I did also add this delay to trigger in reasonable timeframe:

+++ b/block/blk-mq-tag.c
@@ -215,8 +215,11 @@ static bool bt_iter(struct sbitmap *bitmap, 
unsigned int bitnr, void *data)
* We can hit rq == NULL here, because the tagging functions
* test and set the bit before assigning ->rqs[].
*/
-  if (rq && rq->q == hctx->queue && rq->mq_hctx == hctx)
-     return iter_data->fn(hctx, rq, iter_data->data, reserved);
+  if (rq) {
+     mdelay(50);
+     if (rq->q == hctx->queue && rq->mq_hctx == hctx)
+        return iter_data->fn(hctx, rq, iter_data->data, reserved);
+       }
        return true;
}


Thanks,
John