linux-kernel - Re: swapon/913 is trying to acquire lock at zcomp_stream_get+0x5/0x90 [zram] but task is already holding lock at zs_map

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20201016124009.GQ2611@hirez.programming.kicks-ass.net>
Date:   Fri, 16 Oct 2020 14:40:09 +0200
From:   Peter Zijlstra <peterz@...radead.org>
To:     Mikhail Gavrilov <mikhail.v.gavrilov@...il.com>
Cc:     Linux List Kernel Mailing <linux-kernel@...r.kernel.org>,
        linux-block@...r.kernel.org,
        Mike Galbraith <umgwanakikbuti@...il.com>, minchan@...nel.org,
        ngupta@...are.org, sergey.senozhatsky.work@...il.com,
        bigeasy@...utronix.de
Subject: Re: swapon/913 is trying to acquire lock at
 zcomp_stream_get+0x5/0x90 [zram] but task is already holding lock at
 zs_map_object+0x7a/0x2e0

On Fri, Oct 16, 2020 at 11:21:47AM +0500, Mikhail Gavrilov wrote:
> Hi folks,
> today I joined to testing Kernel 5.10 and see that every boot happens
> this warning:
> 
> [    9.032096] ======================================================
> [    9.032097] WARNING: possible circular locking dependency detected
> [    9.032098] 5.10.0-0.rc0.20201014gitb5fc7a89e58b.41.fc34.x86_64 #1 Not tainted
> [    9.032099] ------------------------------------------------------
> [    9.032100] swapon/913 is trying to acquire lock:
> [    9.032101] ffffc984fda4f948 (&zstrm->lock){+.+.}-{2:2}, at: zcomp_stream_get+0x5/0x90 [zram]
> [    9.032106] but task is already holding lock:
> [    9.032107] ffff993c54cdceb0 (&zspage->lock){.+.+}-{2:2}, at: zs_map_object+0x7a/0x2e0
> [    9.032111] which lock already depends on the new lock.
> [    9.032112] the existing dependency chain (in reverse order) is:

> [    9.032112] -> #1 (&zspage->lock){.+.+}-{2:2}:
> [    9.032116]        _raw_read_lock+0x3d/0xa0
> [    9.032118]        zs_map_object+0x7a/0x2e0
> [    9.032119]        zram_bvec_rw.constprop.0.isra.0+0x287/0x730 [zram]
> [    9.032121]        zram_submit_bio+0x189/0x35d [zram]
> [    9.032123]        submit_bio_noacct+0xff/0x650
> [    9.032124]        submit_bh_wbc+0x17d/0x1a0
> [    9.032126]        __block_write_full_page+0x227/0x580
> [    9.032128]        __writepage+0x1a/0x70
> [    9.032129]        write_cache_pages+0x21c/0x540
> [    9.032130]        generic_writepages+0x41/0x60
> [    9.032131]        do_writepages+0x28/0xb0
> [    9.032133]        __filemap_fdatawrite_range+0xa7/0xe0
> [    9.032134]        file_write_and_wait_range+0x67/0xb0
> [    9.032135]        blkdev_fsync+0x17/0x40
> [    9.032137]        __x64_sys_fsync+0x34/0x60
> [    9.032138]        do_syscall_64+0x33/0x40
> [    9.032140]        entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [    9.032140]
>                -> #0 (&zstrm->lock){+.+.}-{2:2}:


> [    9.032169] 1 lock held by swapon/913:
> [    9.032170]  #0: ffff993c54cdceb0 (&zspage->lock){.+.+}-{2:2}, at: zs_map_object+0x7a/0x2e0

> [    9.032176] Call Trace:
> [    9.032179]  dump_stack+0x8b/0xb0
> [    9.032181]  check_noncircular+0xd0/0xf0
> [    9.032183]  __lock_acquire+0x11e3/0x21f0
> [    9.032185]  lock_acquire+0xc8/0x400
> [    9.032187]  ? zcomp_stream_get+0x5/0x90 [zram]
> [    9.032189]  zcomp_stream_get+0x38/0x90 [zram]
> [    9.032190]  ? zcomp_stream_get+0x5/0x90 [zram]
> [    9.032192]  zram_bvec_rw.constprop.0.isra.0+0x4c1/0x730 [zram]
> [    9.032194]  ? __part_start_io_acct+0x4d/0xf0
> [    9.032196]  zram_rw_page+0xa9/0x130 [zram]
> [    9.032197]  bdev_read_page+0x71/0xa0
> [    9.032199]  do_mpage_readpage+0x5a8/0x800
> [    9.032201]  ? xa_load+0xbf/0x140
> [    9.032203]  mpage_readahead+0xfb/0x230
> [    9.032205]  ? bdev_evict_inode+0x1a0/0x1a0
> [    9.032207]  read_pages+0x60/0x1e0
> [    9.032208]  page_cache_readahead_unbounded+0x1da/0x270
> [    9.032211]  generic_file_buffered_read+0x69c/0xe00
> [    9.032213]  new_sync_read+0x108/0x180
> [    9.032215]  vfs_read+0x12e/0x1c0
> [    9.032217]  ksys_read+0x58/0xd0
> [    9.032218]  do_syscall_64+0x33/0x40
> [    9.032219]  entry_SYSCALL_64_after_hwframe+0x44/0xa9


Joy... __zram_bvec_write() and __zram_bvec_read() take these locks in
opposite order.

Does something like the (_completely_) untested below cure things?

---

diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 9100ac36670a..c1e2c2e1cde8 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -1216,10 +1216,11 @@ static void zram_free_page(struct zram *zram, size_t index)
 static int __zram_bvec_read(struct zram *zram, struct page *page, u32 index,
 				struct bio *bio, bool partial_io)
 {
-	int ret;
+	struct zcomp_strm *zstrm;
 	unsigned long handle;
 	unsigned int size;
 	void *src, *dst;
+	int ret;
 
 	zram_slot_lock(zram, index);
 	if (zram_test_flag(zram, index, ZRAM_WB)) {
@@ -1250,6 +1251,9 @@ static int __zram_bvec_read(struct zram *zram, struct page *page, u32 index,
 
 	size = zram_get_obj_size(zram, index);
 
+	if (size != PAGE_SIZE)
+		zstrm = zcomp_stream_get(zram->comp);
+
 	src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO);
 	if (size == PAGE_SIZE) {
 		dst = kmap_atomic(page);
@@ -1257,8 +1261,6 @@ static int __zram_bvec_read(struct zram *zram, struct page *page, u32 index,
 		kunmap_atomic(dst);
 		ret = 0;
 	} else {
-		struct zcomp_strm *zstrm = zcomp_stream_get(zram->comp);
-
 		dst = kmap_atomic(page);
 		ret = zcomp_decompress(zstrm, src, size, dst);
 		kunmap_atomic(dst);