linux-kernel - Re: [PATCH v2] mm: zswap: fix crypto_free_acomp() deadlock in zswap_cpu_comp

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAKEwX=MtmHKnN2Frrny7dZ6=B6d_nzAKeUCwKcMs2zhoDwb3jg@mail.gmail.com>
Date: Tue, 4 Mar 2025 17:14:37 -0800
From: Nhat Pham <nphamcs@...il.com>
To: Yosry Ahmed <yosry.ahmed@...ux.dev>
Cc: Andrew Morton <akpm@...ux-foundation.org>, Johannes Weiner <hannes@...xchg.org>, 
	Chengming Zhou <chengming.zhou@...ux.dev>, "David S. Miller" <davem@...emloft.net>, 
	Herbert Xu <herbert@...dor.apana.org.au>, linux-mm@...ck.org, 
	linux-crypto@...r.kernel.org, linux-kernel@...r.kernel.org, 
	syzkaller-bugs@...glegroups.com, 
	syzbot+1a517ccfcbc6a7ab0f82@...kaller.appspotmail.com, stable@...r.kernel.org
Subject: Re: [PATCH v2] mm: zswap: fix crypto_free_acomp() deadlock in zswap_cpu_comp_dead()

On Wed, Feb 26, 2025 at 10:56 AM Yosry Ahmed <yosry.ahmed@...ux.dev> wrote:
>
> Currently, zswap_cpu_comp_dead() calls crypto_free_acomp() while holding
> the per-CPU acomp_ctx mutex. crypto_free_acomp() then holds scomp_lock
> (through crypto_exit_scomp_ops_async()).
>
> On the other hand, crypto_alloc_acomp_node() holds the scomp_lock
> (through crypto_scomp_init_tfm()), and then allocates memory.
> If the allocation results in reclaim, we may attempt to hold the per-CPU
> acomp_ctx mutex.
>
> The above dependencies can cause an ABBA deadlock. For example in the
> following scenario:
>
> (1) Task A running on CPU #1:
>     crypto_alloc_acomp_node()
>       Holds scomp_lock
>       Enters reclaim
>       Reads per_cpu_ptr(pool->acomp_ctx, 1)
>
> (2) Task A is descheduled
>
> (3) CPU #1 goes offline
>     zswap_cpu_comp_dead(CPU #1)
>       Holds per_cpu_ptr(pool->acomp_ctx, 1))
>       Calls crypto_free_acomp()
>       Waits for scomp_lock
>
> (4) Task A running on CPU #2:
>       Waits for per_cpu_ptr(pool->acomp_ctx, 1) // Read on CPU #1
>       DEADLOCK

Lolll I was scratching my head with this issue while stress-testing
some of my zswap patches. Beat me to it :)

>
> Since there is no requirement to call crypto_free_acomp() with the
> per-CPU acomp_ctx mutex held in zswap_cpu_comp_dead(), move it after the
> mutex is unlocked. Also move the acomp_request_free() and kfree() calls
> for consistency and to avoid any potential sublte locking dependencies
> in the future.
>
> With this, only setting acomp_ctx fields to NULL occurs with the mutex
> held. This is similar to how zswap_cpu_comp_prepare() only initializes
> acomp_ctx fields with the mutex held, after performing all allocations
> before holding the mutex.
>
> Opportunistically, move the NULL check on acomp_ctx so that it takes
> place before the mutex dereference.
>
> Fixes: 12dcb0ef5406 ("mm: zswap: properly synchronize freeing resources during CPU hotunplug")
> Reported-by: syzbot+1a517ccfcbc6a7ab0f82@...kaller.appspotmail.com
> Closes: https://lore.kernel.org/all/67bcea51.050a0220.bbfd1.0096.GAE@google.com/
> Cc: <stable@...r.kernel.org>
> Co-developed-by: Herbert Xu <herbert@...dor.apana.org.au>
> Signed-off-by: Herbert Xu <herbert@...dor.apana.org.au>
> Signed-off-by: Yosry Ahmed <yosry.ahmed@...ux.dev>
> Acked-by: Herbert Xu <herbert@...dor.apana.org.au>

LGTM! Thanks for fixing it.
Reviewed-by: Nhat Pham <nphamcs@...il.com>