linux-kernel - Re: Hard system lock-ups when using encrypted swap and RAM is exhausted

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <32c2c4c6-d8c0-297a-b0eb-f06a939d5e44@redhat.com>
Date: Fri, 28 Nov 2025 14:06:54 +0100 (CET)
From: Mikulas Patocka <mpatocka@...hat.com>
To: Kurt Fitzner <kurt_cryptsetup@...der.ca>
cc: Aaron Rainbolt <arraybolt3@...il.com>, Milan Broz <gmazyland@...il.com>, 
    linux-mm@...ck.org, cryptsetup@...ts.linux.dev, dm-devel@...ts.linux.dev, 
    linux-kernel@...r.kernel.org, adrelanos@...nix.org
Subject: Re: Hard system lock-ups when using encrypted swap and RAM is
 exhausted



On Thu, 27 Nov 2025, Kurt Fitzner wrote:

> On 2025-11-27 13:54, Mikulas Patocka wrote:
> 
> > Encrypted swap file is not supposed to work.
> 
> Do you have a reference for this?  The concept of encrypted swap files has
> been a valid workflow for a very long time.
> 
> > So, this is what happened to you - the machine runs out of memory, it
> > needs to swap out some pages, dm-crypt encrypts the pages and generates
> > write bios, the write bios are directed to the loop device, the loop
> > device directs them to the filesystem, the filesystem attempts to allocate
> > more memory => deadlock.
> 
> If it's the filesystem trying to allocate memory on writes to a swap file that
> is causing a memory allocation/swap race, then any write to a swap file would
> engender the same result, regardless of encryption. The encryption layer is
> redundant under the failure mode you propose.

No - when you swap to unencrypted file, the kernel bypasses the 
filesystem. It creates a block map of the file at swap activation time and 
when it swaps to the file it uses this block map and it doesn't call any 
filesystem functions.

This bypass doesn't work when you put other block device drivers over the 
file.

> I can confirm I have put kernels up to and including 6.14 under heavy memory
> stress and have never encountered anything that feels like a memory allocation
> race.  All my systems have encrypted swap files.
> 
> I can't speak toward later kernels, or any bugs that may or may not be
> presesnt, but I know of nothing to suggest that encrypted swap files remain
> anything other than an intended feature.
> 
>     Kurt

So, it works by chance, not by design.

Mikulas