Message-ID: <sruvxazchu4tnawh7dlz73x5x3iwaqmi5ii33onguu5ozfv4lz@ebecdymmljwa>
Date: Tue, 29 Apr 2025 07:56:13 +0800
From: Coiby Xu <coxu@...hat.com>
To: Arnaud Lefebvre <arnaud.lefebvre@...ver-cloud.com>
Cc: kexec@...ts.infradead.org, Ondrej Kozina <okozina@...hat.com>, 
	Milan Broz <gmazyland@...il.com>, Thomas Staudt <tstaudt@...ibm.com>, 
	Daniel P . Berrangé <berrange@...hat.com>, Kairui Song <ryncsn@...il.com>, 
	Pingfan Liu <kernelfans@...il.com>, Baoquan He <bhe@...hat.com>, Dave Young <dyoung@...hat.com>, 
	linux-kernel@...r.kernel.org, x86@...nel.org, Dave Hansen <dave.hansen@...el.com>, 
	Vitaly Kuznetsov <vkuznets@...hat.com>
Subject: Re: [PATCH v8 0/7] Support kdump with LUKS encryption by reusing
 LUKS volume keys

On Mon, Apr 28, 2025 at 08:40:44PM +0200, Arnaud Lefebvre wrote:
>On Mon, Apr 28, 2025 at 05:02:23PM +0800, Coiby Xu wrote:
>>On Thu, Apr 24, 2025 at 02:08:55AM +0200, Arnaud Lefebvre wrote:
>>>On Fri, Feb 07, 2025 at 04:08:08PM +0800, Coiby Xu wrote:
>>>>LUKS is the standard for Linux disk encryption, widely adopted by users,
>>>>and in some cases, such as Confidential VMs, it is a requirement. With
>>>>kdump enabled, when the first kernel crashes, the system can boot into
>>>>the kdump/crash kernel to dump the memory image (i.e., /proc/vmcore)
>>>>to a specified target. However, there are two challenges when dumping
>>>>vmcore to a LUKS-encrypted device:
>>>>
>>>>- Kdump kernel may not be able to decrypt the LUKS partition. For some
>>>>machines, a system administrator may not have a chance to enter the
>>>>password to decrypt the device in kdump initramfs after the 1st kernel
>>>>crashes; For cloud confidential VMs, depending on the policy the
>>>>kdump kernel may not be able to unseal the keys with TPM and the
>>>>console virtual keyboard is untrusted.
>>>>
>>>>- LUKS2 by default use the memory-hard Argon2 key derivation function
>>>>which is quite memory-consuming compared to the limited memory reserved
>>>>for kdump. Take Fedora example, by default, only 256M is reserved for
>>>>systems having memory between 4G-64G. With LUKS enabled, ~1300M needs
>>>>to be reserved for kdump. Note if the memory reserved for kdump can't
>>>>be used by 1st kernel i.e. an user sees ~1300M memory missing in the
>>>>1st kernel.
>>>>
>>>>Besides users (at least for Fedora) usually expect kdump to work out of
>>>>the box i.e. no manual password input or custom crashkernel value is
>>>>needed. And it doesn't make sense to derivate the keys again in kdump
>>>>kernel which seems to be redundant work.
>>>>
>>>>This patch set addresses the above issues by making the LUKS volume keys
>>>>persistent for kdump kernel with the help of cryptsetup's new APIs
>>>>(--link-vk-to-keyring/--volume-key-keyring). Here is the life cycle of
>>>>the kdump copies of LUKS volume keys,
>>>>
>>>>1. After the 1st kernel loads the initramfs during boot, systemd
>>>> use an user-input passphrase to de-crypt the LUKS volume keys
>>>> or TPM-sealed key and then save the volume keys to specified keyring
>>>> (using the --link-vk-to-keyring API) and the key will expire within
>>>> specified time.
>>>>
>>>>2. A user space tool (kdump initramfs loader like kdump-utils) create
>>>> key items inside /sys/kernel/config/crash_dm_crypt_keys to inform
>>>> the 1st kernel which keys are needed.
>>>>
>>>>3. When the kdump initramfs is loaded by the kexec_file_load
>>>> syscall, the 1st kernel will iterate created key items, save the
>>>> keys to kdump reserved memory.
>>>>
>>>>4. When the 1st kernel crashes and the kdump initramfs is booted, the
>>>> kdump initramfs asks the kdump kernel to create a user key using the
>>>> key stored in kdump reserved memory by writing yes to
>>>> /sys/kernel/crash_dm_crypt_keys/restore. Then the LUKS encrypted
>>>> device is unlocked with libcryptsetup's --volume-key-keyring API.
>>>>
>>>>5. The system gets rebooted to the 1st kernel after dumping vmcore to
>>>> the LUKS encrypted device is finished
>>>>
>>>>After libcryptsetup saving the LUKS volume keys to specified keyring,
>>>>whoever takes this should be responsible for the safety of these copies
>>>>of keys. The keys will be saved in the memory area exclusively reserved
>>>>for kdump where even the 1st kernel has no direct access. And further
>>>>more, two additional protections are added,
>>>>- save the copy randomly in kdump reserved memory as suggested by Jan
>>>>- clear the _PAGE_PRESENT flag of the page that stores the copy as
>>>>suggested by Pingfan
>>>>
>>>>This patch set only supports x86. There will be patches to support other
>>>>architectures once this patch set gets merged.
>>>>
>>>
>>>I'm not sure what's the problem here but I can reliably trigger a kernel
>>>panic on a qemu VM + custom kernel (compiled from
>>>bb066fe812d6fb3a9d01c073d9f1e2fd5a63403b + your patches).
>>
>>Hi Arnaud,
>>
>>Thanks for testing the patches, finding this issue and also sharing the
>>details to reproduce it!
>>
>
>Hello,
>
>You're welcome, thanks to you for this patch series!
>
>>>
>>>When I configure the crash configfs and call kexec in a systemd service
>>>using ExecStart=, the panic occurs when I start the service:
>>>
>>>~ # cat /etc/systemd/system/my-kexec.service
>>>[Unit]
>>>Description=kexec loading for the crash capture kernel
>>>
>>>[Service]
>>>Type=oneshot
>>>ExecStart=/usr/bin/mkdir /sys/kernel/config/crash_dm_crypt_keys/mykey
>>>ExecStart=/usr/bin/echo cryptsetup:mykey >/sys/kernel/config/crash_dm_crypt_keys/mykey/description
>>>ExecStart=/usr/host/bin/kexec --debug --load-panic /linux-hv --initrd /crash-initrd
>>>KeyringMode=shared
>>
>>Can you try putting the above commands into a script e.g.
>>/usr/local/bin/my-kexec.sh and then using
>>ExecStart=/usr/local/bin/my-kexec.sh
>>so I can be more sure that I've reproduced your issue?
>>
>
>I believe that's what I wrote at the end of my previous message
>(see below the panic trace). It works fine using a script like that.
>
>Did you miss it or is there a difference with what you're asking?

Oh, I missed it, thanks for the reminder! Now I'm sure I have reproduced
the issue and understood how the problem happens.  This is because
systemd will ignore the bash redirection ">" as it doesn't invoke bash
to run ExecStart. So you will see systemd logs as follows, 
   warning: ignoring excess arguments, starting with with ‘>/sys/kernel/config/...

So a key configfs item is created but the key description fails to be
set. Unfortunately, the kernel doesn't check if the key description is
null and crashes when trying to copy it. I'll send a new version of
patches to resolve this issue, thanks!

>
> [...]
>>>Kernel panic - not syncing: Fatal exception
>>>Kernel Offset: disabled
>>>
>>>
>>>Calling a script that does the same thing works fine and loads the keys
>>>correctly:
>>>
>>>[Service]
>>>ExecStart=/root/kexec.sh
>>>
>>>~ # cat /root/kexec.sh
>>>#!/bin/bash
>>>
>>>mkdir /sys/kernel/config/crash_dm_crypt_keys/mykey
>>>echo cryptsetup:mykey > /sys/kernel/config/crash_dm_crypt_keys/mykey/description
>>>/usr/host/bin/kexec --debug --load-panic /linux-hv --initrd /crash-initrd
>>>
>>>If that's any help, my crypttab:
>>>
>>>~ # cat /etc/crypttab
>>>root UUID=8001fca4-2e54-48e9-9235-031c19fc6e36 none luks,link-volume-key=@u::%logon:cryptsetup:mykey
>>>
>>>If you can't reproduce, I can help track this. Just let me know if you need
>>>any help.
>>>
>>
>>-- 
>>Best regards,
>>Coiby
>>
>

-- 
Best regards,
Coiby

Powered by blists - more mailing lists