Message-ID: <20241002160722.20025-1-arbn@yandex-team.com>
Date: Wed,  2 Oct 2024 18:07:15 +0200
From: Andrey Ryabinin <arbn@...dex-team.com>
To: linux-kernel@...r.kernel.org
Cc: Alexander Graf <graf@...zon.com>,
	James Gowans <jgowans@...zon.com>,
	Mike Rapoport <rppt@...nel.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	linux-mm@...ck.org,
	Thomas Gleixner <tglx@...utronix.de>,
	Ingo Molnar <mingo@...hat.com>,
	Borislav Petkov <bp@...en8.de>,
	Dave Hansen <dave.hansen@...ux.intel.com>,
	x86@...nel.org,
	"H. Peter Anvin" <hpa@...or.com>,
	Eric Biederman <ebiederm@...ssion.com>,
	kexec@...ts.infradead.org,
	Steven Rostedt <rostedt@...dmis.org>,
	Masami Hiramatsu <mhiramat@...nel.org>,
	Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
	linux-trace-kernel@...r.kernel.org,
	valesini@...dex-team.com,
	Andrey Ryabinin <arbn@...dex-team.com>
Subject: [RFC PATCH 0/7] KSTATE: a mechanism to migrate some part of the kernel state across kexec

kstate (kernel state) is a mechanism to describe internal some part of the
kernel state, save it into the memory and restore the state after kexec
in the new kernel.

This is a very early RFC with a lot of hacks and cut corners with
the purpose to demonstrate the concept itself. Some parts
of this feature isn't well thought trough yet (like dealing with
struct changes between old and new kernel, fixed size of migrate stream
memory and absence of boundary checks, and so on and on).

The end goal here and the main use case for this is to be able to
update host kernel under VMs with VFIO pass-through devices running
on that host.
We are pretty far from that end goal yet. This patchset only tries
to establish some basic infrastructure to describe and migrate complex
in-kernel states.

The inspiration for this came from QEMU and its VMSTATE stuff which
is used to solve similar problem - migrate complex internal state
across different versions of QEMU. So there is a bit of similarity here.

The alternative for the kstate is KHO (Kexec Hand Over) [1].
Since in KHO migrates trace buffers, I decided to choose them
as a victim for kstate too. So we can compare both approaches.

In my very biased opinion with kstate it's much easier to describe
some state to migrate it to new kernel. And seems requires
almost none intervention into existing code paths of the subsystem.icated


So now to the part how this works.

States (usually this is some struct) are described by the
'struct kstate_description' containing the array of individual
fields descpriptions - 'struct kstate_field'.
Fields have different types like:
   KS_SIMPLE  - trivial type that just copied by value

   KS_POINTER - field contains pointer, it will be dereferenced to copy
     the value during save/restore phases.

   KS_STRUCT - contains another struct,
       field->ksd must point to another 'struct kstate_dscription'

   KS_CUSTOM - something that requires fit trivial types as above,
               for this fields the callbacks field->save()/->restore() must
	       do all job
   KS_ARRAY_OF_POINTER - array of pointers, the size of array determined by the
                         field->count() callback
   KS_END - special flag indicating the end of migration stream data.


kstate_register() call accepts kstate_description along with an instance
of an object and registers it in the global 'states' list.

During kexec reboot phase this list iterated, and for each instance
in the list 'struct kstate_entry' formed and saved in the migration stream.
'kstate_entry' contains information like ID of kstate_description, version
of it, size of migration data and the data itself.

After the reboot, when the kstate_register() called it parses migration
stream, finds the appropriate 'kstate_entry' and restores the contents of the
object.

The content of this patchset:

The first patch contains the most of the basic KSTATE infrastructure.

The 2,3 patches are temporary hacks needed to pass the memory used to store
migration data across kexec. Will be completely redone later.

The 4,5 patches are bits needed to preserve pages intact across kexec.

6 is test&playground patch to develop and test kstate itself.

7 is a demonstration of how to migrate trace buffer using kstate.


[1] https://lore.kernel.org/all/20240117144704.602-1-graf@amazon.com/

Andrey Ryabinin (7):
  kstate: Add kstate - a mechanism to migrate some kernel state across
    kexec
  kexec: Hack and abuse crashkernel for the kstate's migration stream
  [hack] purgatory: disable purgatory verification.
  mm/memblock: Add MEMBLOCK_PRSRV flag
  kstate: Add mechanism to preserved specified memory pages across
    kexec.
  kstate, test: add test module for testing kstate subsystem.
  trace: migrate trace buffers across kexec

 arch/x86/kernel/kexec-bzimage64.c  |  36 +++++
 arch/x86/kernel/machine_kexec_64.c |   5 +-
 arch/x86/kernel/setup.c            |  81 ++++++++++++
 arch/x86/purgatory/purgatory.c     |   2 +
 include/linux/kexec.h              |   6 +-
 include/linux/kstate.h             | 129 ++++++++++++++++++
 include/linux/memblock.h           |   7 +
 include/uapi/linux/kexec.h         |   2 +
 kernel/Kconfig.kexec               |  12 ++
 kernel/Makefile                    |   1 +
 kernel/crash_core.c                |   3 +-
 kernel/kexec_core.c                |  10 +-
 kernel/kexec_file.c                |  15 ++-
 kernel/kstate.c                    | 205 +++++++++++++++++++++++++++++
 kernel/trace/ring_buffer.c         | 189 ++++++++++++++++++++++++++
 kernel/trace/trace.c               |  81 ++++++++++++
 lib/Makefile                       |   2 +
 lib/test_kstate.c                  |  89 +++++++++++++
 mm/memblock.c                      |   9 +-
 mm/mm_init.c                       |  19 +++
 20 files changed, 895 insertions(+), 8 deletions(-)
 create mode 100644 include/linux/kstate.h
 create mode 100644 kernel/kstate.c
 create mode 100644 lib/test_kstate.c

-- 
2.45.2

Powered by blists - more mailing lists