Message-ID: <c96203ba-a5b2-4765-9d30-0f4e66c82cd7@amazon.com>
Date: Wed, 17 Jan 2024 15:01:51 +0100
From: Alexander Graf <graf@...zon.com>
To: Rob Herring <robh+dt@...nel.org>
CC: <linux-kernel@...r.kernel.org>, <linux-trace-kernel@...r.kernel.org>,
	<linux-mm@...ck.org>, <devicetree@...r.kernel.org>,
	<linux-arm-kernel@...ts.infradead.org>, <kexec@...ts.infradead.org>,
	<linux-doc@...r.kernel.org>, <x86@...nel.org>, Eric Biederman
	<ebiederm@...ssion.com>, "H. Peter Anvin" <hpa@...or.com>, Andy Lutomirski
	<luto@...nel.org>, Peter Zijlstra <peterz@...radead.org>, Steven Rostedt
	<rostedt@...dmis.org>, Andrew Morton <akpm@...ux-foundation.org>, "Mark
 Rutland" <mark.rutland@....com>, Tom Lendacky <thomas.lendacky@....com>,
	Ashish Kalra <ashish.kalra@....com>, James Gowans <jgowans@...zon.com>,
	Stanislav Kinsburskii <skinsburskii@...ux.microsoft.com>, <arnd@...db.de>,
	<pbonzini@...hat.com>, <madvenka@...ux.microsoft.com>, Anthony Yznaga
	<anthony.yznaga@...cle.com>, Usama Arif <usama.arif@...edance.com>, "David
 Woodhouse" <dwmw@...zon.co.uk>, Benjamin Herrenschmidt
	<benh@...nel.crashing.org>
Subject: Re: [PATCH v2 07/17] kexec: Add documentation for KHO


On 03.01.24 19:48, Rob Herring wrote:
>
> On Fri, Dec 22, 2023 at 12:52 PM Alexander Graf <graf@...zon.com> wrote:
>> With KHO in place, let's add documentation that describes what it is and
>> how to use it.
>>
>> Signed-off-by: Alexander Graf <graf@...zon.com>
>> ---
>>   Documentation/kho/concepts.rst   | 88 ++++++++++++++++++++++++++++++++
>>   Documentation/kho/index.rst      | 19 +++++++
>>   Documentation/kho/usage.rst      | 57 +++++++++++++++++++++
>>   Documentation/subsystem-apis.rst |  1 +
>>   4 files changed, 165 insertions(+)
>>   create mode 100644 Documentation/kho/concepts.rst
>>   create mode 100644 Documentation/kho/index.rst
>>   create mode 100644 Documentation/kho/usage.rst
>>
>> diff --git a/Documentation/kho/concepts.rst b/Documentation/kho/concepts.rst
>> new file mode 100644
>> index 000000000000..8e4fe8c57865
>> --- /dev/null
>> +++ b/Documentation/kho/concepts.rst
>> @@ -0,0 +1,88 @@
>> +.. SPDX-License-Identifier: GPL-2.0-or-later
>> +
>> +=======================
>> +Kexec Handover Concepts
>> +=======================
>> +
>> +Kexec HandOver (KHO) is a mechanism that allows Linux to preserve state -
>> +arbitrary properties as well as memory locations - across kexec.
>> +
>> +It introduces multiple concepts:
>> +
>> +KHO Device Tree
>> +---------------
>> +
>> +Every KHO kexec carries a KHO specific flattened device tree blob that
>> +describes the state of the system. Device drivers can register to KHO to
>> +serialize their state before kexec. After KHO, device drivers can read
>> +the device tree and extract previous state.
> How does this work with kexec when there is also the FDT for the h/w?
> The h/w FDT has a /chosen property pointing to this FDT blob?


Yep, exactly.


>
>> +
>> +KHO only uses the fdt container format and libfdt library, but does not
>> +adhere to the same property semantics that normal device trees do: Properties
>> +are passed in native endianness and standardized properties like ``regs`` and
>> +``ranges`` do not exist, hence there are no ``#...-cells`` properties.
> I think native endianness is asking for trouble. libfdt would need
> different swap functions here than elsewhere in the kernel for example
> which wouldn't even work. So you are just crossing your fingers that
> you aren't using any libfdt functions that swap. And when I sync
> dtc/libfdt and that changes, I might break you.
>
> Also, if you want to dump the FDT and do a dtc DTB->DTS pass, it is
> not going to be too readable given that outputs swapped 32-bit values
> for anything that's a 4 byte multiple.


Yeah, but big endian these days is just a complete waste of brain and 
cpu cycles :). And yes, I don't really want to use any libfdt helper 
functions to read data. I use it only to give me the raw data and take 
it from there.


>
>> +
>> +KHO introduces a new concept to its device tree: ``mem`` properties. A
>> +``mem`` property can inside any subnode in the device tree. When present,
>> +it contains an array of physical memory ranges that the new kernel must mark
>> +as reserved on boot. It is recommended, but not required, to make these ranges
>> +as physically contiguous as possible to reduce the number of array elements ::
>> +
>> +    struct kho_mem {
>> +            __u64 addr;
>> +            __u64 len;
>> +    };
>> +
>> +After boot, drivers can call the kho subsystem to transfer ownership of memory
>> +that was reserved via a ``mem`` property to themselves to continue using memory
>> +from the previous execution.
>> +
>> +The KHO device tree follows the in-Linux schema requirements. Any element in
>> +the device tree is documented via device tree schema yamls that explain what
>> +data gets transferred.
> If this is all separate, then I think the schemas should be too. And
> then from my (DT maintainer) perspective, you can do whatever you want
> here (like FIT images). The dtschema tools are pretty much only geared
> for "normal" DTs. A couple of problems come to mind. You can't exclude
> or change standard properties. The decoding of the DTB to run
> validation assumes big endian. We could probably split things up a
> bit, but you may be better off just using jsonschema directly. I'm not
> even sure running validation here would that valuable. You have 1
> source of code generating the DT and 1 consumer. Yes, there's
> different kernel versions to deal with, but it's not 100s of people
> creating 1000s of DTs with 100s of nodes.
>
> You might look at the netlink stuff which is using its own yaml syntax
> to generate code and jsonschema is used to validate the yaml.


I'm currently a lot more interested in the documentation aspect than in 
the validation, yeah. So I think for v3, I'll just throw the schemas 
into the Documentation/kho directory without any validation. We can 
worry about that later :)

Thanks a lot again for the review!


Alex





Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

Powered by blists - more mailing lists