Date: Wed, 26 Jun 2024 09:24:46 +0200
From: Krzysztof Kozlowski <krzysztof.kozlowski@...aro.org>
To: Rob Herring <robh@...nel.org>, Luca Ceresoli <luca.ceresoli@...tlin.com>
Cc: Conor Dooley <conor@...nel.org>,
 "devicetree@...r.kernel.org" <devicetree@...r.kernel.org>,
 Bartosz Golaszewski <brgl@...ev.pl>, Saravana Kannan <saravanak@...gle.com>,
 linux-kernel@...r.kernel.org, Hervé Codina
 <herve.codina@...tlin.com>, Thomas Petazzoni <thomas.petazzoni@...tlin.com>
Subject: Re: Fixing property memory leaks on device tree overlay removal

On 25/06/2024 19:02, Rob Herring wrote:
> On Mon, Jun 24, 2024 at 3:21 PM Luca Ceresoli <luca.ceresoli@...tlin.com> wrote:
>>
>> Hello,
>>
> 
> This is a great summary of the issues with further overlay support.
> 
>> device tree overlays are implemented in the kernel since a long time,
>> but there is currently no code using them, except unittest. The
>> largest (in terms of code lines involved) blocker for enabling it is the
>> memory leak of properties that happen on overlay removal [0]. The
>> kernel warns about it on overlay insertion:
>>
>>   OF: overlay: WARNING: memory leak will occur if overlay removed, property: /...
> 
> Maybe this needs to just be finer-grained and smarter. For example, we
> know when we've leaked pointers (because you found the cases below).
> We could set a flag in the node when that happens and then warn when
> removing the overlay.
> 
>> We are working on a driver for a proprietary connector allowing a device
>> to connect to an add-on board adding new non-discoverable hardware, and
>> the driver is based on overlays. This has been discussed during ELC last
>> April [1] and I sent a patch series implementing it in May [2].
>>
>> The property memory leak topic was mentioned but we haven't addressed
>> the detail so far. This e-mail is meant to present our proposed plan
>> to fix it.
>>
>> Receiving comments from the kernel community would be very valuable
>> given the amount of work involved.
>>
>> ===================
>> Problem description
>> ===================
>>
>> In the kernel every 'struct device_node' is refcounted so the OF core
>> knows when to free it. There are of course get/put imbalance bugs
>> around, but these are "just" bugs that need to be fixed as they are
>> found.
>>
>> On the other hand, there is no refcounting for 'struct property'. Yet
>> some of the internal kernel APIs to access properties, e.g.
>> of_property_read_string(), return either a 'struct property' pointer or
>> a copy of the 'char *value' field. This is not a bug, it is an API
>> design flaw: any user (e.g. any OF driver) can take a pointer to
>> property data that was allocated and should be deallocated by the OF
>> core, but the OF core has no idea of when that pointer will stop being
>> used.
>>
>> Now, when loading a DT overlay there are three possible cases:
>>
>>  1. both the property and the containing node are in the base tree
>>  2. both the property and the containing node are in the same overlay
>>  3. the property is in an overlay and the containing node is either
>>     in the base tree or in a previously-loaded overlay
>>
>> Cases 1 and 2 are not problematic. In case 1 the data allocated for the
>> properties is never removed. In case 2 the properties are removed when
>> removing the parent node, which gets removed when removing the overlay
>> thanks to 'struct device_node' refcounting, based on the assumption
>> that the property lifetime is a subset of the parent node lifetime. The
>> problem exists in case 3. Properties in case 3 are usually a small part
>> of all the properties but there can be some (and there are some in the
>> product we are working on), and that's what needs to be addressed.
> 
> I'd like to better understand what are the cases where you need to
> change/add properties in a node (other than "status"). I'm not
> entirely convinced we should even allow that.

Just to clarify that I understand the problem correctly - we talk only
about memory leaks, not about accessing released memory (use-after-free)?
I think that during EOSS 2024 discussions we reached consensus that in
general you will not have use-after-free problem with DT properties at
all. If all devices are unbound, their resources get released (including
some core structures registered in subsystems) thus nothing will use any
of properties. With proper kernel code there will be no use of device
node properties after device is unbound.

> 
>> The kernel just doesn't know when to release these property structures,
>> so the OF core code currently stores the properties loaded by overlays
>> in a "deadprops" list within the parent node. They will be freed when
>> the node itself is freed (based on node refcounting) in order to be
>> safe. However it means they are memory leaks for case 3, where the node
>> can be never removed.
>>
>> ====================================
>> Approach 1: adding property refcount
>> ====================================
>>
>> The first option we considered is the obvious one: adding a refcount to
>> 'struct property'. However this does not look like a very good solution
>> as every user of a DT property in the kernel would need to put the
>> property after getting and using it: this is already cumbersome and
>> error-prone for nodes, and properties are usually many more than nodes.
>> Besides it would add a memory and execution overhead to all properties,
>> while only a small fraction is affected.
> 
> I'm glad you reached this conclusion.
> 
>>
>> ===========================================================
>> Approach 2: change the property APIs to not return pointers
>> ===========================================================
>>
>> So we went back to the API design flaw and thought what API we would
>> invent if we had to start from scratch now with the laodable overlay
>> requirement in mind. We concluded the property API should just not
>> return any pointer to 'struct property' or one of its fields. In other
>> words, this is internal data of the OF core.
> 
> I think we also have to consider what Rust DT/property APIs will look
> like. It's the same problem as Rust has clear rules on ownership and
> lifetimes. So far, I've only seen of_find_property() show up, but
> that's not getting accepted on my watch.
> 
>> We think the API to access DT properties should instead do one of these:
>>
>>  1. check whatever is needed and return the outcome, not the internal
>>     data, e.g. like of_property_match_string()
>>  2. if that is not possible, return a copy of the property value, that
>>     the called will be responsible of, and which can outlive the struct
>>     property if that make sense
>>
>> With this approach, it becomes safe to remove the overlay at any moment
>> _unless_ one of these API functions is currently executing on any CPU
>> core. To ensure there is no race condition we need a lock, probably
>> using rwsem or RCU, to guard the APIs accessing properties. This can be
>> coarse-grained (one global lock, or one lock per changeset) and needs no
>> cooperation from callers, being all managed by the OF core.
>>
>> So this is is the option we plan to take.

... and this leads to discussion of use-after-free problem rather, not
memory leaks. Or I miss something here.


....

>> ~~~~~~~~~~~~~~~
>>
>> Transitioning to the new accessors is not going to happen quickly given
>> the amount of call sites, so we devised a strategy to incrementally
>> approach it while avoiding introducing new calls to the old APIs.
>>
>> The transition plan is based on adding a Kconfig symbol (a tentative
>> name is CONFIG_EXPORT_UNSAFE_OF_ACCESSORS) meaning “export old OF
>> accessor functions returning pointers to firmware data” and defaulting
>> to ‘y’ to avoid breaking current usages. When this is set to ‘n’:
>>
>>  - The old accessors returning pointers to struct property values are
>>    not built, or at least not exported outside the OF core
>>  - So any defconfig with at least one driver using the old accessors
>>    would just not build, which allows to easily find out what needs to
>>    be updated if one wants to remove overlays in their specific
>>    configuration, or to disable unneeded drivers that are not yet
>>    updated
>>  - The property leaks under discussion are not possible anymore
>>  - The deadprops list and the warning are not built
>>  - Note: we need to ensure automated test bots won’t set this setting
>>    to ‘n’ or they will fail
>>
>> Any code using DT overlays, including the connector driver proposed in
>> [2], can depend on CONFIG_EXPORT_UNSAFE_OF_ACCESSORS=n, to ensure the
>> leaking APIs are never called when runtime-loadable overlays are in use.
>>
>> Preventing new usages of old accessors will be important. Tools to
>> achieve that:
>>
>>  * Extend checkpatch to report an error on their usage
>>  * Add a 'K:' entry to MAINTAINERS so that patches trying to use them
>>    will be reported (to me at least)

Or just use lore/lei with proper keywords. I track few misuses of kernel
code that way.



Best regards,
Krzysztof

Powered by blists - more mailing lists