Message-ID: <d528111b-4caa-e292-59f4-4ce1eab1f27c@suse.com>
Date:   Tue, 6 Dec 2022 17:57:30 +0100
From:   Petr Pavlu <petr.pavlu@...e.com>
To:     Petr Mladek <pmladek@...e.com>
Cc:     mcgrof@...nel.org, prarit@...hat.com, david@...hat.com,
        mwilck@...e.com, linux-modules@...r.kernel.org,
        linux-kernel@...r.kernel.org, stable@...r.kernel.org
Subject: Re: [PATCH v2] module: Don't wait for GOING modules

On 12/5/22 20:54, Petr Mladek wrote:
> On Mon 2022-12-05 11:35:57, Petr Pavlu wrote:
>> During a system boot, it can happen that the kernel receives a burst of
>> requests to insert the same module but loading it eventually fails
>> during its init call. For instance, udev can make a request to insert
>> a frequency module for each individual CPU when another frequency module
>> is already loaded which causes the init function of the new module to
>> return an error.
>>
>> Since commit 6e6de3dee51a ("kernel/module.c: Only return -EEXIST for
>> modules that have finished loading"), the kernel waits for modules in
>> MODULE_STATE_GOING state to finish unloading before making another
>> attempt to load the same module.
>>
>> This creates unnecessary work in the described scenario and delays the
>> boot. In the worst case, it can prevent udev from loading drivers for
>> other devices and might cause timeouts of services waiting on them and
>> subsequently a failed boot.
>>
>> This patch attempts a different solution for the problem 6e6de3dee51a
>> was trying to solve. Rather than waiting for the unloading to complete,
>> it returns a different error code (-EBUSY) for modules in the GOING
>> state. This should avoid the error situation that was described in
>> 6e6de3dee51a (user space attempting to load a dependent module because
>> the -EEXIST error code would suggest to user space that the first module
>> had been loaded successfully), while avoiding the delay situation too.
>>
>> Fixes: 6e6de3dee51a ("kernel/module.c: Only return -EEXIST for modules that have finished loading")
>> Co-developed-by: Martin Wilck <mwilck@...e.com>
>> Signed-off-by: Martin Wilck <mwilck@...e.com>
>> Signed-off-by: Petr Pavlu <petr.pavlu@...e.com>
>> Cc: stable@...r.kernel.org
>> ---
>>
>> Changes since v1 [1]:
>> - Don't attempt a new module initialization when a same-name module
>>   completely disappeared while waiting on it, which means it went
>>   through the GOING state implicitly already.
>>
>> [1] https://lore.kernel.org/linux-modules/20221123131226.24359-1-petr.pavlu@suse.com/
>>
>>  kernel/module/main.c | 26 +++++++++++++++++++++-----
>>  1 file changed, 21 insertions(+), 5 deletions(-)
>>
>> diff --git a/kernel/module/main.c b/kernel/module/main.c
>> index d02d39c7174e..7a627345d4fd 100644
>> --- a/kernel/module/main.c
>> +++ b/kernel/module/main.c
>> @@ -2386,7 +2386,8 @@ static bool finished_loading(const char *name)
>>  	sched_annotate_sleep();
>>  	mutex_lock(&module_mutex);
>>  	mod = find_module_all(name, strlen(name), true);
>> -	ret = !mod || mod->state == MODULE_STATE_LIVE;
>> +	ret = !mod || mod->state == MODULE_STATE_LIVE
>> +		|| mod->state == MODULE_STATE_GOING;
> 
> There is a actually one more race.
> 
> This function is supposed to wait until load of a particular module
> finishes. But we might find some another module of the same name here.
> 
> Maybe, it is not that bad. If many modules of the same name are loaded
> in parallel then hopefully most of them would wait for the first one
> in add_unformed_module(). And they will never appear in the @modules
> list.

Good point, a load waiting in add_unformed_module() could miss that its older
parallel load already finished if another insert of the same module appears in
the modules list in the meantime. This requires that the new load happens to
arrive just after the old one finishes and before the waiting load makes its
check.

This is somewhat similar to the current state where new same-name insert
requests can skip and starve the ones already waiting in
add_unformed_module().

I think in practice the situation should occur very rarely and be cleared
soon, as long one doesn't continuously try to insert the same module.

> Anyway, to be on the safe side. We might want to pass the pointer
> to the @old module found in add_unformed_module() and make sure
> that we find the same module here. Something like:
> 
> /*
>  * @pending_mod: pointer to module that we are waiting for
>  * @name: name of the module; the string must stay even when
>  *	the pending module goes away completely
>  */
> static bool finished_loading(const struct module *pending_mod,
> 			    const char *name)
> {
> 	struct module *mod;
> 	bool ret = true;
> 
> 	/*
> 	 * The module_mutex should not be a heavily contended lock;
> 	 * if we get the occasional sleep here, we'll go an extra iteration
> 	 * in the wait_event_interruptible(), which is harmless.
> 	 */
> 	sched_annotate_sleep();
> 	mutex_lock(&module_mutex);
> 
> 	mod = find_module_all(name, strlen(name), true);
> 	/* Check if the pending module is still being loaded */
> 	if (mod == pending_mod &&
> 	    (mod->state == MODULE_STATE_UNFORMED ||
> 	       mod->state == MODULE_STATE_COMMING))
> 	       ret = false;
> 	mutex_unlock(&module_mutex);
> 
> 	return ret;
> }

The new pending_mod pointer has no ownership of the target module which can be
then destroyed at any time. While no dereference of pending_mod is made in
finished_loading(), this looks still problematic to me.

I think it is generally good to treat a pointer as invalid and its value as
indeterminate when the object it points to reaches the end of its lifetime.
One specific problem here is that nothing guarantees that a new module doesn't
get allocated at the exactly same address as the old one that got released and
the code is actually waiting on.

Improving this case then likely results in a more complex solution, similar
to the one that was discussed originally.

Thanks,
Petr

Powered by blists - more mailing lists