linux-kernel - Re: [RESEND PATCH] riscv: Fixup boot failure when CONFIG_DEBUG_RT

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <mhng-a0dbdedc-aed2-4dc3-b556-3a81530e7ec4@palmer-ri-x1c9a>
Date: Wed, 11 Dec 2024 13:59:57 -0800 (PST)
From: Palmer Dabbelt <palmer@...belt.com>
To: alex@...ti.fr
CC: guoren@...nel.org, Paul Walmsley <paul.walmsley@...ive.com>,
  Bjorn Topel <bjorn@...osinc.com>, Conor Dooley <conor@...nel.org>, leobras@...hat.com, peterz@...radead.org,
  parri.andrea@...il.com, Will Deacon <will@...nel.org>, longman@...hat.com, boqun.feng@...il.com,
  Arnd Bergmann <arnd@...db.de>, alexghiti@...osinc.com, linux-riscv@...ts.infradead.org,
  linux-kernel@...r.kernel.org, guoren@...ux.alibaba.com
Subject:     Re: [RESEND PATCH] riscv: Fixup boot failure when CONFIG_DEBUG_RT_MUTEXES=y

On Wed, 04 Dec 2024 00:27:00 PST (-0800), alex@...ti.fr wrote:
> Hi Guo,
>
> On 30/11/2024 16:33, guoren@...nel.org wrote:
>> From: Guo Ren <guoren@...ux.alibaba.com>
>>
>> When CONFIG_DEBUG_RT_MUTEXES=y, mutex_lock->rt_mutex_try_acquire
>> would change from rt_mutex_cmpxchg_acquire to
>> rt_mutex_slowtrylock():
>> 	raw_spin_lock_irqsave(&lock->wait_lock, flags);
>> 	ret = __rt_mutex_slowtrylock(lock);
>> 	raw_spin_unlock_irqrestore(&lock->wait_lock, flags);
>>
>> Because queued_spin_#ops to ticket_#ops is changed one by one by
>> jump_label, raw_spin_lock/unlock would cause a deadlock during the
>> changing.
>>
>> That means in arch/riscv/kernel/jump_label.c:
>> 1.
>> arch_jump_label_transform_queue() ->
>> mutex_lock(&text_mutex); +-> raw_spin_lock  -> queued_spin_lock
>> 			 |-> raw_spin_unlock -> queued_spin_unlock
>> patch_insn_write -> change the raw_spin_lock to ticket_lock
>> mutex_unlock(&text_mutex);
>> ...
>>
>> 2. /* Dirty the lock value */
>> arch_jump_label_transform_queue() ->
>> mutex_lock(&text_mutex); +-> raw_spin_lock -> *ticket_lock*
>>                           |-> raw_spin_unlock -> *queued_spin_unlock*
>> 			  /* BUG: ticket_lock with queued_spin_unlock */
>> patch_insn_write  ->  change the raw_spin_unlock to ticket_unlock
>> mutex_unlock(&text_mutex);
>> ...
>>
>> 3. /* Dead lock */
>> arch_jump_label_transform_queue() ->
>> mutex_lock(&text_mutex); +-> raw_spin_lock -> ticket_lock /* deadlock! */
>>                           |-> raw_spin_unlock -> ticket_unlock
>> patch_insn_write -> change other raw_spin_#op -> ticket_#op
>> mutex_unlock(&text_mutex);
>>
>> So, the solution is to disable mutex usage of
>> arch_jump_label_transform_queue() during early_boot_irqs_disabled, just
>> like we have done for stop_machine.
>>
>> Reported-by: Conor Dooley <conor@...nel.org>
>> Signed-off-by: Guo Ren <guoren@...ux.alibaba.com>
>> Signed-off-by: Guo Ren <guoren@...nel.org>
>> Fixes: ab83647fadae ("riscv: Add qspinlock support")
>> Link: https://lore.kernel.org/linux-riscv/CAJF2gTQwYTGinBmCSgVUoPv0_q4EPt_+WiyfUA1HViAKgUzxAg@mail.gmail.com/T/#mf488e6347817fca03bb93a7d34df33d8615b3775
>> Cc: Palmer Dabbelt <palmer@...belt.com>
>> Cc: Alexandre Ghiti <alexghiti@...osinc.com>
>> ---
>>   arch/riscv/kernel/jump_label.c | 12 +++++++++---
>>   1 file changed, 9 insertions(+), 3 deletions(-)
>>
>> diff --git a/arch/riscv/kernel/jump_label.c b/arch/riscv/kernel/jump_label.c
>> index 6eee6f736f68..654ed159c830 100644
>> --- a/arch/riscv/kernel/jump_label.c
>> +++ b/arch/riscv/kernel/jump_label.c
>> @@ -36,9 +36,15 @@ bool arch_jump_label_transform_queue(struct jump_entry *entry,
>>   		insn = RISCV_INSN_NOP;
>>   	}
>>
>> -	mutex_lock(&text_mutex);
>> -	patch_insn_write(addr, &insn, sizeof(insn));
>> -	mutex_unlock(&text_mutex);
>> +	if (early_boot_irqs_disabled) {
>> +		riscv_patch_in_stop_machine = 1;
>> +		patch_insn_write(addr, &insn, sizeof(insn));
>> +		riscv_patch_in_stop_machine = 0;
>> +	} else {
>> +		mutex_lock(&text_mutex);
>> +		patch_insn_write(addr, &insn, sizeof(insn));
>> +		mutex_unlock(&text_mutex);
>> +	}
>>
>>   	return true;
>>   }
>
>
> Sorry for the late answer, I've been sick lately!

Ya, I was also sick -- bad timing.

> Thank you very much for looking into this and finding this not-so-bad
> solution! I remind everyone that this is a temporary solution until we
> can use an alternative instead of a static key.
>
> You can add:
>
> Reviewed-by: Alexandre Ghiti <alexghiti@...osinc.com>
> Tested-by: Alexandre Ghiti <alexghiti@...osinc.com>
>
> The revert is still on the table IMO, let's Palmer decide.

IMO it's fine to just take this as a fix.  Reverting stuff is always a 
nice big hammer to have if we don't get a fix in a reasonable timeframe, 
but with a fix avaliable in the time I was out sick anyway I think it's 
fine to just take this.

So I've got it queued up on fixes.

> Thank you again Guo, really appreciate you took the time to find this
> solution!
>
> Alex