linux-kernel - Re: [RFC] sched,livepatch: call klp_try_switch_task in __cond

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <47440502-930F-4CBD-B859-3AC9BBFF8FC6@fb.com>
Date:   Tue, 10 May 2022 19:45:49 +0000
From:   Song Liu <songliubraving@...com>
To:     Josh Poimboeuf <jpoimboe@...nel.org>
CC:     Rik van Riel <riel@...com>, "song@...nel.org" <song@...nel.org>,
        "joe.lawrence@...hat.com" <joe.lawrence@...hat.com>,
        "peterz@...radead.org" <peterz@...radead.org>,
        "mingo@...hat.com" <mingo@...hat.com>,
        "vincent.guittot@...aro.org" <vincent.guittot@...aro.org>,
        "pmladek@...e.com" <pmladek@...e.com>,
        "live-patching@...r.kernel.org" <live-patching@...r.kernel.org>,
        Kernel Team <Kernel-team@...com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "jpoimboe@...hat.com" <jpoimboe@...hat.com>
Subject: Re: [RFC] sched,livepatch: call klp_try_switch_task in __cond_resched



> On May 10, 2022, at 11:42 AM, Josh Poimboeuf <jpoimboe@...nel.org> wrote:
> 
> On Tue, May 10, 2022 at 06:07:00PM +0000, Rik van Riel wrote:
>> On Tue, 2022-05-10 at 09:52 -0700, Josh Poimboeuf wrote:
>>> On Tue, May 10, 2022 at 04:07:42PM +0000, Rik van Riel wrote:
>>>>> 
>>>> Now I wonder if we could just hook up a preempt notifier
>>>> for kernel live patches. All the distro kernels already
>>>> need the preempt notifier for KVM, anyway...
>>>> 
>>> 
>>> I wouldn't be opposed to that, but how does it solve this problem? 
>>> If
>>> as Peter said cond_resched() can be a NOP, then preemption would have
>>> to
>>> be from an interrupt, in which case frame pointers aren't reliable.
>>> 
>> The systems where we are seeing problems do not, as far
>> as I know, throw softlockup errors, so the kworker
>> threads that fail to transition to the new KLP version
>> are sleeping and getting scheduled out at times.
> 
> Are they sleeping due to an explicit call to cond_resched()?

In this case, yes. The thread calls cond_resched(). 

> 
>> A KLP transition preempt notifier would help those
>> kernel threads transition to the new KLP version at
>> any time they reschedule.
> 
> ... unless cond_resched() is a no-op due to CONFIG_PREEMPT?

Based on my understanding (and a few other folks we chatted with),
a kernel thread can legally run for extended time, as long as it 
calls cond_resched() at a reasonable frequency. Therefore, I 
think we should be able to patch such thread easily, unless it 
calls cond_resched() with being-patched function in the stack, 
of course.

OTOH, Petr's mindset of allowing many minutes for the patch 
transition is new to me. I need to think more about it. 
Josh, what’s you opinion on this? IIUC, kpatch is designed to 
only wait up to 60 seconds (no option to overwrite the time). 

> 
>> How much it will help is hard to predict, but I should
>> be able to get results from a fairly large sample size
>> of systems within a few weeks :)
> 
> As Peter said, keep in mind that we will need to fix other cases beyond
> Facebook, i.e., CONFIG_PREEMPT combined with non-x86 arches which don't
> have ORC so they can't reliably unwind from an IRQ.

I think livepatch transition may fail in different cases, and we
don't need to address all of them in one shoot. Fixing some cases
is an improvement as long as we don't slow down other cases. I 
understand that adding tiny overhead to __cond_resched() may end 
up as a visible regression. But maybe adding it to 
preempt_schedule_common() is light enough?

Did I miss/misunderstand something?

Thanks,
Song