linux-kernel - Re: scheduler problems in -next (was: Re: [PATCH 6.4 000/227] 6.4.7-rc1 review)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20230731143954.GB37820@hirez.programming.kicks-ass.net>
Date:   Mon, 31 Jul 2023 16:39:54 +0200
From:   Peter Zijlstra <peterz@...radead.org>
To:     Guenter Roeck <linux@...ck-us.net>
Cc:     Joel Fernandes <joel@...lfernandes.org>, paulmck@...nel.org,
        Pavel Machek <pavel@...x.de>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        stable@...r.kernel.org, patches@...ts.linux.dev,
        linux-kernel@...r.kernel.org, torvalds@...ux-foundation.org,
        akpm@...ux-foundation.org, shuah@...nel.org, patches@...nelci.org,
        lkft-triage@...ts.linaro.org, jonathanh@...dia.com,
        f.fainelli@...il.com, sudipm.mukherjee@...il.com,
        srw@...dewatkins.net, rwarsow@....de, conor@...nel.org,
        rcu@...r.kernel.org, Ingo Molnar <mingo@...nel.org>
Subject: Re: scheduler problems in -next (was: Re: [PATCH 6.4 000/227]
 6.4.7-rc1 review)

On Mon, Jul 31, 2023 at 04:19:34PM +0200, Peter Zijlstra wrote:
> On Sat, Jul 29, 2023 at 09:00:02PM -0700, Guenter Roeck wrote:
> > On 7/27/23 16:18, Joel Fernandes wrote:
> > 
> > [ ... ]
> > 
> > > > I freely confess that I am having a hard time imagining what would
> > > > be CPU dependent in that code.  Timing, maybe?  Whatever the reason,
> > > > I am not seeing these failures in my testing.
> > > > 
> > > > So which of the following Kconfig options is defined in your .config?
> > > > CONFIG_TASKS_RCU, CONFIG_TASKS_RUDE_RCU, and CONFIG_TASKS_TRACE_RCU.
> > > > 
> > > > If you have more than one of them, could you please apply this patch
> > > > and show me the corresponding console output from the resulting hang?
> > > 
> > > FWIW, I am not able to repro this issue either. If a .config can be shared of the problem system, I can try it out to see if it can be reproduced on my side.
> > > 
> > 
> > I managed to bisect the problem. See bisect log below. Bisect repeated twice.
> > so it should be reliable. I don't really understand it, but the following
> > reverts fix the problem. This is on top of next-20230721 because next-20230728
> > crashes immediately in my tests.
> > 
> > 0caafe9b94ab (HEAD) Revert "sched/fair: Remove sched_feat(START_DEBIT)"
> > 518bdbd39fdb Revert "sched/fair: Add lag based placement"
> > a011162c3e32 Revert "sched/fair: Implement an EEVDF-like scheduling policy"
> > df579720bf98 Revert "sched/fair: Commit to lag based placement"
> > aac459a7e738 Revert "sched/smp: Use lag to simplify cross-runqueue placement"
> > 8d686eb173e1 Revert "sched/fair: Commit to EEVDF"
> > 486474c50f95 Revert "sched/debug: Rename sysctl_sched_min_granularity to sysctl_sched_base_slice"
> > 79e94d67d08a Revert "sched/fair: Propagate enqueue flags into place_entity()"
> > ae867bc97b71 (tag: next-20230721) Add linux-next specific files for 20230721
> > 
> > For context: x86 images (32 and 64 bit) in -next tend to hang at
> > 
> > [    2.309323] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1.
> > [    2.311634] Running RCU-tasks wait API self tests
> > 
> > The hang is not seen with every boot; it happens roughly about once every
> > 10 boot attempts. It is not CPU dependent as I initially thought.
> > 
> > Configuration file is at http://server.roeck-us.net/qemu/x86-next/config.
> > Example qemu command line:
> 
> Hurmph, let me see if I can reproduce on next-20230731 (not having the
> older next thingies around).

I've taken your config above, and the rootfs.ext2 and run-sh from x86/.
I've then modified run-sh to use:

  qemu-system-x86_64 -enable-kvm -cpu host

What I'm seeing is that some boots get stuck at:

[    0.608230] Running RCU-tasks wait API self tests

Is this the right 'problem' ?