linux-kernel - Re: [PATCH v3 0/9] Parallel CPU bringup for x86

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4c92a897249bca15aeea134d0f7945defe66097d.camel@infradead.org>
Date:   Fri, 17 Dec 2021 15:40:32 +0000
From:   David Woodhouse <dwmw2@...radead.org>
To:     Igor Mammedov <imammedo@...hat.com>
Cc:     Tom Lendacky <thomas.lendacky@....com>,
        Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...hat.com>, Borislav Petkov <bp@...en8.de>,
        Dave Hansen <dave.hansen@...ux.intel.com>,
        "x86@...nel.org" <x86@...nel.org>,
        "H . Peter Anvin" <hpa@...or.com>,
        Paolo Bonzini <pbonzini@...hat.com>,
        "Paul E . McKenney" <paulmck@...nel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "rcu@...r.kernel.org" <rcu@...r.kernel.org>,
        "mimoja@...oja.de" <mimoja@...oja.de>,
        "hewenliang4@...wei.com" <hewenliang4@...wei.com>,
        "hushiyuan@...wei.com" <hushiyuan@...wei.com>,
        "luolongjun@...wei.com" <luolongjun@...wei.com>,
        "hejingxian@...wei.com" <hejingxian@...wei.com>
Subject: Re: [PATCH v3 0/9] Parallel CPU bringup for x86_64

On Fri, 2021-12-17 at 11:09 +0100, Igor Mammedov wrote:
> that's most likely the case (there is a race somewhere left).
> To trigger CPU bringup (hotplug) races, I used to run QEMU guest with
> heavy vCPU overcommit. It helps to induce unexpected delays at CPU bringup
> time.

Yeah, I've been doing a fair amount of that but even with Tom's config
I can't reproduce a crash. Have seen this one now though. It's hard to
reproduce, and I suspect it was there already and I've only tweaked the
timing to expose it (or not even that, and just done enough tests that
I've seen it when it's extremely sporadic).

[    0.061937] kvm-clock: cpu 24, msr 31801601, secondary cpu clock
[    0.668842] kvm-guest: stealtime: cpu 24, msr 37c31080
[    0.061937] kvm-clock: cpu 25, msr 31801641, secondary cpu clock
[    0.670557] kvm-guest: stealtime: cpu 25, msr 37c71080
[    0.670557] ------------[ cut here ]------------
[    0.670557] cfs_rq->avg.load_avg || cfs_rq->avg.util_avg || cfs_rq->avg.runnable_avg
[    0.670557] WARNING: CPU: 25 PID: 140 at kernel/sched/fair.c:3299 __update_blocked_fair+0x4b7/0x4d0
[    0.061937] kvm-clock: cpu 26, msr 31801681, secondary cpu clock
[    0.670740] Modules linked in:
[    0.670740] CPU: 25 PID: 140 Comm: kworker/25:0H Not tainted 5.16.0-rc2-sos-testing+ #963
[    0.670740] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
[    0.670740] RIP: 0010:__update_blocked_fair+0x4b7/0x4d0
[    0.670740] Code: 4f fd ff ff 49 8b 96 48 01 00 00 48 89 90 60 09 00 00 e9 e3 fc ff ff 48 c7 c7 30 36 34 b6 c6 05 77 78 ae 01 01 e8 d5 58 96 00 <0f> 0b 41 8b 86 38 01 00 00 e9 aa fc ff ff 66 66 2e 0f 1f 84 00 00
[    0.670740] RSP: 0018:ffffc90000cc7d30 EFLAGS: 00010086
[    0.670740] RAX: 0000000000000000 RBX: 00000000000000c8 RCX: 0000000000000000
[    0.670740] RDX: 0000000000000003 RSI: ffff88803bbfffe8 RDI: 00000000ffffffff
[    0.670740] RBP: ffff888037c6f800 R08: 00000000ffffffea R09: 0000000000000000
[    0.670740] R10: 0000000000000003 R11: 3fffffffffffffff R12: ffff888037c6ff90
[    0.670740] R13: ffff888037c6fe50 R14: ffff888037c6f6c0 R15: 0000000000000000
[    0.670740] FS:  0000000000000000(0000) GS:ffff888037c40000(0000) knlGS:0000000000000000
[    0.670740] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.670740] CR2: 0000000000000000 CR3: 000000003060a000 CR4: 0000000000350ee0
[    0.670740] Call Trace:
[    0.670740]  <TASK>
[    0.670740]  update_blocked_averages+0x98/0x160
[    0.670740]  newidle_balance+0x117/0x390
[    0.670740]  pick_next_task_fair+0x39/0x3c0
[    0.670740]  __schedule+0x156/0x6f0
[    0.670740]  schedule+0x4e/0xc0
[    0.670740]  worker_thread+0xb1/0x300
[    0.670740]  ? rescuer_thread+0x370/0x370
[    0.687790] kvm-guest: stealtime: cpu 26, msr 37cb1080
[    0.061937] kvm-clock: cpu 27, msr 318016c1, secondary cpu clock
[    0.690740] kvm-guest: stealtime: cpu 27, msr 37cf1080
[    0.061937] kvm-clock: cpu 28, msr 31801701, secondary cpu clock
[    0.693781] kvm-guest: stealtime: cpu 28, msr 37d31080
[    0.670740]  kthread+0x158/0x180
[    0.061937] kvm-clock: cpu 29, msr 31801741, secondary cpu clock
[    0.670740]  ? set_kthread_struct+0x40/0x40
[    0.670740]  ret_from_fork+0x22/0x30
[    0.670740]  </TASK>
[    0.670740] ---[ end trace ac8562dd64da6bb5 ]---
[    0.747785] kvm-guest: stealtime: cpu 29, msr 37d71080
[    0.061937] kvm-clock: cpu 30, msr 31801781, secondary cpu clock
[    0.756784] kvm-guest: stealtime: cpu 30, msr 37db1080


Download attachment "smime.p7s" of type "application/pkcs7-signature" (5174 bytes)