linux-kernel - rseq CPU ID not correct on 6.0 kernels for pinned threads

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <87lem9cnxr.fsf@oldenburg.str.redhat.com>
Date:   Wed, 11 Jan 2023 12:26:08 +0100
From:   Florian Weimer <fweimer@...hat.com>
To:     Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
        linux-kernel@...r.kernel.org
Cc:     Ingo Molnar <mingo@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        libc-alpha@...rceware.org
Subject: rseq CPU ID not correct on 6.0 kernels for pinned threads

The glibc test suite contains a test that verifies that sched_getcpu
returns the expected CPU number for a thread that is pinned (via
sched_setaffinity) to a specific CPU.  There are other threads running
which attempt to de-schedule the pinned thread from its CPU.  I believe
the test is correctly doing what it is expected to do; it is invalid
only if one believes that it is okay for the kernel to disregard the
affinity mask for scheduling decisions.

These days, we use the cpu_id rseq field as the return value of
sched_getcpu if the kernel has rseq support (which it has in these
cases).

This test has started failing sporadically for us, some time around
kernel 6.0.  I see failure occasionally on a Fedora builder, it runs:

Linux buildvm-x86-26.iad2.fedoraproject.org 6.0.15-300.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Dec 21 18:33:23 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I think I've seen it on the x86-64 builder only, but that might just be
an accident.

The failing tests log this output:

=====FAIL: nptl/tst-thread-affinity-pthread.out=====
info: Detected CPU set size (in bits): 64
info: Maximum test CPU: 5
error: Pinned thread 1 ran on impossible cpu 0
error: Pinned thread 0 ran on impossible cpu 0
info: Main thread ran on 4 CPU(s) of 6 available CPU(s)
info: Other threads ran on 6 CPU(s)
=====FAIL: nptl/tst-thread-affinity-pthread2.out=====
info: Detected CPU set size (in bits): 64
info: Maximum test CPU: 5
error: Pinned thread 1 ran on impossible cpu 1
error: Pinned thread 2 ran on impossible cpu 0
error: Pinned thread 3 ran on impossible cpu 3
info: Main thread ran on 5 CPU(s) of 6 available CPU(s)
info: Other threads ran on 6 CPU(s)

But I also encountered one local failure, but it is rare.  Maybe it's
load-related.  There shouldn't be any CPU unplug or anything like that
involved here.

I am not entirely sure if something is changing CPU affinities from
outside the process (which would be quite wrong, but not a kernel bug).
But in the past, our glibc test has detected real rseq cpu_id
brokenness, so I'm leaning towards that as the cause this time, too.

Thanks,
Florian