| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <ZqCo7H2gcA1dvIr4@xsang-OptiPlex-9020>
Date: Wed, 24 Jul 2024 15:10:36 +0800
From: Oliver Sang <oliver.sang@...el.com>
To: Dennis Zhou <dennis@...nel.org>
CC: Boqun Feng <boqun.feng@...il.com>, Tejun Heo <tj@...nel.org>, "Suren
Baghdasaryan" <surenb@...gle.com>, <oe-lkp@...ts.linux.dev>, <lkp@...el.com>,
<linux-kernel@...r.kernel.org>, Andrew Morton <akpm@...ux-foundation.org>,
Kent Overstreet <kent.overstreet@...ux.dev>, Kees Cook
<keescook@...omium.org>, Alexander Viro <viro@...iv.linux.org.uk>, "Alex
Gaynor" <alex.gaynor@...il.com>, Alice Ryhl <aliceryhl@...gle.com>, "Andreas
Hindborg" <a.hindborg@...sung.com>, Benno Lossin <benno.lossin@...ton.me>,
Björn Roy Baron <bjorn3_gh@...tonmail.com>, "Christoph
Lameter" <cl@...ux.com>, Gary Guo <gary@...yguo.net>, Miguel Ojeda
<ojeda@...nel.org>, Pasha Tatashin <pasha.tatashin@...een.com>, "Peter
Zijlstra" <peterz@...radead.org>, Vlastimil Babka <vbabka@...e.cz>, "Wedson
Almeida Filho" <wedsonaf@...il.com>, <linux-mm@...ck.org>,
<lkmm@...ts.linux.dev>, <oliver.sang@...el.com>
Subject: Re: [linus:master] [mm] 24e44cc22a:
BUG:KCSAN:data-race_in_pcpu_alloc_noprof/pcpu_block_update_hint_alloc
hi, Dannis,
On Mon, Jul 22, 2024 at 11:13:10PM -0700, Dennis Zhou wrote:
> Hi Oliver,
>
> On Tue, Jul 23, 2024 at 02:09:38PM +0800, Oliver Sang wrote:
> > hi, Dennis Zhou,
> >
> > On Mon, Jul 22, 2024 at 10:50:53PM -0700, Dennis Zhou wrote:
> > > On Mon, Jul 22, 2024 at 01:53:52PM -0700, Boqun Feng wrote:
> > > > On Mon, Jul 22, 2024 at 11:27:48AM -0700, Dennis Zhou wrote:
> > > > > Hello,
> > > > >
> > > > > On Mon, Jul 22, 2024 at 11:03:00AM -0700, Boqun Feng wrote:
> > > > > > On Mon, Jul 22, 2024 at 07:52:22AM -1000, Tejun Heo wrote:
> > > > > > > On Mon, Jul 22, 2024 at 10:47:30AM -0700, Boqun Feng wrote:
> > > > > > > > This looks like a data race because we read pcpu_nr_empty_pop_pages out
> > > > > > > > of the lock for a best effort checking, @Tejun, maybe you could confirm
> > > > > > > > on this?
> > > > > > >
> > > > > > > That does sound plausible.
> > > > > > >
> > > > > > > > - if (pcpu_nr_empty_pop_pages < PCPU_EMPTY_POP_PAGES_LOW)
> > > > > > > > + /*
> > > > > > > > + * Checks pcpu_nr_empty_pop_pages out of the pcpu_lock, data races may
> > > > > > > > + * occur but this is just a best-effort checking, everything is synced
> > > > > > > > + * in pcpu_balance_work.
> > > > > > > > + */
> > > > > > > > + if (data_race(pcpu_nr_empty_pop_pages) < PCPU_EMPTY_POP_PAGES_LOW)
> > > > > > > > pcpu_schedule_balance_work();
> > > > > > >
> > > > > > > Would it be better to use READ/WRITE_ONCE() for the variable?
> > > > > > >
> > > > > >
> > > > > > For READ/WRITE_ONCE(), we will need to replace all write accesses and
> > > > > > all out-of-lock read accesses to pcpu_nr_empty_pop_pages, like below.
> > > > > > It's better in the sense that it doesn't rely on compiler behaviors on
> > > > > > data races, not sure about the performance impact though.
> > > > > >
> > > > >
> > > > > I think a better alternative is we can move it up into the lock under
> > > > > area_found. The value gets updated as part of pcpu_alloc_area() as the
> > > > > code above populates percpu memory that is already allocated.
> > > > >
> > > >
> > > > Not sure I followed what exactly you suggested here because I'm not
> > > > familiar with the logic, but a simpler version would be:
> > > >
> > > >
> > >
> > > I believe that's the only naked access of pcpu_nr_empty_pop_pages. So
> > > I was thinking this'll fix this problem.
> > >
> > > I also don't know how to rerun this CI tho..
> >
> > we could test this patch. what's the base? could we apply it directly upon
> > 24e44cc22a?
> >
> > BTW, our bot is not so clever so far to auto test fix patches, so this is kind
> > of manual effort. due to resource constraint, it will be hard for us to test
> > each patch (we saw several patches in this thread already) or test very fast.
> >
>
> Ah yeah that makes sense. If you don't mind testing the last one I sent,
> the one below, that applies cleanly to 24e44cc22a.
in our tests, you patch could solve the KCSAN issues. thanks
=========================================================================================
compiler/group/kconfig/nr_groups/rootfs/runtime/tbox_group/testcase:
gcc-13/group-04/x86_64-randconfig-013-20240713/5/debian-11.1-i386-20220923.cgz/300s/vm-snb/trinity
commit:
60fa4a9e23231 ("mm: percpu: add codetag reference into pcpuobj_ext")
24e44cc22aa31 ("mm: percpu: enable per-cpu allocation tagging")
dcfbb68202759 <--- your patch
60fa4a9e23231721 24e44cc22aa3112082f2ee23137 dcfbb6820275994e92a9dcf309e
---------------- --------------------------- ---------------------------
fail:runs %reproduction fail:runs %reproduction fail:runs
| | | | |
281:999 -28% :998 -28% :999 dmesg.BUG:KCSAN:data-race_in_pcpu_alloc/pcpu_block_update_hint_alloc
296:999 -30% :998 -30% :999 dmesg.BUG:KCSAN:data-race_in_pcpu_alloc/pcpu_block_update_hint_free
25:999 -3% :998 -3% :999 dmesg.BUG:KCSAN:data-race_in_pcpu_alloc/pcpu_chunk_populated
:999 29% 292:998 0% :999 dmesg.BUG:KCSAN:data-race_in_pcpu_alloc_noprof/pcpu_block_update_hint_alloc
:999 27% 269:998 0% :999 dmesg.BUG:KCSAN:data-race_in_pcpu_alloc_noprof/pcpu_block_update_hint_free
:999 4% 44:998 0% :999 dmesg.BUG:KCSAN:data-race_in_pcpu_alloc_noprof/pcpu_chunk_populated
>
> Thanks,
> Dennis
>
> > >
> > > ---
> > > diff --git a/mm/percpu.c b/mm/percpu.c
> > > index 20d91af8c033..325fb8412e90 100644
> > > --- a/mm/percpu.c
> > > +++ b/mm/percpu.c
> > > @@ -1864,6 +1864,10 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,
> > >
> > > area_found:
> > > pcpu_stats_area_alloc(chunk, size);
> > > +
> > > + if (pcpu_nr_empty_pop_pages < PCPU_EMPTY_POP_PAGES_LOW)
> > > + pcpu_schedule_balance_work();
> > > +
> > > spin_unlock_irqrestore(&pcpu_lock, flags);
> > >
> > > /* populate if not all pages are already there */
> > > @@ -1891,9 +1895,6 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,
> > > mutex_unlock(&pcpu_alloc_mutex);
> > > }
> > >
> > > - if (pcpu_nr_empty_pop_pages < PCPU_EMPTY_POP_PAGES_LOW)
> > > - pcpu_schedule_balance_work();
> > > -
> > > /* clear the areas and return address relative to base address */
> > > for_each_possible_cpu(cpu)
> > > memset((void *)pcpu_chunk_addr(chunk, cpu, 0) + off, 0, size);
Powered by blists - more mailing lists