linux-kernel - Re: [tip: timers/core] hrtimer: Annotate lockless access to timer->state

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89i+8Hq5j234zFRY05QxZU1n=Vr6S-kZCcvn3Z80xYaindg@mail.gmail.com>
Date:   Thu, 7 Nov 2019 07:48:50 -0800
From:   Eric Dumazet <edumazet@...gle.com>
To:     paulmck@...nel.org
Cc:     LKML <linux-kernel@...r.kernel.org>,
        linux-tip-commits@...r.kernel.org,
        syzbot <syzkaller@...glegroups.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...nel.org>, Borislav Petkov <bp@...en8.de>
Subject: Re: [tip: timers/core] hrtimer: Annotate lockless access to timer->state

On Thu, Nov 7, 2019 at 12:53 AM Paul E. McKenney <paulmck@...nel.org> wrote:
>
> On Wed, Nov 06, 2019 at 02:59:36PM -0800, Eric Dumazet wrote:
> > On Wed, Nov 6, 2019 at 2:53 PM Eric Dumazet <edumazet@...gle.com> wrote:
> > >
> > > On Wed, Nov 6, 2019 at 2:24 PM tip-bot2 for Eric Dumazet
> > > <tip-bot2@...utronix.de> wrote:
> > > >
> > > > The following commit has been merged into the timers/core branch of tip:
> > > >
> > > > Commit-ID:     56144737e67329c9aaed15f942d46a6302e2e3d8
> > > > Gitweb:        https://git.kernel.org/tip/56144737e67329c9aaed15f942d46a6302e2e3d8
> > > > Author:        Eric Dumazet <edumazet@...gle.com>
> > > > AuthorDate:    Wed, 06 Nov 2019 09:48:04 -08:00
> > > > Committer:     Thomas Gleixner <tglx@...utronix.de>
> > > > CommitterDate: Wed, 06 Nov 2019 23:18:31 +01:00
> > > >
> > > > hrtimer: Annotate lockless access to timer->state
> > > >
> > >
> > > I guess we also need to fix timer_pending(), since timer->entry.pprev
> > > could change while we read it.
> >
> > It is interesting seeing hlist_add_head() has a WRITE_ONCE(h->first, n);,
> > but no WRITE_ONCE() for the pprev change.
> >
> > The WRITE_ONCE() was added in commit 1c97be677f72b3c338312aecd36d8fff20322f32
> > ("list: Use WRITE_ONCE() when adding to lists and hlists")
>
> The theory is that while the ->next pointer is concurrently accessed by
> RCU readers, the ->pprev pointer is accessed only by updaters, who need
> to supply sufficient synchronization.
>
> But what is this theory missing in practice?

Here is some context : I am helping triaging about 400 KCSAN data-race
splats in syzbot moderation queue.

Take a look at the timer related one in [1]

If we want to avoid potential load/store-tearing, minimall patch would be :

diff --git a/include/linux/list.h b/include/linux/list.h
index 85c92555e31f85f019354e54d6efb8e79c2aee17..9139803b851cc37bb759c8d7c12ee7e36c61f009
100644
--- a/include/linux/list.h
+++ b/include/linux/list.h
@@ -761,7 +761,7 @@ static inline void __hlist_del(struct hlist_node *n)

        WRITE_ONCE(*pprev, next);
        if (next)
-               next->pprev = pprev;
+               WRITE_ONCE(next->pprev, pprev);
 }

 static inline void hlist_del(struct hlist_node *n)
diff --git a/include/linux/timer.h b/include/linux/timer.h
index 1e6650ed066d5d28251b0bd385fc37ef94c96532..c7c8dd89f2797389ca96473e60c7297fd38d8259
100644
--- a/include/linux/timer.h
+++ b/include/linux/timer.h
@@ -164,7 +164,7 @@ static inline void destroy_timer_on_stack(struct
timer_list *timer) { }
  */
 static inline int timer_pending(const struct timer_list * timer)
 {
-       return timer->entry.pprev != NULL;
+       return READ_ONCE(timer->entry.pprev) != NULL;
 }

 extern void add_timer_on(struct timer_list *timer, int cpu);


But really many other WRITE_ONCE() would be needed in include/linux/list.h


[1]

BUG: KCSAN: data-race in del_timer / detach_if_pending

write to 0xffff88808697d870 of 8 bytes by task 10 on cpu 0:
 __hlist_del include/linux/list.h:764 [inline]
 detach_timer kernel/time/timer.c:815 [inline]
 detach_if_pending+0xcd/0x2d0 kernel/time/timer.c:832
 try_to_del_timer_sync+0x60/0xb0 kernel/time/timer.c:1226
 del_timer_sync+0x6b/0xa0 kernel/time/timer.c:1365
 schedule_timeout+0x2d2/0x6e0 kernel/time/timer.c:1896
 rcu_gp_fqs_loop+0x37c/0x580 kernel/rcu/tree.c:1639
 rcu_gp_kthread+0x143/0x230 kernel/rcu/tree.c:1799
 kthread+0x1d4/0x200 drivers/block/aoe/aoecmd.c:1253
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:352

read to 0xffff88808697d870 of 8 bytes by task 12060 on cpu 1:
 del_timer+0x3b/0xb0 kernel/time/timer.c:1198
 sk_stop_timer+0x25/0x60 net/core/sock.c:2845
 inet_csk_clear_xmit_timers+0x69/0xa0 net/ipv4/inet_connection_sock.c:523
 tcp_clear_xmit_timers include/net/tcp.h:606 [inline]
 tcp_v4_destroy_sock+0xa3/0x3f0 net/ipv4/tcp_ipv4.c:2096
 inet_csk_destroy_sock+0xf4/0x250 net/ipv4/inet_connection_sock.c:836
 tcp_close+0x6f3/0x970 net/ipv4/tcp.c:2497
 inet_release+0x86/0x100 net/ipv4/af_inet.c:427
 __sock_release+0x85/0x160 net/socket.c:590
 sock_close+0x24/0x30 net/socket.c:1268
 __fput+0x1e1/0x520 fs/file_table.c:280
 ____fput+0x1f/0x30 fs/file_table.c:313
 task_work_run+0xf6/0x130 kernel/task_work.c:113
 tracehook_notify_resume include/linux/tracehook.h:188 [inline]
 exit_to_usermode_loop+0x2b4/0x2c0 arch/x86/entry/common.c:163

Reported by Kernel Concurrency Sanitizer on:
CPU: 1 PID: 12060 Comm: syz-executor.5 Not tainted 5.4.0-rc3+ #0
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
==================================================================