linux-kernel - Re: [PATCH v3] ring-buffer: Race when writing and swapping cpu buffer in parallel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20140721123019.678788f1@gandalf.local.home>
Date:	Mon, 21 Jul 2014 12:30:19 -0400
From:	Steven Rostedt <rostedt@...dmis.org>
To:	Petr Mládek <pmladek@...e.cz>
Cc:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Ingo Molnar <mingo@...e.hu>,
	Frederic Weisbecker <fweisbec@...il.com>,
	Jiri Kosina <jkosina@...e.cz>, linux-kernel@...r.kernel.org
Subject: Re: [PATCH v3] ring-buffer: Race when writing and swapping cpu
 buffer in parallel

On Mon, 21 Jul 2014 18:18:28 +0200
Petr Mládek <pmladek@...e.cz> wrote:

> On Mon 2014-07-21 08:43:17, Paul E. McKenney wrote:
> > On Mon, Jul 21, 2014 at 04:43:24PM +0200, Petr Mládek wrote:
> > 
> > IIRC, deadlock in the case where two CPUs attempt to invoke
> > smp_call_function_single() at each other, but both have
> > interrupts disabled.  It might be possible to avoid this by telling
> > smp_call_function_single() not to wait for a response, but this often
> > just re-introduces the deadlock at a higher level.
> 
> I thought that IPI used NMI and thus could not be blocked if the
> called function was reasonable. Note that ring_buffer_swap_cpu() does not take
> any lock and can't block anywhere. I am probably too optimistic here.
> 

Heh, that would be a crazy system. No, IPI is a normal maskable
interrupt. It does not use NMIs. In fact, IPI is how irq_work is
implemented to do stuff from an NMI outside of NMI context.

> 
> > > Any pointers or ideas are welcome.
> > 
> > Not immediately.  Mark Batty (mark.batty@...cam.ac.uk) has come up with
> > cute ring-buffer tricks in the past, but would need a clear statement of
> > the problem.  I would be happy to bring him into the discussion if it
> > would help.
> 
> In short. We have two operations: writing and swap. They "block" each
> other by setting the variables "committing" and "record_disabled".
> It is not a real block. The other operation is "nop" when the other
> one is in the critical section.
> 
> We want to keep writing fast and avoid memory barriers there. Writing
> works with per-CPU buffer. It would help when also the swap happens
> on the same CPU.
> 
> More detailed description of the current race can be found at
> https://lkml.org/lkml/2014/7/16/178
> 
> 
> > And yes, my knee-jerk reaction of suggesting RCU runs into the problem
> > that it is not so good to invoke synchronize_rcu() with interrupts
> > disabled.  Might be able to use call_rcu(), but if that worked, then
> > just telling smp_call_function_single() not to wait would probably
> > be a lot simpler.
> 
> I am still not sure if it really has to be called with IRQs disabled.

Yes it has to be. The stop_critical_timings() is the core of the
interrupts off latency tracer and that code is heavily dependent on
interrupts being disabled (for the irqsoff part, preemption must be
off for the preemptoff part).


> 
> BTW: I have just got another idea. If we store pointer to the used
> cpu_buffer into struct ring_buffer_event, it might be possible
> to keep the write operation consistent even when the cpu buffers
> are switched.

Can't. The ring_buffer_event is an ABI and is mapped in the ring buffer
itself which is exported to userspace.

-- Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/