Message-ID: <AANLkTikZXh4NbwTCj22dTAEjnNjtEdGtQpTDChw=fckN@mail.gmail.com>
Date:	Tue, 1 Feb 2011 15:46:26 +1000
From:	Linus Torvalds <torvalds@...ux-foundation.org>
To:	paulmck@...ux.vnet.ibm.com
Cc:	Benjamin Herrenschmidt <benh@...nel.crashing.org>,
	Peter Zijlstra <peterz@...radead.org>,
	Milton Miller <miltonm@....com>, akpm@...ux-foundation.org,
	Anton Blanchard <anton@...ba.org>,
	xiaoguangrong@...fujitsu.com, mingo@...e.hu, jaxboe@...ionio.com,
	npiggin@...il.com, rusty@...tcorp.com.au,
	linux-kernel@...r.kernel.org
Subject: Re: call_function_many: fix list delete vs add race

On Tue, Feb 1, 2011 at 2:45 PM, Paul E. McKenney
<paulmck@...ux.vnet.ibm.com> wrote:
> On Tue, Feb 01, 2011 at 12:43:56PM +1000, Linus Torvalds wrote:
>>
>> That said, I do think that if your memory ordering is much weaker than
>> x86, you are going to see bugs that most testers don't see, and it
>> simply might not be worth it.
>
> IBM's CPUs do split the difference, with s390 having somewhat stronger
> ordering than x86, and with powerpc being rather weaker

I'm not talking about memory ordering as done by the cpu, but as done
by the spinlock operations. They can be arbitrarily strong, even if
the CPU architecture itself might be weakly ordered.

So what I suggest is that even if you have a CPU that can implement
acquire/release operations on the locks, you should at least think
about making a lock/unlock sequence be at least a full memory barrier,
simply because it is on x86.

And on at least alpha, with its famously weak memory ordering, you
actually have to do that, because the weak memory ordering of the
normal memory operations are coupled with rather bad memory ordering
primitives.

So on alpha, both LOCK and UNLOCK  are full memory barriers
_individually_, not even paired. So despite being known for being
weakly ordered, alpha is actually the MOST STRONGLY ordered of all
when it comes to spinlock primitives, and doesn't have any notion at
all of "inside" vs "outside" the lock or anything like that. A
spinlock/unlock sequence contains TWO full memory barriers, not just
one.

So "weak memory ordering" in no way means "weak lock ordering". Quite
often it means the reverse. POWER, as you say, has weak memory
ordering on the actual memory ops (like alpha), but like alpha has
traditionally really quite crappy and bad synchronization primitives.

So don't confuse the two. When talking about spinlocks, don't confuse
the memory ordering for non-locked operations with the memory ordering
for the locks themselves. The two are totally independent, and often
weak ordering on the normal memory operations actually is associated
with just total idiotic crap on the synchronization primitives. Most
of them were designed when threading and locking wasn't seen as a big
issue, and before people understood that serialization is a big deal.

(You can tell how ia64 was designed later, for example, by looking at
its fancy serialization primitives.)

Quite frankly, the POWER case is made worse by the fact that the
synchronization primitives have this total confusion about "pipeline"
synchronization due to historical implementation oddities etc. Talk
about crazy. The whole "isync" vs "sync" vs "lwsync" thing is just an
embarrassment. But this is, after all, a company that ran out of
vowels in instruction naming, so I guess their misleading and confused
naming for memory ordering is just par for the course.

                       Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists