linux-kernel - Re: Leap second insertion causes futex to repeatedly timeout

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Sun, 01 Jul 2012 02:07:42 -0700
From:	John Stultz <johnstul@...ibm.com>
To:	Ben Blum <bblum@...rew.cmu.edu>
CC:	Jan Engelhardt <jengelh@...i.de>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	simon@...e.lp0.eu, Thomas Gleixner <tglx@...utronix.de>
Subject: Re: Leap second insertion causes futex to repeatedly timeout

On 07/01/2012 01:36 AM, Ben Blum wrote:
> On Sun, Jul 01, 2012 at 01:16:13AM -0700, john stultz wrote:
>> On Sat, Jun 30, 2012 at 5:57 PM, Jan Engelhardt <jengelh@...i.de> wrote:
>>> This year's leap second insertion has had the strange effect on at least
>>> Linux versions 3.4.4 (my end) and 3.5-rc4 (Simon's box, Cc) that certain
>>> processes use up all CPU power, because of futexes repeatedly timing
>>> out. This seems to only affect certain processes.
>>>
>>> Simon observes - http://s85.org/owXfmLvt - that
>>> Firefox/Thunderbird/Chrome/Java are affected.
>>>
>>> As for me, it affects VirtualBox, mysqld and ksoftirqd. The processes
>>> continue to run and respond. Most weird: I can stop-start mysqld and the
>>> issue persists. (I would have expected it to go away because the leap
>>> second event would then be in the past that mysqld does not know about
>>> anymore.)
>>>
>>>
>>> Is this a kernel issue? glibc?
>> Some of the reports that the issue is resolved by calling:
>>         $ date -s "`date`"
>> suggests that it might be due to clock_was_set() not being called
>> after the leap second was added, causing some hrtimer confusion.
>>
>> Thomas: does that sound about right?
>>
>> I've got an initial patch to add the clock_was_set() calls where
>> needed, but so far have not been able to reproduce the issue (tried
>> firefox and some simpler futex tests).  I'll keep trying and hopefully
>> have something to send out tomorrow.
>>
>> Again, my apologies for the trouble.
> I can't vouch for whether this is the problem or not, but be very
> careful with clock_was_set()! See this commit:
>
> http://www.mail-archive.com/git-commits-head@vger.kernel.org/msg15039.html
>
> In short, clock_was_set() calls on_each_cpu() which is not allowed to be
> called in atomic context. Watch out for xtime_lock.

Quite right.  The fix is a little awkward due to the need to call it 
outside of holding xtime_lock/timekeeper.lock.

I've just reproduced the issue w/ Thunderbird, and my fix seems to avoid 
the issue. Working up a patch now.

thanks
-john


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/