linux-kernel - Re: New crashes walking proc with Saturday's git

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1416778927.3019.1@mail.thefacebook.com>
Date:	Sun, 23 Nov 2014 16:42:07 -0500
From:	Chris Mason <clm@...com>
To:	Thomas Gleixner <tglx@...utronix.de>
CC:	Borislav Petkov <bp@...en8.de>, <torvalds@...ux-foundation.org>,
	<linux-kernel@...r.kernel.org>, Ingo Molnar <mingo@...nel.org>,
	Stanislaw Gruszka <sgruszka@...hat.com>
Subject: Re: New crashes walking proc with Saturday's git

On Sun, Nov 23, 2014 at 4:38 PM, Thomas Gleixner <tglx@...utronix.de> 
wrote:
> On Sun, 23 Nov 2014, Chris Mason wrote:
>>  On Sun, Nov 23, 2014 at 4:05 PM, Thomas Gleixner 
>> <tglx@...utronix.de> wrote:
>>  > On Sun, 23 Nov 2014, Chris Mason wrote:
>>  > >  On Sun, Nov 23, 2014 at 11:32 AM, Borislav Petkov 
>> <bp@...en8.de> wrote:
>>  > >  > On Sun, Nov 23, 2014 at 11:16:51AM -0500, Chris Mason wrote:
>>  > >  > >  It must be:
>>  > >  > >
>>  > >  > >  commit 6e998916dfe327e785e7c2447959b2c1a3ea4930
>>  > >  > >  Author: Stanislaw Gruszka <sgruszka@...hat.com>
>>  > >  > >  Date:   Wed Nov 12 16:58:44 2014 +0100
>>  > >  > >
>>  > >  > >     sched/cputime: Fix clock_nanosleep()/clock_gettime()
>>  > > inconsistency
>>  > >  > >
>>  > >  > >  I'll do two runs to confirm, but it's the only related 
>> patch between
>>  > > rc5
>>  > >  > > and
>>  > >  > >  now.
>>  > >
>>  > >  I've adding Ingo and Stanislaw to the cc.  With
>>  > >  6e998916dfe327e785e7c2447959b2c1a3ea4930 reverted, I'm no 
>> longer
>>  > > crashing.
>>  > >
>>  > >  Repeating the stack trace for the new cc list.  I see the 
>> crash with atop
>>  > > or
>>  > >  similar walkers of /proc racing against exiting programs.  
>> Given the NULL
>>  > > rip,
>>  > >  this line from the patch is probably broken, but it really 
>> feels like we
>>  > >  should be falling over on p->sched_class and not on the 
>> update_curr func.
>>  > >
>>  > >  +               p->sched_class->update_curr(rq);
>>  > >
>>  > >  I'm leaving my fork bomb running on two machines with the 
>> patch reverted
>>  > > to
>>  > >  make sure.
>>  >
>>  > The sched_class instances which do not have update_curr are 
>> stop_task
>>  > and idle. Patch below.
>>  >
>>  > I'm sure nobody thought about the stats read code path here.
>>  >
>>  > [ 1053.759741]  [<ffffffff81208348>] do_task_stat+0x8b8/0xb00
>>  >
>>  > do_task_stat(()
>>  >  thread_group_cputime_adjusted()
>>  >    thread_group_cputime()
>>  >      task_cputime()
>>  >        task_sched_runtime()
>>  > 	if (task_current(rq, p) && task_on_rq_queued(p)) {
>>  >                 update_rq_clock(rq);
>>  >                 p->sched_class->update_curr(rq);
>>  >         }
>>  >
>>  > Now if the stats are read for a stomp machine task, aka 
>> 'migration/N'
>>  > and that task is current on its cpu. Ooops.
>>  >
>>  > I added the callback for idle tasks as well for completeness sake.
>> 
>>  This does make sense, but it doesn't match with the crash being 
>> much more
>>  likely during the fork bomb.  The difference is crashing within a 
>> few hours vs
>>  crashing within 5 minutes.
> 
> The fork bomb will kick the migration task pretty often into life, so
> the probablity of do_task_stat() to hit a running migration thread is
> higher than on a normaly loaded machine.

Fair enough, I just had crashes_in_proc == races_with_exit stuck in my 
head ;)  I've got a new xfstests run on the second machine to be sure, 
but this is definitely better.

-chris



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/