linux-kernel - Re: [PATCH v8] oom_kill.c: futex: Don't OOM reap the VMA containing the robust_list

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <87h76yff3b.ffs@tglx>
Date:   Tue, 12 Apr 2022 18:20:40 +0200
From:   Thomas Gleixner <tglx@...utronix.de>
To:     Nico Pache <npache@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>
Cc:     linux-mm@...ck.org, linux-kernel@...r.kernel.org,
        Rafael Aquini <aquini@...hat.com>,
        Waiman Long <longman@...hat.com>, Baoquan He <bhe@...hat.com>,
        Christoph von Recklinghausen <crecklin@...hat.com>,
        Don Dutile <ddutile@...hat.com>,
        "Herton R . Krzesinski" <herton@...hat.com>,
        David Rientjes <rientjes@...gle.com>,
        Michal Hocko <mhocko@...e.com>,
        Andrea Arcangeli <aarcange@...hat.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Davidlohr Bueso <dave@...olabs.net>,
        Ingo Molnar <mingo@...hat.com>,
        Joel Savitz <jsavitz@...hat.com>,
        Darren Hart <dvhart@...radead.org>, stable@...nel.org
Subject: Re: [PATCH v8] oom_kill.c: futex: Don't OOM reap the VMA containing
 the robust_list_head

On Mon, Apr 11 2022 at 19:51, Nico Pache wrote:
> On 4/8/22 09:54, Thomas Gleixner wrote:
>> The below reproduces the problem nicely, i.e. the lock() in the parent
>> times out. So why would the OOM killer fail to cause the same problem
>> when it reaps the private anon mapping where the private futex sits?
>> 
>> If you revert the lock order in the child the robust muck works.
>
> Thanks for the reproducer Thomas :)
>
> I think I need to re-up my knowledge around COW and how it effects
> that stack. There are increased oddities when you add the pthread
> library that I cant fully wrap my head around at the moment.

The pthread library functions are just conveniance so I did not have to
hand code the futex and robust list handling.

> My confusion lies in how the parent/child share a robust list here, but they
> obviously do. In my mind the mut_s would be different in the child/parent after
> the fork and pthread_mutex_init (and friends) are done in the child.

They don't share a robust list, each thread has it's own.

The shared mutex mut_s is initialized in the parent before fork and it's
the same address in the child and it's not COWed because the mapping is
MAP_SHARED.

The child allocates private memory and initializes the private mutex in
that private mapping.

So now child does:

   pthread_mutex_lock(mut_s);

That's the mutex in the memory shared with the parent. After that the
childs robusts list head points to mut_s::robust_list.

Now it does:

   pthread_mutex_lock(mut_p);

after that the childs robust list head points to mut_p::robust_list and
mut_p::robust_list points to mut_s::robust_list.

So now the child unmaps the private memory and exists.

The kernel tries to walk the robust list pointer and faults when trying
to access mut_p. End of walk and mut_s stays locked.

So now think about the OOM case. The killed process has a shared mapping
with some other unrelated process (file, shmem) where mut_p sits.

It gets killed after:
		pthread_mutex_lock(mut_s);
		pthread_mutex_lock(mut_p);

So the OOM reaper rips the VMA which contains mut_p and therefore breaks
the chain which is necessary to reach mut_p.

See?

Thanks,

        tglx