linux-kernel - Re: [RFC][RFT] memcg fix cgroup_mutex deadlock when cpuset reclaims memory

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20081210130607.GD25467@balbir.in.ibm.com>
Date:	Wed, 10 Dec 2008 18:36:07 +0530
From:	Balbir Singh <balbir@...ux.vnet.ibm.com>
To:	nishimura@....nes.nec.co.jp
Cc:	KAMEZAWA Hiroyuki <kamezawa.hiroyu@...fujitsu.com>,
	menage@...gle.com, Daisuke Miyakawa <dmiyakawa@...gle.com>,
	YAMAMOTO Takashi <yamamoto@...inux.co.jp>,
	linux-kernel@...r.kernel.org, linux-mm@...ck.org,
	d-nishimura@....biglobe.ne.jp
Subject: Re: [RFC][RFT] memcg fix cgroup_mutex deadlock when cpuset
	reclaims memory

* Daisuke Nishimura <d-nishimura@....biglobe.ne.jp> [2008-12-10 20:53:37]:

> On Wed, 10 Dec 2008 17:18:36 +0900
> KAMEZAWA Hiroyuki <kamezawa.hiroyu@...fujitsu.com> wrote:
> 
> > On Wed, 10 Dec 2008 16:41:26 +0900
> > Daisuke Nishimura <nishimura@....nes.nec.co.jp> wrote:
> > 
> > > On Wed, 10 Dec 2008 15:19:48 +0900, Daisuke Nishimura <nishimura@....nes.nec.co.jp> wrote:
> > > > On Wed, 10 Dec 2008 10:49:47 +0530, Balbir Singh <balbir@...ux.vnet.ibm.com> wrote:
> > > > > Hi,
> > > > > 
> > > > > Here is a proposed fix for the memory controller cgroup_mutex deadlock
> > > > > reported. It is lightly tested and reviewed. I need help with review
> > > > > and test. Is the reported deadlock reproducible after this patch? A
> > > > > careful review of the cpuset impact will also be highly appreciated.
> > > > > 
> > > > > From: Balbir Singh <balbir@...ux.vnet.ibm.com>
> > > > > 
> > > > > cpuset_migrate_mm() holds cgroup_mutex throughout the duration of
> > > > > do_migrate_pages(). The issue with that is that
> > > > > 
> > > > > 1. It can lead to deadlock with memcg, as do_migrate_pages()
> > > > >    enters reclaim
> > > > > 2. It can lead to long latencies, preventing users from creating/
> > > > >    destroying other cgroups anywhere else
> > > > > 
> > > > > The patch holds callback_mutex through the duration of cpuset_migrate_mm() and
> > > > > gives up cgroup_mutex while doing so.
> > > > > 
> > > > I agree changing cpuset_migrate_mm not to hold cgroup_mutex to fix the dead lock
> > > > is one choice, and it looks good to me at the first impression.
> > > > 
> > > > But I'm not sure it's good to change cpuset(other subsystem) code because of memcg.
> > > > 
> > > > Anyway, I'll test this patch and report the result tomorrow.
> > > > (Sorry, I don't have enough time today.)
> > > > 
> > > Unfortunately, this patch doesn't seem enough.
> > > 
> > > This patch can fix dead lock caused by "circular lock of cgroup_mutex",
> > > but cannot that of caused by "race between page reclaim and cpuset_attach(mpol_rebind_mm)".
> > > 
> > > (The dead lock I fixed in memcg-avoid-dead-lock-caused-by-race-between-oom-and-cpuset_attach.patch
> > > was caused by "race between memcg's oom and mpol_rebind_mm, and was independent of hierarchy.)
> > > 
> > > I attach logs I got in testing this patch.
> > > 
> > Hmm, ok then, what you  mention to is this race.
> > --
> > 	cgroup_lock()
> > 		-> cpuset_attach()
> > 			-> down_write(&mm->mmap_sem);
> > 
> > 	down_read()
> > 		-> page fault
> > 			-> reclaim in memcg
> > 				-> cgroup_lock().
> > --
> > What this patch tries to fix is this recursive locks
> > --
> > 	cgroup_lock()
> > 		-> cpuset_attach()
> > 			-> cpuset_migrate_mm()
> > 				-> charge to migration
> > 					-> go to reclaim and meet cgroup_lock.
> > --
> > 
> > 
> > Right ?
> > 
> Yes.
> Thank you for explaining in detail.
> 

Sorry, I don't understand the context, I am unable to figure out

1. How to reproduce the problem that Daisuke-San reported
2. Whether the patch is correct or causing more problems or needs more
   stuff to completely fix the race.

> 
> Daisuke Nishimura.
> 
> > BTW, releasing cgroup_lock() while attach() is going on is finally safe ?
> > If not, can this lock for attach be replaced with (new) cgroup private mutex ?
> > 
> > a new mutex like this ?
> > --
> > struct cgroup {
> > 	.....
> > 	mutex_t		attach_mutex; /* for serializing attach() ops. 
> > 					 while attach() is going on, rmdir() will fail */
> > }
> > --
> > Do we need the big lock of cgroup_lock for attach(), at last ?
> > 
> > -Kame
> > 

-- 
	Balbir
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/