Date:	Fri, 14 Jun 2013 07:00:04 +0800
From:	Fengguang Wu <fengguang.wu@...el.com>
To:	fengguang.wu@...el.com, Tejun Heo <tj@...nel.org>
Cc:	containers@...ts.linux-foundation.org, cgroups@...r.kernel.org,
	linux-kernel@...r.kernel.org
Subject: [cgroup_init_early] WARNING: at mm/percpu.c:292 pcpu_mem_zalloc()

Greetings,

I got the below dmesg and the first bad commit is

commit f284b77333b1320491f138b76f11006bbc962569
Author: Tejun Heo <tj@...nel.org>
Date:   Wed Jun 12 13:50:58 2013 -0700

    cgroup: use percpu refcnt for cgroup_subsys_states
    
    A css (cgroup_subsys_state) is how each cgroup is represented to a
    controller.  As such, it can be used in hot paths across the various
    subsystems different controllers are associated with.
    
    One of the common operations is reference counting, which up until now
    has been implemented using a global atomic counter and can have
    significant adverse impact on scalability.  For example, css refcnt
    can be gotten and put multiple times by blkcg for each IO request.
    For highops configurations which try to do as much per-cpu as
    possible, the global frequent refcnting can be very expensive.
    
    In general, given the various hugely diverse paths css's end up being
    used from, we need to make it cheap and highly scalable.  In its
    usage, css refcnting isn't very different from module refcnting.
    
    This patch converts css refcnting to use the recently added
    percpu_ref.  css_get/tryget/put() directly maps to the matching
    percpu_ref operations and the deactivation logic is no longer
    necessary as percpu_ref already has refcnt killing.
    
    The only complication is that as the refcnt is per-cpu,
    percpu_ref_kill() in itself doesn't ensure that further tryget
    operations will fail, which we need to guarantee before invoking
    ->css_offline()'s.  This is resolved collecting kill confirmation
    using percpu_ref_kill_and_confirm() and initiating the offline phase
    of destruction after all css refcnt's are confirmed to be seen as
    killed on all CPUs.  The previous patches already splitted destruction
    into two phases, so percpu_ref_kill_and_confirm() can be hooked up
    easily.
    
    This patch removes css_refcnt() which is used for rcu dereference
    sanity check in css_id().  While we can add a percpu refcnt API to ask
    the same question, css_id() itself is scheduled to be removed fairly
    soon, so let's not bother with it.  Just drop the sanity check and use
    rcu_dereference_raw() instead.
    
    Signed-off-by: Tejun Heo <tj@...nel.org>
    Cc: Michal Hocko <mhocko@...e.cz>
    Cc: Mike Snitzer <snitzer@...hat.com>
    Cc: Vivek Goyal <vgoyal@...hat.com>
    Cc: "Alasdair G. Kergon" <agk@...hat.com>
    Cc: Jens Axboe <axboe@...nel.dk>
    Cc: Mikulas Patocka <mpatocka@...hat.com>
    Cc: Glauber Costa <glommer@...il.com>
    Cc: Kent Overstreet <koverstreet@...gle.com>


[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] ------------[ cut here ]------------
[    0.000000] WARNING: at /c/kernel-tests/src/tip/mm/percpu.c:292 pcpu_mem_zalloc+0x2f/0x80()
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 3.10.0-rc4-00064-gf284b773 #57
[    0.000000]  0000000000000009 ffffffff82133d58 ffffffff81c259e5 ffffffff82133d98
[    0.000000]  ffffffff810818c6 ffffffff82133da8 0000000000000000 0000000000000004
[    0.000000]  0000000000000082 0000000000000004 0000000000000004 ffffffff82133da8
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff81c259e5>] dump_stack+0x19/0x1b
[    0.000000]  [<ffffffff810818c6>] warn_slowpath_common+0x66/0x90
[    0.000000]  [<ffffffff810819b5>] warn_slowpath_null+0x15/0x20
[    0.000000]  [<ffffffff8113f81f>] pcpu_mem_zalloc+0x2f/0x80
[    0.000000]  [<ffffffff8114077d>] pcpu_alloc+0x28d/0xb30
[    0.000000]  [<ffffffff810de45e>] ? lock_acquired+0x2ae/0x2f0
[    0.000000]  [<ffffffff810dd248>] ? trace_hardirqs_off_caller+0x38/0xc0
[    0.000000]  [<ffffffff810ef0c0>] ? cgroup_pidlist_release+0x50/0x50
[    0.000000]  [<ffffffff811410ab>] __alloc_percpu+0xb/0x10
[    0.000000]  [<ffffffff8152d11b>] percpu_ref_init+0x2b/0x60
[    0.000000]  [<ffffffff810ee670>] init_cgroup_css.isra.24+0x30/0xc0
[    0.000000]  [<ffffffff8255b5a6>] cgroup_init_subsys+0x79/0xe9
[    0.000000]  [<ffffffff8255ba91>] cgroup_init_early+0x1af/0x1bf
[    0.000000]  [<ffffffff82541ae1>] start_kernel+0x1e/0x4a8
[    0.000000]  [<ffffffff81c1ad84>] ? memblock_reserve+0x46/0x4b
[    0.000000]  [<ffffffff825415c0>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff825416ae>] x86_64_start_kernel+0xec/0xfb
[    0.000000] ---[ end trace 4eaa2a86a8e2da22 ]---

git bisect start f284b77333b1320491f138b76f11006bbc962569 f722406faae2d073cc1d01063d1123c35425939e --
git bisect good c7153d0643dd161df9e7a2fcb952418055aa5fca  # 09:39     60+  Merge tag 'driver-core-3.10-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core
git bisect good 088d812fe97a4cc8edaed69f91511ea063b14cef  # 09:54     60+  Merge tag 'for-linus-v3.10-rc3' of git://oss.sgi.com/xfs/xfs
git bisect good 977b55cf988b86cdc929fef730a78aca98a8ebdb  # 10:19     60+  Merge tag 'please-pull-aertracefix' of git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras
git bisect good aa3ae6de36c86994c3446bdbc6b69ece9193732a  # 10:33     60+  Merge branch 'merge' of git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc
git bisect good 0ab60871b451b857206d7afc0d3033be866f76b4  # 11:01     60+  Merge tag 'jfs-3.10-rc5' of git://github.com/kleikamp/linux-shaggy
git bisect good de3c0749e2c1960afcc433fc5da136b85c8bd896  # 11:34     60+  percpu-refcount: implement percpu_tryget() along with percpu_ref_kill_and_confirm()
git bisect good cc5943a7816ba6c00639837a62131386619548dc  # 12:04     60+  cgroup: mark "notify_on_release" and "release_agent" cgroup files insane
git bisect good b5f1b7123762411e6a432f8236304e4dcaf5b4d8  # 12:26     60+  cgroup: use kzalloc() and list_del_init()
git bisect good 919e0c4a614fce2fdf8a57e5500841e18bb523e6  # 12:36     60+  cgroup: drop unnecessary RCU dancing from __put_css_set()
git bisect good fbcc713b2b109bd180688b65c3967d5a1cd46010  # 12:52     60+  cgroup: reorder the operations in cgroup_destroy_locked()
git bisect good 903c63033fff6a0f2f60b7c71831fe3b25ea9abc  # 15:34     60+  cgroup: split cgroup destruction into two steps
git bisect good 903c63033fff6a0f2f60b7c71831fe3b25ea9abc  # 19:12    180+  cgroup: split cgroup destruction into two steps
git bisect  bad f284b77333b1320491f138b76f11006bbc962569  # 19:12      0-  cgroup: use percpu refcnt for cgroup_subsys_states
git bisect good a2c764c7dbc632ab6ac48b2d833f44dce26ba1ed  # 22:43    180+  Revert "cgroup: use percpu refcnt for cgroup_subsys_states"
git bisect good 26e04462c8b78d079d3231396ec72d58a14f114b  # 00:32    180+  Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
git bisect good c04efed734409f5a44715b54a6ca1b54b0ccf215  # 03:51    180+  Add linux-next specific files for 20130607

Thanks,
Fengguang

View attachment "dmesg-kvm-lkp-nex04-41895-20130613035323-3.10.0-rc4-00064-gf284b773-57" of type "text/plain" (56228 bytes)

Download attachment "bisect-f284b77333b1320491f138b76f11006bbc962569-x86_64-randconfig-a02-0612-pcpu_mem_zalloc-68419.log" of type "application/octet-stream" (17777 bytes)

View attachment ".config-bisect" of type "text/plain" (76670 bytes)

Powered by blists - more mailing lists