Message-ID: <1391a759-845e-11d4-09f4-9bd6e498db4f@gmail.com>
Date:   Thu, 5 Mar 2020 21:28:20 +0800
From:   brookxu <brookxu.cn@...il.com>
To:     Michal Hocko <mhocko@...nel.org>
Cc:     hannes@...xchg.org, vdavydov.dev@...il.com,
        akpm@...ux-foundation.org, cgroups@...r.kernel.org,
        linux-mm@...ck.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH] memcg: fix NULL pointer dereference in
 __mem_cgroup_usage_unregister_event

Sorry, the calltrace in the email is from the old system, i will fix it later. this problem also exists in the latest master branch. The
following is the calltrace corresponding to the latest kernel:

[  135.675108] BUG: kernel NULL pointer dereference, address: 0000000000000004
[  135.675350] #PF: supervisor write access in kernel mode
[  135.675579] #PF: error_code(0x0002) - not-present page
[  135.675816] PGD 800000033058e067 P4D 800000033058e067 PUD 3355ce067 PMD 0
[  135.676080] Oops: 0002 [#1] SMP PTI
[  135.676332] CPU: 2 PID: 14012 Comm: kworker/2:6 Kdump: loaded Not tainted 5.6.0-rc4 #3
[  135.676610] Hardware name: LENOVO 20AWS01K00/20AWS01K00, BIOS GLET70WW (2.24 ) 05/21/2014
[  135.676909] Workqueue: events memcg_event_remove
[  135.677192] RIP: 0010:__mem_cgroup_usage_unregister_event+0xb3/0x190
[  135.677473] Code: c0 31 d2 48 63 ca 48 c1 e1 04 4c 39 64 0e 08 0f 95 c1 83 c2 01 0f b6 c9 01 c8 39 fa 75 e5 85 c0 4d 8b 7d 08 0f 84 ad 00 00 00 <41> 89 47 04 41 c7 07 ff ff ff ff 31 c0 49 8b 4d 00 31 d2 8b 71 04
[  135.677825] RSP: 0018:ffffb47e01c4fe18 EFLAGS: 00010202
[  135.678186] RAX: 0000000000000001 RBX: ffff8bb223a8a000 RCX: 0000000000000001
[  135.678548] RDX: 0000000000000001 RSI: ffff8bb22fb83540 RDI: 0000000000000001
[  135.678912] RBP: ffffb47e01c4fe48 R08: 0000000000000000 R09: 0000000000000010
[  135.679287] R10: 000000000000000c R11: 071c71c71c71c71c R12: ffff8bb226aba880
[  135.679670] R13: ffff8bb223a8a480 R14: 0000000000000000 R15: 0000000000000000
[  135.680066] FS:  0000000000000000(0000) GS:ffff8bb242680000(0000) knlGS:0000000000000000
[  135.680475] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  135.680894] CR2: 0000000000000004 CR3: 000000032c29c003 CR4: 00000000001606e0
[  135.681325] Call Trace:
[  135.681763]  memcg_event_remove+0x32/0x90
[  135.682209]  process_one_work+0x172/0x380
[  135.682657]  worker_thread+0x49/0x3f0
[  135.683111]  kthread+0xf8/0x130
[  135.683570]  ? max_active_store+0x80/0x80
[  135.684034]  ? kthread_bind+0x10/0x10
[  135.684506]  ret_from_fork+0x35/0x40
[  135.684981] Modules linked in: netconsole cmac xt_CHECKSUM xt_MASQUERADE tun bridge stp llc ccm ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep sunrpc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm rtl8192cu rtl_usb rtl8192c_common irqbypass rtlwifi mac80211 uvcvideo snd_hda_codec_realtek crct10dif_pclmul snd_hda_codec_hdmi snd_hda_codec_generic iTCO_wdt crc32_pclmul wmi_bmof mei_wdt iTCO_vendor_support snd_hda_intel btusb videobuf2_vmalloc ghash_clmulni_intel snd_intel_dspcfg btrtl videobuf2_memops snd_hda_codec btbcm videobuf2_v4l2 btintel snd_hda_core videobuf2_common bluetooth aesni_intel crypto_simd cryptd cfg80211
videodev
[  135.685003]  snd_hwdep glue_helper snd_seq mc ecdh_generic libarc4 snd_seq_device ecc pcspkr joydev snd_pcm thinkpad_acpi snd_timer ledtrig_audio sg i2c_i801 lpc_ich wmi snd rfkill mei_me soundcore mei ie31200_edac sch_fq_codel ip_tables xfs libcrc32c sr_mod sd_mod cdrom t10_pi i915 i2c_algo_bit drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt serio_raw fb_sys_fops drm e1000e ahci libahci video libata ptp pps_core dm_mirror dm_region_hash dm_log dm_mod
[  135.689733] CR2: 0000000000000004

We can reproduce this problem in the following ways：

1. We create a new cgroup subdirectory and a new eventfd, and then we monitor multiple memory thresholds of the cgroup through this eventfd
2. closing this eventfd, the system will call __mem_cgroup_usage_unregister_event () multiple times to delete all events related to this eventfd.

The first time __mem_cgroup_usage_unregister_event () is called, the system will clear all items related to this eventfd in thresholds-> primary.
Since there is currently only one eventfd, thresholds-> primary becomes empty, so the system will set thresholds-> primary and hresholds-> spare
to NULL. If at this time, the user creates a new eventfd and monitor the memory threshold of this cgroup, then the system will re-initialize
thresholds-> primary. Then when the system call __mem_cgroup_usage_unregister_event () is called for the second time, because thresholds-> primary
is not empty, the system will access thresholds-> spare, but thresholds-> spare is NULL, which will trigger a crash.

In general, the longer it takes to delete all events related to this eventfd, the easier it is to trigger this problem.

thanks

Michal Hocko wrote on 2020/3/5 17:24:
> Thank you for the report!
>
> On Thu 05-03-20 13:52:03, brookxu wrote:
>> One eventfd monitors multiple memory thresholds of cgroup, closing it, the
>> system will delete related events. Before all events are deleted, another
>> eventfd monitors the cgroup's memory threshold.
> Could you describe the race scenario please? Ideally 
>> As a result, thresholds->primary[] is not empty, but thresholds->sparse[]
>> is NULL, __mem_cgroup_usage_unregister_event() leading to a crash:
>>
>> [  138.925809] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
>> [  138.926817] IP: [<ffffffff8116c9b7>] mem_cgroup_usage_unregister_event+0xd7/0x1f0
>> [  138.927701] PGD 73bce067 PUD 76ff3067 PMD 0
>> [  138.928384] Oops: 0002 [#1] SMP
>> [  138.935218] CPU: 1 PID: 14 Comm: kworker/1:0 Not tainted 3.10.107-1-tlinux2-0047 #1
> Also you seem to be running a very old kernel. Does the problem exist in
> the current Vanilla kernel?

Powered by blists - more mailing lists