| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CACT4Y+Yfje1TQNmx+StxaYKB6wi=DGHVRuF6H_DES=OLiqeq2g@mail.gmail.com>
Date: Thu, 27 Sep 2018 18:53:17 +0200
From: Dmitry Vyukov <dvyukov@...gle.com>
To: Eric Dumazet <edumazet@...gle.com>,
Tetsuo Handa <penguin-kernel@...ove.sakura.ne.jp>
Cc: Daniel Borkmann <daniel@...earbox.net>,
Alexei Starovoitov <ast@...nel.org>,
Network Development <netdev@...r.kernel.org>,
"David S. Miller" <davem@...emloft.net>,
Andrew Morton <akpm@...ux-foundation.org>,
Michal Hocko <mhocko@...nel.org>,
John Johansen <john.johansen@...onical.com>,
syzkaller <syzkaller@...glegroups.com>
Subject: Re: bpf: Massive skbuff_head_cache memory leak?
On Thu, Sep 27, 2018 at 6:47 PM, Dmitry Vyukov <dvyukov@...gle.com> wrote:
> On Thu, Sep 27, 2018 at 12:27 PM, Dmitry Vyukov <dvyukov@...gle.com> wrote:
>> On Thu, Sep 27, 2018 at 1:35 AM, John Johansen
>> <john.johansen@...onical.com> wrote:
>>> On 09/26/2018 02:22 PM, Daniel Borkmann wrote:
>>>> On 09/26/2018 11:09 PM, Tetsuo Handa wrote:
>>>>> Hello, Alexei and Daniel.
>>>>>
>>>>> Can you show us how to run testcases you are testing?
>>>>
>>>> Sorry for the delay; currently quite backlogged but will definitely take a look
>>>> at these reports. Regarding your question: majority of test cases are in the
>>>> kernel tree under selftests, see tools/testing/selftests/bpf/ .
>>>>
>>>
>>> Its unlikely to be apparmor. I went through the reports and saw nothing that
>>> would indicate apparmor involvement, but the primary reason is what is being tested
>>> in upstream apparmor atm.
>>>
>>> The current upstream code does nothing directly with skbuffs. Its
>>> possible that the audit code paths (kernel audit does grab skbuffs)
>>> could, but there are only a couple cases that would be triggered in
>>> the current fuzzing so this seems to be an unlikely source for such a
>>> large leak.
>>
>>
>> Ack. There is no direct evidence against apparmor, I am just trying to
>> get at least some hooks re the root cause.
>>
>> From all the weak indirect evidence, I leaning towards skb allocation
>> in an infinite loop (or a timer with infinite rate).
>>
>>>>> On 2018/09/22 22:25, Tetsuo Handa wrote:
>>>>>> Hello.
>>>>>>
>>>>>> syzbot is reporting many lockup problems on bpf.git / bpf-next.git / net.git / net-next.git trees.
>>>>>>
>>>>>> INFO: rcu detected stall in br_multicast_port_group_expired (2)
>>>>>> https://syzkaller.appspot.com/bug?id=15c7ad8cf35a07059e8a697a22527e11d294bc94
>>>>>>
>>>>>> INFO: rcu detected stall in tun_chr_close
>>>>>> https://syzkaller.appspot.com/bug?id=6c50618bde03e5a2eefdd0269cf9739c5ebb8270
>>>>>>
>>>>>> INFO: rcu detected stall in discover_timer
>>>>>> https://syzkaller.appspot.com/bug?id=55da031ddb910e58ab9c6853a5784efd94f03b54
>>>>>>
>>>>>> INFO: rcu detected stall in ret_from_fork (2)
>>>>>> https://syzkaller.appspot.com/bug?id=c83129a6683b44b39f5b8864a1325893c9218363
>>>>>>
>>>>>> INFO: rcu detected stall in addrconf_rs_timer
>>>>>> https://syzkaller.appspot.com/bug?id=21c029af65f81488edbc07a10ed20792444711b6
>>>>>>
>>>>>> INFO: rcu detected stall in kthread (2)
>>>>>> https://syzkaller.appspot.com/bug?id=6accd1ed11c31110fed1982f6ad38cc9676477d2
>>>>>>
>>>>>> INFO: rcu detected stall in ext4_filemap_fault
>>>>>> https://syzkaller.appspot.com/bug?id=817e38d20e9ee53390ac361bf0fd2007eaf188af
>>>>>>
>>>>>> INFO: rcu detected stall in run_timer_softirq (2)
>>>>>> https://syzkaller.appspot.com/bug?id=f5a230a3ff7822f8d39fddf8485931bd06ae47fe
>>>>>>
>>>>>> INFO: rcu detected stall in bpf_prog_ADDR
>>>>>> https://syzkaller.appspot.com/bug?id=fb4911fd0e861171cc55124e209f810a0dd68744
>>>>>>
>>>>>> INFO: rcu detected stall in __run_timers (2)
>>>>>> https://syzkaller.appspot.com/bug?id=65416569ddc8d2feb8f19066aa761f5a47f7451a
>>>>>>
>>>>>> The cause of lockup seems to be flood of printk() messages from memory allocation
>>>>>> failures, and one of out_of_memory() messages indicates that skbuff_head_cache
>>>>>> usage is huge enough to suspect in-kernel memory leaks.
>>>>>>
>>>>>> [ 1554.547011] skbuff_head_cache 1847887KB 1847887KB
>>>>>>
>>>>>> Unfortunately, we cannot find from logs what syzbot is trying to do
>>>>>> because constant printk() messages is flooding away syzkaller messages.
>>>>>> Can you try running your testcases with kmemleak enabled?
>>>>>>
>>>>>
>>>>> On 2018/09/27 2:35, Dmitry Vyukov wrote:
>>>>>> I also started suspecting Apparmor. We switched to Apparmor on Aug 30:
>>>>>> https://groups.google.com/d/msg/syzkaller-bugs/o73lO4KGh0w/j9pcH2tSBAAJ
>>>>>> Now the instances that use SELinux and Smack explicitly contain that
>>>>>> in the name, but the rest are Apparmor.
>>>>>> Aug 30 roughly matches these assorted "task hung" reports. Perhaps
>>>>>> some Apparmor hook leaks a reference to skbs?
>>>>>
>>>>> Maybe. They have CONFIG_DEFAULT_SECURITY="apparmor". But I'm wondering why
>>>>> this problem is not occurring on linux-next.git when this problem is occurring
>>>>> on bpf.git / bpf-next.git / net.git / net-next.git trees. Is syzbot running
>>>>> different testcases depending on which git tree is targeted?
>>>>>
>>> this is another reason that it is doubtful that its apparmor.
>
> On Thu, Sep 27, 2018 at 2:52 PM, edumazet
>> Have you tried kmemleak perhaps, it might give us a clue, but it seems
>> obvious the leak would be in TX path.
>
> So, I've tried. Now what? :)
>
> I've uploaded all reports to:
> https://drive.google.com/file/d/107LUW0zmYbXmxfQCWoLpeenxXJsXIkxj/view?usp=sharing
> This is on net tree d4ce58082f206bf6e7d697380c7bc5480a8b0264
>
> memory leak in __lookup_hash 33 Sep 27 2018 16:35:50
> memory leak in new_inode_pseudo 43 Sep 27 2018 16:41:14
> memory leak in path_openat 1 Sep 27 2018 16:12:53
> memory leak in rhashtable_init 1 Sep 27 2018 16:41:41
> memory leak in shmem_symlink 4 Sep 27 2018 16:34:34
> memory leak in __anon_vma_prepare 1 Sep 27 2018 17:30:02
> memory leak in __do_execve_file 4 Sep 27 2018 18:14:10
> memory leak in __do_sys_perf_event_open 2 Sep 27 2018 17:40:05
> memory leak in __es_insert_extent 3 Sep 27 2018 17:24:52
> memory leak in __getblk_gfp 3 Sep 27 2018 18:19:30
> memory leak in __handle_mm_fault 1 Sep 27 2018 18:11:31
> memory leak in __hw_addr_create_ex 2 Sep 27 2018 18:10:56
> memory leak in __ip_mc_inc_group 13 Sep 27 2018 18:24:31
> memory leak in __khugepaged_enter 1 Sep 27 2018 15:40:25
> memory leak in __list_lru_init 27 Sep 27 2018 17:30:48
> memory leak in __neigh_create 12 Sep 27 2018 17:40:28
> memory leak in __netlink_create 1 Sep 27 2018 15:40:23
> memory leak in __register_sysctl_table 1 Sep 27 2018 17:36:57
> memory leak in __send_signal 1 Sep 27 2018 18:30:48
> memory leak in __sys_socket 7 Sep 27 2018 15:43:20
> memory leak in anon_inode_getfile 4 Sep 27 2018 18:17:29
> memory leak in bpf_prog_store_orig_filter 1 Sep 27 2018 17:59:14
> memory leak in br_multicast_new_group 2 Sep 27 2018 18:16:42
> memory leak in br_multicast_new_port_group 3 Sep 27 2018 18:17:39
> memory leak in build_sched_domains 2 Sep 27 2018 17:28:55
> memory leak in clone_mnt 2 Sep 27 2018 18:18:48
> memory leak in compute_effective_progs 1 Sep 27 2018 18:35:15
> memory leak in create_empty_buffers 1 Sep 27 2018 17:35:42
> memory leak in create_filter_start 2 Sep 27 2018 18:16:00
> memory leak in create_pipe_files 3 Sep 27 2018 18:21:47
> memory leak in do_ip6t_set_ctl 1 Sep 27 2018 15:40:21
> memory leak in do_ipt_set_ctl 1 Sep 27 2018 17:37:35
> memory leak in do_signalfd4 1 Sep 27 2018 18:00:45
> memory leak in do_syslog 2 Sep 27 2018 18:06:22
> memory leak in ep_insert 11 Sep 27 2018 17:39:14
> memory leak in ep_ptable_queue_proc 3 Sep 27 2018 17:37:17
> memory leak in ext4_mb_new_group_pa 1 Sep 27 2018 17:36:04
> memory leak in ext4_mb_new_inode_pa 3 Sep 27 2018 17:39:19
> memory leak in fdb_create 13 Sep 27 2018 18:24:03
> memory leak in fib6_add_1 11 Sep 27 2018 18:19:53
> memory leak in fib_table_insert 1 Sep 27 2018 17:36:49
> memory leak in find_get_context 1 Sep 27 2018 15:42:00
> memory leak in fsnotify_add_mark_locked 11 Sep 27 2018 17:31:41
> memory leak in idr_get_free 2 Sep 27 2018 18:17:34
> memory leak in iget_locked 1 Sep 27 2018 17:52:59
> memory leak in inet_frag_find 3 Sep 27 2018 18:23:53
> memory leak in inotify_update_watch 25 Sep 27 2018 17:30:59
> memory leak in ioc_create_icq 1 Sep 27 2018 15:42:40
> memory leak in ip6_pol_route 4 Sep 27 2018 18:24:59
> memory leak in ip6_route_info_create 1 Sep 27 2018 17:38:09
> memory leak in ip6t_register_table 1 Sep 27 2018 17:35:39
> memory leak in ip_route_output_key_hash_rcu 4 Sep 27 2018 18:21:21
> memory leak in ipt_register_table 3 Sep 27 2018 17:39:58
> memory leak in ipv6_add_addr 1 Sep 27 2018 17:38:49
> memory leak in load_elf_binary 1 Sep 27 2018 18:09:44
> memory leak in map_create 9 Sep 27 2018 18:25:10
> memory leak in memcg_update_all_list_lrus 1 Sep 27 2018 15:39:36
> memory leak in ndisc_send_rs 1 Sep 27 2018 17:50:03
> memory leak in neigh_table_init 6 Sep 27 2018 17:23:41
> memory leak in nf_hook_entries_grow 5 Sep 27 2018 15:43:41
> memory leak in packet_sendmsg 1 Sep 27 2018 18:08:43
> memory leak in pcpu_create_chunk 1 Sep 27 2018 17:48:41
> memory leak in prepare_creds 18 Sep 27 2018 17:29:10
> memory leak in prepare_kernel_cred 15 Sep 27 2018 18:23:42
> memory leak in process_preds 2 Sep 27 2018 18:11:01
> memory leak in rht_deferred_worker 9 Sep 27 2018 18:24:29
> memory leak in sched_init_domains 2 Sep 27 2018 18:12:42
> memory leak in sctp_addr_wq_mgmt 1 Sep 27 2018 17:46:20
> memory leak in sget 5 Sep 27 2018 18:03:40
> memory leak in shmem_symlink 30 Sep 27 2018 17:32:09
> memory leak in skb_clone 3 Sep 27 2018 18:13:22
> memory leak in submit_bh_wbc 1 Sep 27 2018 17:49:06
> memory leak in tracepoint_probe_register_prio 1 Sep 27 2018 17:39:13
> memory leak in xt_replace_table 4 Sep 27 2018 15:43:19
> memory leak in __delayacct_tsk_init 2 Sep 27 2018 17:02:53
> memory leak in disk_expand_part_tbl 1 Sep 27 2018 16:59:05
> memory leak in do_ip6t_set_ctl 14 Sep 27 2018 15:46:37
> memory leak in neigh_table_init 4 Sep 27 2018 17:10:29
> memory leak in do_check 1 Sep 27 2018 18:38:13
I see at least 3 bridge-related:
memory leak in skb_clone
memory leak in br_multicast_new_group
memory leak in br_multicast_new_port_group
Powered by blists - more mailing lists