netdev - Re: [Patch net v2] cgroup: fix cgroup_sk_alloc() for sk_clone

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Thu, 9 Jul 2020 10:10:40 -0700
From:   Guenter Roeck <linux@...ck-us.net>
To:     Cong Wang <xiyou.wangcong@...il.com>
Cc:     Linux Kernel Network Developers <netdev@...r.kernel.org>,
        Cameron Berkenpas <cam@...-zeon.de>,
        Peter Geis <pgwipeout@...il.com>,
        Lu Fengqi <lufq.fnst@...fujitsu.com>,
        Daniël Sonck <dsonck92@...il.com>,
        Zhang Qiang <qiang.zhang@...driver.com>,
        Thomas Lamprecht <t.lamprecht@...xmox.com>,
        Daniel Borkmann <daniel@...earbox.net>,
        Zefan Li <lizefan@...wei.com>, Tejun Heo <tj@...nel.org>,
        Roman Gushchin <guro@...com>
Subject: Re: [Patch net v2] cgroup: fix cgroup_sk_alloc() for sk_clone_lock()

On 7/8/20 12:32 PM, Cong Wang wrote:
> On Wed, Jul 8, 2020 at 12:07 PM Guenter Roeck <linux@...ck-us.net> wrote:
>>
>> On 7/8/20 11:34 AM, Cong Wang wrote:
>>> Hi,
>>>
>>> On Wed, Jul 8, 2020 at 8:33 AM Guenter Roeck <linux@...ck-us.net> wrote:
>>>> This patch causes all my s390 boot tests to crash. Reverting it fixes
>>>> the problem. Please see bisect results and and crash log below.
>>>>
>>> ...
>>>> Crash log:
>>>
>>> Interesting. I don't see how unix socket is any special here, it creates
>>> a peer sock with sk_alloc(), but this is not any different from two separated
>>> sockets.
>>>
>>> What is your kernel config? Do you enable CONFIG_CGROUP_NET_PRIO
>>> or CONFIG_CGROUP_NET_CLASSID? I can see there might be a problem
>>> if you don't enable either of them but enable CONFIG_CGROUP_BPF.
>>>
>>
>> cgroup specific configuration bits:
>>
>> CONFIG_CGROUPS=y
>> CONFIG_BLK_CGROUP=y
>> CONFIG_CGROUP_WRITEBACK=y
>> CONFIG_CGROUP_SCHED=y
>> CONFIG_CGROUP_PIDS=y
>> CONFIG_CGROUP_RDMA=y
>> CONFIG_CGROUP_FREEZER=y
>> CONFIG_CGROUP_HUGETLB=y
>> CONFIG_CGROUP_DEVICE=y
>> CONFIG_CGROUP_CPUACCT=y
>> CONFIG_CGROUP_PERF=y
>> CONFIG_CGROUP_BPF=y
>> # CONFIG_CGROUP_DEBUG is not set
>> CONFIG_SOCK_CGROUP_DATA=y
>> CONFIG_BLK_CGROUP_RWSTAT=y
>> CONFIG_BLK_CGROUP_IOLATENCY=y
>> CONFIG_BLK_CGROUP_IOCOST=y
>> # CONFIG_BFQ_CGROUP_DEBUG is not set
>> # CONFIG_NETFILTER_XT_MATCH_CGROUP is not set
>> CONFIG_NET_CLS_CGROUP=y
>> CONFIG_CGROUP_NET_PRIO=y
>> CONFIG_CGROUP_NET_CLASSID=y
>>
>> This originates from arch/s390/configs/defconfig; I don't touch
>> any cgroup specific configuration in my tests.
> 
> Good to know you enable everything related here.
> 
>>
>>> And if you have the full kernel log, it would be helpful too.
>>>
>>
>> https://kerneltests.org/builders/qemu-s390-pending-fixes/builds/222/steps/qemubuildcommand/logs/stdio
> 
> It looks like cgroup_sk_alloc_disabled is always false in your case.
> This makes the bug even more weird. Unless there is a refcnt bug
> prior to my commit, I don't see how it could happen.
> 
>>
>> Interestingly, enabling CONFIG_BFQ_CGROUP_DEBUG makes the problem disappear.
> 
> Yeah, I guess there might be some cgroup refcnt bug which could
> just paper out with CONFIG_BFQ_CGROUP_DEBUG=y.
> 
> If you can test patches, I can send you a debugging patch for you
> to collect more data. I assume this is 100% reproducible on your
> side?
> 

Something seems fishy with the use of skcd->val on big endian systems.

Some debug output:

[   22.643703] sock: ##### sk_alloc(sk=000000001be28100): Calling cgroup_sk_alloc(000000001be28550)
[   22.643807] cgroup: ##### cgroup_sk_alloc(skcd=000000001be28550): cgroup_sk_alloc_disabled=0, in_interrupt: 0
[   22.643886] cgroup:  #### cgroup_sk_alloc(skcd=000000001be28550): cset->dfl_cgrp=0000000001224040, skcd->val=0x1224040
[   22.643957] cgroup: ###### cgroup_bpf_get(cgrp=0000000001224040)
[   22.646451] sock: ##### sk_prot_free(sk=000000001be28100): Calling cgroup_sk_free(000000001be28550)
[   22.646607] cgroup:  #### sock_cgroup_ptr(skcd=000000001be28550) -> 0000000000014040 [v=14040, skcd->val=14040]
[   22.646632] cgroup: ####### cgroup_sk_free(): skcd=000000001be28550, cgrp=0000000000014040
[   22.646739] cgroup: ####### cgroup_sk_free(): skcd->no_refcnt=0
[   22.646814] cgroup: ####### cgroup_sk_free(): Calling cgroup_bpf_put(cgrp=0000000000014040)
[   22.646886] cgroup: ###### cgroup_bpf_put(cgrp=0000000000014040)

Guenter