netdev - Re: 5.15-rc3+ crash in fq-codel?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 27 Sep 2021 17:04:02 -0700
From:   Ben Greear <greearb@...delatech.com>
To:     Eric Dumazet <eric.dumazet@...il.com>,
        netdev <netdev@...r.kernel.org>
Subject: Re: 5.15-rc3+ crash in fq-codel?

On 9/27/21 4:49 PM, Eric Dumazet wrote:
> 
> 
> On 9/27/21 4:30 PM, Ben Greear wrote:
>> Hello,
>>
>> In a hacked upon kernel, I'm getting crashes in fq-codel when doing bi-directional
>> pktgen traffic on top of mac-vlans.  Unfortunately for me, I've made big changes to
>> pktgen so I cannot easily run this test on stock kernels, and there is some chance
>> some of my hackings have caused this issue.
>>
>> But, in case others have seen similar, please let me know.  I shall go digging
>> in the meantime...
>>
>> Looks to me like 'skb' is NULL in line 120 below.
> 
> 
> pktgen must not be used in a mode where a single skb
> is cloned and reused, if packet needs to be stored in a qdisc.
> 
> qdisc of all sorts assume skb->next/prev can be used as
> anchor in their list.
> 
> If the same skb is queued multiple times, lists are corrupted.
> 
> Please double check your clone_skb pktgen setup.
> 
> I thought we had IFF_TX_SKB_SHARING for this, and that macvlan was properly clearing this bit.

My pktgen config was not using any duplicated queueing in this case.

I changed to pfifo fast and so far it is stable for ~10 minutes, where before it would crash
within a minute.  I'll let it bake overnight....

Thanks,
Ben

> 
>>
>> For help, type "help".
>> Type "apropos word" to search for commands related to "word"...
>> Reading symbols from ./net/sched/sch_fq_codel.ko...done.
>> "/home/greearb/kernel/2.6/linux-5.15.x64/vmlinux" is not a core dump: file format not recognized
>> (gdb) l *(fq_codel_enqueue+0x24b)
>> 0x76b is in fq_codel_enqueue (/home/greearb/git/linux-5.15.dev.y/net/sched/sch_fq_codel.c:120).
>> 115    /* remove one skb from head of slot queue */
>> 116    static inline struct sk_buff *dequeue_head(struct fq_codel_flow *flow)
>> 117    {
>> 118        struct sk_buff *skb = flow->head;
>> 119
>> 120        flow->head = skb->next;
>> 121        skb_mark_not_on_list(skb);
>> 122        return skb;
>> 123    }
>> 124
>> (gdb)
>>
>>
>> BUG: kernel NULL pointer dereference, address: 0000000000000000
>> #PF: supervisor read access in kernel mode
>> #PF: error_code(0x0000) - not-present page
>> PGD 0 P4D 0
>> Oops: 0000 [#1] PREEMPT SMP
>> CPU: 3 PID: 2077 Comm: kpktgend_3 Not tainted 5.15.0-rc3+ #2
>> Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/19/2019
>> RIP: 0010:fq_codel_enqueue+0x24b/0x380 [sch_fq_codel]
>> Code: e0 02 48 89 44 24 08 49 c1 e0 06 4c 03 83 50 01 00 00 45 31 f6 45 31 c9 31 c9 89 74 24 10 eb 04 39 fa 73 33 49 8b 00 83 c13
>> RSP: 0018:ffffc9000030fd10 EFLAGS: 00010202
>> RAX: 0000000000000000 RBX: ffff88810a78f600 RCX: 0000000000000032
>> RDX: 00000000000121ca RSI: ffff88812d716900 RDI: 00000000003b26f5
>> RBP: ffffc9000030fd78 R08: ffff8881311dd340 R09: 00000000000121ca
>> R10: 000000000000034d R11: 0000000001680900 R12: ffffc9000030fde0
>> R13: 000000000001b900 R14: 000000000001b900 R15: 0000000000000040
>> FS:  0000000000000000(0000) GS:ffff888265cc0000(0000) knlGS:0000000000000000
>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 0000000000000000 CR3: 000000000260f003 CR4: 00000000003706e0
>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> Call Trace:
>>   dev_qdisc_enqueue+0x35/0x90
>>   __dev_queue_xmit+0x647/0xb70
>>   macvlan_start_xmit+0x4a/0x110 [macvlan]
>>   pktgen_thread_worker+0x19fe/0x20ed [pktgen]
>>   ? wait_woken+0x60/0x60
>>   ? pktgen_rem_all_ifs+0x70/0x70 [pktgen]
>>   kthread+0x11e/0x150
>>   ? set_kthread_struct+0x40/0x40
>>   ret_from_fork+0x1f/0x30
>>
>>
>> Thanks,
>> Ben
>>
> 


-- 
Ben Greear <greearb@...delatech.com>
Candela Technologies Inc  http://www.candelatech.com