Message-ID: <e37bdc6639d5480fa928a40718630705@huawei.com>
Date:   Fri, 4 Nov 2022 01:39:48 +0000
From:   Caowangbao <caowangbao@...wei.com>
To:     Xin Long <lucien.xin@...il.com>
CC:     "Chenzhen(EulerOS)" <chenzhen126@...wei.com>,
        "vyasevich@...il.com" <vyasevich@...il.com>,
        "nhorman@...driver.com" <nhorman@...driver.com>,
        "marcelo.leitner@...il.com" <marcelo.leitner@...il.com>,
        "linux-sctp@...r.kernel.org" <linux-sctp@...r.kernel.org>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "edumazet@...gle.com" <edumazet@...gle.com>,
        "kuba@...nel.org" <kuba@...nel.org>,
        "pabeni@...hat.com" <pabeni@...hat.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "Yanan (Euler)" <yanan@...wei.com>
Subject: 答复: BUG: kernel NULL pointer dereference in sctp_sched_dequeue_common

It already works.
Thank you very much.

-----邮件原件-----
发件人: Xin Long [mailto:lucien.xin@...il.com] 
发送时间: 2022年11月4日 3:37
收件人: Caowangbao <caowangbao@...wei.com>
抄送: Chenzhen(EulerOS) <chenzhen126@...wei.com>; vyasevich@...il.com; nhorman@...driver.com; marcelo.leitner@...il.com; linux-sctp@...r.kernel.org; davem@...emloft.net; edumazet@...gle.com; kuba@...nel.org; pabeni@...hat.com; netdev@...r.kernel.org; Yanan (Euler) <yanan@...wei.com>
主题: Re: BUG: kernel NULL pointer dereference in sctp_sched_dequeue_common

On Thu, Nov 3, 2022 at 5:53 AM Caowangbao <caowangbao@...wei.com> wrote:
>
> I have reduced the recurrence conditions and can reproduce the problem by running the following statement:
>
> 18:00:56 executing program 0:
> r0 = socket$inet6_sctp(0xa, 0x1, 0x84) 
> setsockopt$inet_sctp6_SCTP_SOCKOPT_BINDX_ADD(r0, 0x84, 0x64, 
> &(0x7f00000001c0)=[@in={0x2, 0x4e20, @empty}], 0x10) (async) 
> getsockopt$inet_sctp6_SCTP_SOCKOPT_CONNECTX3(r0, 0x84, 0x6f, 
> &(0x7f0000000580)={<r1=>0x0, 0x10, &(0x7f0000000540)=[@in={0x2, 
> 0x4e20, @local}]}, &(0x7f0000000600)=0x10) (async)
> r2 = dup2(r0, r0)
> setsockopt$inet_sctp6_SCTP_DEFAULT_PRINFO(r2, 0x84, 0x72, 
> &(0x7f0000000000)={0x0, 0x6, 0x30}, 0xc) (async)
Thanks for the statements.

The crash seems related to SCTP_PR_SCTP_PRIO.
When there is not enough buffer for the out msg, it will prune the out_chunk_list according to the priority set by SCTP_DEFAULT_PRINFO in sctp_prsctp_prune_unsent(). However, it doesn't clear
asoc->stream.out_curr if all frag chunks of current msg are pruned.

Can you apply this patch to your kernel and give it try?

diff --git a/net/sctp/outqueue.c b/net/sctp/outqueue.c index e213aaf45d67..41b8065cfe65 100644
--- a/net/sctp/outqueue.c
+++ b/net/sctp/outqueue.c
@@ -384,6 +384,7 @@ static int sctp_prsctp_prune_unsent(struct sctp_association *asoc,  {
        struct sctp_outq *q = &asoc->outqueue;
        struct sctp_chunk *chk, *temp;
+       struct sctp_stream_out *sout;

        q->sched->unsched_all(&asoc->stream);

@@ -398,12 +399,12 @@ static int sctp_prsctp_prune_unsent(struct sctp_association *asoc,
                sctp_sched_dequeue_common(q, chk);
                asoc->sent_cnt_removable--;
                asoc->abandoned_unsent[SCTP_PR_INDEX(PRIO)]++;
-               if (chk->sinfo.sinfo_stream < asoc->stream.outcnt) {
-                       struct sctp_stream_out *streamout =
-                               SCTP_SO(&asoc->stream, chk->sinfo.sinfo_stream);

-                       streamout->ext->abandoned_unsent[SCTP_PR_INDEX(PRIO)]++;
-               }
+               sout = SCTP_SO(&asoc->stream, chk->sinfo.sinfo_stream);
+               sout->ext->abandoned_unsent[SCTP_PR_INDEX(PRIO)]++;
+               if (asoc->stream.out_curr == sout &&
+                   list_is_last(&chk->frag_list, &chk->msg->chunks))
+                       asoc->stream.out_curr = NULL; /* clear
out_curr if all frag chunks are pruned */

                msg_len -= chk->skb->truesize + sizeof(struct sctp_chunk);
                sctp_chunk_free(chk);

Thanks.

> sendmmsg$sock(r2, ...) (async)
> r3 = socket$inet6_sctp(0xa, 0x1, 0x84)
> r4 = dup2(r2, r3)
> setsockopt$inet_sctp6_SCTP_DEFAULT_PRINFO(r4, 0x84, 0x72, 
> &(0x7f0000000040), 0xc) sendmsg$alg(r4, &(0x7f0000002700)={0x0, 0x0, 
> &(0x7f0000002500)=[{&(0x7f0000004100)='@', 0x1}], 0x1}, 0x0) (async) 
> setsockopt$inet_sctp6_SCTP_DEFAULT_SNDINFO(r2, 0x84, 0x22, 
> &(0x7f0000002480)={0x5, 0x202, 0x9, 0x10001, r1}, 0x10) (async) 
> write$tun(r4, 
> &(0x7f0000002640)=ANY=[@ANYBLOB="000000000000000000000000000006000000a
> aaaaaaaaabb467219e67b1d64441845437b38c12ddeb986e59e82bd4247f1ed8a05309
> a31b9494bda521ffd4b68bf072b030d7ef04cc219c73572fac79f47369d49ae19df016
> 41921e3af34cb84766ede45e4fa9a14460fae51557f643d108ba54f7cb8440ce5aa0e6
> 0d7c2c4da"], 0x1e) (async) write(r0, &(0x7f0000000080)="e4", 0x1)
>
>
>
> -----邮件原件-----
> 发件人: Caowangbao
> 发送时间: 2022年11月3日 10:13
> 收件人: 'Xin Long' <lucien.xin@...il.com>; Chenzhen(EulerOS) 
> <chenzhen126@...wei.com>
> 抄送: 'vyasevich@...il.com' <vyasevich@...il.com>; 
> 'nhorman@...driver.com' <nhorman@...driver.com>; 
> 'marcelo.leitner@...il.com' <marcelo.leitner@...il.com>; 
> 'linux-sctp@...r.kernel.org' <linux-sctp@...r.kernel.org>; 
> 'davem@...emloft.net' <davem@...emloft.net>; 'edumazet@...gle.com' 
> <edumazet@...gle.com>; 'kuba@...nel.org' <kuba@...nel.org>; 
> 'pabeni@...hat.com' <pabeni@...hat.com>; 'netdev@...r.kernel.org' 
> <netdev@...r.kernel.org>; Yanan (Euler) <yanan@...wei.com>
> 主题: 答复: BUG: kernel NULL pointer dereference in 
> sctp_sched_dequeue_common
>
> It can be reproduce by the command " ./syz-execprog -procs=16 -repeat=0 sctp_sched_dequeue_common" with the attachments.
>
> void sctp_sched_dequeue_common(struct sctp_outq *q, struct sctp_chunk *ch) {
>         list_del_init(&ch->list);
>         list_del_init(&ch->stream_list);
>         q->out_qlen -= ch->skb->len;                    // ch->skb is null in the VMCore
> }
>
> The kernel log records：
>         [23411.786575] list_del corruption, ffffa035ddf01c18->next is NULL
>         [23411.787780] WARNING: CPU: 1 PID: 250682 at lib/list_debug.c:49 __list_del_entry_valid+0x59/0xe0
>         ******
>         [23411.830256] Call Trace:
>         [23411.830863]  sctp_sched_dequeue_common+0x17/0x70 [sctp]
>         [23411.831940]  sctp_sched_fcfs_dequeue+0x37/0x50 [sctp]
>         [23411.832967]  sctp_outq_flush_data+0x85/0x360 [sctp] It means "ch->list" has no element.
>
> And in VMCore , there are many calls like:
>         #2 [ffffaf7d84f6bbb8] __lock_sock at ffffffff8ac74ef9
>         #3 [ffffaf7d84f6bc08] lock_sock_nested at ffffffff8ac74f92
>         #4 [ffffaf7d84f6bc20] sctp_wait_for_sndbuf at ffffffffc0c8d9d2 [sctp]
>         #5 [ffffaf7d84f6bc98] sctp_sendmsg_to_asoc at ffffffffc0c8dd1e [sctp]
>         #6 [ffffaf7d84f6bd08] sctp_sendmsg at ffffffffc0c95f55 [sctp]
>         #7 [ffffaf7d84f6bdb8] sock_sendmsg at ffffffff8ac6fd0b
>         #8 [ffffaf7d84f6bdd0] sock_write_iter at ffffffff8ac6fdb7
>         #9 [ffffaf7d84f6be48] new_sync_write at ffffffff8a784021
>         #10 [ffffaf7d84f6bed0] vfs_write at ffffffff8a784d07
>         #11 [ffffaf7d84f6bf08] ksys_write at ffffffff8a78719b
>         #12 [ffffaf7d84f6bf40] do_syscall_64 at ffffffff8ae9a8b3 It may have something to do with these concurrent invocations.
>
> -----邮件原件-----
> 发件人: Xin Long [mailto:lucien.xin@...il.com]
> 发送时间: 2022年11月3日 9:20
> 收件人: Chenzhen(EulerOS) <chenzhen126@...wei.com>
> 抄送: vyasevich@...il.com; nhorman@...driver.com; 
> marcelo.leitner@...il.com; linux-sctp@...r.kernel.org; 
> davem@...emloft.net; edumazet@...gle.com; kuba@...nel.org; 
> pabeni@...hat.com; netdev@...r.kernel.org; Caowangbao 
> <caowangbao@...wei.com>; Yanan (Euler) <yanan@...wei.com>
> 主题: Re: BUG: kernel NULL pointer dereference in 
> sctp_sched_dequeue_common
>
> On Wed, Nov 2, 2022 at 10:29 AM Zhen Chen <chenzhen126@...wei.com> wrote:
> >
> > Hi,all
> >
> > We found the following crash when running fuzz tests on stable-5.10.
> >
> > ------------[ cut here ]------------ list_del corruption, 
> > ffffa035ddf01c18->next is NULL
> > WARNING: CPU: 1 PID: 250682 at lib/list_debug.c:49 __list_del_entry_valid+0x59/0xe0
> > CPU: 1 PID: 250682 Comm: syz-executor.7 Kdump: loaded Tainted: G           O
> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > rel-1.10.2-0-g5f4c7b1-20181220_000000-szxrtosci10000 04/01/2014
> > RIP: 0010:__list_del_entry_valid+0x59/0xe0
> > Code: c0 74 5a 4d 8b 00 49 39 f0 75 6a 48 8b 52 08 4c 39 c2 75 79 b8
> > 01 00 00 00 c3 cc cc cc cc 48 c7 c7 68 ae 78 8b e8 d2 3d 4e 00 <0f> 
> > 0b
> > 31 c0 c3 cc cc cc cc 48 c7 c7 90 ae 78 8b e8 bd 3d 4e 00 0f
> > RSP: 0018:ffffaf7d84a57930 EFLAGS: 00010286
> > RAX: 0000000000000000 RBX: ffffa035ddf01c18 RCX: 0000000000000000
> > RDX: ffffa035facb0820 RSI: ffffa035faca0410 RDI: ffffa035faca0410
> > RBP: ffffa035dddff6f8 R08: 0000000000000000 R09: ffffaf7d84a57770
> > R10: ffffaf7d84a57768 R11: ffffffff8bddc248 R12: ffffa035ddf01c18
> > R13: ffffaf7d84a57af8 R14: ffffaf7d84a57c28 R15: 0000000000000000
> > FS:  00007fb7353ae700(0000) GS:ffffa035fac80000(0000)
> > knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 00007f509a3d0ee8 CR3: 000000010f7c2001 CR4: 00000000001706e0 
> > Call
> > Trace:
> >  sctp_sched_dequeue_common+0x17/0x70 [sctp]
> >  sctp_sched_fcfs_dequeue+0x37/0x50 [sctp]
> >  sctp_outq_flush_data+0x85/0x360 [sctp]
> >  sctp_outq_uncork+0x77/0xa0 [sctp]
> >  sctp_cmd_interpreter.constprop.0+0x164/0x1450 [sctp]
> >  sctp_side_effects+0x37/0xe0 [sctp]
> >  sctp_do_sm+0xd0/0x230 [sctp]
> >  sctp_primitive_SEND+0x2f/0x40 [sctp]
> >  sctp_sendmsg_to_asoc+0x3fa/0x5c0 [sctp]
> >  sctp_sendmsg+0x3d5/0x440 [sctp]
> >  sock_sendmsg+0x5b/0x70
> >  sock_write_iter+0x97/0x100
> >  new_sync_write+0x1a1/0x1b0
> >  vfs_write+0x1b7/0x250
> >  ksys_write+0xab/0xe0
> >  do_syscall_64+0x33/0x40
> >  entry_SYSCALL_64_after_hwframe+0x61/0xc6
> > RIP: 0033:0x461e3d
> > Code: 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 89 f8 48
> > 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 
> > 3d
> > 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
> > RSP: 002b:00007fb7353adc08 EFLAGS: 00000246 ORIG_RAX: 
> > 0000000000000001
> > RAX: ffffffffffffffda RBX: 000000000058c1d0 RCX: 0000000000461e3d
> > RDX: 000000000000001e RSI: 0000000020002640 RDI: 0000000000000004
> > RBP: 000000000058c1d0 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> > R13: 00007fb7353ae700 R14: 00007ffc4c20ce00 R15: 0000000000000fff 
> > ---[ end trace 332cf75246d5ba68 ]---
> > BUG: kernel NULL pointer dereference, address: 0000000000000070
> > #PF: supervisor read access in kernel mode
> > #PF: error_code(0x0000) - not-present page PGD 800000010c0d4067 P4D
> > 800000010c0d4067 PUD 10f275067 PMD 0
> > Oops: 0000 [#1] SMP PTI
> > CPU: 1 PID: 250682 Comm: syz-executor.7 Kdump: loaded Tainted: G        W  O
> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > rel-1.10.2-0-g5f4c7b1-20181220_000000-szxrtosci10000 04/01/2014
> > RIP: 0010:sctp_sched_dequeue_common+0x5c/0x70 [sctp]
> > Code: 5b 08 4c 89 e7 e8 44 c5 cc c9 84 c0 74 0f 48 8b 53 18 48 8b 43
> > 20 48 89 42 08 48 89 10 48 8b 43 38 4c 89 63 18 4c 89 63 20 5b <8b> 
> > 40
> > 70 29 45 20 5d 41 5c c3 cc cc cc cc 66 0f 1f 44 00 00 0f 1f
> > RSP: 0018:ffffaf7d84a57940 EFLAGS: 00010202
> > RAX: 0000000000000000 RBX: ffffaf7d84a579a0 RCX: 0000000000000000
> > RDX: ffffa035ddf01c30 RSI: ffffa035ddf01c30 RDI: ffffa035ddf01c30
> > RBP: ffffa035dddff6f8 R08: ffffa035ddf01c30 R09: ffffaf7d84a57770
> > R10: ffffaf7d84a57768 R11: ffffffff8bddc248 R12: ffffa035ddf01c30
> > R13: ffffaf7d84a57af8 R14: ffffaf7d84a57c28 R15: 0000000000000000
> > FS:  00007fb7353ae700(0000) GS:ffffa035fac80000(0000)
> > knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000000000000070 CR3: 000000010f7c2001 CR4: 00000000001706e0 
> > Call
> > Trace:
> >  sctp_sched_fcfs_dequeue+0x37/0x50 [sctp]
> >  sctp_outq_flush_data+0x85/0x360 [sctp]
> >  sctp_outq_uncork+0x77/0xa0 [sctp]
> >  sctp_cmd_interpreter.constprop.0+0x164/0x1450 [sctp]
> >  sctp_side_effects+0x37/0xe0 [sctp]
> >  sctp_do_sm+0xd0/0x230 [sctp]
> >  sctp_primitive_SEND+0x2f/0x40 [sctp]
> >  sctp_sendmsg_to_asoc+0x3fa/0x5c0 [sctp]
> >  sctp_sendmsg+0x3d5/0x440 [sctp]
> >  sock_sendmsg+0x5b/0x70
> >  sock_write_iter+0x97/0x100
> >  new_sync_write+0x1a1/0x1b0
> >  vfs_write+0x1b7/0x250
> >  ksys_write+0xab/0xe0
> >  do_syscall_64+0x33/0x40
> >  entry_SYSCALL_64_after_hwframe+0x61/0xc6
> > RIP: 0033:0x461e3d
> > Code: 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 89 f8 48
> > 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 
> > 3d
> > 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
> > RSP: 002b:00007fb7353adc08 EFLAGS: 00000246 ORIG_RAX: 
> > 0000000000000001
> > RAX: ffffffffffffffda RBX: 000000000058c1d0 RCX: 0000000000461e3d
> > RDX: 000000000000001e RSI: 0000000020002640 RDI: 0000000000000004
> > RBP: 000000000058c1d0 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> > R13: 00007fb7353ae700 R14: 00007ffc4c20ce00 R15: 0000000000000fff
> >
> >
> > It is quite similar to the issue (See 
> > https://lore.kernel.org/all/CAO4mrfcB0d+qbwtfndzqcrL+QEQgfOmJYQMAdzw
> > xR ePmP8TY1A@...l.gmail.com/ ) , which was addressed by 181d8d2066c0
> > (sctp: leave the err path free in sctp_stream_init to 
> > sctp_stream_free), but unfortunately the patch do not work with this 
> > bug :(
> >
> So this issue is reproducible in your env?
> Can you show what it does in your test or the reproducer if there is one?
>
> Thanks.

Powered by blists - more mailing lists