Date: Thu, 12 Oct 2023 19:49:54 +0300
From: Dmitry Kravkov <dmitryk@...lt.com>
To: Kuniyuki Iwashima <kuniyu@...zon.com>
Cc: edumazet@...gle.com, netdev@...r.kernel.org, slavas@...lt.com
Subject: Re: kernel BUG at net/ipv4/tcp_output.c:2642 with kernel 5.19.0-rc2
 and newer

On Thu, Oct 12, 2023 at 12:19 AM Dmitry Kravkov <dmitryk@...lt.com> wrote:
>
> On Wed, Oct 11, 2023 at 11:54 PM Kuniyuki Iwashima <kuniyu@...zon.com> wrote:
> >
> > From: Dmitry Kravkov <dmitryk@...lt.com>
> > Date: Wed, 11 Oct 2023 23:20:10 +0300
> > > On Wed, Oct 11, 2023 at 5:02 PM Eric Dumazet <edumazet@...gle.com> wrote:
> > > >
> > > > On Wed, Oct 11, 2023 at 12:28 PM Dmitry Kravkov <dmitryk@...lt.com> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > In our try to upgrade from 5.10 to 6.1 kernel we noticed stable crash
> > > > > in kernel that bisected to this commit:
> > > > >
> > > > > commit 849b425cd091e1804af964b771761cfbefbafb43
> > > > > Author: Eric Dumazet <edumazet@...gle.com>
> > > > > Date:   Tue Jun 14 10:17:34 2022 -0700
> > > > >
> > > > >     tcp: fix possible freeze in tx path under memory pressure
> > > > >
> > > > >     Blamed commit only dealt with applications issuing small writes.
> > > > >
> > > > >     Issue here is that we allow to force memory schedule for the sk_buff
> > > > >     allocation, but we have no guarantee that sendmsg() is able to
> > > > >     copy some payload in it.
> > > > >
> > > > >     In this patch, I make sure the socket can use up to tcp_wmem[0] bytes.
> > > > >
> > > > >     For example, if we consider tcp_wmem[0] = 4096 (default on x86),
> > > > >     and initial skb->truesize being 1280, tcp_sendmsg() is able to
> > > > >     copy up to 2816 bytes under memory pressure.
> > > > >
> > > > >     Before this patch a sendmsg() sending more than 2816 bytes
> > > > >     would either block forever (if persistent memory pressure),
> > > > >     or return -EAGAIN.
> > > > >
> > > > >     For bigger MTU networks, it is advised to increase tcp_wmem[0]
> > > > >     to avoid sending too small packets.
> > > > >
> > > > >     v2: deal with zero copy paths.
> > > > >
> > > > >     Fixes: 8e4d980ac215 ("tcp: fix behavior for epoll edge trigger")
> > > > >     Signed-off-by: Eric Dumazet <edumazet@...gle.com>
> > > > >     Acked-by: Soheil Hassas Yeganeh <soheil@...gle.com>
> > > > >     Reviewed-by: Wei Wang <weiwan@...gle.com>
> > > > >     Reviewed-by: Shakeel Butt <shakeelb@...gle.com>
> > > > >     Signed-off-by: David S. Miller <davem@...emloft.net>
> > > > >
> > > > > This happens in a pretty stressful situation when two 100Gb (E810 or
> > > > > ConnectX6) ports transmit above 150Gbps that most of the data is read
> > > > > from disks. So it appears that the system is constantly in a memory
> > > > > deficit. Apparently reverting the patch in 6.1.38 kernel eliminates
> > > > > the crash and system appears stable at delivering 180Gbps
> > > > >
> > > > > [ 2445.532318] ------------[ cut here ]------------
> > > > > [ 2445.532323] kernel BUG at net/ipv4/tcp_output.c:2642!
> > > > > [ 2445.532334] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
> > > > > [ 2445.550934] CPU: 61 PID: 109767 Comm: nginx Tainted: G S         OE
> >
> > It seems 3rd party module is loaded.
> >
> > Just curious if it is possible to reproduce the issue without
> > out-of-tree modules.
> Not sure if ice driver is mature enough there. We will give it a try. Thanks

Happens on not-tained kernel too (we went for 6.1.38 to have more mature ice)

[ 1057.099780] ------------[ cut here ]------------
[ 1057.100389] RSP: 0018:ffffaa4e10093df0 EFLAGS: 00010286
[ 1057.101060] kernel BUG at net/ipv4/tcp_output.c:2645!
[ 1057.122021] RAX: 00000000ffff4000 RBX: ffff8ccad77e3540 RCX: 0000000000000000
[ 1057.122025] RDX: 0000000000000000 RSI: 00000000ffff4000 RDI: ffff8ccad77e3540
[ 1057.122027] RBP: ffff8ccad77e3480 R08: ffff8ccad77e35d4 R09: 0000000080400013
[ 1057.122029] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8ccad77e3480
[ 1057.122031] R13: 7fffffffffffff00 R14: ffff8ccad77e3698 R15: 0000000000000000
[ 1057.122033] FS:  00007fd600d42840(0000) GS:ffff8ce1ffac0000(0000)
knlGS:0000000000000000
[ 1057.122035] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1057.122038] CR2: 00007fd57dacdc80 CR3: 00000041dda7a005 CR4: 0000000000770ee0
[ 1057.122041] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1057.122042] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1057.122044] PKRU: 55555554
[ 1057.122046] Call Trace:
[ 1057.122880] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[ 1057.123683]  <TASK>
[ 1057.124409] CPU: 112 PID: 51072 Comm: nginx Not tainted 6.1.38 #27
[ 1057.125125]  ? show_trace_log_lvl+0x1c4/0x2df
[ 1057.125812] Hardware name: Cisco Systems Inc
UCSC-C220-M6N/UCSC-C220-M6N, BIOS C220M6.4.2.1g.0.1121212157
11/21/2021
[ 1057.125815] RIP: 0010:tcp_write_xmit+0x70f/0x830
[ 1057.126559]  ? show_trace_log_lvl+0x1c4/0x2df



> >
> >
> > > > >     5.19.0-rc2+ #21
> > > > > [ 2445.560127] ------------[ cut here ]------------
> > > > > [ 2445.560565] Hardware name: Cisco Systems Inc
> > > > > UCSC-C220-M6N/UCSC-C220-M6N, BIOS C220M6.4.2.1g.0.1121212157
> > > > > 11/21/2021
> > > > > [ 2445.560571] RIP: 0010:tcp_write_xmit+0x70b/0x830
> > > > > [ 2445.561221] kernel BUG at net/ipv4/tcp_output.c:2642!
> > > > > [ 2445.561821] Code: 84 0b fc ff ff 0f b7 43 32 41 39 c6 0f 84 fe fb
> > > > > ff ff 8b 43 70 41 39 c6 0f 82 ff 00 00 00 c7 43 30 01 00 00 00 e9 e6
> > > > > fb ff ff <0f> 0b 8b 74 24 20 8b 85 dc 05 00 00 44 89 ea 01 c8 2b 43 28
> > > > > 41 39
> > > > > [ 2445.561828] RSP: 0000:ffffc110ed647dc0 EFLAGS: 00010246
> > > > > [ 2445.561832] RAX: 0000000000000000 RBX: ffff9fe1f8081a00 RCX: 00000000000005a8
> > > > > [ 2445.561833] RDX: 000000000000043a RSI: 000002389172f8f4 RDI: 000000000000febf
> > > > > [ 2445.561835] RBP: ffff9fe5f864e900 R08: 0000000000000000 R09: 0000000000000100
> > > > > [ 2445.561836] R10: ffffffff9be060d0 R11: 000000000000000e R12: ffff9fe5f864e901
> > > > > [ 2445.561837] R13: 0000000000000001 R14: 00000000000005a8 R15: 0000000000000000
> > > > > [ 2445.561839] FS:  00007f342530c840(0000) GS:ffff9ffa7f940000(0000)
> > > > > knlGS:0000000000000000
> > > > > [ 2445.561842] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > [ 2445.561844] CR2: 00007f20ca4ed830 CR3: 00000045d976e005 CR4: 0000000000770ee0
> > > > > [ 2445.561846] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > > [ 2445.561847] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > > > > [ 2445.561849] PKRU: 55555554
> > > > > [ 2445.561853] Call Trace:
> > > > > [ 2445.561858]  <TASK>
> > > > > [ 2445.564202] ------------[ cut here ]------------
> > > > > [ 2445.568007]  ? tcp_tasklet_func+0x120/0x120
> > > > > [ 2445.569107] kernel BUG at net/ipv4/tcp_output.c:2642!
> > > > > [ 2445.569608]  tcp_tsq_handler+0x7c/0xa0
> > > > > [ 2445.569627]  tcp_pace_kick+0x19/0x60
> > > > > [ 2445.569632]  __run_hrtimer+0x5c/0x1d0
> > > > > [ 2445.572264] ------------[ cut here ]------------
> > > > > [ 2445.574287] ------------[ cut here ]------------
> > > > > [ 2445.574292] kernel BUG at net/ipv4/tcp_output.c:2642!
> > > > > [ 2445.582581]  __hrtimer_run_queues+0x7d/0xe0
> > > > > --
> > > > > --
> > > > >
> > > > > --
> > > > > --
> > > > >
> > > > > Dmitry Kravkov     Software  Engineer
> > > > > Qwilt | Mobile: +972-54-4839923 | dmitryk@...lt.com
> > > >
> > > > Hi Dmitry, thanks for the report.
> > > >
> > > > Can you post content of /proc/sys/net/ipv4/tcp_wmem and
> > > > /proc/sys/net/ipv4/tcp_rmem ?
> > > Thank you, Eric
> > >
> > > # cat /proc/sys/net/ipv4/tcp_wmem
> > > 786432 1048576 6291456
> > > # cat /proc/sys/net/ipv4/tcp_rmem
> > > 4096 87380 6291456
> > >
> > > >
> > > > Are you using memcg ?
> > > No
> > > >
> > > > Can you try the following patch ?
> > > >
> > > > diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
> > > > index 3f66cdeef7decb5b5d2b84212c623781b8ce63db..d74b197e02e94aa2f032f2c3971969e604abc7de
> > > > 100644
> > > > --- a/net/ipv4/tcp.c
> > > > +++ b/net/ipv4/tcp.c
> > > > @@ -1286,6 +1286,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct
> > > > msghdr *msg, size_t size)
> > > >                 continue;
> > > >
> > > >  wait_for_space:
> > > > +               tcp_remove_empty_skb(sk);
> > > >                 set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
> > > >                 if (copied)
> > > >                         tcp_push(sk, flags & ~MSG_MORE, mss_now,
> > >
> > >
> > > The patched kernel crashed in the same manner:
> > > [ 2214.154278] kernel BUG at net/ipv4/tcp_output.c:2642!
> >
>
>
> --
> --
>
> Dmitry Kravkov     Software  Engineer
> Qwilt | Mobile: +972-54-4839923 | dmitryk@...lt.com



-- 
--

Dmitry Kravkov     Software  Engineer
Qwilt | Mobile: +972-54-4839923 | dmitryk@...lt.com

Powered by blists - more mailing lists