Message-ID: <CADxym3axi_eNtZyP_+6NbxgHaKinBr+=rGvn-LmXnuCxEaHNuA@mail.gmail.com>
Date: Mon, 26 Jun 2023 21:50:31 +0800
From: Menglong Dong <menglong8.dong@...il.com>
To: Eric Dumazet <edumazet@...gle.com>
Cc: ncardwell@...gle.com, davem@...emloft.net, kuba@...nel.org, 
	pabeni@...hat.com, corbet@....net, dsahern@...nel.org, kuniyu@...zon.com, 
	morleyd@...gle.com, imagedong@...cent.com, mfreemon@...udflare.com, 
	mubashirq@...gle.com, netdev@...r.kernel.org, linux-doc@...r.kernel.org, 
	linux-kernel@...r.kernel.org
Subject: Re: [PATCH net-next] net: tcp: support to probe tcp receiver OOM

On Mon, Jun 26, 2023 at 9:27 PM Eric Dumazet <edumazet@...gle.com> wrote:
>
> On Mon, Jun 26, 2023 at 12:01 PM <menglong8.dong@...il.com> wrote:
> >
> > From: Menglong Dong <imagedong@...cent.com>
> >
> > For now, skb will be dropped directly if rmem schedule fails, which means
> > tcp_try_rmem_schedule() returns an error. This can happen on following
> > cases:
> >
> > 1. The total memory allocated for TCP protocol is up to tcp_mem[2], and
> >    the receive queue of the tcp socket is not empty.
> > 2. The receive buffer of the tcp socket is full, which can happen on small
> >    packet cases.
> >
> > If the user hangs and doesn't take away the packet in the receive queue
> > with recv() or read() for a long time, the sender will keep
> > retransmitting until timeout, and the tcp connection will break.
> >
> > In order to handle such case, we introduce the tcp protocol OOM detection
> > in following steps, as Neal Cardwell suggested:
>
> net-next is closed.
>
> I think I suggested something much simpler, and not intrusive like your patch.
> (Your patch adds code in the fast path, and yet another sysctl)
>
> If we can not queue an incoming packet because we are under memory stress,
> simply send an ACK with WIN 0

I tested that simply sending an ACK with WIN 0 does not work.
That's what the commit
b650d953cd39("tcp: enforce receive buffer memory limits by allowing
the tcp window to shrink")
do.

There are 2 reasons:
1. The win in the ACK will be ignored. In the tcp_may_update_window(),
it will check if this ACK can update the window. If the ACK doesn't
acknowledge new data, and doesn't contain data, and doesn't
expand the window, it will be ignored.

2. The window update can't work if the retransmission queue is
not empty, as zero-window probe only happen when the rtx queue
empty. So if the rtx queue is not empty, RTO retransmission can still
happen and timeout.

That commit is similar to the series I sent before:
https://lore.kernel.org/netdev/20230517124201.441634-1-imagedong@tencent.com/

But it seems that it only handles the receiver of the window shrink,
and the sender is not handled yet?

As we already accepted the window shrink, maybe we
handle the sender side and use the previous solution?

Thanks!

Following is the script that I used to reproduce the problem.
On the server side, run:

echo '1024 1500 2048' > /proc/sys/net/ipv4/tcp_mem
./proto-mem-exhaust.py -s -r 1024000

and on the client, run:

./proto-mem-exhaust.py -c -t server_ip -m 100 --data 128

-------------------------------------- proto-mem-exhaust.py
----------------------

#!/bin/python3

import socket
import argparse
import time

parser = argparse.ArgumentParser()
parser.add_argument('-c', '--client', action='store_true',
                    help='run as client')
parser.add_argument('-s', '--server', action='store_true',
                    help='run as server')
parser.add_argument('-t', '--target', help='host address')
parser.add_argument('-m', '--max', type=int, help='max connect count')
parser.add_argument('--data', type=int, help='data in kb a connect send')
parser.add_argument('-r', '--rbuff', type=int, help='receive buff size')
args = parser.parse_args()


def do_client():
    clients = []
    for i in range(0, args.max):
        c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        c.connect((args.target, 9999))
        clients.append(c)
        try:
            data_len = args.data * 1024
            print(f'send {data_len} data in {i} socket')
            c.sendall(bytes(data_len))
        except Exception as e:
            print('error happened: %s' % e)
            time.sleep(60*60*24)


def do_server():
    clients = []
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind(('0.0.0.0', 9999))
    s.listen()
    while True:
        (c, addr) = s.accept()
        c.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, args.rbuff)
        clients.append(c)


if args.client:
    do_client()
elif args.server:
    do_server()


Following is another script to reproduce the problem. On the
server side, run: ./tcp-small.py -s -r 2048
client side, run: ./tcp-small.py -c -t server_ip --data 8

------------------------------------ tcp-small.py ---------------------------

#!/bin/python3

import socket
import argparse
import time

parser = argparse.ArgumentParser()
parser.add_argument('-c', '--client', action='store_true',
                    help='run as client')
parser.add_argument('-s', '--server', action='store_true',
                    help='run as server')
parser.add_argument('-t', '--target', help='host address')
parser.add_argument('--data', type=int, help='data in byte send once')
parser.add_argument('-r', '--rbuff', type=int, help='receive buff size')
args = parser.parse_args()


def do_client():
    c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    c.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
    c.connect((args.target, 9999))
    try:
        while True:
            data_len = args.data
            c.sendall(bytes(data_len))
            time.sleep(0.01)
    except Exception as e:
        print('error happened: %s' % e)
        time.sleep(60*60*24)


def do_server():
    clients = []
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, args.rbuff)
    s.bind(('0.0.0.0', 9999))
    s.listen()
    while True:
        (c, addr) = s.accept()
        clients.append(c)


if args.client:
    do_client()
elif args.server:
    do_server()

Powered by blists - more mailing lists