| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <1d935bfc-50b0-54f3-22f0-d360f8a7c1ac@bytedance.com>
Date: Fri, 8 Sep 2023 15:55:18 +0800
From: Abel Wu <wuyun.abel@...edance.com>
To: "David S. Miller" <davem@...emloft.net>,
Eric Dumazet <edumazet@...gle.com>,
Jakub Kicinski <kuba@...nel.org>,
Paolo Abeni <pabeni@...hat.com>,
Andrew Morton <akpm@...ux-foundation.org>,
Shakeel Butt <shakeelb@...gle.com>,
Roman Gushchin <roman.gushchin@...ux.dev>,
Michal Hocko <mhocko@...e.com>,
Johannes Weiner <hannes@...xchg.org>,
Yosry Ahmed <yosryahmed@...gle.com>,
"Matthew Wilcox (Oracle)" <willy@...radead.org>,
Yu Zhao <yuzhao@...gle.com>,
Kefeng Wang <wangkefeng.wang@...wei.com>,
Yafang Shao <laoar.shao@...il.com>,
Kuniyuki Iwashima <kuniyu@...zon.com>,
Martin KaFai Lau <martin.lau@...nel.org>,
Breno Leitao <leitao@...ian.org>,
Alexander Mikhalitsyn <alexander@...alicyn.com>,
David Howells <dhowells@...hat.com>,
Jason Xing <kernelxing@...cent.com>
Cc: open list <linux-kernel@...r.kernel.org>,
"open list:NETWORKING [GENERAL]" <netdev@...r.kernel.org>,
"open list:MEMORY MANAGEMENT" <linux-mm@...ck.org>
Subject: Re: [RFC PATCH net-next 0/3] sock: Be aware of memcg pressure on
alloc
Friendly ping :)
On 9/1/23 2:21 PM, Abel Wu wrote:
> As a cloud service provider, we encountered a problem in our production
> environment during the transition from cgroup v1 to v2 (partly due to the
> heavy taxes of accounting socket memory in v1). Say one workload behaves
> fine in cgroupv1 with memcg limit configured to 10GB memory and another
> 1GB tcpmem, but will suck (or even be OOM-killed) in v2 with 11GB memory
> due to burst memory usage on socket, since there is no specific limit for
> socket memory in cgroupv2 and relies largely on workloads doing traffic
> control themselves.
>
> It's rational for the workloads to build some traffic control to better
> utilize the resources they bought, but from kernel's point of view it's
> also reasonable to suppress the allocation of socket memory once there is
> a shortage of free memory, given that performance degradation is usually
> better than failure.
>
> This patchset aims to be more conservative on alloc for pressure-aware
> sockets under global and/or memcg pressure, to avoid further memstall or
> possibly OOM in such case. The patchset includes:
>
> 1/3: simple code cleanup, no functional change intended.
> 2/3: record memcg pressure level to enable fine-grained control.
> 3/3: throttle alloc for pressure-aware sockets under pressure.
>
> The whole patchset focuses on the pressure-aware protocols, and should
> have no/little impact on pressure-unaware protocols like UDP etc.
>
> Tested on Intel Xeon(R) Platinum 8260, a dual socket machine containing 2
> NUMA nodes each of which has 24C/48T. All the benchmarks are done inside a
> separate memcg in a clean host.
>
> baseline: net-next c639a708a0b8
> compare: baseline + patchset
>
> case load baseline(std%) compare%( std%)
> tbench-loopback thread-24 1.00 ( 0.50) -0.98 ( 0.87)
> tbench-loopback thread-48 1.00 ( 0.76) -0.29 ( 0.92)
> tbench-loopback thread-72 1.00 ( 0.75) +1.51 ( 0.14)
> tbench-loopback thread-96 1.00 ( 4.11) +1.29 ( 3.73)
> tbench-loopback thread-192 1.00 ( 3.52) +1.44 ( 3.30)
> TCP_RR thread-24 1.00 ( 1.87) -0.87 ( 2.40)
> TCP_RR thread-48 1.00 ( 0.92) -0.22 ( 1.61)
> TCP_RR thread-72 1.00 ( 2.35) +2.42 ( 2.27)
> TCP_RR thread-96 1.00 ( 2.66) -1.37 ( 3.02)
> TCP_RR thread-192 1.00 ( 13.25) +0.29 ( 11.80)
> TCP_STREAM thread-24 1.00 ( 1.26) -0.75 ( 0.87)
> TCP_STREAM thread-48 1.00 ( 0.29) -1.55 ( 0.14)
> TCP_STREAM thread-72 1.00 ( 0.05) -1.59 ( 0.05)
> TCP_STREAM thread-96 1.00 ( 0.19) -0.06 ( 0.29)
> TCP_STREAM thread-192 1.00 ( 0.23) -0.01 ( 0.28)
> UDP_RR thread-24 1.00 ( 2.27) +0.33 ( 2.82)
> UDP_RR thread-48 1.00 ( 1.25) -0.30 ( 1.21)
> UDP_RR thread-72 1.00 ( 2.54) +2.99 ( 2.34)
> UDP_RR thread-96 1.00 ( 4.76) +2.49 ( 2.19)
> UDP_RR thread-192 1.00 ( 14.43) -0.02 ( 12.98)
> UDP_STREAM thread-24 1.00 (107.41) -0.48 (106.93)
> UDP_STREAM thread-48 1.00 (100.85) +1.38 (100.59)
> UDP_STREAM thread-72 1.00 (103.43) +1.40 (103.48)
> UDP_STREAM thread-96 1.00 ( 99.91) -0.25 (100.06)
> UDP_STREAM thread-192 1.00 (109.83) -3.67 (104.12)
>
> As patch 3 moves forward traversal of cgroup hierarchy for pressure-aware
> protocols, which could turn a conditional overhead into constant, tests
> running inside 5-level-depth cgroups are also performed.
>
> case load baseline(std%) compare%( std%)
> tbench-loopback thread-24 1.00 ( 0.59) +0.68 ( 0.09)
> tbench-loopback thread-48 1.00 ( 0.16) +0.01 ( 0.26)
> tbench-loopback thread-72 1.00 ( 0.34) -0.67 ( 0.48)
> tbench-loopback thread-96 1.00 ( 4.40) -3.27 ( 4.84)
> tbench-loopback thread-192 1.00 ( 0.49) -1.07 ( 1.18)
> TCP_RR thread-24 1.00 ( 2.40) -0.34 ( 2.49)
> TCP_RR thread-48 1.00 ( 1.62) -0.48 ( 1.35)
> TCP_RR thread-72 1.00 ( 1.26) +0.46 ( 0.95)
> TCP_RR thread-96 1.00 ( 2.98) +0.13 ( 2.64)
> TCP_RR thread-192 1.00 ( 13.75) -0.20 ( 15.42)
> TCP_STREAM thread-24 1.00 ( 0.21) +0.68 ( 1.02)
> TCP_STREAM thread-48 1.00 ( 0.20) -1.41 ( 0.01)
> TCP_STREAM thread-72 1.00 ( 0.09) -1.23 ( 0.19)
> TCP_STREAM thread-96 1.00 ( 0.01) +0.01 ( 0.01)
> TCP_STREAM thread-192 1.00 ( 0.20) -0.02 ( 0.25)
> UDP_RR thread-24 1.00 ( 2.20) +0.84 ( 17.45)
> UDP_RR thread-48 1.00 ( 1.34) -0.73 ( 1.12)
> UDP_RR thread-72 1.00 ( 2.32) +0.49 ( 2.11)
> UDP_RR thread-96 1.00 ( 2.36) +0.53 ( 2.42)
> UDP_RR thread-192 1.00 ( 16.34) -0.67 ( 14.06)
> UDP_STREAM thread-24 1.00 (106.55) -0.70 (107.13)
> UDP_STREAM thread-48 1.00 (105.11) +1.60 (103.48)
> UDP_STREAM thread-72 1.00 (100.60) +1.98 (101.13)
> UDP_STREAM thread-96 1.00 ( 99.91) +2.59 (101.04)
> UDP_STREAM thread-192 1.00 (135.39) -2.51 (108.00)
>
> As expected, no obvious performance gain or loss observed. As for the
> issue we encountered, this patchset provides better worst-case behavior
> that such OOM cases are reduced at some extent. While further fine-
> grained traffic control is what the workloads need to think about.
>
> Comments are welcomed! Thanks!
>
> Abel Wu (3):
> sock: Code cleanup on __sk_mem_raise_allocated()
> net-memcg: Record pressure level when under pressure
> sock: Throttle pressure-aware sockets under pressure
>
> include/linux/memcontrol.h | 39 +++++++++++++++++++++++++----
> include/net/sock.h | 2 +-
> include/net/tcp.h | 2 +-
> mm/vmpressure.c | 9 ++++++-
> net/core/sock.c | 51 +++++++++++++++++++++++++++++---------
> 5 files changed, 83 insertions(+), 20 deletions(-)
>
Powered by blists - more mailing lists