Message-ID: <40ed29b3-84d7-4812-890d-3676957d503f@linux.dev>
Date: Wed, 3 Sep 2025 22:50:53 -0700
From: Martin KaFai Lau <martin.lau@...ux.dev>
To: Kuniyuki Iwashima <kuniyu@...gle.com>
Cc: Alexei Starovoitov <ast@...nel.org>, Andrii Nakryiko <andrii@...nel.org>,
 Daniel Borkmann <daniel@...earbox.net>,
 John Fastabend <john.fastabend@...il.com>,
 Stanislav Fomichev <sdf@...ichev.me>, Johannes Weiner <hannes@...xchg.org>,
 Michal Hocko <mhocko@...nel.org>, Roman Gushchin <roman.gushchin@...ux.dev>,
 Shakeel Butt <shakeel.butt@...ux.dev>, "David S. Miller"
 <davem@...emloft.net>, Eric Dumazet <edumazet@...gle.com>,
 Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>,
 Neal Cardwell <ncardwell@...gle.com>, Willem de Bruijn <willemb@...gle.com>,
 Mina Almasry <almasrymina@...gle.com>, Kuniyuki Iwashima
 <kuni1840@...il.com>, bpf@...r.kernel.org, netdev@...r.kernel.org
Subject: Re: [PATCH v4 bpf-next/net 5/5] selftest: bpf: Add test for
 SK_BPF_MEMCG_SOCK_ISOLATED.

On 9/3/25 10:08 AM, Kuniyuki Iwashima wrote:
> On Wed, Sep 3, 2025 at 9:59 AM Kuniyuki Iwashima <kuniyu@...gle.com> wrote:
>>
>> On Tue, Sep 2, 2025 at 1:49 PM Kuniyuki Iwashima <kuniyu@...gle.com> wrote:
>>>
>>> On Tue, Sep 2, 2025 at 1:26 PM Martin KaFai Lau <martin.lau@...ux.dev> wrote:
>>>>
>>>> On 8/28/25 6:00 PM, Kuniyuki Iwashima wrote:
>>>>> The test does the following for IPv4/IPv6 x TCP/UDP sockets
>>>>> with/without BPF prog.
>>>>>
>>>>>     1. Create socket pairs
>>>>>     2. Send a bunch of data that requires more than 256 pages
>>>>>     3. Read memory_allocated from the 3rd column in /proc/net/protocols
>>>>>     4. Check if unread data is charged to memory_allocated
>>>>>
>>>>> If BPF prog is attached, memory_allocated should not be changed,
>>>>> but we allow a small error (up to 10 pages) in case other processes
>>>>> on the host use some amounts of TCP/UDP memory.
>>>>>
>>>>> At 2., the test actually sends more than 1024 pages because the sysctl
>>>>> net.core.mem_pcpu_rsv is 256 is by default, which means 256 pages are
>>>>> buffered per cpu before reporting to sk->sk_prot->memory_allocated.
>>>>>
>>>>>     BUF_SINGLE (1024) * NR_SEND (64) * NR_SOCKETS (64) / 4096
>>>>>     = 1024 pages
>>>>>
>>>>> When I reduced it to 512 pages, the following assertion for the
>>>>> non-isolated case got flaky.
>>>>>
>>>>>     ASSERT_GT(memory_allocated[1], memory_allocated[0] + 256, ...)
>>>>>
>>>>> Another contributor to slowness is 150ms sleep to make sure 1 RCU
>>>>> grace period passes because UDP recv queue is destroyed after that.
>>>>
>>>> There is a kern_sync_rcu() in testing_helpers.c.
>>>
>>> Nice helper :)  Will use it.
>>>
>>>>
>>>>>
>>>>>     # time ./test_progs -t sk_memcg
>>>>>     #370/1   sk_memcg/TCP       :OK
>>>>>     #370/2   sk_memcg/UDP       :OK
>>>>>     #370/3   sk_memcg/TCPv6     :OK
>>>>>     #370/4   sk_memcg/UDPv6     :OK
>>>>>     #370     sk_memcg:OK
>>>>>     Summary: 1/4 PASSED, 0 SKIPPED, 0 FAILED
>>>>>
>>>>>     real       0m1.214s
>>>>>     user       0m0.014s
>>>>>     sys        0m0.318s
>>>>
>>>> Thanks. It finished much faster in my setup also comparing with the earlier
>>>> revision. However, it is a bit flaky when I run it in a loop:
>>>>
>>>> check_isolated:FAIL:not isolated unexpected not isolated: actual 861 <= expected 861
>>>>
>>>> I usually can hit this at ~40-th iteration.
>>>
>>> Oh.. I tested ~10 times manually but will try in a tight loop.
>>
>> This didn't reproduce on my QEMU with/without --enable-kvm.
>>
>> Changing the assert from _GT to _GE will address the very case
>> above, but I'm not sure if it's enough.
> 
> I doubled NR_SEND and it was still faster with kern_sync_rcu()
> than usleep(), so I'll simply double NR_SEND in v5
> 
> # time ./test_progs -t sk_memcg
> ...
> Summary: 1/4 PASSED, 0 SKIPPED, 0 FAILED
> real 0m0.483s
> user 0m0.010s
> sys 0m0.191s
> 
> 
>>
>> Does the bpf CI run tests repeatedly or is this only a manual
>> scenario ?

I haven't seen bpf CI hit it yet. It is in my manual bash while loop. It should 
not be dismissed so easily. Some flaky CI tests were eventually reproduced in a 
loop before and fixed. I kept the bash loop continue this time until grep-ed a 
"0" from the error output:

check_isolated:FAIL:not isolated unexpected not isolated: actual 0 <= expected 256

The "long memory_allocated[2]" read from /proc/net/protocols are printed as 0 
but it is probably actually negative:

static inline long
proto_memory_allocated(const struct proto *prot)
{
         return max(0L, atomic_long_read(prot->memory_allocated));
}

prot->memory_allocated could be negative afaict but printed as 0 in 
/proc/net/protocols. Even the machine is network quiet after test_progs started, 
the "prot->memory_allocated" and the "proto->per_cpu_fw_alloc" could be in some 
random states before the test_progs start.  When I hit "0", it will take some 
efforts to send some random traffic to the machine to get the test working again. :(

Also, after reading the selftest closer, I am not sure I understand why "+ 256". 
The "proto->per_cpu_fw_alloc" can start with -255 or +255.

I don't think changing NR_SEND help here. It needs a better way. May be some 
functions can be traced such that prot->memory_allocated can be read directly? 
If fentry and fexit of that function has different memory_allocated values, then 
the test could also become more straight forward.

Powered by blists - more mailing lists