netdev - Re: issue with inflight pages from page

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4520b112-96c7-2dd8-b2c0-027961eb3a7c@engleder-embedded.com>
Date:   Mon, 17 Apr 2023 20:24:11 +0200
From:   Gerhard Engleder <gerhard@...leder-embedded.com>
To:     Lorenzo Bianconi <lorenzo@...nel.org>,
        Eric Dumazet <edumazet@...gle.com>
Cc:     netdev@...r.kernel.org, hawk@...nel.org,
        ilias.apalodimas@...aro.org, davem@...emloft.net, kuba@...nel.org,
        pabeni@...hat.com, bpf@...r.kernel.org,
        lorenzo.bianconi@...hat.com, nbd@....name
Subject: Re: issue with inflight pages from page_pool

On 17.04.23 20:17, Lorenzo Bianconi wrote:
>> On Mon, Apr 17, 2023 at 7:53 PM Lorenzo Bianconi <lorenzo@...nel.org> wrote:
>>>
>>> Hi all,
>>>
>>> I am triggering an issue with a device running the page_pool allocator.
>>> In particular, the device is running an iperf tcp server receiving traffic
>>> from a remote client. On the driver I loaded a simple xdp program returning
>>> xdp_pass. When I remove the ebpf program and destroy the pool, page_pool
>>> allocator starts complaining in page_pool_release_retry() that not all the pages
>>> have been returned to the allocator. In fact, the pool is not really destroyed
>>> in this case.
>>> Debugging the code it seems the pages are stuck softnet_data defer_list and
>>> they are never freed in skb_defer_free_flush() since I do not have any more tcp
>>> traffic. To prove it, I tried to set sysctl_skb_defer_max to 0 and the issue
>>> does not occur.
>>> I developed the poc patch below and the issue seems to be fixed:
>>
>> I do not see why this would be different than having buffers sitting
>> in some tcp receive
>> (or out or order) queues for a few minutes ?
> 
> The main issue in my tests (and even in mt76 I think) is the pages are not returned
> to the pool for a very long time (even hours) and doing so the pool is like in a
> 'limbo' state where it is not actually deallocated and page_pool_release_retry
> continues complaining about it. I think this is because we do not have more tcp
> traffic to deallocate them, but I am not so familiar with tcp codebase :)
> 
> Regards,
> Lorenzo
> 
>>
>> Or buffers transferred to another socket or pipe (splice() and friends)

I'm not absolutely sure that it is the same problem, but I also saw some
problems with page_pool destroying and page_pool_release_retry(). I did
post it, but I did not get any reply:

https://lore.kernel.org/bpf/20230311213709.42625-1-gerhard@engleder-embedded.com/T/

Could this be a similar issue?

Gerhard