lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <A8918103-4A1D-43FF-9490-1E26ABEDA748@gmail.com>
Date: Tue, 24 Sep 2024 00:12:34 +0800
From: Miao Wang <shankerwangmiao@...il.com>
To: Jijie Shao <shaojijie@...wei.com>
Cc: netdev@...r.kernel.org,
 Shengqi Chen <harry-chen@...look.com>,
 Yuxiang Zhang <zz593141477@...il.com>,
 Jiajie Chen <jiegec@...com>,
 Mirror Admin Tuna <mirroradmin@...a.tsinghua.edu.cn>,
 Salil Mehta <salil.mehta@...wei.com>
Subject: Re: [BUG Report] hns3: tx_timeout on high memory pressure


> 2024年9月23日 21:37,Jijie Shao <shaojijie@...wei.com> 写道:
> 
> 
> on 2024/9/23 21:11, Miao Wang wrote:
>> 
>>> 2024年9月23日 20:58,Jijie Shao <shaojijie@...wei.com> 写道:
>>> 
>>>    Hi:
>>> 
>>> in dmesg, we can see:
>>> tx_timeout count: 35, queue id: 1, SW_NTU: 0x346, SW_NTC: 0x334, napi state: 17
>>> BD_NUM: 0x7f HW_HEAD: 0x346, HW_TAIL: 0x346, BD_ERR: 0x0, INT: 0x0
>>> 
>>> Because HW_HEAD==HW_TAIL, the hardware has sent all the packets.
>>> napi state: 17, Therefore, the TX interrupt is received and npai scheduling is triggered.
>>> However, napi scheduling is not complete, Maybe napi.poll() is not executed.
>>> Is npai not scheduled in time due to high CPU load in the environment?
>> Thanks for your analysis. I wonder how can I verify the scheduling of NAPI.
> 
> You can use napi trace to verify it:
> echo 1 > /sys/kernel/debug/tracing/events/napi/napi_poll/enable
> cat /sys/kernel/debug/tracing/trace

I managed to make a reproduce. Attached is the dmesg and the trace log. It seems
that in the trace log, napi_poll() is kept called. 

My reproducing environment is a smart git http server, which is using nginx as
the frontend, git-http-backend as the CGI server and fastcgiwrapper for
connecting them, running on a Taishan server. The served git repo is linux.git.
At the same time, start some programs taking up about 70% of the system memory.
Using several other hosts as git client, start as many git clone processes as
possible on the client hosts, about 2000 in total, at the same time, pointing
to the git server, letting the forked git processes on the server side take up
all the memory left, and causing OOM on the server.

Cheers,

Miao Wang



View attachment "dmesg.txt" of type "text/plain" (257529 bytes)

View attachment "trace.txt" of type "text/plain" (1487170 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ