netdev - Debugging stuck tcp connection across localhost

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Thu, 6 Jan 2022 06:59:58 -0800
From:   Ben Greear <greearb@...delatech.com>
To:     netdev <netdev@...r.kernel.org>
Subject: Debugging stuck tcp connection across localhost

Hello,

I'm working on a strange problem, and could use some help if anyone has ideas.

On a heavily loaded system (500+ wifi station devices, VRF device per 'real' netdev,
traffic generation on the netdevs, etc), I see cases where two processes trying
to communicate across localhost with TCP seem to get a stuck network
connection:

[greearb@...dt7 ben_debug]$ grep 4004 netstat.txt |grep 127.0.0.1
tcp        0 7988926 127.0.0.1:4004          127.0.0.1:23184         ESTABLISHED
tcp        0  59805 127.0.0.1:23184         127.0.0.1:4004          ESTABLISHED

Both processes in question continue to execute, and as far as I can tell, they are properly
attempting to read/write the socket, but they are reading/writing 0 bytes (these sockets
are non blocking).  If one was stuck not reading, I would expect netstat
to show bytes in the rcv buffer, but it is zero as you can see above.

Kernel is 5.15.7+ local hacks.  I can only reproduce this in a big messy complicated
test case, with my local ath10k-ct and other patches that enable virtual wifi stations,
but my code can grab logs at time it sees the problem.  Is there anything
more I can do to figure out why the TCP connection appears to be stuck?

Thanks,
Ben

-- 
Ben Greear <greearb@...delatech.com>
Candela Technologies Inc  http://www.candelatech.com