netdev - socket leaks observed in Linux kernel's passive close path

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <SJ0PR84MB1847204B80E86F8449DE1AAAB2259@SJ0PR84MB1847.NAMPRD84.PROD.OUTLOOK.COM>
Date:   Thu, 13 Oct 2022 06:47:56 +0000
From:   "Arankal, Nagaraj" <nagaraj.p.arankal@....com>
To:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: socket leaks observed in Linux kernel's passive close path

Description:
We have observed a strange race condition , where sockets are not freed in kernel in the following condition.
We have a kernel module , which monitors the TCP connection state changes , as part of the functionality it replaces the default sk_destruct function of all TCP sockets with our module specific routine.  Looks like sk_destruct() is not invoked in following condition and hence the sockets are leaked despite receiving RESET from the remote.

1.	Establish a TCP connection between Host A and Host B.
2.	Make the client at B to initiate the CLOSE() immediately after 3-way handshake.
3.	Server end sends huge amount of data to client and does close on FD.
4.	FIN from the client is not ACKED, and server is busy sending the data.
5.	RESET is received from the remote client.
6.	Sk_destruct() is not invoked due to non-null sk_refcnt or sk_wmem_alloc count.

Kernel version: Debian Linux 4.19.y(238,247)

Please find below tcpdump 

No.             Source       Destination                  Protocol     Info
97              10.10.10.41                 10.10.10.21                 TCP            [TCP Port numbers reused] 33968 → 6570 [SYN] Seq=74596442 Win=43800 Len=0 MSS=1460 SACK_PERM=1 TSval=466120930 TSecr=0 WS=32
98              10.10.10.21                 10.10.10.41                 TCP            6570 → 33968 [SYN, ACK] Seq=2529360114 Ack=74596443 Win=65535 Len=0 MSS=1460 SACK_PERM=1 TSval=2085271968 TSecr=466120930 WS=32
99              10.10.10.41                 10.10.10.21                 TCP            33968 → 6570 [ACK] Seq=74596443 Ack=2529360115 Win=43808 Len=0 TSval=466120930 TSecr=2085271968
100            10.10.10.41                 10.10.10.21                 TCP            33968 → 6570 [FIN, ACK] Seq=74596443 Ack=2529360115 Win=43808 Len=0 TSval=466120930 TSecr=2085271968
101            10.10.10.21                 10.10.10.41                 TCP            6570 → 33968 [ACK] Seq=2529360115 Ack=74596443 Win=65536 Len=1448 TSval=2085271969 TSecr=466120930
102            10.10.10.21                 10.10.10.41                 TCP            6570 → 33968 [ACK] Seq=2529361563 Ack=74596443 Win=65536 Len=1448 TSval=2085271969 TSecr=466120930
103            10.10.10.21                 10.10.10.41                 TCP            6570 → 33968 [ACK] Seq=2529363011 Ack=74596443 Win=65536 Len=1448 TSval=2085271969 TSecr=466120930
104            10.10.10.21                 10.10.10.41                 TCP            6570 → 33968 [ACK] Seq=2529364459 Ack=74596443 Win=65536 Len=1448 TSval=2085271969 TSecr=466120930
105            10.10.10.21                 10.10.10.41                 TCP            6570 → 33968 [ACK] Seq=2529365907 Ack=74596443 Win=65536 Len=1448 TSval=2085271969 TSecr=466120930
106            10.10.10.21                 10.10.10.41                 TCP            6570 → 33968 [ACK] Seq=2529367355 Ack=74596443 Win=65536 Len=1448 TSval=2085271969 TSecr=466120930
107            10.10.10.21                 10.10.10.41                 TCP            6570 → 33968 [ACK] Seq=2529368803 Ack=74596443 Win=65536 Len=1448 TSval=2085271969 TSecr=466120930
108            10.10.10.21                 10.10.10.41                 TCP            6570 → 33968 [ACK] Seq=2529370251 Ack=74596443 Win=65536 Len=1448 TSval=2085271969 TSecr=466120930
109            10.10.10.21                 10.10.10.41                 TCP            6570 → 33968 [ACK] Seq=2529371699 Ack=74596443 Win=65536 Len=1448 TSval=2085271969 TSecr=466120930
110            10.10.10.21                 10.10.10.41                 TCP            6570 → 33968 [ACK] Seq=2529373147 Ack=74596443 Win=65536 Len=1448 TSval=2085271969 TSecr=466120930
111            10.10.10.41                 10.10.10.21                 TCP            33968 → 6570 [RST] Seq=74596443 Win=0 Len=0
112            10.10.10.41                 10.10.10.21                 TCP            33968 → 6570 [RST] Seq=74596443 Win=0 Len=0
113            10.10.10.41                 10.10.10.21                 TCP            33968 → 6570 [RST] Seq=74596443 Win=0 Len=0
114            10.10.10.41                 10.10.10.21                 TCP            33968 → 6570 [RST] Seq=74596443 Win=0 Len=0


Bisecting the state of one of the leaked socket.

crash> p *(struct sock *) 0xffff926f465aa200| grep state
    skc_state = 7 '\a', << TCP_CLOSE
..
  skc_refcnt = {
      refs = {
        counter = 1
....
  sk_wmem_alloc = {
    refs = {
      counter = 3

sk_err = 104,
sk_destruct = 0xffffffffc06d6240 <socket_destruct_func>,

}

 tcp_header_len = 32,
  gso_segs = 15,
  pred_flags = 1493504128,
  bytes_received = 1,
  segs_in = 4,
  data_segs_in = 0,
  rcv_nxt = 74596444,
  copied_seq = 74596443,
  rcv_wup = 74596444,
  snd_nxt = 2529374595,
  segs_out = 11,
  data_segs_out = 10,
  bytes_sent = 14480,
  bytes_acked = 0,
  dsack_dups = 0,
  snd_una = 2529360115,
  snd_sml = 2529360115,
  rcv_tstamp = 521240444,
  lsndtime = 521240445,

Regards,
Nagaraj P Arankal