netdev - Re: /proc/net/sockstat invalid memory accounting or memory leak in latest kernels?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <ca939b03f0ca33f4e1c043774288bd82@visp.net.lb>
Date:	Thu, 17 Jul 2014 15:13:10 +0300
From:	Denys Fedoryshchenko <nuclearcat@...learcat.com>
To:	Eric Dumazet <eric.dumazet@...il.com>
Cc:	netdev@...r.kernel.org, kaber@...sh.net, davem@...emloft.net
Subject: Re: /proc/net/sockstat invalid memory accounting or memory leak in
 latest kernels?

On 2014-07-17 14:51, Eric Dumazet wrote:
> On Thu, 2014-07-17 at 13:52 +0300, Denys Fedoryshchenko wrote:
>> Hi
>> 
>> I noticed TCP transfer rate slowdown after few days of operation on
>> kernel 3.15.3, after some digging found out this:
> 
> What was previous version you were using without this problem ?
It is fresh installations, just noticed on units with heavy load 
(200-300k sockets at peak time) after few days it will exhaust memory 
completely, in one case it makes unit
completely unreachable, while trying to ssh there i can establish it 
establish tcp, but no ssh banner message shown from server side.
But because unit is remote, and no kvm - i can't see what is really 
wrong there, seems i should try netconsole.
On other units it just causes tcp performance degradation.
> 
>> 
>> balancer-backup ~ # cat /proc/net/sockstat
>> sockets: used 118236
>> TCP: inuse 122958 orphan 4986 tw 108010 alloc 123179 mem 1955339
>> UDP: inuse 1 mem 0
>> UDPLITE: inuse 0
>> RAW: inuse 0
>> FRAG: inuse 1 memory 2
>> 
>> after shutting down program
>> balancer-backup ~ # cat /proc/net/sockstat
>> sockets: used 47
>> TCP: inuse 10552 orphan 10547 tw 142645 alloc 10552 mem 1877061
>> UDP: inuse 0 mem 0
>> UDPLITE: inuse 0
>> RAW: inuse 0
>> FRAG: inuse 0 memory 0
>> 
>> sysctl settings:
>> net.ipv4.tcp_mem = 1767103      2045612 3068412
>> 
>> I restarted recently process, and mem value didnt changed (while 
>> because
>> it is sockets should release all memory), also it looks incorrect,
>> because at same time:
>> balancer-backup ~ # cat /proc/meminfo
>> MemTotal:       32939492 kB
>> MemFree:        29876564 kB
>> 
>> While 1955339 * 4096 should be around 8GB.
>> Probably it is just accounting issue or is it real memory leak?
>> What other info i can provide to troubleshoot this info more properly?
>> I will upgrade to 3.15.5 also now, to see if issue persist there.
> 
> I see nothing really wrong in your report.
> 
> It looks like you have a lot of sockets around after shutdown of the
> program. Each FIN-WAIT socket might consume a lot of buffers in its
> write queue, unless you use/force SO_LINGER or something.
> 
> If you try the following command, you might see how many sockets have
> outstanding data.
> 
> ss -amn
I will check that next time, thanks a lot!


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html