netdev - Re: cat /proc/net/tcp takes 0.5 seconds on x86

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <48B46E89.4030104@hhs.nl>
Date:	Tue, 26 Aug 2008 22:58:49 +0200
From:	Hans de Goede <j.w.r.degoede@....nl>
To:	Eric Dumazet <dada1@...mosbay.com>
CC:	Dave Jones <davej@...hat.com>, netdev@...r.kernel.org
Subject: Re: cat /proc/net/tcp takes 0.5 seconds on x86_64

Eric Dumazet wrote:
> Hans de Goede a écrit :
>> Eric Dumazet wrote:
>>> Dave Jones a écrit :
>>>> Just had this bug reported against our development tree..
>> <snip>
>>>>  > [hans@...alhost devel]$ time cat /proc/net/tcp
>>>>  > <snip>
>>>>  > real    0m0.520s
>>>>  > user    0m0.000s
>>>>  > sys     0m0.446s
>>>>  >  > Thats amazingly slow, esp as I only have 8 tcp connections open.
>>>>  >  > Some maybe usefull info: top reports a very high load (50%) 
>>>> from soft IRQ's.
>>>>  >  > Anyways changing this to a kernel bug.
>>>>
>>>
>>> I wonder why this qualifies as a "kernel bug". This is a well known 
>>> problem.
>>>
>>
>> No its not, /proc/net/tcp may be slow in general but not *this* slow ...
>>
>> <snip>
>>
>>>
>>> Time difference between /proc/net/tcp and netlink on a 4GB x86_64 
>>> machine :
>>>
>>> # dmesg | grep "TCP established hash"
>>> TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
>>> # time cat /proc/net/tcp >/dev/null
>>>
>>> real    0m0.091s
>>> user    0m0.001s
>>> sys     0m0.090s
>>
>> As quoted above my idle x86_64, using the exact same hash table size, 
>> running 2.6.27-rc2.git1 uses 0.520 seconds for that same command, 
>> thats a difference of more then a factor 50 !!
>>
>> This is not about /proc/net/tcp not being fast, this is about it haven 
>> gotten slower by a factor of 50!
>>
>> Also notice that this slowdown does not happen on i386.
> 
> And your .config files on i386 and x86_64 are ?
> Some configuration options can slow down all lock/unlock operations 
> (CONFIG_SMP, CONFIG_PREEMPT, CONFIG_PROVE_LOCKING, 
> CONFIG_DEBUG_SPINLOCK, CONFIG_NR_CPUS ...)
> 

Attached

> If you TCP hash table has 512.000 slots (I am just guessing, you didnt 
> provide this information), it can make a huge difference.

I did provide that information: "using the exact same hash table size" and then 
quoting your first mail in this thread:
"TCP established hash table entries: 262144 (order: 10, 4194304 bytes)"

>>
>> Anyways I'll try 2.6.27-rc4 and report back with its results.
>>
> 
> Yes, please, but nothing really changed in this area in the recent times...
> 

I'm afraid that atleast the Fedora rc4 build won't boot on my machine ...

> We added some checks so that softirqs can preempt us.
> Latencies used to be very high, and are now bonded, at the price of 
> potential slowdown for the /proc/net/tcp reader.

Slowdown as in 2x or 4x as slow I presume, not 50x ?

Regards,

Hans

View attachment "config-2.6.27-0.244.rc2.git1.fc10.x86_64" of type "text/plain" (84929 bytes)