| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <51990dad-3dfe-1206-5f78-74ff3d068b40@candelatech.com>
Date: Mon, 22 Jan 2018 10:30:05 -0800
From: Ben Greear <greearb@...delatech.com>
To: Eric Dumazet <eric.dumazet@...il.com>,
netdev <netdev@...r.kernel.org>
Subject: Re: TCP many-connection regression between 4.7 and 4.13 kernels.
On 01/22/2018 10:16 AM, Eric Dumazet wrote:
> On Mon, 2018-01-22 at 09:28 -0800, Ben Greear wrote:
>> My test case is to have 6 processes each create 5000 TCP IPv4 connections to each other
>> on a system with 16GB RAM and send slow-speed data. This works fine on a 4.7 kernel, but
>> will not work at all on a 4.13. The 4.13 first complains about running out of tcp memory,
>> but even after forcing those values higher, the max connections we can get is around 15k.
>>
>> Both kernels have my out-of-tree patches applied, so it is possible it is my fault
>> at this point.
>>
>> Any suggestions as to what this might be caused by, or if it is fixed in more recent kernels?
>>
>> I will start bisecting in the meantime...
>>
>
> Hi Ben
>
> Unfortunately I have no idea.
>
> Are you using loopback flows, or have I misunderstood you ?
>
> How loopback connections can be slow-speed ?
>
I am sending to self, but over external network interfaces, by using
routing tables and rules and such.
On 4.13.16+, I see the Intel driver bouncing when I try to start 20k
connections. In this case, I have a pair of 10G ports doing 15k, and then
I try to start 5k on two of the 1G ports....
Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Down
Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Down
Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Down
Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 22 10:15:43 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Down
Jan 22 10:15:45 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 22 10:15:51 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_s...es: 1
Jan 22 10:15:51 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
System reports 10+GB RAM free in this case, btw.
Actually, maybe the good kernel was even older than 4.7...I see same resets and inability to do a full 20k
connections on 4.7 too. I double-checked with system-test and it seems 4.4 was a good kernel. I'll test
that next. Here is splat from 4.7:
[ 238.921679] ------------[ cut here ]------------
[ 238.921689] WARNING: CPU: 0 PID: 3 at /home/greearb/git/linux-bisect/net/sched/sch_generic.c:272 dev_watchdog+0xd4/0x12f
[ 238.921690] NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out
[ 238.921691] Modules linked in: nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv4 cfg80211 macvlan pktgen bnep bluetooth fuse coretemp intel_rapl
ftdi_sio x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm iTCO_wdt iTCO_vendor_support joydev ie31200_edac ipmi_devintf irqbypass serio_raw ipmi_si edac_core
shpchp fjes video i2c_i801 tpm_tis lpc_ich ipmi_msghandler tpm nfsd auth_rpcgss nfs_acl lockd grace sunrpc mgag200 i2c_algo_bit drm_kms_helper ttm drm i2c_core
e1000e ixgbe mdio hwmon dca ptp pps_core ipv6 [last unloaded: nf_conntrack]
[ 238.921720] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted 4.7.0 #62
[ 238.921721] Hardware name: Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b 09/17/2012
[ 238.921723] 0000000000000000 ffff88041cdd7cd8 ffffffff81352a23 ffff88041cdd7d28
[ 238.921725] 0000000000000000 ffff88041cdd7d18 ffffffff810ea5dd 000001101cdd7d90
[ 238.921727] ffff880417a84000 0000000000000100 ffffffff8163ecff ffff880417a84440
[ 238.921728] Call Trace:
[ 238.921733] [<ffffffff81352a23>] dump_stack+0x61/0x7d
[ 238.921736] [<ffffffff810ea5dd>] __warn+0xbd/0xd8
[ 238.921738] [<ffffffff8163ecff>] ? netif_tx_lock+0x81/0x81
[ 238.921740] [<ffffffff810ea63e>] warn_slowpath_fmt+0x46/0x4e
[ 238.921741] [<ffffffff8163ecf2>] ? netif_tx_lock+0x74/0x81
[ 238.921743] [<ffffffff8163edd3>] dev_watchdog+0xd4/0x12f
[ 238.921746] [<ffffffff8113cfbb>] call_timer_fn+0x65/0x11b
[ 238.921748] [<ffffffff8163ecff>] ? netif_tx_lock+0x81/0x81
[ 238.921749] [<ffffffff8113d73e>] run_timer_softirq+0x1ad/0x1d7
[ 238.921751] [<ffffffff810ee701>] __do_softirq+0xfb/0x25c
[ 238.921752] [<ffffffff810ee87b>] run_ksoftirqd+0x19/0x35
[ 238.921755] [<ffffffff81105ae8>] smpboot_thread_fn+0x169/0x1a9
[ 238.921756] [<ffffffff8110597f>] ? sort_range+0x1d/0x1d
[ 238.921759] [<ffffffff811031a1>] kthread+0xa0/0xa8
[ 238.921763] [<ffffffff816ce19f>] ret_from_fork+0x1f/0x40
[ 238.921764] [<ffffffff81103101>] ? init_completion+0x24/0x24
[ 238.921765] ---[ end trace 933912956c6ee5ff ]---
[ 238.961672] e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
Thanks,
Ben
--
Ben Greear <greearb@...delatech.com>
Candela Technologies Inc http://www.candelatech.com
Powered by blists - more mailing lists