lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAO-X30uMA7=DTm5KqqvYC5RVTM0bg9pKBO1nX1+6x_2pF_fWfA@mail.gmail.com> Date: Tue, 3 Feb 2015 22:50:54 -0800 From: Avery Fay <avery@...panel.com> To: netdev@...r.kernel.org Subject: Invalid timestamp? causing tight ack loop (hundreds of thousands of packets / sec) Hello, Let me say first: if there's a better place to ask this, please point me in that direction. We've been having huge packets / sec spikes in the past few days. After some investigation, it looks like single connections are getting stuck in a loop (see tcpdump below). Each "stuck" connection will generate about 200kpps. It looks like our side is rejecting packets with "packets rejects in established connections because of timestamp" from netstat -s (internally PAWSEstab counter) and then generating an additional packet that we send out. All of these connections originate from georgia tech, but so far (not completely verified) it doesn't seem like there's any pattern to the client/os other than the fact that they're trying to make an https request to us. As a temporary countermeasure, we've disabled net.ipv4.tcp_timestamps, which solves the immediate problem. Our server is 174.36.240.86 running Ubuntu 12.04 with kernel 3.13.0-35-generic The client is 128.61.57.205 and in this case almost certainly has user agent (we found successful requests 10 seconds before the tcpdump with same ip): Dalvik/2.1.0 (Linux; U; Android 5.0; XT1095 Build/LXE22.46-11) Beginning of tcpdump: 05:36:10.723423 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [S], seq 4192140517, win 65535, options [mss 1380,sackOK,TS val 3985707 ecr 0,nop,wscale 8], length 0 05:36:10.723431 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [S.], seq 1191654135, ack 4192140518, win 28960, options [mss 1460,sackOK,TS val 3311875276 ecr 3985707,nop,wscale 7], length 0 05:36:10.742480 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [.], ack 1, win 343, options [nop,nop,TS val 3985710 ecr 3311875276], length 0 05:36:10.742534 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [.], ack 1, win 343, options [nop,nop,TS val 3985710 ecr 3311875276], length 0 05:36:10.761060 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], seq 1:227, ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 226 05:36:10.761067 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 227, win 235, options [nop,nop,TS val 3311875285 ecr 3985712], length 0 05:36:10.761122 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], seq 1:2737, ack 227, win 235, options [nop,nop,TS val 3311875285 ecr 3985712], length 2736 05:36:10.761129 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [P.], seq 2737:3256, ack 227, win 235, options [nop,nop,TS val 3311875285 ecr 3985712], length 519 05:36:10.798133 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [P.], seq 2737:3256, ack 227, win 235, options [nop,nop,TS val 3311875295 ecr 3985712], length 519 05:36:11.018130 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], seq 1:1369, ack 227, win 235, options [nop,nop,TS val 3311875350 ecr 3985712], length 1368 05:36:11.458134 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], seq 1:1369, ack 227, win 235, options [nop,nop,TS val 3311875460 ecr 3985712], length 1368 05:36:12.338130 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], seq 1:1369, ack 227, win 235, options [nop,nop,TS val 3311875680 ecr 3985712], length 1368 05:36:14.102128 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], seq 1:1369, ack 227, win 235, options [nop,nop,TS val 3311876121 ecr 3985712], length 1368 05:36:17.634130 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], seq 1:1369, ack 227, win 235, options [nop,nop,TS val 3311877004 ecr 3985712], length 1368 05:36:20.820850 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [F.], seq 227, ack 1, win 343, options [nop,nop,TS val 3986717 ecr 3311875285], length 0 05:36:20.820879 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [F.], seq 3256, ack 228, win 235, options [nop,nop,TS val 3311877800 ecr 3986717], length 0 05:36:20.835368 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276,nop,nop,sack 1 {3256:3256}], length 0 05:36:20.835373 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877804 ecr 3986717], length 0 05:36:20.835375 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.835378 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877804 ecr 3986717], length 0 05:36:20.849735 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276,nop,nop,sack 2 {3257:3257}{3256:3256}], length 0 05:36:20.849740 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877807 ecr 3986717], length 0 05:36:20.849783 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.849785 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877807 ecr 3986717], length 0 05:36:20.849787 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.849788 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877807 ecr 3986717], length 0 05:36:20.849789 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.849790 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877807 ecr 3986717], length 0 05:36:20.864132 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.864137 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877811 ecr 3986717], length 0 05:36:20.864185 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.864189 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877811 ecr 3986717], length 0 05:36:20.864226 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.864228 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877811 ecr 3986717], length 0 05:36:20.864229 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.864230 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877811 ecr 3986717], length 0 05:36:20.864231 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.864233 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877811 ecr 3986717], length 0 05:36:20.864282 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.864285 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877811 ecr 3986717], length 0 05:36:20.878523 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.878530 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717], length 0 05:36:20.878532 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.878534 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717], length 0 05:36:20.878620 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.878624 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717], length 0 05:36:20.878625 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.878627 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717], length 0 05:36:20.878628 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.878629 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717], length 0 05:36:20.878630 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.878632 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717], length 0 05:36:20.878632 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.878634 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717], length 0 05:36:20.878635 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.878636 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717], length 0 05:36:20.878672 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.878674 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717], length 0 05:36:20.878678 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.878679 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717], length 0 05:36:20.878724 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.878726 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717], length 0 05:36:20.878816 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276], length 0 05:36:20.878818 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.], ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717], length 0 At this point, it just repeats until some timeout is hit. I haven't timed it, but probably one or two minutes. I guess I have a few questions: 1.) What's going on here? It looks like maybe there's some packet loss and then connection termination gets stuck in a loop because the client timestamp went down? 2.) Is there a better way to mitigate this other than disabling tcp_timestamps or blocking gatech ips? 3.) Is this our problem (ok, obviously our problem since we're affected but...), a kernel problem, or a gatech problem? I'd really appreciate any help on this, Avery -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists