full-disclosure - Flaw in Syn Attack Protection on non-updated Microsoft OSes can lead to DoS

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date: Mon Nov 28 23:46:00 2005
From: lm at symbolic.it (Luigi Mori)
Subject: Flaw in Syn Attack Protection on non-updated
 Microsoft OSes can lead to DoS

Flaw in Syn Attack Protection on non-updated Microsoft OSes, can lead to DoS

Summary

It is possible to mount a DoS attack against Windows 2000/2003 hosts where
the SYN attack protection has been enabled. The attacker can consume all CPU
resources of the victim host making it unresponsive.
While a standard SYN flood attack can make a single application server
unavailable, this attack can make the whole host unreachable.

Systems Affected

Windows 2003 without SP1
Windows 2000 SP4 without Update Roll-Up

Description

On Windows 2000/2003 the system administrator can enable a SYN Attack
protection mechanism on the TCP/IP by adding the value SynAttackProtect in
the registry key HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters.
If the value of SynAttackProtect is 2 the TCP/IP stack notifies a listening
socket only when the 3-way handshake has been completed and tracks the
ongoing 3-way handshakes by storing them in an hash table.
This way the backlog of the socket is defended from the SYN floods attacks.

SynAttackProtect is not enabled by default on the affected systems but has
been recommended by a number of articles:
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q315669&sd=tech
http://www.microsoft.com/technet/security/topics/networksecurity/secdeny.mspx
http://msdn.microsoft.com/library/en-us/dnnetsec/html/HTHardTCP.asp
http://support.microsoft.com/default.aspx?scid=kb;en-us;142641
http://www.securityfocus.net/infocus/1729
http://www.awprofessional.com/articles/article.asp?p=371702

The vulnerability resides in the hash table management, in fact the
hash function used by the TCP/IP stack works only on some fields of the
incoming SYN packet and is thus predictable. An attacker can generate
a large number of SYN packets with the same hash value to target the same
hash table bucket. When the victim machine receives them, it stores them
in just one bucket of the hash table. The chain attached to this bucket
keeps growing, and the more it grows, the slower the lookup algorithm
becomes.

Vendor response

I've notified Microsoft of the vulnerability 2 years ago, when
the attack was possible on the Windows 2000 version (SP3) in production
at that time.
They confirmed the vulnerability but didn't release a patch because
the correction needed extensive changes in the code of the TCP/IP stack.
Microsoft has patched the vulnerability in Windows 2003 SP1 and
Windows 2000 Update Roll-up but it has inadvertently forgot to notify me.
The new version of TCPIP.SYS has this Syn Attack Protection enabled by
default but uses a crypto hash function (MD5) for the table lookup.
The hash material is the source port, dest port, source ip, dest ip of the
SYN packet and some pseudo random material extracted at startup.
This way the hash function is not easily predictable.

-- 
Luigi Mori

Symbolic S.p.A.
W: www.symbolic.it
T: +390521708811