Message-ID: <556824452.1725880.1452192051219.JavaMail.yahoo@mail.yahoo.com>
Date: Thu, 7 Jan 2016 18:40:51 +0000 (UTC)
From: Eitan Caspi <eitanc@...oo.com>
To: "cert@...t.org" <cert@...t.org>, "vuln@...unia.com" <vuln@...unia.com>,
  "bugs@...uritytracker.com" <bugs@...uritytracker.com>,
  "submissions@...ketstormsecurity.org" <submissions@...ketstormsecurity.org>,
  "bugtraq@...urityfocus.com" <bugtraq@...urityfocus.com>
Subject: Possible vulnerability in F5 BIG-IP LTM - Improper input validation
 of the HTTP version number of the HTTP reqest allows any payload size and
 conent to pass through

Initial note: The vendor has graded this issue as a vulnerability graded as "High" in my email exchange with it, but eventually posted the issue as a "Know Issue", so some of this issue's characteristic that follows can be treated as initial ones, as I ask the IS community to look into this issue and give a "second opinion" about it. Thank you.


Suggested severity level: High (per the vendor's initial response)

Local / Remote activated: Remote

Exploit Code: No need for experiencing the issue, but may be needed to realize an exploit via this issue

Assumed cause for the issue: Improper input validation

Assumed types of Possible Risks/Abuses (all assumed as possible directions, since I did not have the time nor resources to look into them):
Denial of Service
Code Injection
(See more at "CWE-20: Improper Input Validation" - https://cwe.mitre.org/data/definitions/20.html)

Affected Software:
Vendor: F5 Networks
Product: BIG-IP
Module: LTM (Local Traffic Manager)
Versions: 11.6.0, 11.5.3, 11.5.2, 11.5.1, 11.5.0, 11.4.1, 11.4.0, 11.3.0, 11.2.1, 11.2.0, 11.1.0, 11.0.0, 10.2.4, 10.2.3, 10.2.2, 10.2.1, 10.2.0, 10.1.0

Summary:
Input data accepted for the HTTP version number section of the HTTP request is not enforced to be in the correct format, hence any payload content/format and size is getting through without being blocked immediately by an error or security reply, and only when the underlying TCP timeout is reached – only then the base TCP connection is ending by the server side (i.e. no HTTP response is accepted from the remote server side)

Description, reproduction instructions and communication with the vendor (it is taken from the blog post I published about this issue, http://fudie.net/possible-vulnerability-in-f5-big-ip-ltm/, so it is somewhat written as a kind of a "story"):
"
About a year ago, while I was performing a web site penetration test for a customer, I run a manual fuzzing phase, where I like to “question” even the most basic networking and application conventions, and this time it paid off more than the usual…

The site was behind a “F5 Networks” BIG-IP device, running the modules of LTM (load-balancer, https://f5.com/products/modules/local-traffic-manager) and ASM (WAF (Web Application Firewall, https://f5.com/products/modules/application-security-manager)).

Generally, each HTTP request made by the client towards a web site (normally using a web browser) should have its first line in the following format:
<Method> <URL> <HTTP Version>

– The <Method> part is replaced with one of a fixed set of words that describe the action the client wish to perform on the target site (usually GET for viewing a web page) – The <URL> is the site address the client wishes to access – The <HTTP Version> must be in the format of HTTP/<version number> (usually 0.9 or 1.0 or 1.1 which is the most common these days, e.g. HTTP/1.1)

For example:
GET http://www.somesite.com/ HTTP/1.1

I decided to attempt and disregard this format, so first I carefully tried messing with the HTTP version number – first I tried 0.1 (a non-existing HTTP version number) and sent something like:
GET http://somesite.com/ HTTP/0.1

To my surprise, instead of getting a reply of either an error message from the back-end web server or a security blocking message from ASM, for trying to be naughty – simply NOTHING came back, no reply arrived from the server side and the HTTP session ended only when its underlying TCP session timed out.

So, I dug dipper, to try and find what caused this phenomena, to see if there is some logic for this glitch – so… – I tried different numbers (e.g. 1000, 0.5, -4, etc.) and the result was the same.
– I tried without any number (e.g. HTTP/ ) – the result was the same.
– I tried omitting the forward slash – the result was the same.
– I changed HTTP to be helloworld (GET http://somesite.com/ helloworld) – the result was still the same.

So, it looked like there is no format verification and enforcement being made at the BIG-IP end regarding the HTTP version part of the request, a situation that looked risky to me, as the content/payload being pushed from the client to the server is not sanitized properly, possibly being accumulated at some memory, either on any relevant F5 BIG-IP module or at the back-end application server, which may lead to server resources being exhausted or possibly overflowing the memory, which can be much more dangerous, as it may allow the attacker to run there his/her own malicious code at the remote server side.

I tried to run some basic low-power DoS (Denial-of-Service) attacks using this issue, to see at what level, if at all, this issue can harm the target system – but since I didn’t have the sufficient resources to perform a decent large scale DoS attack, I wasn’t able to spot anything more than a minor traffic delay, but not something that I can be sure that this issue was the cause of it.
So, I had to stop my research at this point and turn to the vendor, “F5 Networks”.



First of all, I found the web page in which F5 states about its security vulnerability response policy (“sol4602: Overview of the F5 security vulnerability response policy“,https://support.f5.com/kb/en-us/solutions/public/4000/600/sol4602.html). Unfortunately this page did not include any public key (using PGP) or S/MIME certificate to encrypt the email exchange between me and F5 nor any SSL based web form.
So, since I wanted to report my finding securely, I sent on the 8-Feb-2015 the following email to the email address mentioned at the above web page as the one for reporting security issues – security-reporting@...com. I signed this email with my S/MIME certificate, so F5 will have a way to securely reply to me.
1. I wrote:
”
Hello,

I wish to report something to F5, but at https://support.f5.com/kb/en-us/solutions/public/4000/600/sol4602.html?sr=12234302 there is no mention of any way to security encrypt the email sent to F5, either PGP or s/mime – do you have something in place for this goal? (s/mime is preferred)

Thanks.
”

2. The next day, 9-Feb-2015, I was replied by F5 Support Reply <SupportOne@...com>:
”
Hello Eitan,

I have forwarded your request to our IT department, as I spoke with one of the security people and he does know what you are asking for, someone will be back in contact with you soon in regards to your request.

Respectfully,

<employee-name> | Technical Support Coordinator II ”

3. I promptly replied on the same day:
”
Cool, thanks.

I will be waiting for your move.
”

4. As no one approached me by the 14-Feb-2015, I sent the following email to the support email address I was replied from initially:
”
Hello F5,

Any news on this?
”

5. I was replied on the same day with:
”
Hello, Eitan:

I spoke with our Duty Manager about your security concerns.  He followed up with our IT Security Department.  I was told that they do not have an answer at this moment, but they are working on the problem.

Meanwhile, I am leaving a note for <employee-name>, the woman with whom you communicated last week.  She will be in tomorrow, and can get in touch with the team originally tasked with this.

I apologize for the length of time it is taking to resolve this, but I assure you that we are taking measures.

Regards,

<employee-name>

F5 Networks Support | www.f5.com
”

As no one replied to me by the 21-Feb-2015 I sent the following email (which included all of the above correspondence) to Mr. Manny Rivelo, who at the time was the company’s senior vice president in charge of security (and now he is the former-CEO, https://f5.com/about-us/news/press-releases/f5-networks-announces-appointment-of-long-time-f5-executive-john-mcadam-as-president-and-ceo), and I was in touch with him a few times during the time I worked at F5:
”
Guys, are you serious?! How long does it take?

You don’t treat security seriously.
”

On the same day he replied from his phone:
”
Eitan, I will reach out to our CIO to see what the issue is.
”

I replied on the same day:
”
Cool, it will help my will to securely hand F5 via email a possible security issue and also for others in the future.
Consider also adding a secured web form for this purpose, as a parallel and quick way to submit security issues.

In security response time is crucial.
”

Two days later, on the 23-Feb-2015 I was replied by M., a senior security supporter, who from now on will be my point of contact, and this was the first time I received an official support case number, to track this case:
”
Eitan,

First, apologies for the delay in responding to you.  I wasn’t involved, but I’ve been told there was some confusion with our first tier of support which caused a delay in getting this into the proper channels.

I’ll email you directly in a moment.  I don’t have S/MIME, but we do have two options for secure communication.  The first, and easiest, is <a service F5 uses. I will omit it as it is not relevant here and it is a quite long text. Eitan>

The second method we can use is PGP.  My public key is listed on MIT’s keyserver: <link>

I’ll email you directly twice – once with each method – to make things easier for you.  Then you can respond using your preferred method.

Thank you.

-M.
”

I replied to M. on the same day:
”
Hello M.,

Thank you for jumping in to save the day… 😉

Got your key and added to my keyring (btw, consider giving it an expiration date).
I don’t publish my PK on the net, but I added it here for you.

I will re-verify the issue, write my report about it and send it your way in the coming days, now that I am sure we have a secure (relatively, these days…) communication channel.

Will be in touch soon.

Cheers!
”

I sent M. the report using an encrypted email and on the 2-March-2015 he emailed me:
”
Eitan,
I received and successfully decrypted your email from Saturday.  I’ll attempt to produce the issue in our lab and I will let you know the status.
Thank you
”

On 8-May-2015 M. wrote to me:
”
Eitan,

We worked on this with our product development group and we’ve opened ID518020 for this issue.  We are classifying it as a Vulnerability, with a Severity of High under our policy – covered in SOL4602.  We will be fixing this in our next major release, 12.0.0, due out later this year.  We’re also planning on pulling the fix back to future Hotfix Rollups on our current supported releases.  That work is already underway.

>From our testing, overall it appears that the behavior is basically similar to what we do if we get a connection that doesn’t send a complete request; we’ll eventually disconnect the client via the TCP profile timeout.  The reason these requests aren’t rejected outright is that the profile currently looks for a valid HTTP/1.0 or 1.1 request format – basically <method> <uri> HTTP1.[0|1] – but if it isn’t one of those it presumes it must be HTTP/0.9, which was much more forgiving with respect to the headers.  (HTTP/2 is experimental in the latest 11.6.0 releases, but that’s a different beast entirely.)  So the current profile errs on the side of trying to service the request and falls back to 0.9; a server which strictly enforces 1.0 or 1.1 is likely to reject them outright.

We try to process this HTTP/0.9 request and look for a response from the server, but things break down when we try to proxy the bad request.  The short version is that we send a request to a pool member, but it sees it as HTTP/1.0 and since it isn’t a complete 1.0 header it is waiting for more.  We end up sitting in the middle until our timer fires and we close the connection or one of the endpoints times out and closes it, whichever comes first.  So this all stems from trying to be too accommodating with support for HTTP/0.9 and the fix is to get stricter about what we’ll accept.

The good news is that we’ve conducted volumetric automated testing and we haven’t been able to produce any noticeable issue with the BIG-IP; it has absorbed the queries as intended.  In theory a very high number of requests, from a larger number of attacking systems, could create a DoS situation, but the behavior appears to be the same as other known DoS methods in being related to the TCP profile timeout.  Even with 65,000 simultaneous connections (using all available source ports, in other words) from one host it barely moved the needle on the BIG-IPs resource utilization, so it would need to be a DDoS type of attack from a large number of attack hosts.  The actual cost per connection to the BIG-IP is negligible, it isn’t consuming much in the way of resources.

Overall we do see a few ways we could enhance the BIG-IP’s internal filters to more strictly enforce the request format and reject this kind of exploit in general more readily, and we’re looking into what improvements can be made in future releases.   Based on our testing, this attack has about the same impact on the BIG-IP as opening the TCP connection and sending nothing, until the TCP profile closes the connection.  It isn’t as resource intensive as Slowloris for example, though that’s obviously true on the client side as well, making this attack lower cost.  We tested a number of variations with different types of malformed headers, looking for any different characteristics.

We do appreciate your bringing this to our attention as it has given us a few ideas on ways we can further improve the product.  I’ll be working with our AskF5 team to prepare a Solution article for our site, and we can coordinate the publication with your timing as well.

I hope this explanation addresses the concerns you’ve raised.  Do you have any questions I might address?

Thank you.
”

Following this email we had some more emails exchange regarding various possible risk and mitigations about this issue, and I was asked to supply the title I wished to be presented in the acknowledgment part of the vulnerability support post.

At the 27-May-2015 I received an email from M. that eventually they published the issue at the support site, titled “SOL16672: An improperly formatted HTTP request-line may cause connections to hang and eventually timeout“,https://support.f5.com/kb/en-us/solutions/public/16000/600/sol16672.html.

According to this KB this issue is affecting versions ranging from version 10.1.0 (from December 2009) until a very recent version – 11.6.0 (August 2015) (Click the “Show Version” part on the upper right side of the KB post).

To my surprise the issue was not marked as a vulnerability but as a “Known Issue”, although during my discussions with “F5 Networks” they classified it as a “High” grade vulnerability (see above), and no one updated me, before publishing the support post – that they decided to demote it from a vulnerability.
Even the “Acknowledgments” section was deviated from the normal text pattern they use for vulnerabilities posts (see here an example here - https://support.f5.com/kb/en-us/solutions/public/16000/700/sol16728.html)

I asked M. why, after they admitted it is a “High” grade vulnerability by their own policy – they eventually  played it down to a “Known Issue”?

He replied:
”
Eitan,

There was a lot of discussion about this internally, but in the end the developers and management decided this is not a vulnerability in BIG-IP, but a defect.  The ID was re-categorized as Defect and is being addressed as such.  Hence there will be no CVE requested and the SOL format used was the Known Issue template used for product defects.

Through the testing that was done they were never able to produce a real DoS/DDoS effect on the BIG-IP and they didn’t feel that it rose to the level to be classified as a vulnerability.  But they did feel that it was not the proper handling of these requests, so it is considered a defect to be resolved in upcoming HF releases.  It was a very involved debate, with a lot of arguments made for both sides.  In the end the ‘Defect’ arguments carried the day.

The central point was really trying to rectify the test results, wherein little impact was observed on the BIG-IP, with calling it a Vulnerability and then trying to explain the minimal impact in a SOL.  There was concern that labeling it as such would cause undue concern for customers in light of the negligible impact observed, as there is something of a strong reaction to any Vulnerability SOL independent of the content.  It was agreed however that customers should be made aware of the issue in any case, and the Known Issue SOL was expedited for that purpose.

I had initially argued to handle it as a Vulnerability, but in the end I agreed with the consensus to handle it as a Defect after all of the arguments had been presented.
”

I publish this post because I guess most people are not aware of this issue as it was not flagged as a vulnerability hence folks who collect and react to vulnerabilities report could not possibly know about it.

I hope that with this post more “F5 Networks” customers will be aware of this issue and patch their systems with the fixes mentioned at the KB post (or instead use the suggested mitigation steps offered there as well); and that the security researchers community will learn about this issue, thus researchers with better tools, knowledge and experience than I have – will look deeper into this issue and give this issue a “second opinion”, whether it can be exploited as a vulnerability or not.
"

Direct binary based resolution: Download and upgrade to the latest main version of the LTM module, begging from version 12.0.0 or install a hotfix for matching earlier versions, per the vendor's support article of "sol16672: An improperly formatted HTTP request-line may cause connections to hang and eventually timeout" at https://support.f5.com/kb/en-us/solutions/public/16000/600/sol16672.html

Workaround (in case you cannot apply the above binary code or do not wish to do so):
1.    Create an iRule (an internal BIG-IP script) to enforce correct HTTP version input format validation 2.    In addition it is possible to lower the TCP timeout value Exact details about these workarounds can be found at the vendor's support article of "sol16672: An improperly formatted HTTP request-line may cause connections to hang and eventually timeout" at https://support.f5.com/kb/en-us/solutions/public/16000/600/sol16672.html

Credit: Eitan Caspi, Israel

Past security advisories:

1. CVE-2002-0049 - Exchange 2000 System Attendant Incorrectly Sets Remote Registry Permissions http://www.microsoft.com/technet/security/bulletin/MS02-003.mspx
http://support.microsoft.com/kb/315085/en-us
http://online.securityfocus.com/bid/4053
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CAN-2002-0049

2. CVE-2002-1932 - Defined Actions for Administrative Alerts Do Not Occur When the Security Log Is Full
http://support.microsoft.com/?kbid=329350
http://online.securityfocus.com/bid/5972
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2002-1932

3. User downgraded from Administrator to User retains the ability to list other user's running tasks
http://www.securityfocus.com/archive/1/301624
http://online.securityfocus.com/bid/6280

4. "Compaq Web Agent" management session can be re-used without the need to perform authentication
http://online.securityfocus.com/archive/1/309442
http://online.securityfocus.com/bid/6736

5. Windows XP "welcome screen" exposes the names of all the members of the local administrators group
http://www.securityfocus.com/archive/1/314361
http://www.securityfocus.com/bid/7046

6. Symantec Antivirus client locally created scheduled scan is not running if the local console is logged off
http://www.securityfocus.com/archive/1/393800

7. CVE-2006-2612 - Novell Client login form enables reading and writing from and to the clipboard of the logged-in user http://www.securityfocus.com/archive/1/archive/1/434704/100/0/
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-2612

8. CVE-2006-4886 - McAfee VirusScan Enterprise - disabling the client side "On-Access Scan"
http://www.securityfocus.com/archive/1/archive/1/446220/100/0/
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-4886

9. CVE-2007-0833 - VMware workstation guest isolation weaknesses (clipboard transfer) http://www.securityfocus.com/archive/1/459140/30/90/
http://www.securityfocus.com/bid/22413
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-0833

10. CVE-2007-1056 - VMware Workstation multiple denial of service and isolation manipulation vulnerabilities http://www.securityfocus.com/archive/1/460664/30/60/
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-1056

11. "run as" local denial-of-service enables administrative account processes to be killed www.securityfocus.com/archive/1/472216/100/0/

Email: eitancaspi (at) yahoo (dot) com or via the "Contact" form at the following blogs

LinkedIn Profie: https://www.linkedin.com/in/eitancaspi

Information Security blogs:
FUD for thought (English) - http://fudie.net Not Safe/Sure (Hebrew) - http://security.caspi.org.il

Articles: You can find several IT, business and security articles I wrote some time ago at http://www.themarker.com/misc/search-results?searchType=textSearch&text=eitan+caspi&simpleSearch=simpleSearch

"Technology is like sex. No hands on - No fun." (Eitan Caspi)

Powered by blists - more mailing lists