Message-ID: <4C2DE5BB.6000703@vsecurity.com>
Date: Fri, 02 Jul 2010 09:12:27 -0400
From: VSR Advisories <advisories@...curity.com>
To: bugtraq@...urityfocus.com
Subject: VSR Advisory: Multiple Cisco CSS / ACE Client Certificate and HTTP
 Header Manipulation Vulnerabilities


                   Virtual Security Research, LLC.
                      http://www.vsecurity.com/
                         Security Advisory


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Advisory Name: Multiple Cisco CSS / ACE Client Certificate and HTTP Header
           Manipulation Vulnerabilities
 Release Date: 2010-07-02
  Application: Cisco Content Services Switch (CSS) / ACE Products
     Versions: Cisco CSS 11500 - 08.20.1.01
                Cisco ACE 4710 - Version A3(2.5) [build 3.0(0)A3(2.5)
                (Other versions may be affected)
     Severity: High (in specific configurations)
       Author: George D. Gal <ggal (a) vsecurity . com>
Vendor Status: Cisco CSS vulnerability remains unpatched, workarounds
available
           Cisco ACE workarounds available
CVE Candidate: CVE-2010-1575 - Certificate Spoofing Flaw
               CVE-2010-1576 - HTTP Request Parsing Flaw
    Reference: http://www.vsecurity.com/resources/advisory/20100702-1/

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


Product Description
-------------------
>From [1]:
 "The Cisco CSS 11500 Series Content Services Switch is a high-performance,
  high-availability modular architecture for Web infrastructures. As the
  premiere switch for the Cisco Web Network Services Software, the Cisco
  CSS 11500 Series helps businesses to build global Web networks
  optimized for content delivery and e-commerce. By activating HTTP
  headers, the CSS 11500 Series helps to ensure availability, optimize
  utilization, reduce latency, increase scalability, and enhance security
  for Websites, server farms, cache clusters, and firewall systems."

>From [2]:
  "Cisco(R) ACE Application Control Engine application switches represent
  the state of the art in next-generation application switches for
  increasing the availability, performance, and security of data center
  applications.
  The Cisco ACE family of application switches includes the Cisco ACE
  Service Module for the Cisco Catalyst(R) 6500 Series Switches and Cisco
  7600 Series Routers, as well as the Cisco ACE 4710 Appliance in a
  standalone form factor for discrete data center deployments."


Vulnerability Overview
----------------------
On June 4th 2009, VSR identified multiple weaknesses in the Cisco CSS
11500's handling of HTTP header interpretation and client-side SSL
certificates.  Individually, these issues may be considered minor, but
combined they could allow for the compromise of an application that
relies on a vulnerable CSS to assist in authenticating clients.  If
successfully exploited, an attacker could spoof another application
user's identity without possession of the victim's client certificate.

Additionally, due to the fact that the Cisco CSS product has been
effectively superceded by the Cisco ACE, VSR has also identified
similar issues to those described below in the ACE in particular
configurations.

These issues may affect any CSS installation, but would have the
greatest impact on deployments that have the following feature enabled
in the configuration:

 ssl-server <context> http-header client-cert

Similarly, on the Cisco ACE, these issues may manifest themselves when
using a policy map with a class-default class, as shown below:

 policy-map type loadbalance first-match SLB-VIP-REDIRECT
   class class-default
     serverfarm TEST-FARM
     action DO-SOMETHING-WITH-HEADERS
     insert-http X-SRC-IP header-value "%is"


Issue 1: Weak Enforcement of Authority in HTTP Certificate Headers
------------------------------------------------------------------
Cisco Bug Id - CSCSZ04690
Affects      - Cisco CSS

The first weakness affecting the Cisco CSS is that, in a typical client
certificate configuration, HTTP clients may confuse web applications by
injecting their own certificate headers.  When utilizing the CSS to
terminate SSL communications, SSL client certificates are first
authenticated by the CSS.  From there, the CSS will normally pass the
client's identity to the back-end web server in the form of several HTTP
headers as shown below:

ClientCert-Subject: XXX
ClientCert-Subject-CN: XXX
ClientCert-Fingerprint: XXX
ClientCert-Subject-CN: XXX
ClientCert-Issuer-CN: XXX
ClientCert-Certificate-Version: XXX
ClientCert-Serial-Number: XXX
ClientCert-Data-Signature-Algorithm: XXX
ClientCert-Subject: XXX
ClientCert-Issuer: XXX
ClientCert-Not-Before: XXX
ClientCert-Not-After: XXX
ClientCert-Public-Key-Algorithm: XXX
ClientCert-RSA-Modulus-Size: XXX
ClientCert-RSA-Modulus: XXX
ClientCert-RSA-Exponent: XXX
ClientCert-X509v3-Subject-Key-Identifier: XXX
ClientCert-X509v3-Authority-Key-Identifier: XXX
ClientCert-Signature-Algorithm: XXX
ClientCert-Signature: XXX


However, there is no attempt by the CSS to prevent clients from
supplying their own ClientCert-* headers.  Depending on how application
developers handle multiple copies of these headers, an attacker may be
able to impersonate other users.

For example, assuming that a back-end web application simply trusts
the user identity supplied by the CSS in the ClientCert-Subject-CN
header and userX wants to impersonate userY, he may simply insert
the following HTTP header(s) in the HTTP request issued to the
server:

 ClientCert-Subject-CN: CN=userY

or

 ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userY

Upon injecting the attacker-supplied HTTP headers the application
would receive an HTTP request similar to that shown below:

POST /targetapp HTTP/1.1
Content-Type: text/xml; charset=utf-8
ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userY
ClientCert-Subject-CN: CN=userY
Host: test.vsecurity.com
Content-Length: 1024
ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userX
ClientCert-Subject-CN: CN=userX
ClientCert-Fingerprint: XXX
ClientCert-Subject-CN: XXX
ClientCert-Issuer-CN: XXX
ClientCert-Certificate-Version: XXX
ClientCert-Serial-Number: XXX
ClientCert-Data-Signature-Algorithm: XXX
ClientCert-Subject: XXX
ClientCert-Issuer: XXX
ClientCert-Not-Before: XXX
ClientCert-Not-After: XXX
ClientCert-Public-Key-Algorithm: XXX
ClientCert-RSA-Modulus-Size: XXX
ClientCert-RSA-Modulus: XXX
ClientCert-RSA-Exponent: XXX
ClientCert-X509v3-Subject-Key-Identifier: XXX
ClientCert-X509v3-Authority-Key-Identifier: XXX
ClientCert-Signature-Algorithm: XXX
ClientCert-Signature: XXX


Since existing ClientCert-* headers are left intact, application
developers are expected to trust only the last instance of a given
certificate header.  This approach is clearly prone to error if
application developers do not carefully test this attack scenario.

An alternative approach to securing these headers can be achieved
through an optional configuration where the CSS places an additional
prefix string on the inserted certificate headers [4].  For instance, a
server administrator could select a random header prefix through a
command such as:

 ssl-server <context> http-header prefix "<random_prefix>"

This would cause the new certificate headers to be included with the
form: <random_prefix>-ClientCert-*

So long as an attacker could not discover this random prefix, then there
would be no way to spoof these headers.  However, this solution is far
from ideal, since there may be several ways for an attacker to obtain a
copy of these headers.  Examples include TRACE/TRACK requests being
honored by the back-end web servers or debugging components in web
applications which echo client headers.


Issue 2: Lack of HTTP Request Validation
----------------------------------------
Cisco Bug Id - CSCTA04885
Affects      - Cisco CSS & ACE

A second weakness that manifests itself on the CSS and ACE through
different interpretation of HTTP newline sequences between the content
switch and commonly used web servers.  RFC 2616 [3] defines the US ASCII
carriage return/line feed (CRLF) sequence as the end-of-line marker for
protocol elements (excluding the entity-body). Indeed, the CSS and ACE
appear to adhere relatively closely to this requirement.

Popular web servers, however, permit various permutations of the CRLF
sequence as end-of-line markers, including: LF, CR, and LFCR.  This
difference in interpretation could lead to serious consequences if the
device relies on any of these headers and end-of-line delimiters or
makes changes to any of them.  This has been well documented in the past
in relation to HTTP request smuggling and related attacks [5].


Discussion
----------
It is difficult to consider either of these two issues, taken alone, as
extremely critical flaws.  In the case of certificate header handling,
there are clearly work-arounds for header spoofing attacks, even if they
are error-prone or imperfect.  For HTTP newline interpretation, it is
difficult to fault Cisco for adhering strictly to the RFC.  However, in
combining these issues in typical deployment scenarios the end result
could be catastrophic for an application using a CSS and relying on
client certificates for user identification and authorization.

During testing, VSR found that use of invalid newline sequences caused
the CSS to fail to insert it's own ClientCert-* headers, though the
back-end Apache web server accepted these newline sequences.  This
clearly defeats the approach that some application developers might take
in trying to rely only on the last set of certificate headers. 

The following simple HTTP request demonstrates how a client could cause
the CSS to omit its insertion of the ClientCert-* headers:

 GET /protected_resource HTTP/1.0\x0a
 ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userX\x0a
 ClientCert-Subject-CN: CN=userX\x0a\x0a

However, various combinations of end-of-line delimiters should be
possible. In fact, Cisco describes their logic for the fix as follows:

 "The HTTP request detection will attempt to determine the end-of-line
  marker following the HTTP/1.x line.  If it consists of a single LF,
  then the logic will attempt to identify a LFLF."

Unfortunately, this logic may be problematic when attempting to parse
HTTP requests consisting of various combinations of end-of-line markers
within the same HTTP request.  For instance, many web servers accept
requests with mixed newline sequences such as:

 GET /protected_resource HTTP/1.0\x0d\x0a
 ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userX\x0a
 ClientCert-Subject-CN: CN=userX\x0a\x0a


While testing a Cisco ACE appliance, VSR was able to cause rules that
normally add or remove headers in HTTP requests to be omitted when
the class-default class was used in a policy map as opposed to a layer 7
class for http communications.  It is possible that if a class-default
class were used for client certificate processing on the Cisco ACE
Module, that client certificate headers could also possibly be omitted,
allowing an attacker to insert their own headers used to set the
authenticated user context.

Cisco's recommendation to utilize a layer 7 policy class map as shown
below appears to mitigate this issue, preventing malformed or unexpected
end-of-line delimiters:

 class-map type http loadbalance match-any http-cm
   2 match http url .*

 policy-map type loadbalance http first-match http-pm
   class http-cm
     serverfarm TEST-FARM
     action DO-SOMETHING-WITH-HEADERS
     insert-http X-SRC-IP header-value "%is"


Recommendations for Cisco and Competing Vendors
-----------------------------------------------
VSR recommends that Cisco (and any other vendors who provide similar
products) implement more stringent request validation and/or corrections
when receiving requests which do not utilize HTTP-compliant newlines.
While failing to validate requests forwarded to back-end servers is a
reasonable approach when requests are not interpreted or modified, VSR
believes that some responsibility for correctness should be assigned
when HTTP requests are modified in transit.

Three primary approaches are possible for Cisco CSS/ACE devices and
similar load balancers upon receiving invalid newlines (CR which lack a
trailing LF and LFs which lack a preceeding CR):

 * Requests with invalid newlines could be rejected outright

 * Invalid newlines could be stripped prior to header interpretation or
   modification 

 * Invalid newlines could be converted to valid newlines prior to
   header interpretation or modification

Each of these approaches carries its own set of security, speed and
compatibility risks which should be weighed carefully.  However, given
that load balancers are border devices which commonly terminate SSL,
some level of validation should be implemented to help protect internal
systems.

In addition to addressing newline issues, Cisco CSS devices should
provide better tools and documentation on securing certificate headers.
VSR recommends one or both of the following approaches be adopted:

 * The CSS and ACE should strip any client certificate headers provided
   by clients prior to adding new ones.  Note that this approach cannot
   be guaranteed to work if CR/LF ambiguities are not first corrected.

   Note: Cisco has added a new command to the CSS firmware, currently
   only available by TAC to address this issue via the following command:

    ssl pre-remove-http-hdr

 * The CSS and ACE should require device administrators to specify
   a random header prefix when configuring client certificates, but
   provide a mechanism for existing deployments to disable or opt
   out of a secure by default configuration to support exisitng or
   legacy applications. One way to accomplish this would be to prevent
   the client-cert header insertion configuration from taking effect
   until a device administrator has configured the header prefix using
   the following command syntax:
  
     ssl-server <context> http-header prefix "<random_prefix>"

   This <random_prefix> parameter should be documented as
   "password" or "key" and users should be urged to select an
   unpredictable one, known only by the CSS / ACE administrators
   and the back-end applications.


Recommendations for Web Server Vendors
--------------------------------------
Web server vendors should provide options for requiring strict HTTP
compliance in HTTP headers.  That is, instances of bare CR or bare LF
characters should not be accepted as delimiters of HTTP headers.  These
should either be ignored (and considered a part of the HTTP header
content) or requests containing bare CR and/or bare LF characters should
be rejected.  Eventually, such a configuration option should be enabled
by default.

It is impossible to predict the number of combinations of HTTP proxies,
load balancers and web servers which, in combination, could allow for
serious vulnerabilities along the lines of HTTP request smuggling, or in
this case, certificate header spoofing.  Web server vendors should stop
accepting non-compliant requests in order to help head off future
attacks.



Recommendations for Customers
-----------------------------
There are currently no known workarounds for the end-of-line marker
parsing vulnerability on the Cisco CSS.  On the Cisco ACE customers are
encouraged to utilize a layer 7 class map rather than simply using a
VIP redirect.

In order to mitigate the risk of the ClientCert header insertion
that may lead to user impersonation VSR recommends the use of the
following CSS/ACE command:

 ssl-server <context> http-header prefix "<random_prefix>"

Where <random_prefix> is a difficult to guess value that is
comprised of a combination of 3 of the 4 following character classes:
lowercase alphabetic, uppercase alphabetic, numeric and special
characters.  The value should consist of a minimum of 8 characters,
decreasing the likelihood of successful brute-force attacks.

VSR strongly recommends disabling the TRACE/TRACK HTTP methods on the
target web/application server and periodically changing the random
prefix.


Other work arounds are possible if application administrators/developers
are able to perform additional validation of user identities.  One
possibility would be to simply disable SSL termination at the Cisco CSS,
instead performing certificate validation at the application server
level.  However, this may not be feasible in many environments.

As an alternative, user certificate headers could undergo a second round
of validation at the application server level.  For instance, randomized
user IDs or user-specific secret values could be embedded in user
certificate headers which would be passed along by the CSS to
applications.  Once received, application servers would validate this
additional parameter to help prevent header spoofing in a user-specific
way.  However, while this could prevent blind certificate spoofing, an
attacker who could perform man-in-the-middle attacks would likely be
able to obtain certificate headers of other users and subsequently
bypass this protection on the server side.


Versions Affected
-----------------
The described end-of-line interpretation behavior exists in all versions
of the Cisco CSS and Cisco ACE (Application Control Engine) and has
only been partially updated in version 8.20.4.02 and ACE A2(3.0).

Cisco has stated that the new behavior to address this issue in the CSS
applicance is to look for the terminator of LFLF if the separator that
follows the HTTP/1.X is a single LF.  However if it is a CRLF pair the
CSS will only search for CRLF as end-of-line markers, however it does
not currently address situations where mixed end-of-line markers are
used.

The client supplied header insertion vulnerability continues to affect
all versions of the Cisco CSS and ACE (Application Control Engine),
however a mitigation exists and is described above.


Vendor Response
---------------
The following timeline details Cisco's response to the reported issue:

2009-06-05    VSR submitted a security bug report to Cisco PSIRT
2009-06-06    Cisco confirmed receipt of bug report
2009-07-02    Cisco acknowledged the presence of VSR submitted
              vulnerabilities
2009-08-04    Cisco confirmed release plans for end-of-line marker
          parsing vulnerability
2009-10-14    Cisco provided update on defect notes and remediation
          approach
2010-03-11    VSR reviewed the Cisco release notes on potential CSS bug fix
          and provided Cisco with notice indicating that these
          fixes are inadequate
2010-03-12    Cisco confirmed receipt of correspondence
2010-04-07    Conference call between VSR and Cisco to discuss security
          ramifications and understand implementation specifics of Cisco
          ACE
2010-05-21    VSR performed testing against Cisco ACE 4710 in a lab
verifying
          end-of-line parsing issue in default class map configuration
2010-05-26    Cisco provided guidance on utilizing a layer 7 class map to
          address issue on Cisco ACE
2010-06-28    VSR performed verification of layer 7 class map in a lab
          verifying correct device behavior
2010-06-29    Cisco and VSR begin coordinating advisory release
2010-07-01    Cisco was provided a draft advisory
2010-07-02    VSR advisory released


Common Vulnerabilities and Exposures (CVE) Information
------------------------------------------------------
The Common Vulnerabilities and Exposures (CVE) project has assigned
the numbers CVE-2010-1575 and CVE-2010-1576 to these issues.  These
are candidates for inclusion in the CVE list (http://cve.mitre.org),
which standardizes names for security problems.


Acknowledgements
----------------
Thanks to Cisco for response and cooperation, Tim Morgan for assistance in
attempting to verify this issue on other competing load-balancers, and
Ken Pierce for providing a lab environment to conduct Cisco ACE testing.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

References:

1. Cisco CSS 11500 Series Content Services Switches
   http://www.cisco.com/en/US/products/hw/contnetw/ps792/

2. Cisco ACE Application Control Engine Application Switches
  
http://www.cisco.com/en/US/prod/collateral/modules/ps2706/ps6906/AAG_Cisco_ACE_Application_Control_Engine_Application_Switches.pdf

3. RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1
   http://tools.ietf.org/html/rfc2616

4. Configuring SSL Termination:
   Adding a Prefix to the Fields Inserted in the HTTP Header
  
http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/css11500series/v8.10/configuration/ssl/guide/terminat.html#wp1026217

5. HTTP Request Smuggling
   http://www.cgisecurity.com/lib/HTTP-Request-Smuggling.pdf

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
    Copyright 2010 Virtual Security Research, LLC.  All rights reserved.

Powered by blists - more mailing lists