Message-ID: <03c901c6767e$d3e5fde0$2201a8c0@ngssoftware.com>
Date: Sat, 13 May 2006 12:17:31 +0100
From: "David Litchfield" <davidl@...software.com>
To: "Ferguson, Justin (IARC)" <FergusonJ@...doe.gov>,
	"'Adam Shostack'" <adam@...eport.org>
Cc: full-disclosure@...ts.grok.org.uk, ntbugtraq@...tserv.ntbugtraq.com,
	bugtraq@...urityfocus.com, dbsec@...elists.org
Subject: Re: How secure is software X?


Hi Justin,
> One thing you have to keep in mind is that a lot of things are incredibly
> variable when dealing with this subject. For instance, suppose you want to
> ensure that the URI in a web server is not overflowable. So you test with
> something like
>
> GET /[AAAAAAAAA x 4096] HTTP/1.1
> Host: foobar.com
> Connection: close
>
> This is all fine and well, unless at 8192 is where the overflow takes 
> place,
> or if I can stick as many characters as I want in, so long as I am using
> HTTP 1.1 and not HTTP 0.9, or if I am using HTTP/1.1 and Host doesn't
> contain a 36 backslashes, et cetera.
>
> This is generally why fuzzing is mostly inconclusive because it often 
> misses
> a lot of conditions and you have essentially assured nothing. Without
> in-depth analysis of each software package you are basically pushing snake
> oil. There are just far too many variables to really standardize such a
> thing.


There are a few things to remember:
1) There are still too many products that fall to simple fuzzing. Having a 
standard that employs fuzzing as part of it means that (hopefully!) vendors 
will develop at least to that level - this raises the bar so to speak.
2) Not all fuzzers are born equal. Having written a fair few in my time I do 
realize that condition based fuzzing is important. A very simple but quite 
common example, to add to the ones you given, is with SMTP fuzzing. Some 
overflows only trigger after an EHLO greeting but not after a HELO. A good 
fuzzer and a good fuzzing process should take into consideration as many 
conditions as possible.
3) Fuzzing would only be part of the standard to be proposed. There are 
code/assembly scanning tools which can be incorporated amongst other things.


Cheers,
David


>
> Best Regards,
>
> Justin Ferguson
> Reverse Engineer
> NNSA IARC
> 702.942.2539
>
> "It is a capital mistake to theorize before one has data. Insensibly one
> begins to twist facts to suit theories, instead of theories to suit 
> facts."
> -- Sir Arthur Conan Doyle
>
>> -----Original Message-----
>> From: Adam Shostack [mailto:adam@...eport.org]
>> Sent: Friday, May 12, 2006 11:35 AM
>> To: David Litchfield
>> Cc: bugtraq@...urityfocus.com;
>> full-disclosure@...ts.grok.org.uk;
>> ntbugtraq@...tserv.ntbugtraq.com; dbsec@...elists.org
>> Subject: Re: How secure is software X?
>>
>>
>> Hi David,
>>
>> Very briefly because I'm swamped today:  Please consider
>> bringing some of this to Metricon
>> (https://securitymetrics.org/content/Wiki.jsp?page=Welcome)
>>
>> Also there's a project of US DHS/NIST and probably others
>> called SAMATE Software Assurance Metrics and Tool Evaluation
>> http://samate.nist.gov/index.php/Main_Page
>>
>> which might be of interest.
>>
>> Adam
>>
>> On Fri, May 12, 2006 at 02:59:17AM +0100, David Litchfield wrote:
>> | How secure is software X?
>> |
>> | At least as secure as Vulnerability Assessment Assurance
>> Level P; or Q
>> | or
>> | R. Well, that's what I think we should be able to say. What
>> we need is an
>> | open standard, that has been agreed upon by recognized
>> experts, against
>> | which the absence of software security vulnerability can be
>> measured -
>> | something which improves upon the failings of the Common
>> Criteria. Let's
>> | choose web server software as an example. When looking for
>> flaws in a new
>> | piece of web server software there are a bunch of well
>> known checks that
>> | one would throw at it first. Try directory traversal
>> attacks and the
>> | several variations. Try overflowing the request method, the
>> URI, the query
>> | string, the host header field and so on. Try cross site
>> scripting attacks
>> | in server error pages and file not found messages. As I
>> said, there's a
>> | bunch of checks and I've mentioned but a few. If these were
>> all written
>> | down and labelled with as a "standard" then one could say
>> that web server
>> | software X is at least as secure as the standard -
>> providing of course the
>> | server stands up.
>> |
>> | For products that are based upon RFCs it would be trivial
>> to write a
>> | simple
>> | criteria that tests every aspect of the software as per the
>> RFCs. This
>> | would be called Vulnerability Assessment Assurance Level:
>> Protocol. If a
>> | bit of software was accredited at VAAL:Protocol  then it
>> would given a
>> | level of assurance that it at least stood up to those attacks.
>> |
>> | Not all products are RFC compliant however. Sticking with
>> web servers,
>> | one
>> | bit of software might have a bespoke request method of
>> "FOOBAR". This opens
>> | up a whole new attack surface that's not covered by the
>> VAAL:Protocol
>> | standard. There are two aspects to this. Anyone with a
>> firewall capable of
>> | blocking non-RFC compliant requests could configure it to
>> do so - thus
>> | closing off the attack surface - from the outside at least.
>> As far as the
>> | standards go however - you'd have to introduce criteria to
>> cover that
>> | specific functionality. And what about different
>> application environments
>> | running on top of the web server? And what about more
>> complex products such
>> | as database servers? I suppose at a minimum for DB software
>> you could at
>> | least have a standard that simply checks if the server
>> falls to a long
>> | username or password buffer overflow attempt and then fuzz
>> SQL-92 language
>> | elements. It certainly makes standardization much more
>> difficult but I
>> | think by no means impossible.
>> |
>> | Clearly, what is _easy_ is writing and agreeing upon a VAAL:Protocol
>> | standard for many different types of servers. You could
>> then be assured
>> | that any server that passes is at least as secure as
>> VAAL:Protocol and for
>> | those looking for more "comfort" then they can at least
>> block non-RFC
>> | compliant traffic.
>> |
>> | Having had a chat with Steve Christey about this earlier
>> today I know
>> | there
>> | are other people thinking along the same lines and I bet
>> there are more
>> | projects out there being worked on that are attempting to
>> achieve the same
>> | thing. If anyone is currently working on this stuff or
>> would like to get
>> | involved in thrashing out some ideas then please mail me -
>> I'd love to hear
>> | from you.
>> |
>> | Cheers,
>> | David Litchfield
>> | http://www.databasesecurity.com/
>> | http://www.ngssoftware.com/
>>
>
> 

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Powered by blists - more mailing lists