Date: Thu, 13 Aug 2009 18:36:51 +0300
From: Gichuki John Chuksjonia <chuksjonia@...il.com>
Cc: full-disclosure@...ts.grok.org.uk
Subject: Re: [Professional IT Security Providers -
	Exposed] Redspin, Inc. (C+)

Just read this.

What happened to your blog, http://secreview.blogspot.com?



On 8/11/09, secreview <secreview@...hmail.com> wrote:
> We received 22 requests from different people to perform a review of
> Redspin! Their website can be found at http://www.redspin.com. We
> haven’t done a review of anyone in quite a while, the last review that
> we did was for Pivot Point Security who got an A (we still recommend
> them). We apologize for this long delay but we have been very busy
> traveling (yes we still have jobs doing consulting work sometimes).
>
> As you can see from the comments that we received in other posts we
> have a lot of catch up work to do, but to be honest we are not sure
> that we will be able to do it. This review might be our final and last
> review depending on how much more travel we have. (We have lives, some
> of us have families, and we can’t keep doing this for free even though
> we feel that this is a great service).
>
> We did a lot of research on Redspin and we managed to get a copy of two
> reports that they did for two different customers. We won’t share those
> reports with you because that would be unethical, don’t ask.
>
> Redspin claims that it is a “pure penetration testing firm”. What they
> mean by “pure penetration testing” is that they do not resell third
> party software or hardware. They also say that “don't find problems on
> your network so that [they] can make more money; [their] penetration
> testing services reveal vulnerabilities, [that] will help you become
> more secure.”
>
> We verified their claim with our own research. Redspin will not try to
> sell you software or hardware… but they might try to sell you software
> as a service. (see their www.jetmetric.com website).
>
> Redspin takes it a step further and is brutally honest about their
> methodology for delivering penetration-testing services. They openly
> admit that their services rely on automated vulnerability scanners
> (Nessus) and are enhanced by manual testing. In fact, Redspin says that
> automated scanners “can miss about 40% of the security risk so they
> alone do not adequately assess risk. Furthermore, about half of the
> findings from a vulnerability scan are false positives”.
>
> Any security company that relies on automated scanners can weed out
> false positives, but doing that doesn’t really increase the depth and
> accuracy of testing. A false positive, also known as an error of the
> first kind, or a Type I Error, is the rejection of a null hypothesis
> when it is in fact true. In more simple terms, this is the error of
> observing a difference when in fact there isn’t one. Identifying false
> positives is fairly easily done, as it only requires inspecting the
> results produced by a scanner.
>
> But what about False Negatives? A False Negative, also called a Type II
> Error, or an error of the second kind, is the error of failing to
> reject a null hypothesis when it is in fact not true. More simply, a
> False Negative is the error of failing to observe a difference when in
> truth there is one. So, if an automated vulnerability scanner tests a
> vulnerable service (a known vulnerability) but the scanner doesn’t
> detect the vulnerability then the vulnerability is excluded from the
> report. If this is the case then Redspin’s methodology will break down
> because there will be no result in the report for Redspin to manually
> test. That vulnerability will fly under the Redspin radar but might not
> be missed by a hacker. So how many vulnerabilities does Redspin miss?
> It’s a question worth asking.
>
> Redspin does say that “vulnerability scanning is not suitable on its
> own as a complete or billable service offering, it does provides some
> value in the early reconnaissance phase of a more comprehensive
> External Network Security Assessment”. They have a typo in that
> sentence, but other than that, they are right. Vulnerability scanning
> does have a position in the industry and is a huge time saver,
> especially when testing large numbers of systems. Just don’t rely on
> one vulnerability scanner like Redspin does, use two or more like the
> OSSTMM proposes.
>
> Redspin says “manual analysis is at the heart of all of [their]
> assessments which not only gives you confidence that you have a
> complete view of your security risk, but provides tailored reporting
> and recommendations enabling simple work-arounds and cost-effective
> mitigation strategies for most security issues.” Based on our research
> Redspin’s “manual analysis” isn’t what we expected it to be. It is not
> based on vulnerability research and is strictly based on the inspection
> and verification of scanner output.
>
> What we can say is that their “manual analysis” doesn’t produce the
> highest quality reports that ever we’ve seen, but it does produce
> reports that are higher than average quality. The Redspin reports have
> very few, if any, False Positives but will contain more False Negatives
> than a report that is centered on solid (vulnerability) research.
>
> One thing that Redspin does that we really don’t like is to ask their
> customers to lower their defenses before they do testing. That’s right,
> they ask their customers to white list their scanner’s IP addresses so
> that the customer’s Intrusion Prevention System doesn’t block the
> scanner. We verified this during 3 different interviews on three
> different dates. We even talked to one Redspin customer to confirm it,
> and they did. We think that a security testing company should be able
> to test around a customer Intrusion Prevention System. If they can’t
> then that really brings their capabilities into question.
>
> We feel this way because Intrusion Prevention Systems are a part of the
> networks defenses and they should be tested. Disabling them for a
> security test prevents them from being tested. If they aren’t tested
> then how does one know how effective they are? It just doesn’t’ make
> sense. On top of that, the test won’t properly reflect the actual
> security level of the network being tested.
>
> Something that Redspin claims is that they’ve done is “ground breaking
> security research”. We’ve searched high and low for this “ground
> breaking security research” but haven’t found it anywhere, so we’re not
> sure what they are talking about. When looking at the research page on
> their website we see white papers that might make good blog entries,
> but we don’t see any “ground breaking security research”.
>
> When we’re told that a company does “ground breaking security research”
> we expect to see things like them finding security bugs in critical
> systems, or publishing professional security advisories, and maybe even
> publishing proof of concept code. Redspin doesn’t do any of that. The
> only thing that we were able to find was an “Ultr@ VNC 1.0.1 viwer PoC”
> (and what’s the point of that?).
>
> In conclusion, Redspin’s services are slightly better than average.
> Their manual testing isn’t true manual testing at all; it’s the
> inspection of output from scanners and the elimination of false
> positives. We don’t like the fact that Redspin asks its customers to
> disable their IPS before being tested, and Redspin doesn’t seem to have
> any Vulnerability Research capability.
>
> Its not all bad, Redspin is very honest about their methodology, they
> are focused on quality, and they are passionate about what they do.
> We’d recommend Redspin to people with testing requirements that do not
> require extreme depth and that can afford some False Negatives. By no
> means is Redspin a company that we’d suggest you stay clear of, but
> they’re certainly not the best in the industry.
>
> As normal, if there are any issues with this review and its
> truthfulness please let us know and please provide proof. We will make
> changes if we need to and we strive to be as honest and fair as we can
> be. Thanks for reading!
>
> Score Card (Click to Enlarge)
>
>
>
>
> --
> Posted By secreview to Professional IT Security Providers - Exposed at
> 8/10/2009 08:51:00 PM


-- 
-- 
Gichuki John Ndirangu, C.E.H , C.P.T.P, O.S.C.P
I.T Security Analyst and Penetration Tester
infosigmer@...ox.com

{FORUM}http://lists.my.co.ke/pipermail/security/
http://nspkenya.blogspot.com/
http://chuksjonia.blogspot.com/

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Powered by blists - more mailing lists