Date: Mon, 11 Jan 2010 12:45:33 +0100
From: Thierry Zoller <Thierry@...ler.lu>
To: Anthony Desnos <desnos@...ea.fr>
Cc: full-disclosure@...ts.grok.org.uk
Subject: Re: iAWACS 2010 : Rules of the PWN2KILL contest

Hi Anthony,

AD> The PWN2KILL Contest aims at performing a comparative evaluation of
AD> commercial
AD> antivirus software against actual threats.
AD> An actual threat can be defined as any threat that is operationnally
AD> viable.

The challenge is rather large and the goals not really clear, based of
above,  "pwn" includes dropping custom malware and checking whether
it    is    detected. Installing a rootkit is counted as pwned? You do
not include  the hardware details of the machine for instance if there
are cpu vitalization features supported?

Apparently  proactive  detection  rules can simply be ignored based on
the  assumption  a  grandma will click yes anyways.(below) I am not
sure  thought a grandma really provides the incentive to create custom
code in real life ;)

Will this really will prove anything, from my experience all
an  every  anti-virus  software can be pwned (as per your definition)
with custom unknown code. What  is left are the Windows7 ACLs which
you need to bypass also, these can be more of a problem then
"bypassing" AV.

I  am  with  Sergio,  what is there to gain for somebody that spends x
weeks  on  targets  ?  Apart  from having their name displayed on your
website, that might not be enough for anybody ;)

Regards,
Thierry


AD> As an
AD>        example, warnings like ``an application is attempting to become
AD> resident.
AD>        Do you allow it?'' has no meaning for a grandmother using a
AD> computer.
AD>        She is free to allow it!


AD>    2.- Each participant will come with his (malware) code(s) to test
AD> against
AD>        the antivirus software. He can perform any action that a normal
AD> user can
AD>        do (including rebooting the computer, closing a session, using USB
AD>        devices...). In case of ``proactive'' warning from the operating
AD> system
AD>        or from any application, the user is free to follow them or not.
AD> Any user
AD>        has not to be an expert in computers in order to evaluate and
AD> interpret
AD>        technical warnings that sometimes refers to normal behaviours. As an
AD>        example, warnings like ``an application is attempting to become
AD> resident.
AD>        Do you allow it?'' has no meaning for a grandmother using a
AD> computer.
AD>        She is free to allow it!

AD>    3.- In order to make a comparative and fair testing, any code must be
AD> tested
AD>        against ALL antivirus selected for the challenge. The test will
AD> consist
AD>        in two step~: first the code(s) will be scanned (on demand analysis)
AD>        then used as intended (on-access analysis).

AD>    4.- Any participant will have first to announce what effect/attack he
AD> intends
AD>        to perform. The board will decide whether this attack is
AD> admissible or
AD>        not. An admissible attack is an attack which affect availability,
AD>        integrity and/or confidentiality of the system and/or the data (data
AD>        system, user data...).

AD>    5.- Any participant will have to write a short technical summary of his
AD>        attack(s) which will be published on the iAWACS website. He will
AD> have to
AD>        present his attack(s) during the contest debriefing. A copy of
AD> its code
AD>        will be given to the organizers of the challenge.

AD> For fairness purposes, no participants working for any AV company or any
AD> company sharing common interest with AV companies, will be allowed to
AD> participate. Any participant will thus have to sign an assessment form
AD> confirming he is not working for such companies.
AD>  
AD> The organizers of iAWACS 2010 and of the PWN2KILL challenge have
AD> selected the
AD> following antivirus software:
AD>    -- Avast
AD>    -- AVG
AD>    -- Avira
AD>    -- BitDefender
AD>    -- DrWeb
AD>    -- FSecure
AD>    -- GData
AD>    -- Kasperky
AD>    -- McAfee
AD>    -- Microsoft AV
AD>    -- NOD 32
AD>    -- Norton Symantec
AD>    -- Trend Micro
AD>  
AD> Only commercial licences will be tested -- in other words they will be
AD> anonymously bought in public stores/website (no demo or free version).
AD> The antivirus will be updated right before the beginning of the challenge.

AD> The organizers will publish a technical summary of the results once
AD> validated
AD> by the contest board. No communication will be done directly towards the AV
AD> vendors. Only a technical communication and press conference will be
AD> organized
AD> during the iAWACS event. A technical summary will be available on the
AD> iAWACS
AD> website. The complete data and codes collected will be communicated only
AD> to the
AD> French CERT-A for analysis and feedbacks. No code will be neither
AD> published nor
AD> distributed.

AD> Any participant is free to communicate later on about his test/code/attack
AD> performed during the contest. In this case, iAWACS organizers are not
AD> responsible for that communication.

AD> _______________________________________________
AD> Full-Disclosure - We believe in it.
AD> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
AD> Hosted and sponsored by Secunia - http://secunia.com/



-- 
http://blog.zoller.lu
Thierry Zoller


_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

Powered by blists - more mailing lists