Date: Wed, 11 Feb 2015 22:38:27 +0100
From: Jean-Philippe Aumasson <jeanphilippe.aumasson@...il.com>
To: discussions@...sword-hashing.net
Subject: Re: [PHC] PHC status report

On Wed, Feb 11, 2015 at 10:31 PM, Somitra Sanadhya <somitra@...td.ac.in> wrote:
> In all academic paper submissions in conferences, researchers submit their
> papers and "unpaid volunteers" review the papers.

Reviewing papers is part of an academic's job. And most
conferences/journals do at least single-blind reviews (that is,
reviewers are almost always anonymous). There's been some rigorous
studies showing that it's better than non-blind reviews, btw.



>
> On Wed, Feb 11, 2015 at 11:24 PM, Jean-Philippe Aumasson
> <jeanphilippe.aumasson@...il.com> wrote:
>>
>> Succinct response (not much time to elaborate):
>>
>> * You will find "secretive selection" and non-public discussions of
>> the committee panel in all crypto competitions: AES, NESSIE, eSTREAM,
>> SHA-3, CAESAR, and PHC.
>>
>> * A non-negligible difference of PHC compared to other projects: we're
>> all unpaid volunteers, working on this project on our free time, where
>> NIST had a dedicated team for AES and SHA-3, a budget to organize
>> workshops, etc.
>>
>> * We wished we had time and resources to prepare a lengthy and
>> detailed report, but that wasn't the case. We felt that it was better
>> to publish such a summary than nothing at all.
>>
>>
>> On Wed, Feb 11, 2015 at 10:14 PM, Somitra Sanadhya <somitra@...td.ac.in>
>> wrote:
>> > Dear all,
>> >
>> > I think the discussion is not leading to a meaningful and factual
>> > direction.
>> > Let me humbly try to steer the discussion towards concrete measures and
>> > stop
>> > this blame game. There are some TO-DO items in the comments, which can
>> > help
>> > restore the credibility of the competition (at least among some of us
>> > who
>> > are not happy about it).
>> >
>> > 1. The panel should make the voting and discussion on the selection
>> > criteria
>> > public.
>> >
>> > I refer to the AES competition archived selection criteria here:
>> > http://csrc.nist.gov/archive/aes/pre-round1/aes_9709.htm Note the
>> > language
>> > from section 4: "In order to provide a basis for the analysis and
>> > evaluation
>> > of encryption algorithms submitted to be considered for incorporation
>> > into
>> > the FIPS for AES, evaluation criteria will be used to review candidate
>> > algorithms.  All of NIST’s analysis results will be made publicly
>> > available."
>> >
>> > It is in the spirit of an open competition to keep the process
>> > transparent.
>> > All discussion leading to the selection should have been public already.
>> >
>> > 2. I had pointed this earlier that the comment on our design Rig is as
>> > follows: "Similar to Catena, but received less attention (cf. bugs found
>> > in
>> > the specification and code)".
>> >
>> > There are two parts to this comment. (a) Similar to Catena. (b) bugs
>> > found.
>> >
>> > I would have liked the panel to clarify each of the two points with
>> > respect
>> > to the revised submission Rig v2 which is the one the PHC website lists
>> > since October 2014. Clearly, (b) is false for this. Further, is (a)
>> > measurable by any means ?  Aren't there novel ideas in the design of
>> > Rig,
>> > apart from ideas from Catena ? Given the publicly available eprint
>> > report
>> > which I referred to in my previous mail, is it fair to dismiss the
>> > design in
>> > this single sentence ? If Rig v2 was not being considered, shouldn't the
>> > designers have been informed at the time of the revised submission
>> > itself
>> > that this version will not be evaluated any more ? (Related: weren't
>> > other
>> > design also allowed to be modified around the same time ? The changes
>> > were
>> > not overhauling the design, these were minor modification to handle the
>> > issues which Bill Cox found. I don't think there was any comment on the
>> > design after that. If the panel was not looking at the revised
>> > submission
>> > then we could have as well saved our time to do other things, rather
>> > than
>> > investing our time in something which no one was interested in looking
>> > at.)
>> >
>> > 3. It is not just our design. Most designs have one line comments on
>> > them in
>> > the document shared earlier. To say that the panel could not prepare a
>> > detailed document is mocking the competition. As pointed by Krisztian
>> > earlier, many of these one liners are actually not factual but based on
>> > opinions. The report should have had meaningful comparison of the
>> > candidates, not just one liners on the entries. Dismissing entries by
>> > such
>> > one liners is devaluing the effort put by so many designers in the
>> > competition.
>> >
>> > If you want some specific metrics, then here is a randomly thought list
>> > which is not exhaustive: performance on various platforms, cryptographic
>> > strength, memory hardness, .... (add whatever else you like, make a
>> > baseline
>> > and compare all entries on some rational basis).
>> >
>> > In my humble opinion, the bitterness which we are witnessing in the
>> > mailing
>> > list is due to the secretive selection and the improper rationale for
>> > selection in the document. If these were public and based on detailed
>> > discussions, I don't think anyone would have complained. IMHO, the panel
>> > members should have already realized that there is a lot to blame
>> > themselves
>> > rather than the people questioning their decision now. To blame the
>> > questions on the "frustration of non-finalists" is not showing the
>> > maturity
>> > expected from a panel, which contains many good people whom many of us
>> > trusted (if that was not the case then you wouldn't have received so
>> > many
>> > submissions in the first place). Honestly, please discuss with some
>> > researchers in universities around you about the way the selection has
>> > happened so far, showing them the "selection rationale document" and
>> > "the
>> > process followed" (the secret voting, and not even following that voting
>> > perfectly;  claiming that this was in addition to the private discussion
>> > etc). I am quite certain that none of them will favor the process as
>> > followed. All of it was easily avoided by keeping the process in public
>> > domain and having a well thought out selection document.
>> >
>> > To quote Bruce Schneier on the AES competition (a losing finalist for
>> > the
>> > competition): "I have nothing but good things to say about NIST and the
>> > AES
>> > process".
>> >
>> > We were expecting a competition in the spirit of AES, but unfortunately
>> > things haven't gone that way.
>> >
>> > 4. I deviate slightly to the "flexibility" and "simplicity" ideas of the
>> > AES
>> > competition. The AES competition had a criteria termed "simplicity", but
>> > it
>> > already created lots of discussion/confusion that time. To quote a few
>> > lines
>> > from the TwoFish team (https://www.schneier.com/paper-twofish-final.pdf)
>> > "
>> > "Simplicity” is the NIST criterion that’s hardest to describe. Lines of
>> > pseudocode, number of mathematical equations, density of lines in a
>> > block
>> > diagram: these are all potential measures of simplicity. Our worry about
>> > simplicity as a measure is that the simplest algorithms are often the
>> > easiest to break. ...".
>> >
>> >  If things are hazy then people will question them.
>> >
>> > Regards.
>> > Somitra
>> >
>
>

Powered by blists - more mailing lists