Message-ID: <20150213153354.GC9198@openwall.com>
Date: Fri, 13 Feb 2015 18:33:54 +0300
From: Solar Designer <solar@...nwall.com>
To: discussions@...sword-hashing.net
Subject: Re: [PHC] PHC status report

On Fri, Feb 13, 2015 at 09:49:45AM +0100, Kriszti??n Pint??r wrote:
> On Thu, Feb 12, 2015 at 5:58 PM, Solar Designer <solar@...nwall.com> wrote:
> > The panel has already considered then-available information on the
> > candidates' security and performance.

I'll add: ... to the extent the individual panel members chose to.

I do not claim the reviews so far were necessarily to the maximum extent
possible as limited only by information availability; they were also
limited by the panel members' time allocations.  e.g. if a panel member
concluded early on that they would definitely not support some candidate
as a potential winner, per whatever combination of allowed criteria,
they may very well have skipped analyzing its security and performance
further and instead suggested that the candidate not be made a finalist.
Similarly, for the finalists further security and performance analysis
may have been postponed to the post-tweaks period (now).

That's just rational use of time on the panel members' behalf, but this
does result in us not being able to produce a report that would state
specific security and performance characteristics among reasons for
selecting or not selecting each of the 22 non-withdrawn candidates.  And
in a way that's right: in some cases the primary reasons were different.

> or at least you say so. we can choose to trust your word, or not. we
> can not verify.

Given the explanation above, there's not much to verify.  I agree you
could have better transparency into just how much or how little effort
went into the decision-making so far, but whatever it was, it was the
panel's choice to decide on the finalists in time given whatever it took
into consideration by that point.

I do think there would be a benefit to having the panel's discussion
occur on the public list right away, where the submitters would be
invited to comment.  As JP explained, there were also valid reasons to
handle the discussion in private, though.  Each approach has its pros
and cons.  Personally, I lean towards the public discussion approach,
but I accept that it has drawbacks and that a different choice was made
for PHC so far.

> > In other words, is this just to do the 22 candidates justice (while
> > distracting us from selecting the winners to doing this previous step in
> > what might or might not be a better way)
> 
> uhm, eeer, without proper analysis, you are in no position to base
> your choice on anything.

It was really up to the panel members to decide on how much weight to
give to the different criteria.  If in some cases multiple panel members
deemed a candidate e.g. "not understood well enough" to be a potential
winner, and thus suggested that it not be a finalist, and other panel
members did not object, then so it happened.  (e.g. I do feel bad about
TwoCats not being a finalist despite of its excellent performance.  But
I accept this sort of rationale as valid, so I did not object.)

Personally, I do wish we had more optimized implementations and more
performance metrics, and then we could have given these more weight.
But we had what we had, and I don't find it unreasonable that other
criteria were used.  It's similar with adoption of existing crypto
primitives, etc: people cite many reasons beyond (measurable) security
and performance.  If there's a substantial difference in support for one
scheme vs. another by the panel members now (obviously not including
co-authors of the schemes in question), quite likely it'd be similar for
adoption of the corresponding schemes later.

> this sounds like you have a winner already,
> and the other candidates are just obstacles in your way.

Given the selection of finalists having already been made, PHC's focus
at this time should certainly be on selecting winners from them rather
than on looking back at the non-finalists (although we do that as well,
in this discussion). (*)

I don't see how/why you interpret it as sounding that we "have a winner
already".  No, beyond limiting ourselves to the selected finalists, we
haven't made a choice yet, and I'd expect multiple winners.  Personally,
I can imagine some groups of finalists that would likely make good
winners.  I guess other panel members have their thoughts on this, too.
But nothing is decided yet, and we haven't reviewed the tweaks yet.
(Personally, I am most curious about the tweaks to POMELO - if the
tweaked POMELO is as good as I think it might be, it might win my
preference over specific one or two other finalists.  But that's just me.)

The selection of finalists does limit us to just 1 remaining candidate
in some categories, though.  (And in some of them it was 1 candidate
from the start.)  This does mean that either that 1 is selected as a
winner, or there's no winner fitting that category.  The latter might
happen if we choose to limit the number of winners or/and if the
corresponding finalist is found to be unsatisfactory or/and the category
not important enough.  We'll see.

(*) I think some further references to non-finalists may be desirable in
PHC context - in particular, I think we should compare the finalists'
optimized implementations performance against TwoCats' and hopefully
ensure that they have convincing justifications for running slower (or
we'd be less happy with them when they don't).

Alexander

Powered by blists - more mailing lists