linux-kernel - Re: AUTOSEL process

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <ZAzrsgAMX+LC9E5A@sashalap>
Date:   Sat, 11 Mar 2023 15:59:30 -0500
From:   Sasha Levin <sashal@...nel.org>
To:     Willy Tarreau <w@....eu>
Cc:     Eric Biggers <ebiggers@...nel.org>, Theodore Ts'o <tytso@....edu>,
        Matthew Wilcox <willy@...radead.org>,
        Pavel Machek <pavel@....cz>, linux-kernel@...r.kernel.org,
        stable@...r.kernel.org, viro@...iv.linux.org.uk,
        linux-fsdevel@...r.kernel.org
Subject: Re: AUTOSEL process

On Sat, Mar 11, 2023 at 09:19:54PM +0100, Willy Tarreau wrote:
>On Sat, Mar 11, 2023 at 11:46:05AM -0800, Eric Biggers wrote:
>> (And please note, the key word here is *confidence*.  We all agree that it's
>> never possible to be absolutely 100% sure whether a commit is appropriate for
>> stable or not.  That's a red herring.
>
>In fact even developers themselves sometimes don't know, and even when they
>know, sometimes they know after committing it. Many times we've found that
>a bug was accidently resolved by a small change. Just for this it's important
>to support a post-merge analysis.
>> And I would assume, or at least hope, that the neural network thing being used
>> for AUTOSEL outputs a confidence rating and not just a yes/no answer.  If it
>> actually just outputs yes/no, well how is anyone supposed to know that and fix
>> that, given that it does not seem to be an open source project?)
>
>Honestly I don't know. I ran a few experiments with natural language
>processors such as GPT-3 on commit messages which contained human-readable
>instructions, and asking "what am I expected to do with these patches", and
>seeing the bot respond "you should backport them to this version, change
>this and that in that version, and preliminary take that patch". It
>summarized extremely well the instructions delivered by the developer,
>which is awesome, but was not able to provide any form of confidence
>level. I don't know what Sasha uses but wouldn't be surprised it shares
>some such mechanisms and that it might not always be easy to get such a
>confidence level. But I could be wrong.

It's actually pretty stupid: it uses the existence of ~10k of the most
common words in commit messages + metrics from cqmetrics
(github.com/dspinellis/cqmetrics) as input.

Although I get a score, which is already set pretty high, confidence is
really non-existant here: at the end it depends mostly on the writing
style of said commit author more than anything.

-- 
Thanks,
Sasha