linux-kernel - Re: [PATCH v9 0/5] rust: adds Bitmap API, ID pool and bindings

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CACQBu=WL=RLN_9sU-DcJaRxCrCcokFhHvOF4K+=Ui+=POH9jVQ@mail.gmail.com>
Date: Mon, 2 Jun 2025 11:52:38 +0200
From: Burak Emir <bqe@...gle.com>
To: Yury Norov <yury.norov@...il.com>
Cc: Kees Cook <kees@...nel.org>, Rasmus Villemoes <linux@...musvillemoes.dk>, 
	Viresh Kumar <viresh.kumar@...aro.org>, Miguel Ojeda <ojeda@...nel.org>, 
	Alex Gaynor <alex.gaynor@...il.com>, Boqun Feng <boqun.feng@...il.com>, 
	Gary Guo <gary@...yguo.net>, Björn Roy Baron <bjorn3_gh@...tonmail.com>, 
	Benno Lossin <benno.lossin@...ton.me>, Andreas Hindborg <a.hindborg@...nel.org>, 
	Alice Ryhl <aliceryhl@...gle.com>, Trevor Gross <tmgross@...ch.edu>, 
	"Gustavo A . R . Silva" <gustavoars@...nel.org>, rust-for-linux@...r.kernel.org, 
	linux-kernel@...r.kernel.org, linux-hardening@...r.kernel.org
Subject: Re: [PATCH v9 0/5] rust: adds Bitmap API, ID pool and bindings

On Tue, May 27, 2025 at 4:27 PM Yury Norov <yury.norov@...il.com> wrote:

> So, 95% CI means z=1.96, isn't? And to me it should be, for example for
> the first line: 5.18 +- 1.96*0.32/sqrt(32) = 5.18 +- 0.11 = [5.07, 5.29].
> Can you check your math please?

Facepalm... I used scipy to get the CI, and passed alpha (5%) instead
of 1-alpha (95%) for confidence level.
I am very sorry...

> > Results for sparse bitmap:
> > +------------+------+-----------+--------------+-----------+-----------+
> > | Benchmark  | Code | Mean (ns) | Std Dev (ns) | 95% CI Lo | 95% CI Hi |
> > +------------+------+-----------+--------------+-----------+-----------+
> > | find_bit/  | C    |     22.51 |        12.34 |     22.38 |     22.65 |
> > | next_bit   | Rust |     30.53 |        20.44 |     30.30 |     30.75 |
> > +------------+------+-----------+--------------+-----------+-----------+
> > | find_zero/ | C    |      5.69 |         0.22 |      5.68 |      5.69 |
> > | next_zero  | Rust |      5.68 |         0.29 |      5.68 |      5.68 |
> > +------------+------+-----------+--------------+-----------+-----------+
>
> Your numbers look pretty weird. I wrote the test such that on a typical
> x86 system it takes milliseconds for each subtest to pass.  Here you
> report nanoseconds-scaled times. Did you divide by the number of
> iterations? If so, please mention it.

Yes, I had divided by number of iterations.
Now that you are questioning this, I have second thoughts.
I will repeat the benchmark, without dividing by iterations.

> Please print raw output of your test in patch #4 which adds the test.
> Because the test is tightly coupled to it's C version, we need to make
> sure it hast the same format - fields alignment, etc.

Will do. In v10, I have removed the module name from the output and
the lines match exactly now.

> I would prefer to have detailed performance discussion in the
> corresponding patch (#4), and here in cover letter you'd just mention
> the overall performance difference - 2%, as I can see.
>
ok.

cheers,
Burak