linux-kernel - Re: Applying pipe fix this merge window?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20200213225952.GA5902@localhost>
Date:   Thu, 13 Feb 2020 22:08:34 -0800
From:   Josh Triplett <josh@...htriplett.org>
To:     Linus Torvalds <torvalds@...ux-foundation.org>
Cc:     Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: Applying pipe fix this merge window?

On Wed, Feb 12, 2020 at 12:03:26PM -0800, Linus Torvalds wrote:
> And I realized that I find it surprising that it makes your build
> times noticeably better.
> 
> Yes, I have that silly example program to show the issue in the commit
> message, and yes, the exclusive directed write->read wakeups should
> most definitely improve by that commit.
> 
> But the make jobserver code ends up using "poll()/pselect()" and
> non-blocking reads, because of how it handles the child death signals.
> 
> Which means that none of the nice exclusive directed write->read
> wakeups should even trigger in the first place, because the readers
> never block, and he poll/pselect code doesn't use exclusive wakeups
> (because it can't - it doesn't actually consume the data).
> 
> So I was looking at it, and going "it should actually not help GNU
> jobserver at all" in the fixed jobserver case.

I dug into this a little further yesterday and today:

- With hindsight, I realized that the performance improvements I
  observed for GNU make didn't measure the pipe fix in isolation; they
  measured 5.4 versus ~5.5-rc4 plus the pipe fix, which would include
  all the other pipe work in 5.5 and potentially other optimizations.
  *That* showed substantial performance improvements in GNU make, on the
  order of a couple of seconds in a 30-60 second kernel build. ("5.5-rc4
  plus pipe fix" is what I hammered on for a month on various systems.)

- Measuring the pipe fix patch in isolation
  (0bf999f9c5e74c7ecf9dafb527146601e5c848b9, with and without the pipe
  fix reverted, with nothing else changed), GNU make performance indeed
  doesn't show any difference.

- Other things that use the GNU make jobserver (with pipe fds in
  blocking mode) benefit much more heavily, in wall-clock time and in
  total CPU time. I saw jobs that involved just a minute or two of
  wall-clock time, where the total CPU time went down by *minutes*.

Hope that helps,
Josh Triplett