linux-kernel - Re: I disabled more compiler warnings..

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <73dace5aca21bee09ce12aa8dcfd50daa2cd6051.camel@gnu.org>
Date:   Tue, 12 May 2020 10:35:57 -0400
From:   Paul Smith <psmith@....org>
To:     David Laight <David.Laight@...LAB.COM>,
        'Linus Torvalds' <torvalds@...ux-foundation.org>
Cc:     'Arnd Bergmann' <arnd@...db.de>,
        'Masahiro Yamada' <yamada.masahiro@...ionext.com>,
        'Linux Kernel Mailing List' <linux-kernel@...r.kernel.org>
Subject: Re: I disabled more compiler warnings..

On Tue, 2020-05-12 at 07:55 +0000, David Laight wrote:
> > One problem is ensuring that all the recursive makes actually
> > use the same token queue.
> > The Linux kernel build acts as though the sub-makes have their
> > own queue - I certainly had to fix that as well.

I don't understand this... I guess I'm not familiar enough with the
kernel build system.

> I think I've remembered the obvious thing that made it work better.
> 
> When a job ends it is important to get a new token from the jobserver
> rather than reusing the one to hand.
> Otherwise you don't seen the 'abort' marker for ages.

If GNU make retrieved a token then it will always put that token back
into the jobserver pipe when the job ends, and get another one when the
next job is to start.  To do otherwise would mean that some makes could
hoard tokens.

However, the jobserver is implemented such that make itself is not
considered a job, even a sub-make.  The way it works is that when you
invoke a recursive make the parent make will obtain a jobserver token
for that recursive invocation (like it does for every job), then that
sub-make can "pass on" that token: in other words, the sub-make has a
free token that it can always use without querying the jobserver.

This way every invocation of recursive make can always make progress,
at least serially.

I can see that in the "fast fail" model this could be problematic, but
it should only ever be an issue in situations where a sub-make was
running serially for some reason: either the structure of the
prerequisites means it's naturally serial, or else someone added
.NOTPARALLEL to the makefile or something.  As soon as make wants to
run a second job in parallel it will go to the jobserver and discover
the "failure" token.

Changing this will require thought.  We can't just skip the free token
otherwise you can get into a state where all your tokens are used by
recursive makes and no make can get a new token to run a job.

I can see two possible solutions:

First, when a sub-make starts it could put back one token into the
jobserver, representing the token the parent make obtained for it, then
proceed to always get a token before every job (no free token).  This
means that sometimes a sub-make won't be able to run any jobs at all:
it can get locked out waiting for a token.  Maybe that's not a problem.

The other idea is to keep the free token but make it a last resort
rather than a first resort.  This has the nice properties that (a)
we'll see failures fast and (b) we still have a free token, but the
code is more complex: basically we'd need to perform a non-blocking
read on the jobserver FD and if we didn't get anything back, we'd use
our free token if it's still available: if not we'd do a blocking read
on the jobserver FD to wait for a new token.