linux-kernel - Re: I disabled more compiler warnings..

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 11 May 2020 13:58:52 -0400
From:   Paul Smith <psmith@....org>
To:     Linus Torvalds <torvalds@...ux-foundation.org>,
        David Laight <David.Laight@...lab.com>
Cc:     Arnd Bergmann <arnd@...db.de>,
        Masahiro Yamada <yamada.masahiro@...ionext.com>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: I disabled more compiler warnings..

On Mon, 2020-05-11 at 10:41 -0700, Linus Torvalds wrote:
> On Mon, May 11, 2020 at 12:43 AM David Laight <
> David.Laight@...lab.com> wrote:
> > 
> > I've not looked inside gmake, but I fixed nmake so that it
> > properly used a single job token pipe for the entire (NetBSD)
> > build and then flushed and refilled it with 'abort' tokens
> > when any command failed.
> > That made the build stop almost immediately.
> 
> The GNU jobserver doesn't have anything like that, afaik.
> 
> I think it always writes a '+' character as a token, so I guess it
> could be extended to write something else for the "abort now"
> situation (presumably a '-' character).

That was exactly my plan.

> But at least for external jobserver clients (of which I am not aware
> of any, I think we only depend on the internal GNU make behavior),
> the documentation states that you should just write back the same
> token you read.

I wrote this text precisely because I intended to support using other
tokens to specify other situations, such as failure :).

As a note, the GCC project has a GSoC project approved for this summer,
for GCC and/or binutils to participate in the jobserver protocol when
they do multithreading in the compiler/linker.  I think they are
planning on creating a generic "jobserver library" but I'm not
mentoring (I don't have the bandwidth for GSoC mentoring).  I do hope
to stay abreast of their work and perhaps toss in suggestions however.

> Paul - the issue is that most of us build the kernel with a "make
> -j<bignum>" (in my case "-j32") and if an error happens during the
> make, it can take a _looong_ time for make to react. And if there are
> warnings in the build, they can hide the actual error fairly easily).

Yes, I believe I have an enhancement in Savannah about this.

> I can trivially see how to do it in the jobserver code itself (just
> see if the token we get was '-', and if it was, write it back for the
> next user and return error), but it's the downstream make code I'm
> entirely unfamiliar with.

That's necessary, but my thinking is that more could be done.  What I
was going to do was if we write back an error token then we would also
go into a loop trying to read all the tokens off the jobserver and
write back error tokens, until we read an error token, then we'd exit
(after writing it back obviously).

If you don't do that you'll have to go through an entire set of builds
after the failure before everyone notices the failure.  If every
instance of make does this then you can propogate the error to everyone
more quickly.

My current work on GNU make is fixing the atrocious mess it has with
signal handling: don't even look and if you do, remember I inherited
this code (yeah, yeah, a long time ago but still... :)).  Right now
it's not so hard (especially with large -j) to have make instances
hanging when ^C is used due to race conditions in the signal handling.

As with all single-threaded applications, though, the problem is the
difficulty (in a portable way) of handling both signals and wait*(2)
reliably...

Anyway, this shouldn't be too difficult.