linux-kernel - Re: [GIT PULL] pin control bulk changes for v4.16

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CA+55aFxdK3CJiAN_yytM7nYubYOeShvCNFGpEU9SR1HC0b8WiQ@mail.gmail.com>
Date:   Mon, 5 Feb 2018 08:55:15 -0800
From:   Linus Torvalds <torvalds@...ux-foundation.org>
To:     David Laight <David.Laight@...lab.com>
Cc:     Linus Walleij <linus.walleij@...aro.org>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Ingo Molnar <mingo@...nel.org>,
        linux-kernel <linux-kernel@...r.kernel.org>,
        "linux-gpio@...r.kernel.org" <linux-gpio@...r.kernel.org>
Subject: Re: [GIT PULL] pin control bulk changes for v4.16

On Mon, Feb 5, 2018 at 2:09 AM, David Laight <David.Laight@...lab.com> wrote:
> From: Linus Torvalds
>> Sent: 02 February 2018 22:57
> ...
>> I also wonder if there are any automated tools that try to find these
>> kinds of crazy things. I suspect a lot of our build times is the poor
>> compiler just reading and parsing header files over and over again,
>> and a lot of them are probably not needed.
>
> I've counted system calls during a NetBSD kernel build, I imagine Linux is
> much the same.
> Most of the calls were open(), and most of those failing opens.
> I suspected that most came from searching the -I path to find headers.
> Build over NFS and the cost is even more significant (every directory
> name in the path (used to) require an NFS message exchange).

I suspect that Linux is actually very _different_ in this area,
because Linux has a very extensive name cache for pathname lookup that
also captures negative path component entries.

End result: opening a file - whether it exists or not - doesn't
actually go down to the filesystem at all when things are cached. My
kernel profiles also show that very clearly, there's absolutely no
filesystem component to the build at all (but there is a noticeable
VFS component to it, and __d_lookup_rcu is generally one of the
hottest kernel functions along with the system call entry/exit code).

Now, on NFS the caching will be a bit less effective because we have a
timeout that forces re-validation of the caches, but even there at
least the "look up the same header pathname" will be captured very
well. So the most common case - of doing the include path lookup for
all those really core header files that _everbody_ ends up including -
ends up being captured pretty much 100% in the caches even on NFS. On
a local filesystem, as long as you have enough RAM, those lookups will
be cached even across multiple kernel compiles.

At least if you are like me, and all you do all day long is build the kernel ;)

                 Linus