Date:   Sat, 23 Apr 2022 10:16:37 -0400
From:   James Bottomley <James.Bottomley@...senPartnership.com>
To:     Kent Overstreet <kent.overstreet@...il.com>
Cc:     Steven Rostedt <rostedt@...dmis.org>,
        Christoph Hellwig <hch@....de>, linux-kernel@...r.kernel.org,
        linux-mm@...ck.org, linux-fsdevel@...r.kernel.org,
        hannes@...xchg.org, akpm@...ux-foundation.org,
        linux-clk@...r.kernel.org, linux-tegra@...r.kernel.org,
        linux-input@...r.kernel.org, roman.gushchin@...ux.dev
Subject: Rust and Kernel Vendoring [Was Re: [PATCH v2 1/8] lib/printbuf: New
 data structure for heap-allocated strings]

[change of subject for an easy thread kill, since I'm sure most people
on cc don't want to follow this]
On Fri, 2022-04-22 at 17:13 -0400, Kent Overstreet wrote:
> On Fri, Apr 22, 2022 at 04:03:12PM -0400, James Bottomley wrote:
> > Hey, I didn't say that at all.  I said vendoring the facebook
> > reference implementation wouldn't work (it being 74k lines and us
> > using 300) but that facebook was doing the right thing for us with
> > zstd because they were maintaining the core code we needed, even if
> > we couldn't vendor it from their code base:
> > 
> > https://lore.kernel.org/rust-for-linux/ea85b3bce5f172dc73e2be8eb4dbd21fae826fa1.camel@HansenPartnership.com/
> > 
> > You were the one who said all that about facebook, while
> > incorrectly implying I said it first (which is an interesting
> > variation on the strawman fallacy):
> 
> Hey, sorry for picking on you James - I didn't mean to single you
> out.
> 
> Let me explain where I'm coming from - actually, this email I
> received puts it better than I could:
> 
> From: "John Ericson" <mail@...nericson.me>
> To: "Kent Overstreet" <kent.overstreet@...il.com>
> Subject: Nice email on cargo crates in Linux
> 
> Hi. I saw your email linked from https://lwn.net/Articles/889924/.
> Nicely put!
> 
> I was involved in some of the allocating try_* function work to avoid
> nasty panics, and thus make sure Rust in Linux didn't have to
> reinvent the wheel redoing the alloc library. I very much share your
> goals but suspect this will happen in a sort of boil-the-frog way
> over time. The basic portability method of "incentivize users to
> write code that's *more* portable than what they currently
> need" is a culture shock for user space and kernel space alike. But
> if we get all our ducks in a row on the Rust side, eventually it
> should just be "too tempting", and the code sharing will happen for
> economic reasons (rather than ideological ones about trying to smash
> down the arbitrary dichotomies :) going it alone).
> 
> The larger issue is major projects closed off unto themselves such as
> Linux, PostgreSQL, or even the Glasgow Haskell Compiler (something I
> am currently writing a plan to untangle) have fallen too deep in
> Conway's law's embrace, and thus are full of bad habits like
> https://johno.com/composition-over-configuration is arguing against.
> Over time, I hope using external libraries will not only result in
> less duplicated work, but also "pry open" their architecture a bit,
> tamping down on crazy configurability and replacing it with more
> principled composition.
> 
> I fear this is all way too spicy to bring up on the LKML in 2022, at
> least coming from someone like me without any landed kernel patches
> under his belt, but I wanted to let you know other people have
> similar thoughts.
> 
> Cheers,
> 
> John
> 
> -------
> 
> Re: Rust in Linux, I get frustrated when I see senior people tell the
> new Rust people "don't do that" to things that are standard practice
> in the outside world.

You stripped the nuance of that.  I said many no_std crates could be
used in the kernel.  I also said that the async crate couldn't because
the rust compiler itself would have to support the kernel threading
model.

> I think Linus said recently that Rust in the kernel is something that
> could fail, and he's right - but if it fails, it won't just be the
> failure of the Rust people to do the required work, it'll be _our_
> failure too, a failure to work with them.

The big risk is that rust needs to adapt to the kernel environment. 
This isn't rust specific, llvm had similar issues as an alternative C
compiler.  I think rust in the kernel would fail if it were only the
rust kernel people asking.  Fortunately the pressure to support rust in
embedded leading to the rise in no_std crates is a force which can also
get rust in the kernel over the finish line because of the focus it
puts on getting the language and crates to adapt to non standard
environments.

[...]
> The kernel community has a lot of that going on here. Again, sorry to
> pick on you James, but I wanted to make the argument that - maybe the
> kernel _should_ be adopting a more structured way of using code from
> outside repositories, like cargo, or git submodules (except I've
> never had a positive experience with git submodules, so ignore that
> suggestion, unless I've just been using them wrong, in which case
> someone please teach me). To read you and Greg saying "nah, just
> copy code from other repos, it's fine" - it felt like being back in
> the old days when we were still trying to get people to use source
> control, and having that one older colleague who _insisted_ on not
> using source control of any kind, and that's a bit disheartening.

Even in C terms, the kernel is a nostdlib environment.  If a C project
has too much libc dependency it's not going to work directly in the
kernel, nor should it.  Let's look at zstd (which is pretty much a
nostdlib project) as a great example: the facebook people didn't
actually port the top of their tree (1.5) to the kernel, they
backported bug fixes to the 1.4 branch and made a special release
(1.4.10) just for us.  Why did they do this?  It was because the 1.5
version vastly increased stack use to the extent it would run off the
end of the limited kernel stack so couldn't be ported directly into the
kernel.  A lot of C libraries that are nostdlib have problems like this
as well (you can use recursion, but not in the kernel).  There's no
easy way of shimming environmental constraints like this.

The lesson: it is possible to make the core of a project mobile, but
only if you're aware of all the environmental constraints it will run
into as it gets ported.  The list of possible environments is huge:
kernel, embedded, industrial control ..., so naturally not every (or
more accurately hardly any) project wants to do this.

James

Powered by blists - more mailing lists