linux-kernel - Re: [PATCH v4 0/4] Don't create Python bytecode when building the kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <aA8vs8gw75aAfwYb@smile.fi.intel.com>
Date: Mon, 28 Apr 2025 10:35:15 +0300
From: Andy Shevchenko <andriy.shevchenko@...el.com>
To: Mauro Carvalho Chehab <mchehab+huawei@...nel.org>
Cc: Akira Yokosawa <akiyks@...il.com>, airlied@...il.com, corbet@....net,
	dri-devel@...ts.freedesktop.org, intel-gfx@...ts.freedesktop.org,
	jani.nikula@...ux.intel.com, joonas.lahtinen@...ux.intel.com,
	linux-doc@...r.kernel.org, linux-kbuild@...r.kernel.org,
	linux-kernel@...r.kernel.org, maarten.lankhorst@...ux.intel.com,
	masahiroy@...nel.org, mripard@...nel.org, nathan@...nel.org,
	nicolas.schier@...ux.dev, rodrigo.vivi@...el.com, simona@...ll.ch,
	tursulin@...ulin.net, tzimmermann@...e.de
Subject: Re: [PATCH v4 0/4] Don't create Python bytecode when building the
 kernel

On Sat, Apr 26, 2025 at 08:57:08PM +0800, Mauro Carvalho Chehab wrote:
> Em Sat, 26 Apr 2025 11:39:05 +0900
> Akira Yokosawa <akiyks@...il.com> escreveu:
> 
> > Bothering with might-become-incompatilbe-in-the-future python environment
> > variables in kernel Makefiles looks over-engineering to me.
> > Also, as Mauro says in 3/4, it is incomplete in that it does not cover
> > the cases where those scripts are invoked outside of kernel build.
> > And it will interfere with existing developers who want the benefit of
> > bytecode caching.
> > 
> > I'm not precluding the possibility of incoherent bytecode cache; for example
> > by using a shared kernel source tree among several developers, and only
> > one of them (owner) has a write permission of it.  In that case, said
> > owner might update the tree without running relevant python scripts.
> > 
> > I don't know if python can notice outdated cache and disregard it.
> > 
> > In such a situation, setting PYTHONPYCACHEPREFIX as an environment
> > variable should help, for sure, but only in such special cases.
> > 
> > Andy, what do you say if I ask reverts of 1/4, 2/4/, and 3/4?
> 
> Patches 1 and 2 are, IMO, needed anyway, as they fix a problem:
> KERNELDOC environment is not used consistently.
> 
> Now, patch 3 is the one that may require more thinking.
> 
> I agree with Andy that, when O=<dir> is used, nothing shall be
> written to source dir.
> 
> There are a couple of reasons for that:
> 
> 1. source dir may be read only;
> 2. one may want to do cross compilation and use multiple output
>    directories, one for each version;
> 3. the source dir could be mapped via NFS to multiple machines
>    with different architectures.
> 
> For (3), it could mean that multiple machines may have different
> Python versions, so, sharing the Python bytecode from source dir doesn't
> sound a good idea. Also, I'm not sure if the pyc from different archs
> would be identical.
> 
> With that, there are two options:
> 
> a. disable cache;
> b. set PYTHONCACHEPREFIX.

Thanks, Mauro, for replying. I'm with you on all of it.

> We're currently doing (a). I guess everybody agrees that this is
> is not ideal.

Yes, I also prefer to have cache working if it's possible. The only BUT here is
that users should not suffer from it.

> So, ideally, we should move to (b). For Spinx, the easiest solution
> is just to place it under Documentation/output, but this is not
> generic enough: ideally, we should revert patch 3 and set
> PYTHONCACHEPREFIX when O is used. Eventually, we can apply my
> patch for Documentation/output, while we craft such logic.

-- 
With Best Regards,
Andy Shevchenko