linux-kernel - Re: [PATCH v3 0/2] Don't create Python bytecode when building the kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250424083125.1c8a0d24@sal.lan>
Date: Thu, 24 Apr 2025 08:31:25 +0800
From: Mauro Carvalho Chehab <mchehab+huawei@...nel.org>
To: Jonathan Corbet <corbet@....net>
Cc: Dmitry Baryshkov <dmitry.baryshkov@....qualcomm.com>, Linux Doc Mailing
 List <linux-doc@...r.kernel.org>, linux-kernel@...r.kernel.org, Andy
 Shevchenko <andriy.shevchenko@...el.com>, David Airlie <airlied@...il.com>,
 Jani Nikula <jani.nikula@...ux.intel.com>, Joonas Lahtinen
 <joonas.lahtinen@...ux.intel.com>, Maarten Lankhorst
 <maarten.lankhorst@...ux.intel.com>, Masahiro Yamada
 <masahiroy@...nel.org>, Maxime Ripard <mripard@...nel.org>, Nathan
 Chancellor <nathan@...nel.org>, Nicolas Schier <nicolas.schier@...ux.dev>,
 Rodrigo Vivi <rodrigo.vivi@...el.com>, Simona Vetter <simona@...ll.ch>,
 Thomas Zimmermann <tzimmermann@...e.de>, Tvrtko Ursulin
 <tursulin@...ulin.net>, dri-devel@...ts.freedesktop.org,
 intel-gfx@...ts.freedesktop.org, linux-kbuild@...r.kernel.org
Subject: Re: [PATCH v3 0/2] Don't create Python bytecode when building the
 kernel

Em Mon, 21 Apr 2025 10:35:29 -0600
Jonathan Corbet <corbet@....net> escreveu:

> Dmitry Baryshkov <dmitry.baryshkov@....qualcomm.com> writes:
> 
> > On Wed, Apr 16, 2025 at 03:51:03PM +0800, Mauro Carvalho Chehab wrote:  
> >> 
> >> As reported by Andy, the Kernel build system runs kernel-doc script for DRM,
> >> when W=1. Due to Python's normal behavior, its JIT compiler will create
> >> a bytecode and store it under scripts/lib/*/__pycache__. 
> >> 
> >> As one may be using O= and even having the sources on a read-only mount
> >> point, disable its creation during build time.  
> >
> > Would it be possible to properly support O= and create pyc / pycache
> > inside the object/output dir?  
> 
> I have to confess, I've been wondering if we should be treating the .pyc
> files like we treat .o files or other intermediate products.  Rather
> than trying to avoid their creation entirely, perhaps we should just be
> sure they end up in the right place and are properly cleaned up...?

I sent a v4 of the series disabling *.pyc creation, basically placing
*.pyc at the right place on .gitignore.

That's said, I agree that the best would be to use PYTHONPYCACHEPREFIX,
placing the intermediate products altogether with O= results. There is
however something we need to deal with. To follow our building system
to the letter, if we have *.py files at scripts/lib/kdoc, the intermediate
files should be at: either:

	- {outdir}/scripts/lib/kdoc; or: 
	- {outdir}/scripts/lib/kdoc/__pycache__ 

The same shall happen for all other places including Documentation/sphinx.
In the specific case of Sphinx extensions, it would mean that it would
produce *.pyc for both scripts/lib/*/*.py and Documentation/sphinx.
No idea how to make Python to do that without doing some horrible tricks.

An alternate approach would be to store all of them at the same place, like:

	- {outdir}/__pycache__ 

The problem is that, if we end having two scripts with the same name, and
using PYTHONPYCACHEPREFIX would place both of them at the same place, we'll
have troubles.

IMO, let's apply this series first, and then see if we can replace patch 3
with another one using PYTHONPYCACHEPREFIX, after we're sure that there's
a way for it to do the right thing.

> 
> To answer Dmitry's question, it seems that setting PYTHONPYCACHEPREFIX
> should do the trick?

Regards,
Mauro