Message-ID: <CAK7LNAQFk1UNARu4HMcAfpykpw6u3ex_PHOLzfnAo2CoOzBT8w@mail.gmail.com>
Date: Sun, 10 Nov 2024 20:18:15 +0900
From: Masahiro Yamada <masahiroy@...nel.org>
To: HONG Yifan <elsk@...gle.com>
Cc: enh@...gle.com, ccross@...gle.com, Miguel Ojeda <ojeda@...nel.org>, 
	Matthias Maennich <maennich@...gle.com>, kernel-team@...roid.com, linux-kernel@...r.kernel.org
Subject: Re: [PATCH v1] kheaders: prevent `find` from seeing perl temp files

On Thu, Nov 7, 2024 at 9:58 AM HONG Yifan <elsk@...gle.com> wrote:
>
> Symptom:
>
> The command
>
>     find ... | xargs ... perl -i
>
> occasionally triggers error messages like the following, with the build
> still succeeding:
>
>     Can't open <redacted>/kernel/.tmp_cpio_dir/include/dt-bindings/clock/XXNX4nW9: No such file or directory.


I tested this patch on Alpine Linux
with "for i in $(seq 100); do" loop.


[Without this patch]

  CHK     kernel/kheaders_data.tar.xz
  GEN     kernel/kheaders_data.tar.xz
  AR      kernel/module/built-in.a
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXKOEPLf:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXFEJdaa:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXdcljBm:
No such file or directory
Can't open /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXlLaJJF:
No such file or directory.
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXELBDIO:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXoHheAk:
No such file or directory
Can't open /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXeMGhpj:
No such file or directory.
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXbJmIOD:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXiHiAAd:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXjmiien:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXHDjNNM:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXlOHclm:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXbOGokp:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXfiMojG:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXhjfhgd:
No such file or directory
Can't open /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXiHbNEg:
No such file or directory.
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXKiEgne:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXLNlene:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXmKpBbk:
No such file or directory
Can't open /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXdgokkN:
No such file or directory.
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXDpfMGd:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXhAMmbc:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXpaKiKk:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXJNPgAH:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXlkCjbB:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXmkfeAA:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXFFbADB:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXEJlBBd:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXPHMiNP:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXeiMaiG:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXpfopfE:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXLOHHKc:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXiaBiIc:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXJainoG:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXJFhLEI:
No such file or directory
Can't open /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXhPACEf:
No such file or directory.
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXpMhaAc:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXNdoMNL:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXgKMLkg:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXaKkEKh:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXmdOFEo:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXmjKioE:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXPeEhNj:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXJCnpLp:
No such file or directory
Can't open /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXHNOiKd:
No such file or directory.
  CC      kernel/kheaders.o




[With this patch]


  CHK     kernel/kheaders_data.tar.xz
  GEN     kernel/kheaders_data.tar.xz
  AR      kernel/module/built-in.a
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXeDFldH:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXLJMCah:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXckkiOH:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXNggIFH:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXnjOibB:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXkHOnGO:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXMogCin:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXogNJeJ:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXBAgJfo:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXfgFjJH:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXKiiEfj:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXkOEknj:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXjbddCE:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXDHGOlH:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXMGAdak:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXhfAChH:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXmGoEnP:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXGJEpaH:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXkKODpJ:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXDEfBDk:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXmgjJkj:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXBjhcbB:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXHoLKEE:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXOgcfAB:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXdjoEEc:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXFdEDFO:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXiNpHAN:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXopbKfc:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXlJnGkN:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXKfPIJB:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXDfbPkb:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXEiOgDb:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXdCdCIK:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXdoCePE:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXnDjKlh:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXEEcAMM:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXjlDBLj:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXEFPFLj:
No such file or directory
find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXpoFkCA:
No such file or directory
  CC      kernel/kheaders.o




Indeed, the error messages from perl were fixed, but
I still see a bunch of error messages,
which were presumably emitted by 'find'.


Do you know how to fix them?









> Analysis:
>
> With strace, the root cause has been identified to be `perl -i` creating
> temporary files inside $cpio_dir, which causes `find` to see the
> temporary files and emit the names. `find` is likely implemented with
> readdir. POSIX `readdir` says:
>
>     If a file is removed from or added to the directory after the most
>     recent call to opendir() or rewinddir(), whether a subsequent call
>     to readdir() returns an entry for that file is unspecified.
>
> So if the libc that `find` links against choose to return that entry
> in readdir(), a possible sequence of events is the following:
>
> 1. find emits foo.h
> 2. xargs executes `perl -i foo.h`
> 3. perl (pid=100) creates temporary file `XXXXXXXX`
> 4. find sees file `XXXXXXXX` and emit it
> 5. PID 100 exits, cleaning up the temporary file `XXXXXXXX`
> 6. xargs executes `perl -i XXXXXXXX`
> 7. perl (pid=200) tries to read the file, but it doesn't exist any more.
>
> ... triggering the error message.
>
> One can reproduce the bug with the following command (assuming PWD
> contains the list of headers in kheaders.tar.xz)
>
>     for i in $(seq 100); do
>         find -type f -print0 |
>             xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;';
>     done
>
> With a `find` linking against musl libc, the error message is emitted
> 6/100 times.
>
> The fix:
>
> This change modifies the command so `find` only emits .h names, thereby
> skipping the temporary files. Another possible fix would be to store
> the results of `find` before feeding them into xargs; I didn't take this
> approach because the current approach appears marginally more optimized
> and involves a smaller change.
>
> Signed-off-by: HONG Yifan <elsk@...gle.com>
> ---
>  kernel/gen_kheaders.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> index 383fd43ac612..271e0145f406 100755
> --- a/kernel/gen_kheaders.sh
> +++ b/kernel/gen_kheaders.sh
> @@ -84,7 +84,7 @@ for f in $dir_list;
>  done | cpio --quiet -pdu $cpio_dir >/dev/null 2>&1
>
>  # Remove comments except SDPX lines
> -find $cpio_dir -type f -print0 |
> +find $cpio_dir -type f -name '*.h' -print0 |
>         xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;'
>
>  # Create archive and try to normalize metadata for reproducibility.
> --
> 2.47.0.199.ga7371fff76-goog
>


-- 
Best Regards
Masahiro Yamada

Powered by blists - more mailing lists