linux-kernel - Re: [PATCH v4] coccinelle: fix parallel build with CHECK=scripts/coccicheck

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAK7LNASJQ5Pr2f+JNuENZEsb3vEU1oD=EO-iV5+L-f_YHPkbzw@mail.gmail.com>
Date:   Thu, 23 Nov 2017 23:18:13 +0900
From:   Masahiro Yamada <yamada.masahiro@...ionext.com>
To:     Julia Lawall <julia.lawall@...6.fr>
Cc:     Linux Kbuild mailing list <linux-kbuild@...r.kernel.org>,
        cocci@...teme.lip6.fr, Nicolas Palix <nicolas.palix@...g.fr>,
        Gilles Muller <Gilles.Muller@...6.fr>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Michal Marek <michal.lkml@...kovi.net>
Subject: Re: [PATCH v4] coccinelle: fix parallel build with CHECK=scripts/coccicheck

2017-11-14 21:35 GMT+09:00 Julia Lawall <julia.lawall@...6.fr>:
>
>
> On Tue, 14 Nov 2017, Masahiro Yamada wrote:
>
>> The command "make -j8 C=1 CHECK=scripts/coccicheck COCCI=..." produces
>> lots of "coccicheck failed" error messages.
>>
>> Julia Lawall explained the Coccinelle behavior as follows:
>> "The problem on the Coccinelle side is that it uses a subdirectory
>> with the name of the semantic patch to store standard output and
>> standard error for the different threads.  I didn't want to use a
>> name with the pid, so that one could easily find this information
>> while Coccinelle is running.  Normally the subdirectory is cleaned
>> up when Coccinelle completes, so there is only one of them at a time.
>> Maybe it is best to just add the pid.  There is the risk that these
>> subdirectories will accumulate if Coccinelle crashes in a way such
>> that they don't get cleaned up, but Coccinelle could print a warning
>> if it detects this case, rather than failing."
>>
>> When scripts/coccicheck is used as CHECK tool and -j option is given
>> to Make, the whole of build process runs in parallel.  So, multiple
>> processes try to get access to the same subdirectory.
>>
>> I notice spatch creates the subdirectory only when it runs in parallel
>> (i.e. --jobs <N> is given and <N> is greater than 1).
>>
>> Setting NPROC=1 is a reasonable solution; spatch does not create the
>> subdirectory.  Besides, ONLINE=1 mode takes a single file input for
>> each spatch invocation, so there is no reason to parallelize it in
>> the first place.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@...ionext.com>
>
> Acked-by: Julia Lawall <Julia.Lawall@...6.fr>
>
>> ---
>>
>> Changes in v4:
>>   - Remove unnecessary comments.  One line comment will be enough.
>>
>> Changes in v3:
>>   - Set NPROC=1 because this is a more sensible solution
>>     given that there is no reason to run coccinelle in parallel
>>     for ONLINE=1 mode.
>>   - Move J= option handling to a proper place.
>>   - Add more detailed explanation
>>
>> Changes in v2:
>>   - Grep '-j' instead of '--jobserver-auth'.
>>     '--jobserver-*' is not a stable option flag.
>>     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
>>   - Add -q option to grep
>>
>>  scripts/coccicheck | 15 +++++++++------
>>  1 file changed, 9 insertions(+), 6 deletions(-)
>>
>> diff --git a/scripts/coccicheck b/scripts/coccicheck
>> index 040a8b1..fefda40 100755
>> --- a/scripts/coccicheck
>> +++ b/scripts/coccicheck
>> @@ -30,12 +30,6 @@ else
>>       VERBOSE=0
>>  fi
>>
>> -if [ -z "$J" ]; then
>> -     NPROC=$(getconf _NPROCESSORS_ONLN)
>> -else
>> -     NPROC="$J"
>> -fi
>> -
>>  FLAGS="--very-quiet"
>>
>>  # You can use SPFLAGS to append extra arguments to coccicheck or override any
>> @@ -70,6 +64,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
>>      # Take only the last argument, which is the C file to test
>>      shift $(( $# - 1 ))
>>      OPTIONS="$COCCIINCLUDE $1"
>> +
>> +    # No need to parallelize Coccinelle since this mode takes one input file.
>> +    NPROC=1
>>  else
>>      ONLINE=0
>>      if [ "$KBUILD_EXTMOD" = "" ] ; then
>> @@ -77,6 +74,12 @@ else
>>      else
>>          OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE"
>>      fi
>> +
>> +    if [ -z "$J" ]; then
>> +        NPROC=$(getconf _NPROCESSORS_ONLN)
>> +    else
>> +        NPROC="$J"
>> +    fi
>>  fi
>>
>>  if [ "$KBUILD_EXTMOD" != "" ] ; then
>> --
>> 2.7.4
>>
>>

Applied to linux-kbuild.


-- 
Best Regards
Masahiro Yamada