linux-kernel - Re: [PATCH 4/4] coccicheck: add indexing enhancement options

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <alpine.DEB.2.02.1606102301090.2070@localhost6.localdomain6>
Date:	Fri, 10 Jun 2016 23:02:38 +0200 (CEST)
From:	Julia Lawall <julia.lawall@...6.fr>
To:	"Luis R. Rodriguez" <mcgrof@...nel.org>
cc:	Gilles Muller <Gilles.Muller@...6.fr>, nicolas.palix@...g.fr,
	mmarek@...e.com, linux-kernel@...r.kernel.org,
	cocci@...teme.lip6.fr
Subject: Re: [PATCH 4/4] coccicheck: add indexing enhancement options



On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:

> Enable indexing optimizations heuristics. Coccinelle has
> support to make use of its own enhanced "grep" mechanisms
> instead of using regular grep for searching code 'coccigrep',
> in practice though this seems to not perform better than
> regular grep however its expected to help with some use cases
> so we use that if you have no other indexing options in place
> available.
> 
> Since git has its own index, support for using 'git grep' has been
> added to Coccinelle, that should on average perform better than
> using the internal cocci grep, and regular grep. Lastly, Coccinelle
> has had support for glimpseindex for a long while, however the
> tool was previously closed source, its now open sourced, and
> provides the best performance, so support that if we can detect
> you have a glimpse index.
> 
> These tests have been run on an 8 core system:
> 
> Before:
> 
> $ export COCCI=scripts/coccinelle/free/kfree.cocci
> $ time make coccicheck MODE=report
> 
> Before this patch with no indexing or anything:
> 
> real    16m22.435s
> user    128m30.060s
> sys     0m2.712s
> 
> Using coccigrep (after this patch if you have no .git):
> 
> real    16m27.650s
> user    128m47.904s
> sys     0m2.176s
> 
> If you have .git and therefore use gitgrep:
> 
> real    16m21.220s
> user    129m30.940s
> sys     0m2.060s
> 
> And if you have a .glimpse_index:
> 
> real    16m14.794s
> user    128m42.356s
> sys     0m1.880s

I don't see any convincing differences in these times.

I believe that Coccinelle's internal grep is always used, even with no 
option.

I'm puzzled why glimpse gives no benefit.

julia


> 
> Signed-off-by: Luis R. Rodriguez <mcgrof@...nel.org>
> ---
>  scripts/coccicheck | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
> 
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index eeb5fdc142ca..f31c9a152559 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -5,6 +5,8 @@
>  # version 1.0.0-rc11.
>  #
>  
> +DIR=$(dirname $(readlink -f $0))
> +DIR="${DIR}/../"
>  SPATCH="`which ${SPATCH:=spatch}`"
>  
>  if [ ! -x "$SPATCH" ]; then
> @@ -15,6 +17,20 @@ fi
>  USE_JOBS="no"
>  $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
>  
> +# 0. --use-glimpse currently outperforms all. Refer
> +#    to scripts/glimpse.sh for details.
> +# 1. Second best is --use-gitgrep, this is very comparable to --use-glimpse
> +# 2. Use --use-coccigrep if no indexing options are available and your
> +#    version of coccinelle supports it
> +USE_GLIMPSE="no"
> +$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null && [ -f $DIR/.glimpse_index ] && USE_GLIMPSE="yes"
> +
> +USE_GITGREP="no"
> +$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null && [ -d $DIR/.git ] && USE_GITGREP="yes"
> +
> +USE_COCCIGREP="no"
> +$SPATCH --help | grep "\-\-use\-coccigrep" > /dev/null && USE_COCCIGREP="yes"
> +
>  # The verbosity may be set by the environmental parameter V=
>  # as for example with 'make V=1 coccicheck'
>  
> @@ -89,6 +105,14 @@ else
>  	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
>  fi
>  
> +if [ "$USE_GLIMPSE" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-glimpse"
> +elif [ "$USE_GITGREP" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-gitgrep"
> +elif [ "$USE_COCCIGREP" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-coccigrep"
> +fi
> +
>  run_cmd_paramap() {
>  	if [ $VERBOSE -ne 0 ] ; then
>  		echo "Running ($NPROC in parallel): $@"
> -- 
> 2.8.2
> 
>