linux-kernel - [PATCH 4/4] coccicheck: add indexing enhancement options

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 10 Jun 2016 13:42:12 -0700
From:	"Luis R. Rodriguez" <mcgrof@...nel.org>
To:	Julia.Lawall@...6.fr, Gilles.Muller@...6.fr, nicolas.palix@...g.fr,
	mmarek@...e.com
Cc:	linux-kernel@...r.kernel.org, cocci@...teme.lip6.fr,
	"Luis R. Rodriguez" <mcgrof@...nel.org>
Subject: [PATCH 4/4] coccicheck: add indexing enhancement options

Enable indexing optimizations heuristics. Coccinelle has
support to make use of its own enhanced "grep" mechanisms
instead of using regular grep for searching code 'coccigrep',
in practice though this seems to not perform better than
regular grep however its expected to help with some use cases
so we use that if you have no other indexing options in place
available.

Since git has its own index, support for using 'git grep' has been
added to Coccinelle, that should on average perform better than
using the internal cocci grep, and regular grep. Lastly, Coccinelle
has had support for glimpseindex for a long while, however the
tool was previously closed source, its now open sourced, and
provides the best performance, so support that if we can detect
you have a glimpse index.

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report

Before this patch with no indexing or anything:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

Using coccigrep (after this patch if you have no .git):

real    16m27.650s
user    128m47.904s
sys     0m2.176s

If you have .git and therefore use gitgrep:

real    16m21.220s
user    129m30.940s
sys     0m2.060s

And if you have a .glimpse_index:

real    16m14.794s
user    128m42.356s
sys     0m1.880s

Signed-off-by: Luis R. Rodriguez <mcgrof@...nel.org>
---
 scripts/coccicheck | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index eeb5fdc142ca..f31c9a152559 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -5,6 +5,8 @@
 # version 1.0.0-rc11.
 #
 
+DIR=$(dirname $(readlink -f $0))
+DIR="${DIR}/../"
 SPATCH="`which ${SPATCH:=spatch}`"
 
 if [ ! -x "$SPATCH" ]; then
@@ -15,6 +17,20 @@ fi
 USE_JOBS="no"
 $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
+# 0. --use-glimpse currently outperforms all. Refer
+#    to scripts/glimpse.sh for details.
+# 1. Second best is --use-gitgrep, this is very comparable to --use-glimpse
+# 2. Use --use-coccigrep if no indexing options are available and your
+#    version of coccinelle supports it
+USE_GLIMPSE="no"
+$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null && [ -f $DIR/.glimpse_index ] && USE_GLIMPSE="yes"
+
+USE_GITGREP="no"
+$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null && [ -d $DIR/.git ] && USE_GITGREP="yes"
+
+USE_COCCIGREP="no"
+$SPATCH --help | grep "\-\-use\-coccigrep" > /dev/null && USE_COCCIGREP="yes"
+
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
 
@@ -89,6 +105,14 @@ else
 	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
 fi
 
+if [ "$USE_GLIMPSE" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-glimpse"
+elif [ "$USE_GITGREP" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-gitgrep"
+elif [ "$USE_COCCIGREP" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-coccigrep"
+fi
+
 run_cmd_paramap() {
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
-- 
2.8.2