linux-kernel - Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20160618012243.GD11948@wotan.suse.de>
Date:	Sat, 18 Jun 2016 03:22:43 +0200
From:	"Luis R. Rodriguez" <mcgrof@...nel.org>
To:	Julia Lawall <julia.lawall@...6.fr>
Cc:	"Luis R. Rodriguez" <mcgrof@...nel.org>, Gilles.Muller@...6.fr,
	nicolas.palix@...g.fr, mmarek@...e.com,
	linux-kernel@...r.kernel.org, akpm@...ux-foundation.org,
	gregkh@...uxfoundation.org, markivx@...eaurora.org,
	stephen.boyd@...aro.org, zohar@...ux.vnet.ibm.com,
	broonie@...nel.org, ming.lei@...onical.com, tiwai@...e.de,
	johannes@...solutions.net, chunkeey@...glemail.com,
	hauke@...ke-m.de, jwboyer@...oraproject.org,
	dmitry.torokhov@...il.com, dwmw2@...radead.org, jslaby@...e.com,
	torvalds@...ux-foundation.org, cocci@...teme.lip6.fr
Subject: Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

On Fri, Jun 17, 2016 at 05:35:26PM +0200, Julia Lawall wrote:
> On Fri, 17 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Fri, Jun 17, 2016 at 11:44:26AM +0200, Julia Lawall wrote:
> > > I'm not sure that this is worth it.  It adds a dependency on a tool that
> > > seems not to be well maintained.  In terms of Coccinelle, I'm not sure
> > > that it gives a big benefit.
> > >
> > > Attached is a graph showing the file selection time for Coccinelle for a
> > > selection of fairly complex semantic patches.  Coccigrep is just a
> > > line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
> > > In most cases, glimpse is clearly faster.
> > >
> > > On the other hand, it seems that glimpse often selects more files.
> > > Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
> > > vs 236.  I suspect that this is because glimpse considers _ to be a space,
> > > and thus it can have many false positives.  There are, however, a few
> > > cases where glimpse also selects fewer files.
> > >
> > > The file processing time (ie parsing the file, searching for, matches of
> > > the semantic patch in the file, and performing the transformation) is
> > > normally much higher than the file selection time.
> > >
> > > So it seems that git grep is currently a better option for the kernel.
> >
> > Great, thanks, consider this patch dropped, do we want the heuristics
> > for the cache index in place though or should I drop that as well ?
> 
> I assume you mean this patch:
> 
> [PATCH v2 3/8] coccicheck: add indexing enhancement options
> 
> I think it should be dropped.  It adds complexity and git grep works
> pretty well.

Hmm but coccicheck does not make use of --git-grep even.

> If people want to use something else, they can use SPARGS,
> or a .cocciconfig file, eg:
> 
> [spatch]
> options = --use-glimpse

Neat will these be used last and thus override anything? If so, what about just
adding an upstream .cocciconfig with --use-gitgrep -- only issue then is
what if a user wants to use idutils ? How do we let them override?

  Luis