linux-kernel - Re: [Cocci] [PATCH] coccinelle: add pycocci wrapper for multithreaded support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 10 Apr 2014 21:32:34 +0200 (CEST)
From:	Julia Lawall <julia.lawall@...6.fr>
To:	"Luis R. Rodriguez" <mcgrof@...e.com>
cc:	Johannes Berg <johannes@...solutions.net>,
	linux-kernel@...r.kernel.org, backports@...r.kernel.org,
	cocci@...teme.lip6.fr
Subject: Re: [Cocci] [PATCH] coccinelle: add pycocci wrapper for	multithreaded
 support



On Thu, 10 Apr 2014, Luis R. Rodriguez wrote:

> On Thu, Apr 10, 2014 at 07:51:29PM +0200, Johannes Berg wrote:
> > On Thu, 2014-04-10 at 10:48 -0700, Luis R. Rodriguez wrote:
> > 
> > > You just pass it a cocci file, a target dir, and in git environments
> > > you always want --in-place enabled. Experiments and profiling random
> > > cocci files with the Linux kernel show that using just using number of
> > > CPUs doesn't scale well given that lots of buckets of files don't require
> > > work, as such this uses 10 * number of CPUs for its number of threads.
> > > For work that define more general ruler 3 * number of CPUs works better,
> > > but for smaller cocci files 3 * number of CPUs performs best right now.
> > > To experiment more with what's going on with the multithreading one can enable
> > > htop while kicking off a cocci task on the kernel, we want to keep
> > > these CPUs busy as much as possible. 
> > 
> > That's not really a good benchmark, you want to actually check how
> > quickly it finishes ... If you have some IO issues then just keeping the
> > CPUs busy trying to do IO won't help at all.
> 
> I checked the profile results, the reason the jobs finish is some threads
> had no work or little work. Hence why I increased the number of threads,
> depending on the context (long or short cocci expected, in backports
> at least, the long being all cocci files in one, the short being --test-cocci
> flag to gentree.py). This wrapper uses the short assumption with 10 * num_cpus
> 
> > > Since its just a helper I toss it into the python directory but don't
> > > install it. Hope is that we can evolve it there instead of carrying this
> > > helper within backports.
> > 
> > If there's a plan to make coccinelle itself multi-threaded, what's the
> > point?
> 
> To be clear, Coccinelle *has* a form of multithreaded support but requires manual
> spawning of jobs with references to the max count and also the number thread
> that this new process you are spawning belongs to. There's plans to consider
> reworking things to handle all this internally but as I discussed with Julia
> the changes required would require some structural changes, and as such we
> need to live with this for a bit longer. I need to use Coccinelle daily now,
> so figured I'd punt this out there in case others might make use of it.

I agree with Luis.  Multithreading inside Coccinelle is currently a 
priority task, but not a highest priority one.

julia
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/