linux-kernel - Re: [PATCH v3] tags: much faster, parallel "make tags"

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20150510135309.GD1717@p183.telecom.by>
Date:	Sun, 10 May 2015 16:53:09 +0300
From:	Alexey Dobriyan <adobriyan@...il.com>
To:	Pádraig Brady <P@...igBrady.com>
Cc:	akpm@...ux-foundation.org, mmarek@...e.cz,
	linux-kernel@...r.kernel.org
Subject: Re: [PATCH v3] tags: much faster, parallel "make tags"

[fix Andrew's email]

On Sun, May 10, 2015 at 04:26:34PM +0300, Alexey Dobriyan wrote:
> On Sat, May 09, 2015 at 06:07:18AM +0100, Pádraig Brady wrote:
> > On 08/05/15 14:26, Alexey Dobriyan wrote:
> 
> > >  exuberant()
> > >  {
> > > -	all_target_sources | xargs $1 -a                        \
> > > +	rm -f .make-tags.*
> > > +
> > > +	all_target_sources >.make-tags.src
> > > +	NR_CPUS=$(getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1)
> > 
> > `nproc` is simpler and available since coreutils 8.1 (2009-11-18)
> 
> nproc was discarded because getconf is standartized.
> 
> > > +	NR_LINES=$(wc -l <.make-tags.src)
> > > +	NR_LINES=$((($NR_LINES + $NR_CPUS - 1) / $NR_CPUS))
> > > +
> > > +	split -a 6 -d -l $NR_LINES .make-tags.src .make-tags.src.
> > 
> > `split -d -nl/$(nproc)` is simpler and available since coreutils 8.8 (2010-12-22)
> 
> -nl/ can't count and always make first file somewhat bigger, which is
> suspicious. What else it can't do right?
> 
> > > +	sort .make-tags.* >>$2
> > > +	rm -f .make-tags.*
> > 
> > Using sort --merge would speed up significantly?
> 
> By ~1 second, yes.
> 
> > Even faster would be to get sort to skip the header lines, avoiding the need for sed.
> > It's a bit awkward and was discussed at:
> > http://lists.gnu.org/archive/html/coreutils/2013-01/msg00027.html
> > Summarising that, is if not using merge you can:
> > 
> >   tlines=$(($(wc -l < "$2") + 1))
> >   tail -q -n+$tlines .make-tags.* | LC_ALL=C sort >>$2
> > 
> > Or if merge is appropriate then:
> > 
> >   tlines=$(($(wc -l < "$2") + 1))
> >   eval "eval LC_ALL=C sort -m '<(tail -n+$tlines .make-tags.'{1..$(nproc)}')'" >>$2
> 
> Might as well teach ctags to do real parallel processing.
> LC_* are set by top level Makefile.
> 
> > p.p.s. You may want to `trap EXIT cleanup` to rm -f .make-tags.*
> 
> The real question is how to kill ctags reliably.
> Naive
> 
> 	trap 'kill $(jobs -p); rm -f .make-tags.*' TERM INT
> 
> doesn't work.
> 
> Files are removed, but processes aren't.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/