linux-kernel - [PATCH] tags: much faster, parallel "make tags"

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20150414172047.GA5641@p183.telecom.by>
Date:	Tue, 14 Apr 2015 20:20:47 +0300
From:	Alexey Dobriyan <adobriyan@...il.com>
To:	mmarek@...e.cz, akpm@...ux-foundation.org
Cc:	linux-kernel@...r.kernel.org
Subject: [PATCH] tags: much faster, parallel "make tags"

ctags is single-threaded program. Split list of files to be tagged into
equal parts, 1 part for each CPU and then merge the results.

Speedup on one 2-way box I have is ~143 s => ~99 s (-31%).
On another 4-way box: ~120 s => ~65 s (-46%!).

Resulting "tags" files aren't byte-for-byte identical because ctags
program numbers anon struct and enum declarations with "__anonNNN"
symbols. If those lines are removed, "tags" file becomes byte-for-byte
identical with those generated with current code.

Signed-off-by: Alexey Dobriyan <adobriyan@...il.com>
---

 scripts/tags.sh |   34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

--- a/scripts/tags.sh
+++ b/scripts/tags.sh
@@ -152,7 +152,24 @@ dogtags()
 
 exuberant()
 {
-	all_target_sources | xargs $1 -a                        \
+	NR_CPUS=1
+	if [ -e /proc/cpuinfo ]; then
+		NR_CPUS=$(grep -e '^processor	: ' /proc/cpuinfo | wc -l)
+	fi
+
+	rm -f .make-tags.src.* .make-tags.*
+
+	all_target_sources >.make-tags.src
+	# seems like Useless Use of cat(1) but not really
+	NR_LINES=$(cat .make-tags.src | wc -l)
+	NR_LINES=$((($NR_LINES + $NR_CPUS - 1) / $NR_CPUS))
+
+	split -a 6 -d -l $NR_LINES .make-tags.src .make-tags.src.
+
+	for i in .make-tags.src.*; do
+		N=$(echo $i | sed -e 's/.*\.//')
+		# -u: don't sort now, sort later
+		cat $i | xargs $1 -a -f .make-tags.$N -u	\
 	-I __initdata,__exitdata,__initconst,			\
 	-I __cpuinitdata,__initdata_memblock			\
 	-I __refdata,__attribute,__maybe_unused,__always_unused \
@@ -211,7 +228,20 @@ exuberant()
 	--regex-c='/DEFINE_PCI_DEVICE_TABLE\((\w*)/\1/v/'		\
 	--regex-c='/(^\s)OFFSET\((\w*)/\2/v/'				\
 	--regex-c='/(^\s)DEFINE\((\w*)/\2/v/'				\
-	--regex-c='/DEFINE_HASHTABLE\((\w*)/\1/v/'
+	--regex-c='/DEFINE_HASHTABLE\((\w*)/\1/v/'			\
+	&
+	done
+	wait
+	rm -f .make-tags.src .make-tags.src.*
+
+	# write header
+	$1 -f tags /dev/null
+	# remove header
+	for i in .make-tags.*; do
+		sed -i -e '/^!/d' $i
+	done
+	sort .make-tags.* >>tags
+	rm -f .make-tags.*
 
 	all_kconfigs | xargs $1 -a                              \
 	--langdef=kconfig --language-force=kconfig              \
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/