linux-kernel - Re: [PATCH 0/8] (REBASED) get_abi.pl undefined: improve precision and performance

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <YVHntS2e3FTWIjds@kroah.com>
Date:   Mon, 27 Sep 2021 17:48:05 +0200
From:   Greg Kroah-Hartman <gregkh@...uxfoundation.org>
To:     Mauro Carvalho Chehab <mchehab+huawei@...nel.org>
Cc:     Linux Doc Mailing List <linux-doc@...r.kernel.org>,
        linux-kernel@...r.kernel.org, Jonathan Corbet <corbet@....net>
Subject: Re: [PATCH 0/8] (REBASED) get_abi.pl undefined: improve precision
 and performance

On Mon, Sep 27, 2021 at 03:39:42PM +0200, Mauro Carvalho Chehab wrote:
> Em Mon, 27 Sep 2021 11:23:20 +0200
> Greg Kroah-Hartman <gregkh@...uxfoundation.org> escreveu:
> 
> > On Mon, Sep 27, 2021 at 10:55:53AM +0200, Mauro Carvalho Chehab wrote:
> > > Em Thu, 23 Sep 2021 19:13:04 +0200
> > > Greg Kroah-Hartman <gregkh@...uxfoundation.org> escreveu:
> > >   
> > > > On Thu, Sep 23, 2021 at 05:41:11PM +0200, Mauro Carvalho Chehab wrote:  
> > > > > Hi Greg,
> > > > > 
> > > > > As requested, this is exactly the same changes, rebased on the top of
> > > > > driver-core/driver-core-next.
> > > > > 
> > > > > -
> > > > > 
> > > > > It follows a series of improvements for get_abi.pl. it is on the top of driver-core/driver-core-next.
> > > > > 
> > > > > With such changes, on my development tree, the script is taking 6 seconds to run 
> > > > > on my desktop:
> > > > > 
> > > > > 	$ !1076
> > > > > 	$ time ./scripts/get_abi.pl undefined |sort >undefined_after && cat undefined_after| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined_after undefined_symbols
> > > > > 
> > > > > 	real	0m6,292s
> > > > > 	user	0m5,640s
> > > > > 	sys	0m0,634s
> > > > > 	  6838 undefined_after
> > > > > 	   808 undefined_symbols
> > > > > 	  7646 total
> > > > > 
> > > > > And 7 seconds on a Dell Precision 5820:
> > > > > 
> > > > > 	$ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols
> > > > > 
> > > > > 	real	0m7.162s
> > > > > 	user	0m5.836s
> > > > > 	sys	0m1.329s
> > > > > 	6548 undefined
> > > > > 	772 undefined_symbols
> > > > > 
> > > > > Both tests were done against this tree (based on today's linux-next):
> > > > > 
> > > > > 	$ https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/log/?h=get_abi_undefined-latest
> > > > > 
> > > > > It should be noticed that, as my tree has several ABI fixes,  the time to run the
> > > > > script is likely less than if you run on your tree, as there will be less symbols to
> > > > > be reported, and the algorithm is optimized to reduce the number of regexes
> > > > > when a symbol is found.
> > > > > 
> > > > > Besides optimizing and improving the seek logic, this series also change the
> > > > > debug logic. It how receives a bitmap, where "8" means to print the regexes
> > > > > that will be used by "undefined" command:
> > > > > 
> > > > > 	$ time ./scripts/get_abi.pl undefined --debug 8 >foo
> > > > > 	real	0m17,189s
> > > > > 	user	0m13,940s
> > > > > 	sys	0m2,404s
> > > > > 
> > > > > 	$wc -l foo
> > > > > 	18421939 foo
> > > > > 
> > > > > 	$ cat foo
> > > > > 	...
> > > > > 	/sys/kernel/kexec_crash_loaded =~ /^(?^:^/sys/.*/iio\:device.*/in_voltage.*_scale_available$)$/
> > > > > 	/sys/kernel/kexec_crash_loaded =~ /^(?^:^/sys/.*/iio\:device.*/out_voltage.*_scale_available$)$/
> > > > > 	/sys/kernel/kexec_crash_loaded =~ /^(?^:^/sys/.*/iio\:device.*/out_altvoltage.*_scale_available$)$/
> > > > > 	/sys/kernel/kexec_crash_loaded =~ /^(?^:^/sys/.*/iio\:device.*/in_pressure.*_scale_available$)$/
> > > > > 	...
> > > > > 
> > > > > On other words, on my desktop, the /sys match is performing >18M regular 
> > > > > expression searches, which takes 6,2 seconds (or 17,2 seconds, if debug is 
> > > > > enabled and sent to an area on my nvme storage).    
> > > > 
> > > > Better, it's down to 10 minutes on my machine now:
> > > > 
> > > > 	real	10m39.218s
> > > > 	user	10m37.742s
> > > > 	sys	0m0.775s  
> > > 
> > > A lot better, but not clear why it is still taking ~40x more than here...
> > > It could well be due to the other ABI changes yet to be applied
> > > (I'll submit it probably later today), but it could also be related to
> > > something else. Could this be due to disk writes?  
> > 
> > Disk writes to where for what?  This is a very fast disk (nvme raid
> > array)  It's also a very "big" system, with lots of sysfs files:
> > 
> > 	$ find /sys/devices/ -type f | wc -l
> > 	44334
> 
> Ok. Maybe that partially explains why it is taking so long, as the
> number of regex to compare will increase (not linearly).

No idea.  I just ran it on my laptop and it took only 5 seconds.

Hm, you aren't reading the values of the sysfs files, right?

Anything I can do to run to help figure out where the script is taking
so long?

> > And any Documentation/ABI/ updates you might have I'll gladly
> > take as well.
> 
> I'll be submitting it soon enough. Got sidetracked by a regression
> on my INBOX due to a fetchmail regression[1].

Ick, fetchmail.  I recommend getmail instead, much more robust and a
sane maintainer :)

I'll take a look at those patches now.

thanks,

greg k-h