| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20210928235425.02ffa1a4@coco.lan>
Date: Tue, 28 Sep 2021 23:54:25 +0200
From: Mauro Carvalho Chehab <mchehab+huawei@...nel.org>
To: Greg Kroah-Hartman <gregkh@...uxfoundation.org>
Cc: Linux Doc Mailing List <linux-doc@...r.kernel.org>,
linux-kernel@...r.kernel.org, Jonathan Corbet <corbet@....net>
Subject: Re: [PATCH 0/3] get_abi: improve message output and fix a
regression
Em Tue, 28 Sep 2021 19:18:31 +0200
Greg Kroah-Hartman <gregkh@...uxfoundation.org> escreveu:
> On Tue, Sep 28, 2021 at 02:27:39PM +0200, Mauro Carvalho Chehab wrote:
> > Em Tue, 28 Sep 2021 13:04:22 +0200
> > Greg Kroah-Hartman <gregkh@...uxfoundation.org> escreveu:
> >
> > > On Tue, Sep 28, 2021 at 12:14:01PM +0200, Mauro Carvalho Chehab wrote:
> > > > Hi Greg,
> > > >
> > > > As promised on
> > > >
> > > > https://lore.kernel.org/lkml/20210928120304.62319fba@coco.lan/T/#u
> > > >
> > > > I'm adding progress info when get_abi.pl is checking for undefined ABI symbols
> > > > on patches 1 and 2.
> > > >
> > > > That will help not only to identify what is causing delays on the script, but also
> > > > to notify the user that processing it could take some time on some systems.
> > > >
> > > > If you run it on your big server with:
> > > >
> > > > scripts/get_abi.pl undefined 2>logs
> > > >
> > > > The "logs" file will contain timestamps relative to the time the script started to
> > > > do the regex matches for sysfs files. It should be printing one line every
> > > > time the progress completes 1% or one second after the last progress output.
> > >
> > > Adding more debugging and tweaking the script a bit to show the file it
> > > is about to check, not the one it finished checking,
> >
> > Feel free to modify the script and add such debug/tweaks if you find
> > it useful.
> >
> > > I got the following
> > > debug output that seems to pinpoint the problem file.
> > >
> > > The sysfs file that is causing problems is:
> > > /sys/devices/pci0000:40/0000:40:00.2/iommu/ivhd1/amd-iommu/cap
> > >
> > > and here's some debugging output for the regex it needs to search for
> > > this:
> > >
> > > /sys/devices/pci0000:40/0000:40:00.2/iommu/ivhd1/amd-iommu/cap =~ /^(?^:^/sys/class/iommu/.*/amd\-iommu/cap$)$/
> > > /sys/devices/pci0000:40/0000:40:00.2/iommu/ivhd1/amd-iommu/cap =~ /^(?^:^/sys/class/iommu/.*/intel\-iommu/cap$)$/
> > > /sys/devices/pci0000:40/0000:40:00.2/iommu/ivhd1/amd-iommu/cap =~ /^(?^:^/sys/devices/pci.*.*.*.*\:.*.*/0000\:.*.*\:.*.*..*/dma/dma.*chan.*/quickdata/cap$)$/
> > > /sys/devices/pci0000:40/0000:40:07.0/iommu/amd-iommu/cap =~ /^(?^:^/sys/class/iommu/.*/amd\-iommu/cap$)$/
> > > /sys/devices/pci0000:40/0000:40:07.0/iommu/amd-iommu/cap =~ /^(?^:^/sys/class/iommu/.*/intel\-iommu/cap$)$/
> > > /sys/devices/pci0000:40/0000:40:07.0/iommu/amd-iommu/cap =~ /^(?^:^/sys/devices/pci.*.*.*.*\:.*.*/0000\:.*.*\:.*.*..*/dma/dma.*chan.*/quickdata/cap$)$/
> > > /sys/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:02/device:7a/physical_node/iommu/ivhd1/amd-iommu/cap =~ /^(?^:^/sys/class/iommu/.*/amd\-iommu/cap$)$/
> > > /sys/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:02/device:7a/physical_node/iommu/ivhd1/amd-iommu/cap =~ /^(?^:^/sys/class/iommu/.*/intel\-iommu/cap$)$/
> > > /sys/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:02/device:7a/physical_node/iommu/ivhd1/amd-iommu/cap =~ /^(?^:^/sys/devices/pci.*.*.*.*\:.*.*/0000\:.*.*\:.*.*..*/dma/dma.*chan.*/quickdata/cap$)$/
> > > /sys/devices/pci0000:40/0000:40:01.3/0000:4a:00.0/0000:4b:0a.0/0000:50:00.0/iommu/amd-iommu/cap =~ /^(?^:^/sys/class/iommu/.*/amd\-iommu/cap$)$/
> > > /sys/devices/pci0000:40/0000:40:01.3/0000:4a:00.0/0000:4b:0a.0/0000:50:00.0/iommu/amd-iommu/cap =~ /^(?^:^/sys/class/iommu/.*/intel\-iommu/cap$)$/
> > > /sys/devices/pci0000:40/0000:40:01.3/0000:4a:00.0/0000:4b:0a.0/0000:50:00.0/iommu/amd-iommu/cap =~ /^(?^:^/sys/devices/pci.*.*.*.*\:.*.*/0000\:.*.*\:.*.*..*/dma/dma.*chan.*/quickdata/cap$)$/
> >
> > Hmm... interesting. Perhaps the problem is on regexes like this:
> >
> > /^(?^:^/sys/devices/pci.*.*.*.*\:.*.*/0000\:.*.*\:.*.*..*/dma/dma.*chan.*/quickdata/cap$)$/
> >
> > Which actually represents this What:
> > /sys/devices/pciXXXX:XX/0000:XX:XX.X/dma/dma<n>chan<n>/quickdata/cap
> >
> > The script could have done a better job escaping "." character on
> > it (but that is not too trivial) and grouping altogether ".*"
> > repetitions, although, in this specific case, probably the best
> > regex would be, instead:
> >
> > /sys/devices/pci[\da-f:\.]+/dma/dma\d+chan\d+/quickdata/cap
> >
> > One possible long-term solution would be to directly use regexes
> > directly on "What" fields inside Documentation/ABI, but on some parts
> > this would require some changes, like, for instance:
> >
> > /sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/kone/roccatkone<minor>/weight
> >
> > Ok, we could likely use capture groups like:
> >
> > (?<NAME>pattern)
> >
> > but IMO that would make it a lot harder to be understood by humans.
> >
> >
> > > And sometimes this thing finishes in 20 seconds, and others, many many
> > > minutes. It's not deterministic at all, which is odd. Is the sysfs
> > > tree being sorted so that this should always have the same search order?
> >
> > No, because it uses a lot of hashes in order to speed it up. Yet,
> > it shouldn't be hard - nor it would significantly affect the processing
> > time - to make it more deterministic. See the enclosed path.
>
> That patch solved everything.
>
> It now only takes 10 seconds. Every time. Without the patch, it feels
> hung for some reason.
The explanation is simple - still weird :-) - basically, when an ABI
symbol is found, the regex test loop stops.
Sorting the regexes probably placed the slowest regex to happen
*after* the one that matched the ABI symbol.
> Care to turn that into a patch that I can take?
Sure. Just sent it.
>
> > > Anyway, I've applied this series as well, this helps in finding the
> > > problems :)
> >
> > Thanks!
> >
> > > Note, I can provide an off-list tarball of /sys/ if that would help in
> > > debugging anything on your end.
> >
> > Yeah, that can help. Feel free to send it to me.
> >
> > Btw, I just got an arm64 server with 128 CPUs for testing. I'm trying
> > to allocate also a big x86 server here, but I'm not sure if it is AMD or
> > Intel.
>
> This is on perl 5.34 here.
>
> thanks,
>
> greg k-h
Thanks,
Mauro
Powered by blists - more mailing lists