linux-kernel - Re: sched/debug: CPU hotplug operation suffers in a large cpu systems

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <Y06B0pr8hpwzxEzI@li-05afa54c-330e-11b2-a85c-e3f3aa0db1e9.ibm.com>
Date:   Tue, 18 Oct 2022 16:07:06 +0530
From:   Vishal Chourasia <vishalc@...ux.vnet.ibm.com>
To:     Greg Kroah-Hartman <gregkh@...uxfoundation.org>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        linux-kernel@...r.kernel.org, mingo@...hat.com,
        vincent.guittot@...aro.org, vschneid@...hat.com,
        srikar@...ux.vnet.ibm.com, sshegde@...ux.ibm.com,
        linuxppc-dev@...ts.ozlabs.org, ritesh.list@...il.com,
        aneesh.kumar@...ux.ibm.com
Subject: Re: sched/debug: CPU hotplug operation suffers in a large cpu systems

On Mon, Oct 17, 2022 at 04:54:11PM +0200, Greg Kroah-Hartman wrote:
> On Mon, Oct 17, 2022 at 04:19:31PM +0200, Peter Zijlstra wrote:
> > 
> > +GregKH who actually knows about debugfs.
> > 
> > On Mon, Oct 17, 2022 at 06:40:49PM +0530, Vishal Chourasia wrote:
> > > smt=off operation on system with 1920 CPUs is taking approx 59 mins on v5.14
> > > versus 29 mins on v5.11 measured using:
> > > # time ppc64_cpu --smt=off
> > > 
> > > 
> > > |--------------------------------+----------------+--------------|
> > > | method                         | sysctl         | debugfs      |
> > > |--------------------------------+----------------+--------------|
> > > | unregister_sysctl_table        |   0.020050 s   | NA           |
> > > | build_sched_domains            |   3.090563 s   | 3.119130 s   |
> > > | register_sched_domain_sysctl   |   0.065487 s   | NA           |
> > > | update_sched_domain_debugfs    |   NA           | 2.791232 s   |
> > > | partition_sched_domains_locked |   3.195958 s   | 5.933254 s   |
> > > |--------------------------------+----------------+--------------|
> > > 
> > > Note: partition_sched_domains_locked internally calls build_sched_domains
> > >       and calls other functions respective to what's being currently used to
> > >       export information i.e. sysctl or debugfs
> > > 
> > > Above numbers are quoted from the case where we tried offlining 1 cpu in system
> > > with 1920 online cpus.
> > > 
> > > From the above table, register_sched_domain_sysctl and
> > > unregister_sysctl_table collectively took ~0.085 secs, whereas
> > > update_sched_domain_debugfs took ~2.79 secs. 
> > > 
> > > Root cause:
> > > 
> > > The observed regression stems from the way these two pseudo-filesystems handle
> > > creation and deletion of files and directories internally.  
> 
> Yes, debugfs is not optimized for speed or memory usage at all.  This
> happens to be the first code path I have seen that cares about this for
> debugfs files.
> 
> You can either work on not creating so many debugfs files (do you really
> really need all of them all the time?)  Or you can work on moving
> debugfs to use kernfs as the backend logic, which will save you both
> speed and memory usage overall as kernfs is used to being used on
> semi-fast paths.
> 
> Maybe do both?
> 
> hope this helps,
> 
> greg k-h

Yes, we need to create 7-8 files per domain per CPU, eventually ending up
creating a lot of files. 

Peter,

Is there a possibility of reverting back to /proc/sys/kernel/sched_domain/?

-- vishal.c
Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)