linux-kernel - Re: [RFC 00/14] Dynamic Kernel Stacks

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date: Thu, 14 Mar 2024 15:49:22 -0400
From: Kent Overstreet <kent.overstreet@...ux.dev>
To: Pasha Tatashin <pasha.tatashin@...een.com>
Cc: "H. Peter Anvin" <hpa@...or.com>, linux-kernel@...r.kernel.org, 
	linux-mm@...ck.org, akpm@...ux-foundation.org, x86@...nel.org, bp@...en8.de, 
	brauner@...nel.org, bristot@...hat.com, bsegall@...gle.com, 
	dave.hansen@...ux.intel.com, dianders@...omium.org, dietmar.eggemann@....com, 
	eric.devolder@...cle.com, hca@...ux.ibm.com, hch@...radead.org, 
	jacob.jun.pan@...ux.intel.com, jgg@...pe.ca, jpoimboe@...nel.org, jroedel@...e.de, 
	juri.lelli@...hat.com, kinseyho@...gle.com, kirill.shutemov@...ux.intel.com, 
	lstoakes@...il.com, luto@...nel.org, mgorman@...e.de, mic@...ikod.net, 
	michael.christie@...cle.com, mingo@...hat.com, mjguzik@...il.com, mst@...hat.com, 
	npiggin@...il.com, peterz@...radead.org, pmladek@...e.com, 
	rick.p.edgecombe@...el.com, rostedt@...dmis.org, surenb@...gle.com, tglx@...utronix.de, 
	urezki@...il.com, vincent.guittot@...aro.org, vschneid@...hat.com
Subject: Re: [RFC 00/14] Dynamic Kernel Stacks

On Thu, Mar 14, 2024 at 03:34:03PM -0400, Pasha Tatashin wrote:
> On Thu, Mar 14, 2024 at 3:29 PM Kent Overstreet
> <kent.overstreet@...ux.dev> wrote:
> >
> > On Thu, Mar 14, 2024 at 03:23:08PM -0400, Pasha Tatashin wrote:
> > > > >
> > > > > My point is that what matters is total memory use, not just memory used in
> > > > > the kernel. Amdahl's law.
> > > >
> > > > If userspace is running a few processes with many threads and the
> > > > userspace stacks are small, kernel stacks could end up dominating.
> > > >
> > > > I'd like to see some numbers though.
> > >
> > > The unused kernel stack pages occupy petabytes of memory across the fleet [1].
> >
> > Raw number doesn't mean much here (I know how many machines Google has,
> > of course it's going to be petabytes ;), percentage of system memory
> > would be better.
> >
> > What I'd _really_ like to see is raw output from memory allocation
> > profiling, so we can see how much memory is going to kernel stacks vs.
> > other kernel allocations.
> 
> I've heard there is memory profiling working that can help with that...

I heard you've tried it out, too :)

> While I do not have the data you are asking for, the other kernel
> allocations might be useful, but this particular project is targeted
> to help with reducing overhead where the memory is not used, or used
> in very extreme rare cases.

Well, do you think you could gather it? We shouldn't be blindly applying
performance optimizations; we need to know where to focus our efforts.

e.g. on my laptop I've currently got 356 processes for < 6M of kernel
stack out of 32G total ram, so clearly this isn't much use to me. If the
ratio is similar on your servers - nah, don't want it. I expect the
ratio is not similar and you are burning proportially more memory on
kernel stacks, but we still need to gather the data and do the math :)

> 
> > Number of kernel threads vs. number of user threads would also be good
> > to know - I've been seeing ps output lately where we've got a lot more
> > workqueue workers than we should, perhaps that's something that could be
> > addressed.
> 
> Yes, doing other optimizations make sense, reducing the total number
> kernel threads if possible might help as well. I will look into this
> as well to see how many user threads vs kernel threads we have.

Great, that will help too.