netdev - Re: Receive side performance issue with multi-10-GigE and NUMA

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20090814164412.be5daa74.billfink@mindspring.com>
Date:	Fri, 14 Aug 2009 16:44:12 -0400
From:	Bill Fink <billfink@...dspring.com>
To:	Neil Horman <nhorman@...driver.com>
Cc:	Linux Network Developers <netdev@...r.kernel.org>, brice@...i.com,
	gallatin@...i.com
Subject: Re: Receive side performance issue with multi-10-GigE and NUMA

On Fri, 7 Aug 2009, Neil Horman wrote:

> On Fri, Aug 07, 2009 at 08:54:42PM -0400, Bill Fink wrote:
> > On Fri, 7 Aug 2009, Neil Horman wrote:
> > 
> > > You're timing is impeccable!  I just posted a patch for an ftrace module to help
> > > detect just these kind of conditions:
> > > http://marc.info/?l=linux-netdev&m=124967650218846&w=2
> > > 
> > > Hope that helps you out
> > > Neil
> > 
> > Thanks!  It could be helpful.  Do you have a pointer to documentation
> > on how to use it?  And does it require the latest GIT kernel or could
> > it possibly be used with a 2.6.29.6 kernel?
> > 
> > 						-Bill
> 
> It should apply to 2.6.29.6 no problem (might take a little massaging, but not
> much).

It doesn't look like I can apply your patches to my 2.6.29.6 kernel.

For starters, there's no include/trace/events directory, so there's
no include/trace/events/skb.h.  There is an include/trace/skb.h file,
but there's no TRACE_EVENT defined anywhere in the kernel.

I don't suppose it's as simple as defining (from include/linux/tracepoint.h
from Linus's GIT tree):

#define PARAMS(args...) args

#define TRACE_EVENT(name, proto, args, struct, assign, print)   \
	DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))

So do you still think it's reasonable to try applying your patches
to my 2.6.29.6 kernel, or should I get a newer kernel like 2.6.30.4
or 2.6.31-rc6?

						-Thanks

						-Bill



> No docs I'm afraid (sorry, I'm horrible about that)
> 
> Using it is easy though:
> 
> 1) Patch, build and boot the kernel (make sure to have
> CONFIG_SKB_SOURCES_TRACER, along with the other FTRACE requisite options)
> 
> 2) mount -t debugfs nodev /sys/kernel/debug
> 
> 3) cd /sys/kernel/debug/tracing
> 
> 4) echo skb_sources > ./current_tracer
> 
> 5) echo 1 > trace
> 
> 6) cat ./trace
> 
> Step 5 clears the trace buffer.  Step 6 provides you a list list this
> 
> 
> PID	ANID	CNID	RXQ	CCPU	LEN
> 
> 
> Where:
> PID - The process receiving an skb
> ANID - The node which the skb being received was allocated on
> CNID - The node which the process is running when it read this skb
> RQQ - The NIC receive queue that received this skb
> CCPU - The cpu the process was running on when it read the skb in question
> LEN - The length of the skb being received
> 
> Each entry in the list denotes a unique skb (obviously), and with a clever awk
> script you can identify which nodes each process in your system is receiving
> frames from, so that you can use numactl or taskset to bias that process to run
> on the same nodes cpus.
> 
> Note that step (6) wil show a larger list each time you cat that file (as trace
> records aren't removed during a read.  Step 5 is what actually clears the trace
> buffer and resets the list length to zero.
> 
> Hope that helps. Please feel free to email me if you have any questions.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html