linux-kernel - Re: RFC: THE OFFLINE SCHEDULER

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <1251408785.3700.22.camel@raz>
Date:	Fri, 28 Aug 2009 00:33:05 +0300
From:	raz ben yehuda <raziebe@...il.com>
To:	Chris Friesen <cfriesen@...tel.com>
Cc:	Andrew Morton <akpm@...ux-foundation.org>, mingo@...e.hu,
	peterz@...radead.org, maximlevitsky@...il.com, efault@....de,
	riel@...hat.com, wiseman@...s.biu.ac.il,
	linux-kernel@...r.kernel.org, linux-rt-users@...r.kernel.org
Subject: Re: RFC: THE OFFLINE SCHEDULER


On Thu, 2009-08-27 at 10:51 -0600, Chris Friesen wrote:
> On 08/26/2009 03:37 PM, raz ben yehuda wrote:
> > 
> > On Wed, 2009-08-26 at 15:15 -0600, Chris Friesen wrote:
> 
> >> We gave it as close to a whole cpu as we could using cpu and irq
> >> affinity and we used message queues in shared memory to allow another
> >> cpu to handle I/O.  In our case we still had kernel threads running on
> >> the app cpu, but if we'd had a straightforward way to avoid them we
> >> would have used it.
> 
> > Chris. I offer myself to help anyone wishes to apply OFFSCHED.
> 
> I just went and read the docs.  One of the things I noticed is that it
> says that the offlined cpu cannot run userspace tasks.  For our
> situation that's a showstopper, unfortunately.
Given that your entire software is T size , and T' is the amount of real
time size, what is the relation T'/T ?
If T'/T << 1 then dissect it, and put the T' in OFFSCHED. 
My software T's is about 100MB while the real time section is about 60K.
They communicate through a simple ioctls.  

              CPU isolation example: a transmission engine.

In the image bellow, I am presenting 4 streaming engines, over 4 Intels
82598EB 10Gbps. A streaming engine is actually a Xeon E5420 2.5GHz.
Each engine has ***full control*** over its own interface. So you can:

1. fully control the processor's usage.

2. know **exactly*** how much each single packet transmission costs. for
example, in this case in processor 3 a single packet average
transmission is 1974tscs, which is ~700ns.

3. know how many packets fails to transmit right **on time** ( the Lates
counter) . and on time in this case means within the 122us jitter. 

4. There are 8 cores in this machine. The rest 4 OS cores are ~95% idle.
The only resource these cores share is the bus. 



State: kstreamer UP. Started at October 05 05:19:51
******************************************************
CPU 3,63% usage,Sessions 1499,6124301 kbps 
CPU 5,77% usage,Sessions 1499,6123859 kbps 
CPU 6,78% usage,Sessions 1498,6123709 kbps 
CPU 7,73% usage,Sessions 1498,6117766 kbps 
Summary: Throughput=24.489Gbps Sessions =5994
******************************************************

Streaming Processor 3
Tx Count       : Tot=399990164  Good=399990164 Bad=0 ERR=0(LOC=0,FULL=0)
Time 	       : GoodSendTsc( Max 1565895 Avg 1974) Lates=649
Flow Errors    : Underflow (0,0) NotResched=0 GenErr=0
Sessions       : Cur 1499(RTP=1499,UDP=0,MCAST=0,ENCRYPT=0) PAUSED=0 
CPU            : 63% usage
Queue          : Max Size 92 Avg 69 Csc=79
Throughput (bps) : Tot  6271285040 MPEG 5988905440
Throughput (Mbps): Tot Mbps 6271 MPEG Mbps 5988
Throughput       : Packets/sec 568855 

Streaming Processor 5
Tx Count       : Tot=399944597  Good=399944595 Bad=2 ERR=2(LOC=0,FULL=2)
Time 	       : GoodSendTsc( Max 1566052 Avg 2464) Lates=5521
Flow Errors    : Underflow (0,0) NotResched=0 GenErr=0
Sessions       : Cur 1499(RTP=1499,UDP=0,MCAST=0,ENCRYPT=0) PAUSED=0 
CPU            : 77% usage
Queue          : Max Size 95 Avg 69 Csc=79
Throughput (bps) : Tot  6270832416 MPEG 5988473792
Throughput (Mbps): Tot Mbps 6270 MPEG Mbps 5988
Throughput       : Packets/sec 568814 

Streaming Processor 6
Tx Count       : Tot=399898586  Good=399898585 Bad=0 ERR=0(LOC=0,FULL=0)
Time 	       : GoodSendTsc( Max 1661385 Avg 2474) Lates=8064
Flow Errors    : Underflow (0,0) NotResched=0 GenErr=0
Sessions       : Cur 1498(RTP=1498,UDP=0,MCAST=0,ENCRYPT=0) PAUSED=0 
CPU            : 78% usage
Queue          : Max Size 91 Avg 69 Csc=87
Throughput (bps) : Tot  6270678560 MPEG 5988326400
Throughput (Mbps): Tot Mbps 6270 MPEG Mbps 5988
Throughput       : Packets/sec 568800 

Streaming Processor 7
Tx Count       : Tot=399845166 Good=399845100 Bad=66
ERR=66(LOC=0,FULL=66)
Time 	       : GoodSendTsc( Max 2962620 Avg 2377) Lates=42626
Flow Errors    : Underflow (0,0) NotResched=0 GenErr=0
Sessions       : Cur 1498(RTP=1498,UDP=0,MCAST=0,ENCRYPT=0) PAUSED=0 
CPU            : 73% usage
Queue          : Max Size 94 Avg 69 Csc=66
Throughput (bps) : Tot  6264592672 MPEG 5982514944
Throughput (Mbps): Tot Mbps 6264 MPEG Mbps 5982
Throughput       : Packets/sec 568248 
--------------- Reservation Load Balancer ------------------
eth2 : 5994501 kbps
eth3 : 5994501 kbps
eth4 : 5990502 kbps
eth5 : 5990502 kbps














 

> Chris
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/