linux-kernel - Re: [KORG] Re: kernel.org lies about latest -mm kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 19 Dec 2006 14:32:56 +0100
From:	Willy Tarreau <w@....eu>
To:	"J.H." <warthog9@...nel.org>
Cc:	Matti Aarnio <matti.aarnio@...iler.org>,
	Randy Dunlap <randy.dunlap@...cle.com>,
	Andrew Morton <akpm@...l.org>, Pavel Machek <pavel@....cz>,
	kernel list <linux-kernel@...r.kernel.org>, hpa@...or.com,
	webmaster@...nel.org
Subject: Re: [KORG] Re: kernel.org lies about latest -mm kernel

On Mon, Dec 18, 2006 at 11:39:51PM -0800, J.H. wrote:
> > If the frontend machines are not taken off-line too often, it should
> > be no big deal for them to handle something such as LVS, and would
> > help spreding the load.
> 
> I'll have to look into it - but by and large the round robining tends to
> work.  Specifically as I am writing this the machines are both pushing
> right around 150mbps, however the load on zeus1 is 170 vs. zeus2's 4.
> Also when we peak the bandwidth we do use every last kb we can get our
> hands on, so doing any tunneling takes just that much bandwidth away
> from the total.

Indeed.

> 	Number of Processes running
> process		#1	#2
> ------------------------------------
> rsync		162	69
> http		734	642
> ftp		353	190
> 
> as a quick snapshot.  I would agree with HPA's recent statement - that
> people who are mirroring against kernel.org have probably hard coded the
> first machine into their scripts, combine that with a few dns servers
> that don't honor or deal with round robining and you have the extra load
> on the first machine vs. the second.

I've also already experienced I/O loads due to rsync. The most annoying
part certainly being that most of the connections see nothing new, but
the disks are seeked anyway, and the cache always gets trashed. A dirty
but probably efficient emergency workaround would be to randomly refuse
a few rsync connections on www1. It would make the mirroring tools fail
once in a while, and the data would be mirrored in larger batches, so
all in all, it would reduce the rate of useless disk seeks.

Since I suspect that the volume of data transferred by rsync is fairly
moderate, it might be interesting to load balance the rsync between the
two machines, even if that involves making the data transit via the net
twice. I can help setting up a reverse proxy setup if you want to give
a try to such a setup.

Best regards,
Willy

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/