linux-kernel - Re: 20% performance drop on PostgreSQL 9.2 from kernel 3.5.3 to 3.6-rc5 on AMD chipsets

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Sat, 15 Sep 2012 16:47:39 +0200
From:	Mike Galbraith <efault@....de>
To:	Borislav Petkov <bp@...en8.de>
Cc:	Nikolay Ulyanitsky <lystor@...il.com>,
	linux-kernel@...r.kernel.org,
	Andreas Herrmann <andreas.herrmann3@....com>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Ingo Molnar <mingo@...nel.org>
Subject: Re: 20% performance drop on PostgreSQL 9.2 from kernel 3.5.3 to
 3.6-rc5 on AMD chipsets - bisected

On Sat, 2012-09-15 at 12:44 +0200, Borislav Petkov wrote: 
> On Sat, Sep 15, 2012 at 06:11:02AM +0200, Mike Galbraith wrote:
> > My wild (and only) theory is that this is userspace spinlock related.
> > If so, starting the server and benchmark SCHED_BATCH should not only
> > kill the regression, but likely improve throughput as well.
> 
> FWIW,
> 
> I went and tried it. Here are the exact steps:
> 
> $ ps ax | grep postgres
>  2066 ?        S      0:01 /usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main -c config_file=/etc/postgresql/9.1/main/postgresql.conf
>  2070 ?        Ss     0:07 postgres: writer process
>  2071 ?        Ss     0:05 postgres: wal writer process
>  2072 ?        Ss     0:01 postgres: autovacuum launcher process
>  2073 ?        Ss     0:01 postgres: stats collector process
>  5788 pts/0    S+     0:00 grep postgres
> 
> # set to SCHED_BATCH
> $ schedtool -B 2066 2070 2071 2072 2073
> 
> # verify:
> $ schedtool 2066 2070 2071 2072 2073
> PID  2066: PRIO   0, POLICY B: SCHED_BATCH   , NICE   0, AFFINITY 0x3f
> PID  2070: PRIO   0, POLICY B: SCHED_BATCH   , NICE   0, AFFINITY 0x3f
> PID  2071: PRIO   0, POLICY B: SCHED_BATCH   , NICE   0, AFFINITY 0x3f
> PID  2072: PRIO   0, POLICY B: SCHED_BATCH   , NICE   0, AFFINITY 0x3f
> PID  2073: PRIO   0, POLICY B: SCHED_BATCH   , NICE   0, AFFINITY 0x3f
> 
> $ su - postgres
> postgres@...st:~$ export PATH=$PATH:/usr/lib/postgresql/9.1/bin/
> postgres@...st:~$ schedtool -B -e pgbench -i pgbench && pgbench -c 10 -t 10000 pgbench
> 
> ...
> 
> tps = 4388.118940 (including connections establishing)
> tps = 4391.771875 (excluding connections establishing)
> 
> => even better than the results with 3.5 (had something around 3900ish
> on that particular configuration).

Increasing /proc/sys/kernel/sched_min_granularity_ns to roughly half of
sched_latency_ns should also help.  That will allow LAST_BUDDY to do
it's job, try to hand the CPU back to a preempted task if possible.  The
change that increased sched_nr_latency to 8 should have injured
postgress as well.  ATM, it's disabled unless you're massively loaded.

I _think_ it's about preemption, but it doesn't matter, patch is toast.

-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/