linux-kernel - Re: Kernel migration eat CPUs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20130904185315.GB13294@beaver>
Date:	Wed, 4 Sep 2013 22:53:15 +0400
From:	Alexey Vlasov <renton@...ton.name>
To:	Peter Zijlstra <peterz@...radead.org>
Cc:	Andrew Morton <akpm@...ux-foundation.org>,
	Ingo Molnar <mingo@...e.hu>,
	Mike Galbraith <bitbucket@...ine.de>,
	linux-kernel@...r.kernel.org
Subject: Re: Kernel migration eat CPUs

On Sun, Aug 25, 2013 at 04:28:37PM +0200, Peter Zijlstra wrote:

> and a function (or function-graph)
> trace for when this happens?

Unfortunately I could not make trace.
First when migration threads start to eat CPUs and when I turn on trace:
# echo function > current_tracer
# echo 1 > tracing_on
then server is down at once. LA comes up to several thousands, it stops
to receive commands. Sometimes I even have to reboot it.

And second when I manage to get trace then it shows little useful data.

https://www.dropbox.com/s/kxgwr9fuckddtgf/trace.txt.gz

Probably this junk appears from grsec patch? And I don't see migration
threads in it.

> Perhaps perf top -p <pick-a-thread> while the problem is happening will
> show something.

Also "perf top" doesn't show anything. It gives same values when
migration threads eat CPUs and when everything is OK on server.

=========================================
PerfTop: 68932 irqs/sec kernel:60.3%  exact: 0.0% [4000Hz cycles], (all, 32 CPUs)
---------------------------------------------------------------------------------

64.11%  [netconsole]           [k] 0xffffffff81000de3
34.19%  [unknown]              [.] 0x0000000000585c45
 0.40%  libc-2.15.so           [.] 0x0000000000124b10
 0.23%  perf                   [.] 0x000000000004df4f
 0.14%  ld-2.15.so             [.] 0x00000000000183e7
 0.08%  libc-2.15.so           [.] _nss_files_parse_grent
 0.06%  libc-2.15.so           [.] _nss_files_parse_pwent
 0.04%  libc-2.15.so           [.] memchr
 0.04%  php-cgi                [.] match
 0.03%  libcrypt-2.15.so       [.] 0x0000000000004eab
 0.03%  bash                   [.] 0x000000000001fe5a
 0.02%  libc-2.15.so           [.] _IO_getline_info
 0.02%  php-cgi                [.] php_pcre_exec
 0.02%  libc-2.15.so           [.] _dl_addr
 0.02%  eaccelerator.so        [.] eaccelerator_crc32
 0.02%  php-cgi                [.] zend_hash_quick_find
 0.02%  libnss_compat-2.15.so  [.] 0x00000000000055f1
 0.01%  php-cgi                [.] zendparse
 0.01%  libc-2.15.so           [.] fgets_unlocked
 0.01%  libnss_compat-2.15.so  [.] _nss_compat_initgroups_dyn
=========================================

Now server is more loaded and eating CPUs by migration threads happens more often
and it lasts longer.

-- 
BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/