linux-kernel - CPU eaten by unknown code

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date:	Thu, 24 Dec 2009 16:12:21 +0200
From:	Carl <carlr@...email.gr>
To:	linux-kernel@...r.kernel.org
Subject: CPU eaten by unknown code

Hi.

I was benchmarking two programs and at some points i was getting
completely crazy results. Upon futher investigation it was discovered
that at some times the benchmarked programs would take much longer
to run because "somebody" was eating CPU.

Naturally i tried to stop any other processes that could be running
(also note that this is a dual-core system and the benchmarks used
only the one CPU, so theoretically, even another process like cron
might run on the other CPU without interrupting the benchmarks).
No result.

So i wrote a reduced testcase that would exploit the problem.
It's a small python script that keeps printing the time it takes
to run a tripple for-loop, forever.
On my system (Dual core Athlon 64, 32bit kernel+user) the standard
time to run the loop is 1.2 secs. However, this is what happens:

The loop takes 1.2 seconds for about 138 seconds.
The loop takes 2.4 second for about 45 seconds.
And this repeats forever.

In other words, every 138 seconds something is "running" for
45 seconds and it takes 50% of my CPU time!

I have been completely unable to discover what is this thing.
It is not some process as I would see it with 'ps' and 'top'.
It could not be something like flushing the disc cache because
45 seconds is just too much.
The network cable is completely unpluged.
The thing is not visible through /proc/stat cpuload: the system
load is 0% and the only way to understand this thing is eating
CPU is by looking that the for-loop in the python script takes
double time to run.
I cannot completely exclude the possibility that I've been rootkitted
although I believe this is a pretty secure system.

tested on linux 2.6.31.5 and 2.6.29.4

This is a real WTF. 45 seconds with 50% of the CPU is a *lot* of
processing power for a 2.5GHz system. What could be running all this
time? Anybody else seeing similar results? Ideas on how to disable
this so one can run benchmarks on a really "idle" system?

The script:
It prints the time to run the for-loop. It +/-5% deviation from
the previous value it prints a dot. Run for at least 5 minutes on
an idle system...

======================================================>8
# idle.py

import time, sys

def r ():
	for i in xrange (10):
		for i in xrange (1000):
			for i in xrange (1000):
				i = i + 1

def t ():
	t0 = time.time ()
	r ()
	return int (10000*(time.time () - t0))

l = None
while 1:
	x = t ()
	if l is not None:
		if -dt <= x - l <= dt:
			sys.stdout.write (".")
			sys.stdout.flush ()
		else:
			print "(%i secs)"%(time.time () - lt)
			lt = time.time ()
			print "Time to run loop:", x/10000.0
	else:
		dt = x / 20
		lt = time.time ()
		print "Time to run loop:", x/10000.0
	l = x

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/