linux-kernel - [regression, 3.15-rc1] vdso_gettimeofday hogs all my CPU

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20140415075335.GE31578@dastard>
Date:	Tue, 15 Apr 2014 17:53:36 +1000
From:	Dave Chinner <david@...morbit.com>
To:	linux-kernel@...r.kernel.org
Cc:	"H. Peter Anvin" <hpa@...or.com>
Subject: [regression, 3.15-rc1] vdso_gettimeofday hogs all my CPU

Hi Peter,

I'm guessing that x86 vdso problems are in your area of expertise,
if not can you point me at the right person to bug?

I just spun up my XFS performance tests on 3.15-rc1, the first of
which runs a concurrent fsmark workload. fsmark uses gettimeofday to
do per-operation timing and performance is horrendous. 3.14.0 on this
workload does around 300,000 file creates per second. 3.15-rc1
does:

#  ./fs_mark  -D  10000  -S0  -n  100000  -s  0  -L  32  -d  /mnt/scratch/0  -d  /mnt/scratch/1  -d  /mnt/scratch/2  -d  /mnt/scratch/3  -d  /mnt/scratch/4  -d  /mnt/scratch/5  -d  /mnt/scratch/6  -d  /mnt/scratch/7  -d  /mnt/scratch/8  -d  /mnt/scratch/9  -d  /mnt/scratch/10  -d  /mnt/scratch/11  -d  /mnt/scratch/12  -d  /mnt/scratch/13  -d  /mnt/scratch/14  -d  /mnt/scratch/15
#       Version 3.3, 16 thread(s) starting at Tue Apr 15 17:37:20 2014
#       Sync method: NO SYNC: Test does not issue sync() or fsync() calls.
#       Directories:  Time based hash between directories across 10000 subdirectories with 180 seconds per subdirectory.
#       File names: 40 bytes long, (16 initial bytes of time stamp with 24 random bytes at end of name)
#       Files info: size 0 bytes, written with an IO size of 16384 bytes per write
#       App overhead is time in microseconds spent in the test not doing file writing related system calls.

FSUse%        Count         Size    Files/sec     App Overhead
     0      1600000            0      70950.5        155219877
     0      3200000            0      71832.4        150138649
     0      4800000            0      71195.5        150935842
     0      6400000            0      71215.4        150906768
     0      8000000            0      63707.7        166569461
.....

about 70,000 files/sec, so it's way, way down on 3.14. The load has
pegged all 16 CPUs in the VM, and perf top tells me:

-  63.11%  [vdso]              [.] __vdso_gettimeofday
   - __vdso_gettimeofday
        44.24% stop
        28.18% do_run
        13.98% write_file
        13.60% setup_file_name

that it's the gettimeofday calls that are causing the excessive CPU
load.

I haven't changed anything in userspace for the past couple of
months, the only thing that has changed is the kernel. I haven't
looked at this any further and have no idea what to do to debug it
further, so let me know if you need more information....

Cheers,

Dave.
-- 
Dave Chinner
david@...morbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/