linux-kernel - Re: Linux 2.6.31 - very swap-happy with plenty of free RAM

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <2f11576a0911190636vd21069bv2fe4f22a57b3d333@mail.gmail.com>
Date:	Thu, 19 Nov 2009 23:36:04 +0900
From:	KOSAKI Motohiro <kosaki.motohiro@...il.com>
To:	Dan Merillat <dan.merillat@...il.com>
Cc:	linux-kernel@...r.kernel.org, Rik van Riel <riel@...hat.com>,
	Norbert Preining <preining@...ic.at>,
	Tomasz Chmielewski <mangoo@...g.org>,
	Sven-Haegar Koch <haegar@...net.de>,
	Dave Chinner <david@...morbit.com>
Subject: Re: Linux 2.6.31 - very swap-happy with plenty of free RAM

Hi Dan,

2009/11/18 Dan Merillat <dan.merillat@...il.com>:
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
>  1  1      0 774404  49644 1713932    0    0     8    44 2680 5105 23 42  0 35
>  2  1      0 774900  49448 1713668    0    0     8    28 2870 5101 22 41  0 36
>
> swapon /dev/vg1/swap
>
>  2  2  34460 770808  47824 1709852    0 34460   952 34472 3174 5601 21 48  2 29
>  1  1 129716 759904  47460 1699364    0 95316   272 95532 3093 5641 16 49  7 29
>  1  4 196728 756680  47436 1693616   64 67080    92 67104 2980 5569 18 49 24  8
>  2  1 246212 754076  47396 1689240   32 49652   220 49760 3241 5405 19 51  1 29
>  2  3 282404 791648  47272 1686252   16 36284   240 36392 3088 6281 18 52  9 22
>  1  4 299464 847200  47260 1685208  324 17296   324 17320 3190 6199 23 46  0 30
>  2  5 302316 854384  47256 1685884  944 3804  1948  3812 2723 5297 20 45  5 31
> 11  3 303436 861700  47384 1686400 1188 1900  2084  1912 2615 4593 21 51  9 20
> swapoff -a
>  2  6 301740 863436  47384 1687048 1860    4  2368   128 2541 4591 21 63  0 15
>  3  4 300076 865916  47384 1687604 1672    0  2120   156 2673 5208 19 63  0 18
>  2  5 297676 866288  47380 1687988 2396    0  2668   188 2632 5259 17 70  2 12
>  2  2 295556 866784  47380 1687956 2352    0  2352     0 2621 5316 16 72  9  4
>  5  5 293168 867156  47384 1688120 2344    0  2344    20 2651 5469 18 79  0  4
>  2  4 291108 870504  47396 1688260 2036    0  2172   160 2381 4817 15 67  0 18
>  3  3 289416 870612  47400 1688476 2028    0  2260   148 2602 5214 18 69  0 13
>  3  3 287528 871928  47400 1688556 1928    0  2136     0 2470 5096 19 68  3 10
>  2  4 285740 873896  47408 1688892 2352    0  2764   172 2490 5194 18 66  0 17
>
> 1.7gb of disk cache, 750 meg of free RAM, and it swaps out 300meg in a
> matter of seconds when given access to a swapfile, then thrashes the
> disk like crazy because that RAM was actively in use. From an
> interactivity standpoint the machine is unusable - 1-2 second pauses
> on mouse movement, 10-15 seconds for the window manager to change
> window focus, keypresses take 1-2 seconds to show up, etc.  When I
> issue a swapoff it takes 10-15  minutes for it to slowly pull it all
> back in - once it's done I run fine.
>
> The machine has 4gb of RAM (64bit), and runs 3 VMware VMs with a total
> of 1024mb allocated to them.   VMware uses file-backed RAM (for some
> ungodly stupid reason) which is on a separate spindle (/dev/sdb) from
> swap (/dev/sda).  I've got one VM pointed to a file-backed RAM on a
> tmpfs (512mb) that's probably got quite a bit of memory activity.
>
> The other main RAM hogs are the usual suspects - firefox, Xorg,
> thunderbird, openoffice, pidgin, etc, the usual junk on a busy Linux
> desktop.
>
> All the sysctl tuneables should be defaults, there's no distro or
> local changes to any vm settings (mmap_min_addr=0 for Wine,
> shmmax=256mb, some ipv6 tweaks)
>
> Ideas on where to start looking?  If this is a vmware issue I'll take
> it to them.

Umm, very strange.
I made two debug patch. can you please apply it and post following
command output?

% cat /proc/meminfo
% cat /proc/vmstat
% cat /proc/zoneinfo
# cat /proc/filecache | sort -nr -k3 |head -30

Download attachment "proc-filecache-v2.patch" of type "application/octet-stream" (10880 bytes)

Download attachment "0001-Add-recent-rotated-scanned-info-to-proc-zoneinfo.patch" of type "application/octet-stream" (1049 bytes)