Message-ID: <4D64F26D.1050803@internode.on.net>
Date:	Wed, 23 Feb 2011 22:11:33 +1030
From:	Arthur Marsh <arthur.marsh@...ernode.on.net>
To:	Mel Gorman <mel@....ul.ie>
CC:	Andrea Arcangeli <aarcange@...hat.com>,
	Clemens Ladisch <cladisch@...glemail.com>,
	alsa-user@...ts.sourceforge.net, linux-kernel@...r.kernel.org
Subject: Re: [Alsa-user] new source of MIDI playback slow-down identified
 -	5a03b051ed87e72b959f32a86054e1142ac4cf55 thp: use compaction in	kswapd
 for GFP_ATOMIC order > 0



Mel Gorman wrote, on 23/02/11 19:45:
> On Tue, Feb 22, 2011 at 08:43:25PM +0100, Andrea Arcangeli wrote:
>> On Wed, Feb 23, 2011 at 04:17:44AM +1030, Arthur Marsh wrote:
>>> OK, these patches applied together against upstream didn't cause a crash
>>> but I did observe:
>>>
>>> significant slowdowns of MIDI playback (moreso than in previous cases,
>>> and with less than 20 Meg of swap file in use);
>>>
>
> Can this be replicated with on-board audio hardware or is there something
> specific about the card? e.g. those typically driven by snd-intel8x0 or
> snd_hda_intel

This card (Soundblaster Audigy 2 ZS - SB0350) has an on-board wavetable 
synthesiser. To perform a test with sound hardware included on a 
motherboard, one would need to connect the pc to an external synthesiser 
to play the MIDI signal from the sound hardware on the motherboard.

To quote an earlier posting from Clemens Ladisch:

"The ALSA sequencer uses either the system timer or an HR timer at 1 kHz
to deliver MIDI commands (notes); the wavetable driver requires its own
interrupts in regular 5.3 ms intervals."

The wavetable interrupts are apparently being lost. I am not getting 
lost MIDI events (ie notes going missing or being stuck on), but 
slowdown in play-back.

>
> Unfortunately the original mail is a bit light on details on how this was
> reproduced and I didn't find a thread with more details. It looks like it's
> simply playing a midi file while the system is under load but less clear
> on what the symptoms are (audio skipping maybe?). I'll start with using
> irqs-off tracer to see can I replicate a similar style of issue without
> depending on sound.

Clemens, would this work to identify the problem without relying on a 
device such as a sound card with a wavetable synthesiser or external 
synthesiser receiving MIDI signals from the pc?

>
>>> kswapd0 sharing equal top place in CPU usage at times (e.g. 20 percent).
>>>
>>> If I should try only one of the patches or something else entirely,
>>> please let me know.
>>
>> For Mel: with z1, kswapd used only 0.1-3.9 percent of CPU while he
>> loaded other applications.
>>
>
> What's the usage otherwise?
>
> As that patch has been NAKd by Rik on the grounds it eliminates the "balance
> gap" entirely, can it be checked if the patch below keeps the CPU usage low
> as well please?

Unfortunately, when I loaded KDE 3.5.X, konversation, aptitude -u, and 
icedove with Mel's patch in the last few minutes, kswapd0 ran up to 17.5 
percent CPU usage and the playback using aplaymidi slowed down. Load was 
around 6 and swap usage had barely started when I noticed the slowdown 
in MIDI playback.

By contrast, with the "z1" patch I could load the pc as above and also 
starting iceweasel with a few dozen tabs open, and play the MIDI file by 
using the QT application Rosegarden instead of aplaymidi, and not have 
any slowdown, even though the swap usage was up around 200 MB and load 
was over 10.0.

>
>> We may need a way to put kswapd in all uncompactable mode to solve
>> this, logic 3 just trying not to disable the all unreclaimable logic
>> seems not enough. I.e. if compact_zone_order doesn't return
>> COMPACT_COMPLETE, stop the compaction loop in kswapd. Then we can put
>> back in the COMPACT_MODE_KSWAPD return COMPACT_CONTINUE in
>> compact_finished as the caller will throttle it (and it won't run more
>> than one scan before putting kswapd to sleep in all uncompactable mode).
>>
>
> Should be possible. I'm not going to develop such a patch now though and
> instead have a stab at replicating some of the exhibited problems (high
> kswapd CPU usage, long interrupt disabled times etc).
>
> Can the following patch be tested as a potential replacement for z1
> please?
>
> ==== CUT HERE ====
> vmscan: kswapd should not free an excessive number of pages when balancing small zones
>
> When reclaiming for order-0 pages, kswapd requires that all zones be
> balanced. Each cycle through balance_pgdat() does background ageing on all
> zones if necessary and applies equal pressure on the inactive zone unless
> a lot of pages are free already.
>
> A "lot of free pages" is defined as a "balance gap" above the high watermark
> which is currently 7*high_watermark. Historically this was reasonable as
> min_free_kbytes was small. However, on systems using huge pages, it is
> recommended that min_free_kbytes is higher and it is tuned with hugeadm
> --set-recommended-min_free_kbytes. With the introduction of transparent
> huge page support, this recommended value is also applied. On X86-64 with
> 4G of memory, min_free_kbytes becomes 67584 so one would expect around 68M
> of memory to be free. The Normal zone is approximately 35000 pages so under
> even normal memory pressure such as copying a large file, it gets exhausted
> quickly. As it is getting exhausted, kswapd applies pressure equally to all
> zones, including the DMA32 zone. DMA32 is approximately 700,000 pages with
> a high watermark of around 23,000 pages. In this situation, kswapd will
> reclaim around (23000*8 where 8 is the high watermark + balance gap of 7 *
> high watermark) pages or 718M of pages before the zone is ignored. What
> the user sees is that free memory far higher than it should be.
>
> To avoid an excessive number of pages being reclaimed from the larger zones,
> explicitely defines the "balance gap" to be either 1% of the zone or the
> low watermark for the zone, whichever is smaller.  While kswapd will check
> all zones to apply pressure, it'll ignore zones that meets the (high_wmark +
> balance_gap) watermark.
>
> To test this, 80G were copied from a paritition and the amount of memory
> being used was recorded. A comparison of a patch and unpatched kernel
> can be seen at
> http://www.csn.ul.ie/~mel/postings/minfree-20110222/memory-usage-hydra.ps
> and shows that kswapd is not reclaiming as much memory with the patch
> applied.
>
> Signed-off-by: Andrea Arcangeli<aarcange@...hat.com>
> Signed-off-by: Mel Gorman<mel@....ul.ie>
> ---
>   include/linux/swap.h |    9 +++++++++
>   mm/vmscan.c          |   16 +++++++++++++---
>   2 files changed, 22 insertions(+), 3 deletions(-)
>
> diff --git a/include/linux/swap.h b/include/linux/swap.h
> index 4d55932..a57c6e7 100644
> --- a/include/linux/swap.h
> +++ b/include/linux/swap.h
> @@ -155,6 +155,15 @@ enum {
>   #define SWAP_CLUSTER_MAX 32
>   #define COMPACT_CLUSTER_MAX SWAP_CLUSTER_MAX
>
> +/*
> + * Ratio between the present memory in the zone and the "gap" that
> + * we're allowing kswapd to shrink in addition to the per-zone high
> + * wmark, even for zones that already have the high wmark satisfied,
> + * in order to provide better per-zone lru behavior. We are ok to
> + * spend not more than 1% of the memory for this zone balancing "gap".
> + */
> +#define KSWAPD_ZONE_BALANCE_GAP_RATIO 100
> +
>   #define SWAP_MAP_MAX	0x3e	/* Max duplication count, in first swap_map */
>   #define SWAP_MAP_BAD	0x3f	/* Note pageblock is bad, in first swap_map */
>   #define SWAP_HAS_CACHE	0x40	/* Flag page is cached, in first swap_map */
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 17497d0..0c83530 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -2388,6 +2388,7 @@ loop_again:
>   			int compaction;
>   			struct zone *zone = pgdat->node_zones + i;
>   			int nr_slab;
> +			unsigned long balance_gap;
>
>   			if (!populated_zone(zone))
>   				continue;
> @@ -2404,11 +2405,20 @@ loop_again:
>   			mem_cgroup_soft_limit_reclaim(zone, order, sc.gfp_mask);
>
>   			/*
> -			 * We put equal pressure on every zone, unless one
> -			 * zone has way too many pages free already.
> +			 * We put equal pressure on every zone, unless
> +			 * one zone has way too many pages free
> +			 * already. The "too many pages" is defined
> +			 * as the high wmark plus a "gap" where the
> +			 * gap is either the low watermark or 1%
> +			 * of the zone, whichever is smaller.
>   			 */
> +			balance_gap = min(low_wmark_pages(zone),
> +				(zone->present_pages +
> +					KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> +				KSWAPD_ZONE_BALANCE_GAP_RATIO);
>   			if (!zone_watermark_ok_safe(zone, order,
> -					8*high_wmark_pages(zone), end_zone, 0))
> +					high_wmark_pages(zone) + balance_gap,
> +					end_zone, 0))
>   				shrink_zone(priority, zone,&sc);
>   			reclaim_state->reclaimed_slab = 0;
>   			nr_slab = shrink_slab(sc.nr_scanned, GFP_KERNEL,
>

Regards,

Arthur.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists