Message-ID: <3553647.iZASKD2KPV@natalenko.name>
Date: Tue, 12 Aug 2025 09:37:22 +0200
From: Oleksandr Natalenko <oleksandr@...alenko.name>
To: David Rientjes <rientjes@...gle.com>, Damien Le Moal <dlemoal@...nel.org>
Cc: linux-kernel@...r.kernel.org, linux-block@...r.kernel.org,
 Jens Axboe <axboe@...nel.dk>, John Garry <john.g.garry@...cle.com>,
 Christoph Hellwig <hch@....de>,
 "Martin K. Petersen" <martin.petersen@...cle.com>, linux-mm@...ck.org,
 Lorenzo Stoakes <lorenzo.stoakes@...cle.com>,
 Shakeel Butt <shakeel.butt@...ux.dev>, Qi Zheng <zhengqi.arch@...edance.com>,
 Michal Hocko <mhocko@...nel.org>, David Hildenbrand <david@...hat.com>,
 Johannes Weiner <hannes@...xchg.org>,
 Andrew Morton <akpm@...ux-foundation.org>
Subject:
 Re: [REGRESSION][BISECTED] Unexpected OOM instead of reclaiming inactive file
 pages

Hello.

On úterý 12. srpna 2025 2:45:02, středoevropský letní čas Damien Le Moal wrote:
> On 8/12/25 5:42 AM, Oleksandr Natalenko wrote:
> > On pondělí 11. srpna 2025 18:06:16, středoevropský letní čas David Rientjes wrote:
> >> On Mon, 11 Aug 2025, Oleksandr Natalenko wrote:
> >>> I'm fairly confident that the following commit
> >>>
> >>> 459779d04ae8d block: Improve read ahead size for rotational devices
> >>>
> >>> caused a regression in my test bench.
> >>>
> >>> I'm running v6.17-rc1 in a small QEMU VM with virtio-scsi disk. It has got 1 GiB of RAM, so I can saturate it easily causing reclaiming mechanism to kick in.
> >>>
> >>> If MGLRU is enabled:
> >>>
> >>> $ echo 1000 | sudo tee /sys/kernel/mm/lru_gen/min_ttl_ms
> >>>
> >>> then, once page cache builds up, an OOM happens without reclaiming inactive file pages: [1]. Note that inactive_file:506952kB, I'd expect these to be reclaimed instead, like how it happens with v6.16.
> >>>
> >>> If MGLRU is disabled:
> >>>
> >>> $ echo 0 | sudo tee /sys/kernel/mm/lru_gen/min_ttl_ms
> >>>
> >>> then OOM doesn't occur, and things seem to work as usual.
> >>>
> >>> If MGLRU is enabled, and 459779d04ae8d is reverted on top of v6.17-rc1, the OOM doesn't happen either.
> >>>
> >>> Could you please check this?
> >>>
> >>
> >> This looks to be an MGLRU policy decision rather than a readahead 
> >> regression, correct?
> >>
> >> Mem-Info:
> >> active_anon:388 inactive_anon:5382 isolated_anon:0
> >>  active_file:9638 inactive_file:126738 isolated_file:0
> >>
> >> Setting min_ttl_ms to 1000 is preserving the working set and triggering 
> >> the oom kill is the only alternative to free memory in that configuration.  
> >> The oom kill is being triggered by kswapd for this purpose.
> >>
> >> So additional readahead would certainly increase that working set.  This 
> >> looks working as intended.
> > 
> > OK, this makes sense indeed, thanks for the explanation. But is inactive_file explosion expected and justified?
> > 
> > Without revert:
> > 
> > $ echo 3 | sudo tee /proc/sys/vm/drop_caches; free -m; sudo journalctl -kb >/dev/null; free -m
> > 3
> >                total        used        free      shared  buff/cache   available
> > Mem:             690         179         536           3          57         510
> > Swap:           1379          12        1367
> > /* OOM happens here */
> >                total        used        free      shared  buff/cache   available
> > Mem:             690         177          52           3         561         513
> > Swap:           1379          17        1362 
> > 
> > With revert:
> > 
> > $ echo 3 | sudo tee /proc/sys/vm/drop_caches; free -m; sudo journalctl -kb >/dev/null; free -m
> > 3
> >                total        used        free      shared  buff/cache   available
> > Mem:             690         214         498           4          64         476
> > Swap:           1379           0        1379
> > /* no OOM */
> >                total        used        free      shared  buff/cache   available
> > Mem:             690         209         462           4         119         481
> > Swap:           1379           0        1379
> > 
> > The journal folder size is:
> > 
> > $ sudo du -hs /var/log/journal
> > 575M    /var/log/journal
> > 
> > It looks like this readahead change causes far more data to be read than actually needed?
> 
> For your drive as seen by the VM, what is the value of
> /sys/block/sdX/queue/optimal_io_size ?
> 
> I guess it is "0", as I see on my VM.

Yes, it's 0.

> So before 459779d04ae8d, the block device read_ahead_kb was 128KB only, and
> 459779d04ae8d switched it to be 2 times the max_sectors_kb, so 8MB. This change
> significantly improves file buffered read performance on HDDs, and HDDs only.

Right, max_sectors_kb is 4096.

> This means that your VM device is probably being reported as a rotational one
> (/sys/block/sdX/queue/rotational is 1), which is normal if you attached an
> actual HDD. If you are using a qcow2 image for that disk, then having
> rotational==1 is questionable...

Yes, it's reported as rotational by default.

I've just set -device scsi-hd,drive=hd1,rotation_rate=1 so that guest will see the drive as non-rotational from now on, which brings old behaviour back.

> The other issue is the device driver for the device reporting 0 for the optimal
> IO size, which normally happens only for SATA drives. I see the same with
> virtio-scsi, which is also questionable given that the maximum IO size with it
> is fairly limited. So virtio-scsi may need some tweaking.
> 
> The other thing to question, I think, is setting read_ahead_kb using the
> optimal_io_size limit (io_opt), which can be *very large*. For most SCSI
> devices, it is 16MB, so you will see a read_ahead_kb of 32 MB. But for SCSI
> devices, optimal_io_size indicates a *maximum* IO size beyond which performance
> may degrade. So using any value lower than this, but still reasonably large,
> would be better in general I think. Note that lim->io_opt for RAID arrays
> actually indicates the stripe size, so generally a lot smaller than the
> component drives io_opt. And this use changes the meaning of that queue limit,
> which makes things even more confusing and finding an adequate default harder.

Thank you for the explanation.

-- 
Oleksandr Natalenko, MSE
Download attachment "signature.asc" of type "application/pgp-signature" (834 bytes)

Powered by blists - more mailing lists