linux-ext4 - Re: fsck memory usage

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-Id: <307CE092-F24C-4F8A-8AA9-B9E15C03BF3E@dilger.ca>
Date:	Sun, 5 May 2013 19:27:57 -0600
From:	Andreas Dilger <adilger@...ger.ca>
To:	Subranshu Patel <spatel.ml@...il.com>
Cc:	"Theodore Ts'o" <tytso@....edu>, linux-ext4@...r.kernel.org
Subject: Re: fsck memory usage

On 2013-04-30, at 8:42 PM, Subranshu Patel wrote:
>> What version of e2fsprogs are you using? There has been a number of
>> changes made to improve both CPU and memory utilization in more recent versions of e2fsprogs.
> 
> I am using version 1.41.12

Could you please retest with a recent release like 1.42.7?  That
would allow us to compare the memory usage of the newer bitmap
code.  To make it fair, it would probably be best to run the
1.41.12 and 1.42.7 e2fsck on the same image, so you should make
a copy of the block device after corrupting it, but before the
first e2fsck.

>>> Then I performed metadata corruption - 10% of the files, 10% of the directories and some superblock attributes using debugfs.
>>> Then I executed fsck to find a memory usage of around 8GB, a
>>> much larger value.
>> 
>> It's going to depend on what sort of metadata corruption was suffered.  If you need to do pass 1b/c/d fix ups, it will need
>> more memory.
>> 
>> That's pretty much unavoidable, but it's also not the common case.
>> In most use cases, if those cases require using swap, that's
>> generally OK if it's the rare case, and not the common case.
>> That's why it's not something I've really been worried about.
> 
> I used the sar command for tracking memory usage. The total memory
> usage reported by sar command is around 8GB, but it includes the
> buffer and cache memory.
> 
> memused = 8GB
> 
> buffer = 6.7GB
> 
> cache = negligible (some MBs)
> 
> So I think the effective memory usage will be 1.3GB (8 - 6.7). So the
> memory reported under buffer and cache is available for use (if any
> other process requires it). Please correct my understanding.

It would also be useful to compare the "sar" memory usage to the
usage reported by e2fsck itself with "-ttt" to see if they match
relatively well or not.

Cheers, Andreas





--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html