linux-ext4 - Re: Significant difference in 'file size' and 'disk usage' for single files

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <07f4b566-7ec5-6d36-74bb-d41b694b3194@clearskydata.com>
Date:   Mon, 6 Nov 2017 10:35:54 -0500
From:   "Mattthew L. Martin" <linux@...arskydata.com>
To:     mfe555 <mfe555@....de>, linux-ext4@...r.kernel.org
Subject: Re: Significant difference in 'file size' and 'disk usage' for single
 files

I filed a bug for this a while ago:

https://bugzilla.kernel.org/show_bug.cgi?id=151491

We would be happy to help track this down as it is a pain to manage this 
on running servers.

Matthew


On 11/5/17 06:16, mfe555 wrote:
> Some follow-up:
>
> The issue only occurs with "bigalloc" enabled.
>
>     echo 3 > /proc/sys/vm/drop_caches
>
> seems to detach the blocked disk space from the files (so that 'du 
> file' no longer includes the offset), but it does not free the space, 
> 'df' still shows all file overheads as used disk space.
>
>
>
> Am 02.11.2017 um 20:17 schrieb mfe555:
>> Hi, I'm using ext4 on a Linux based Enigma2 set-top box, kernel 4.8.3.
>>
>> When creating a fresh file, there is a significant difference in file 
>> size (ls -la) and disk usage (du). When making two copies of the file ..
>>
>> gbquad:/hdd/test# cp file file.copy1
>> gbquad:/hdd/test# cp file file.copy2
>> gbquad:/hdd/test# ls -la
>> -rw-------    1 root     root     581821460 Nov  1 18:52 file
>> -rw-------    1 root     root     581821460 Nov  1 18:56 file.copy1
>> -rw-------    1 root     root     581821460 Nov  1 18:57 file.copy2
>> gbquad:/hdd/test# du *
>> 607232  file
>> 658176  file.copy1
>> 644864  file.copy2
>>
>> ... all three files show an overhead in the ~10% range, and the 
>> overhead is different for these files although their md5sums are equal.
>>
>> When deleting a file (rm), the overhead remains occupied on the disk. 
>> For example, after deleting "file", "df" reports approx. 581821460 
>> more bytes free, not 607232 kbytes more free space. The overhead 
>> (607232 kB - 581821460 B =pprox. 39 MB) remains blocked.
>>
>> When re-booting, the blocked space becomes free again, and in 
>> addition the overhead of those files that were not deleted also 
>> disappears, so that after a reboot the'file size' and 'disk usage' 
>> match for all files (except for rounding up to some block size).
>>
>> A colleague and I have observed this on two different "kernel 4.8.3" 
>> boxes and three ext4 disks, but not on a "kernel 3.14" box also using 
>> ext4.
>>
>> Can anyone help me with this ?
>>
>> Thanks a lot
>> Lukas
>>
>