linux-ext4 - ext4 fallocate+truncate not freeing up allocated space

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 29 Oct 2010 11:26:39 -0700
From:	"George Kola" <georgekola@...il.com>
To:	<linux-ext4@...r.kernel.org>
Subject: ext4 fallocate+truncate not freeing up allocated space

I am using RHEL (CentOS) 5.5 with kernel  2.6.18-194.17.4.el5  and ext4. I
posted to RHEL bugzilla and they requested me to post it here.

I wrote a program that fallocates space for a file. It does not know the
file
size apriori and uses a heuristic. If the final file size is less than the
allocated space, I free up the extra space with truncate. I found that
freeing
up did not work so I ended up doing two truncates -- truncate(size+1)
followed
by truncate(size) as suggested in
http://kerneltrap.org/mailarchive/linux-ext4/2009/7/21/6243463. 

I now find that the freed space is not given back till a unmount+ mount. 
I detected this by finding disks full even though the used space was lesser.

I did the standard tests
1. Stopped all program having handles open to files on that partition
2. Verified with lsof that there are no open handles to files in that
partition
     Still there was a mismatch. e.g. see below

e.g. 
bash-3.2# mount | grep hadoop_data
/dev/mapper/vg0-data on /hadoop_data type ext4
(rw,noatime,barrier=0,journal_checksum)
bash-3.2# mount | grep hadoop_data
/dev/mapper/vg0-data on /hadoop_data type ext4
(rw,noatime,barrier=0,journal_checksum)
bash-3.2# du -h /hadoop_data/ | tail -1
328G    /hadoop_data/
bash-3.2# df -h /hadoop_data/
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg0-data  745G  592G  117G  84% /hadoop_data

         After unmount + mount, I find this

bash-3.2# umount /hadoop_data/
bash-3.2# mount /hadoop_data/
bash-3.2# df -h /hadoop_data/
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg0-data  745G  328G  380G  47% /hadoop_data

Version-Release number of selected component (if applicable):

RHEL (CentOS) 5.5 with kernel   2.6.18-194.17.4.el5  and ext4.

How reproducible:
 All the time.

Steps to Reproduce:
  Have a program 
1.  Open a file on ext4 partition
2. Fallocate space for that file  say 20 GB
3. Write data to the file  (but less than the allocated space) say 10 GB
4. Close the file
5. Call truncate(file-path, size+1), truncate(file-path,size) on that file
to
give up the space (observed that the space is given up using filefrag)
6. Use du and df on that ext4 partition to find that the mismatch

-George

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html