linux-ext4 - Lots of free extents, lots of contiguous free space, and fragmentation lvl is ~15%?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date:   Mon, 10 Aug 2020 16:00:42 +0200
From:   Mikhail Morfikov <mmorfikov@...il.com>
To:     linux-ext4@...r.kernel.org
Subject: Lots of free extents, lots of contiguous free space, and
 fragmentation lvl is ~15%?

I have a 1G HDD which is used in LUKSv2 + LVM setup (10 logical disks). One of 
the LVM disks is 320G in size. I had to split this partition into two smaller 
ones of about 160G each. The partition was filled with 75G of data, so over 50%
was free.

To shrinked the partition using the following command:

	# lvreduce -L -160G -r /dev/mapper/wd_blue_label-android

and after about 1 hour, the process was completed. When it was done, I initiated 
e4defrag, to defrag the files, and I noticed, that even very small files 
(10-100K in size) have 2+ extents:

	[591612/591617]/media/Android/AIO-TWRP-Compiler/compile.log:    100%  extents: 2 -> 2   [ OK ]

	# ls -alh /media/Android/AIO-TWRP-Compiler/compile.log
	-rw-r--r-- 1 morfik morfik 34K 2019-02-12 18:22:48 /media/Android/AIO-TWRP-Compiler/compile.log

	# filefrag -ve /media/Android/AIO-TWRP-Compiler/compile.log
	Filesystem type is: ef53
	File size of /media/Android/AIO-TWRP-Compiler/compile.log is 33999 (9 blocks of 4096 bytes)
	 ext:     logical_offset:        physical_offset: length:   expected: flags:
	   0:        0..       7:      10684..     10691:      8:
	   1:        8..       8:      10692..     10692:      1:             last,eof
	/media/Android/AIO-TWRP-Compiler/compile.log: 1 extent found

And there are many files like this one. 

When the defragmentation process finished it's job, there was this summary at 
the end:

        Success:                        [ 510319/591617 ]
        Failure:                        [ 81298/591617 ]
        Total extents:                  593408->593437
        Fragmented percentage:           15%->15%

Fsck says something like this:

	# fsck.ext4 -Dvf /dev/mapper/wd_blue_label-android
	e2fsck 1.45.6 (20-Mar-2020)
	Pass 1: Checking inodes, blocks, and sizes
	Pass 2: Checking directory structure
	Pass 3: Checking directory connectivity
	Pass 3A: Optimizing directories
	Pass 4: Checking reference counts
	Pass 5: Checking group summary information

	android: ***** FILE SYSTEM WAS MODIFIED *****

		  591626 inodes used (5.64%, out of 10485760)
			1311 non-contiguous files (0.2%)
			 265 non-contiguous directories (0.0%)
				 # of inodes with ind/dind/tind blocks: 0/0/0
				 Extent depth histogram: 587843/69
		20222806 blocks used (48.21%, out of 41943040)
			   0 bad blocks
			   3 large files

		  516113 regular files
		   69686 directories
			   0 character device files
			   0 block device files
			   0 fifos
			   0 links
			5818 symbolic links (3706 fast symbolic links)
			   0 sockets
	------------
		  591617 files
 
and the free space looks like this:

	# e2freefrag /dev/mapper/wd_blue_label-android
	Device: /dev/mapper/wd_blue_label-android
	Blocksize: 4096 bytes
	Total blocks: 41943040
	Free blocks: 21720234 (51.8%)

	Min. free extent: 4 KB
	Max. free extent: 2064256 KB
	Avg. free extent: 8972 KB
	Num. free extent: 9681

	HISTOGRAM OF FREE EXTENT SIZES:
	Extent Size Range :  Free extents   Free Blocks  Percent
	    4K...    8K-  :           306           306    0.00%
	    8K...   16K-  :           432           997    0.00%
	   16K...   32K-  :          1127          5672    0.03%
	   32K...   64K-  :          3587         42393    0.20%
	   64K...  128K-  :          1306         30025    0.14%
	  128K...  256K-  :           837         37784    0.17%
	  256K...  512K-  :           411         36730    0.17%
	  512K... 1024K-  :           270         48248    0.22%
	    1M...    2M-  :           243         86984    0.40%
	    2M...    4M-  :           227        164786    0.76%
	    4M...    8M-  :           239        343081    1.58%
	    8M...   16M-  :           196        548225    2.52%
	   16M...   32M-  :           149        833919    3.84%
	   32M...   64M-  :           126       1389825    6.40%
	   64M...  128M-  :            95       2199517   10.13%
	  128M...  256M-  :            49       2322803   10.69%
	  256M...  512M-  :            38       3497585   16.10%
	  512M... 1024M-  :            31       5627846   25.91%
	    1G...    2G-  :            12       4503508   20.73%

So how to read the data? Are the files in this partition really fragmented or 
not?

--
Debian Sid
Linux version 5.8.0-amd64 (morfik@...fikownia) (gcc (Debian 10.2.0-3) 10.2.0, GNU ld (GNU Binutils for Debian) 2.35) #1 SMP PREEMPT Wed Aug 5 15:35:00 CEST 2020
e4defrag 1.45.6 (20-Mar-2020)




Download attachment "signature.asc" of type "application/pgp-signature" (229 bytes)