linux-ext4 - Re: Ext4 devel interlock meeting minutes (April 23, 2007)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <462E13BF.7040703@redhat.com>
Date:	Tue, 24 Apr 2007 09:27:11 -0500
From:	Eric Sandeen <sandeen@...hat.com>
To:	Avantika Mathur <mathur@...ux.vnet.ibm.com>
CC:	linux-ext4@...r.kernel.org
Subject: Re: Ext4 devel interlock meeting minutes (April 23, 2007)

Avantika Mathur wrote:
> - large filesystem
>     - We would like to perform more testing on large (>16TB) filesystems
>     - currently hardware limitations are preventing this testing.  We 
> have tested 10TB raid dists, and 16TB loopback devices.  Avantika will 
> look into creating very large sparse devices for testing.

I've been hacking up some ext3@16T testing scripts to use sparse
devicemapper devices which make use of snapshots... loopback files don't
work for testing, at least not hosted on ext[234], because we still
can't do these large file offsets.

(Documentation/device-mapper/zero.txt in the kernel tree describes these
sparse dm devices)

Testing the whole range as a sparse snapshot can be slow, since
devicemapper has to do all the exception handling etc, and I think
essentially creates a fragmented block device.

I've been playing with something like this:

# 90% of the real device size is used for a "real" 1:1 mapping
# The other 10% is sparsely mapped out to add up to totalsize.
# i.e. -

#                          [large sparse-ish device]
#
# +----------------------~  ~-----------------------------------------+
# |                     sparse                |         real          |
# +----------------------~  ~-----------------------------------------+
#
# |<------------ SPARSE_SIZE ---------------->|<----- REAL_SIZE ----->|

# is mapped on top of:

#                           [real block device]
#                      +----------------------------+
#                      | sp |       real            |
#                      +----------------------------+

and then marking the sparse range as full (maybe via lazy_bg, or other
methods).  You could then also put a dm-error target under the "full"
sections so that any IO that may stray there will fail.

This way you can direct the real IO to the 1:1 mapping portion of the
large dm device, and shouldn't get the snapshot slowdowns.

Anyway, just something I've been playing with...

-eric
-
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html