lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Tue,  1 Oct 2019 00:40:49 -0700
From:   Harshad Shirwadkar <>
Cc:     Harshad Shirwadkar <>
Subject: [PATCH v3 00/13] ext4: add fast commit support

This patch series adds support for fast commits which is a simplified
version of the scheme proposed by Park and Shin, in their paper,
"iJournaling: Fine-Grained Journaling for Improving the Latency of
Fsync System Call"[1]. The basic idea of fast commits is to make JBD2
give the client file system an opportunity to perform a faster
commit. Only if the file system cannot perform such a commit
operation, then JBD2 should fall back to traditional commits.

Because JBD2 operates at block granularity, for every file system
metadata update it commits all the changed blocks are written to the
journal at commit time. This is inefficient because updates to some
blocks that JBD2 commits are derivable from some other blocks. For
example, if a new extent is added to an inode, then corresponding
updates to the inode table, the block bitmap, the group descriptor and
the superblock can be derived based on just the extent information and
the corresponding inode information. So, if we take this relationship
between blocks into account and replay the journalled blocks smartly,
we could increase performance of file system commits significantly.

Fast commits introduced in this patch have two main contributions:

(1) Making JBD2 fast commit aware, so that clients of JBD2 can
    implement fast commits

(2) Add support in ext4 to use JBD2's new interfaces and implement
    fast commits


e2fsprogs was updated to set fast commit feature flag and to ignore
fast commit blocks during e2fsck.

After applying all the patches in this series, following runs of
xfstests were performed:

- -g log -c 4k
- smoke

All the log tests were successful and smoke tests didn't introduce any
additional failures.

Performance Evaluation

Ext4 file system performance was tested with and without fast commit
using fs_mark benchmark. Following was the command used:

Command: ./fs_mark -t 8 -n 1024 -s 65536 -w 4096 -d /mnt

Without Fast Commit: 1501.2 files/sec
With Fast commits: 3055 files/sec
~103% write performance improvement

Changes since V2:

- Added ability to support new file creation in fast commits. This
  allows us to use fs_mark benchmark for performance testing

- Added support for asynchronous fast commits

- Many cleanups and bug fixes

- Re-organized the patch set, moved most of the new code to
  ext4_jbd2.c instead of super.c

- Handling of review comments on previous patchset

Harshad Shirwadkar(13):
 docs: Add fast commit documentation
 ext4: add support for asynchronous fast commits
 ext4: fast-commit recovery path changes
 ext4: fast-commit commit path changes
 ext4: fast-commit commit range tracking
 ext4: track changed files for fast commit
 ext4: add fields that are needed to track changed files
 jbd2: fast-commit recovery path changes
 jbd2: fast-commit commit path new APIs
 jbd2: fast-commit commit path changes
 jbd2: fast commit setup and enable
 ext4: add handling for extended mount options
 ext4: add fast commit support

 Documentation/filesystems/ext4/journal.rst |   98 +-
 Documentation/filesystems/journalling.rst  |   22
 fs/ext4/acl.c                              |    1
 fs/ext4/balloc.c                           |    7
 fs/ext4/ext4.h                             |   86 +
 fs/ext4/ext4_jbd2.c                        |  902 +++++++++++++++++++
 fs/ext4/ext4_jbd2.h                        |   98 ++
 fs/ext4/extents.c                          |   43
 fs/ext4/fsync.c                            |    7
 fs/ext4/ialloc.c                           |   60 -
 fs/ext4/inline.c                           |   14
 fs/ext4/inode.c                            |   77 +
 fs/ext4/ioctl.c                            |    9
 fs/ext4/mballoc.c                          |   83 +
 fs/ext4/mballoc.h                          |    2
 fs/ext4/migrate.c                          |    1
 fs/ext4/namei.c                            |   16
 fs/ext4/super.c                            |   55 +
 fs/ext4/xattr.c                            |    1
 fs/jbd2/commit.c                           |   98 ++
 fs/jbd2/journal.c                          |  343 ++++++-
 fs/jbd2/recovery.c                         |   63 +
 fs/jbd2/transaction.c                      |    3
 include/linux/jbd2.h                       |  117 ++
 include/trace/events/ext4.h                |   61 +
 include/trace/events/jbd2.h                |    9
 26 files changed, 2170 insertions(+), 106 deletions(-)

Powered by blists - more mailing lists