Message-Id: <cover.1596213437.git.dsterba@suse.com>
Date:   Fri, 31 Jul 2020 18:42:03 +0200
From:   David Sterba <dsterba@...e.com>
To:     torvalds@...ux-foundation.org
Cc:     David Sterba <dsterba@...e.com>, linux-btrfs@...r.kernel.org,
        linux-kernel@...r.kernel.org
Subject: [GIT PULL] Btrfs updates for 5.9

Hi,

we don't have any big feature updates this time, there are lots of small
enhacements or fixes. A hilight perhaps are the parallel fsync
performance improvements, numbers below.

Regarding the dio/iomap that was reverted last time, the required API
changes are likely to land in the upcoming cycle, the btrfs part will be
updated afterwards.

Please pull, thanks.

---

User visible changes:

- new mount option rescue= to group all recovery-related mount options
  so we don't have many specific options, currently introducing only
  aliases for existing options, future extensions are in development to
  allow read-only mount with partially damaged structures
  - usebackuproot is an alias for rescue=usebackuproot
  - nologreplay is an alias for rescue=nologreplay

- start deprecation of mount option inode_cache, removal scheduled to
  5.11

- removed deprecated mount options alloc_start and subvolrootid

- device stats corruption counter gets incremented when a checksum
  mismatch is found

- qgroup information exported in sysfs /sys/fs/btrfs/<UUID>/qgroups/<id>

- add link /sys/fs/btrfs/<UUID>/bdi pointing to the associated backing
  dev info

- FS_INFO ioctl enhancements:
  - add flags to request/describe newly added items
  - new item: numeric checksum type and checksum size
  - new item: generation
  - new item: metadata_uuid

- seed device: with one new read-write device added, print the new
  device information in /proc/mounts

- balance: detect cancellation by Ctrl-C in existing cancellation points

Performance improvements:

- optimized versions of various helpers on little-endian architectures,
  where we don't have to do LE/BE conversion from on-disk format

- tree-log/fsync optimizations leading to lower max latency reported by
  dbench, reduced by about 12%

- all chunk tree leaves are prefetched at mount time, can improve mount
  time on large (terabyte-sized) filesystems

- speed up parallel fsync of files with reflinked/deduped extents, with
  jobs 16 to 1024 the throughput gets improved roughly by 50% on average
  and runtime decreased roughly by 30% on average, notable outlier is
  128 jobs with +121.2% on throughput and -54.6% runtime

- another speed up of parallel fsync, reduce number of checksum tree
  lookups and contention, the improvements start to show up with 2
  tasks with +20% throughput and -16% runtime up to 64 with +200%
  throughput and -66% runtime

Core:

- umount-time qgroup leak checker

- qgroups
  - add a way to unreserve partial range after failure, avoiding some
    EDQUOT errors
  - improved flushing logic when EDQUOT is hit

- possible EINTR interruption caused by failed reservations after
  transaction start is better handled and documented

- transaction abort errors are unified to EROFS in case it's not the
  original reason of abort or we don't have other way to determine the
  reason

Fixes:

- make truncate succeed on a NOCOW file even if data space is exhausted

- fix cancelling balance on filesystem with exhausted metadata space

- anon block device:
  - preallocate anon bdev when subvolume is created to report failure
    early
  - shorten time the anon bdev id is allocated
  - don't allocate anon bdev for internal roots

- minor memory leak in ref-verify

- refuse invalid combinations of compression and NOCOW file flags

- lockdep fixes, updating the device locks

- remove obsolete fallback logic for block group profile adjustments
  when switching from 1 to more devices, causing allocation of unwanted
  block groups

Other:

- cleanups, refactoring, simplifications
  - conversions from struct inode to struct btrfs_inode in internal
    functions
  - removal of unused struct members

----------------------------------------------------------------
The following changes since commit 92ed301919932f777713b9172e525674157e983d:

  Linux 5.8-rc7 (2020-07-26 14:14:06 -0700)

are available in the Git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-5.9-tag

for you to fetch changes up to 5e548b32018d96c377fda4bdac2bf511a448ca67:

  btrfs: do not set the full sync flag on the inode during page release (2020-07-27 12:55:48 +0200)

----------------------------------------------------------------
Anand Jain (3):
      btrfs: let btrfs_return_cluster_to_free_space() return void
      btrfs: use helper btrfs_get_block_group
      btrfs: don't traverse into the seed devices in show_devname

David Sterba (18):
      btrfs: scrub: remove kmap/kunmap of pages
      btrfs: scrub: unify naming of page address variables
      btrfs: scrub: simplify superblock checksum calculation
      btrfs: scrub: remove temporary csum array in scrub_checksum_super
      btrfs: scrub: clean up temporary page variables in scrub_checksum_super
      btrfs: scrub: simplify data block checksum calculation
      btrfs: scrub: clean up temporary page variables in scrub_checksum_data
      btrfs: scrub: simplify tree block checksum calculation
      btrfs: scrub: clean up temporary page variables in scrub_checksum_tree_block
      btrfs: add little-endian optimized key helpers
      btrfs: don't use UAPI types for fiemap callback
      btrfs: remove unused btrfs_root::defrag_trans_start
      btrfs: start deprecation of mount option inode_cache
      btrfs: allow use of global block reserve for balance item deletion
      btrfs: remove deprecated mount option alloc_start
      btrfs: remove deprecated mount option subvolrootid
      btrfs: prefetch chunk tree leaves at mount
      btrfs: add missing check for nocow and compression inode flags

Denis Efremov (1):
      btrfs: tests: remove if duplicate in __check_free_space_extents()

Filipe Manana (13):
      btrfs: remove no longer necessary chunk mutex locking cases
      btrfs: remove the start argument from btrfs_free_reserved_data_space_noquota()
      btrfs: use btrfs_alloc_data_chunk_ondemand() when allocating space for relocation
      btrfs: remove no longer used log_list member of struct btrfs_ordered_extent
      btrfs: remove no longer used trans_list member of struct btrfs_ordered_extent
      btrfs: only commit the delayed inode when doing a full fsync
      btrfs: only commit delayed items at fsync if we are logging a directory
      btrfs: stop incremening log_batch for the log root tree when syncing log
      btrfs: remove no longer needed use of log_writers for the log root tree
      btrfs: reduce contention on log trees when logging checksums
      btrfs: fix race between page release and a fast fsync
      btrfs: release old extent maps during page release
      btrfs: do not set the full sync flag on the inode during page release

Johannes Thumshirn (7):
      btrfs: get mapping tree directly from fsinfo in find_first_block_group
      btrfs: factor out reading of bg from find_frist_block_group
      btrfs: use free_root_extent_buffer to free root
      btrfs: pass checksum type via BTRFS_IOC_FS_INFO ioctl
      btrfs: add filesystem generation to FS_INFO ioctl
      btrfs: add metadata_uuid to FS_INFO ioctl
      btrfs: open-code remount flag setting in btrfs_remount

Josef Bacik (10):
      btrfs: convert block group refcount to refcount_t
      btrfs: don't WARN if we abort a transaction with EROFS
      btrfs: document special case error codes for fs errors
      btrfs: return EROFS for BTRFS_FS_STATE_ERROR cases
      btrfs: sysfs: use NOFS for device creation
      btrfs: open device without device_list_mutex
      btrfs: move the chunk_mutex in btrfs_read_chunk_tree
      btrfs: fix lockdep splat from btrfs_dump_space_info
      btrfs: don't adjust bg flags and use default allocation profiles
      btrfs: if we're restriping, use the target restripe profile

Liao Pingfang (1):
      btrfs: check-integrity: remove unnecessary failure messages during memory allocation

Marcos Paulo de Souza (1):
      btrfs: add multi-statement protection to btrfs_set/clear_and_info macros

Nikolay Borisov (74):
      btrfs: don't balance btree inode pages from buffered write path
      btrfs: read stripe len directly in btrfs_rmap_block
      btrfs: simplify checks when adding excluded ranges
      btrfs: make __btrfs_add_ordered_extent take struct btrfs_inode
      btrfs: make get_extent_allocation_hint take btrfs_inode
      btrfs: make btrfs_lookup_ordered_extent take btrfs_inode
      btrfs: make btrfs_reloc_clone_csums take btrfs_inode
      btrfs: make create_io_em take btrfs_inode
      btrfs: make extent_clear_unlock_delalloc take btrfs_inode
      btrfs: make btrfs_csum_one_bio takae btrfs_inode
      btrfs: make __btrfs_drop_extents take btrfs_inode
      btrfs: remove hole check in prealloc_file_extent_cluster
      btrfs: perform data management operations outside of inode lock
      btrfs: use for loop in prealloc_file_extent_cluster
      btrfs: tracepoints: fix btrfs_trigger_flush symbolic string for flags
      btrfs: tracepoints: fix extent type symbolic name print
      btrfs: tracepoints: move FLUSH_ACTIONS define
      btrfs: tracepoints: fix qgroup reservation type printing
      btrfs: tracepoints: switch extent_io_tree_owner to using EM macro
      btrfs: tracepoints: convert flush states to using EM macros
      btrfs: make qgroup_free_reserved_data take btrfs_inode
      btrfs: make __btrfs_qgroup_release_data take btrfs_inode
      btrfs: make btrfs_qgroup_free_data take btrfs_inode
      btrfs: make cow_file_range_inline take btrfs_inode
      btrfs: make btrfs_add_ordered_extent take btrfs_inode
      btrfs: make cow_file_range take btrfs_inode
      btrfs: make btrfs_add_ordered_extent_compress take btrfs_inode
      btrfs: make btrfs_submit_compressed_write take btrfs_inode
      btrfs: make submit_compressed_extents take btrfs_inode
      btrfs: make btrfs_qgroup_release_data take btrfs_inode
      btrfs: make insert_reserved_file_extent take btrfs_inode
      btrfs: make fallback_to_cow take btrfs_inode
      btrfs: make run_delalloc_nocow take btrfs_inode
      btrfs: make cow_file_range_async take btrfs_inode
      btrfs: make btrfs_dec_test_first_ordered_pending take btrfs_inode
      btrfs: make __endio_write_update_ordered take btrfs_inode
      btrfs: make btrfs_cleanup_ordered_extents take btrfs_inode
      btrfs: make inode_can_compress take btrfs_inode
      btrfs: make inode_need_compress take btrfs_inode
      btrfs: make need_force_cow take btrfs_inode
      btrfs: make btrfs_run_delalloc_range take btrfs_inode
      btrfs: make btrfs_add_ordered_extent_dio take btrfs_inode
      btrfs: make btrfs_create_dio_extent take btrfs_inode
      btrfs: make btrfs_new_extent_direct take btrfs_inode
      btrfs: make __extent_writepage_io take btrfs_inode
      btrfs: make writepage_delalloc take btrfs_inode
      btrfs: make btrfs_set_extent_delalloc take btrfs_inode
      btrfs: make btrfs_dirty_pages take btrfs_inode
      btrfs: make btrfs_qgroup_reserve_data take btrfs_inode
      btrfs: make btrfs_free_reserved_data_space_noquota take btrfs_fs_info
      btrfs: make btrfs_free_reserved_data_space take btrfs_inode
      btrfs: make btrfs_delalloc_release_space take btrfs_inode
      btrfs: make btrfs_check_data_free_space take btrfs_inode
      btrfs: make btrfs_delalloc_reserve_space take btrfs_inode
      btrfs: remove BTRFS_I calls in btrfs_writepage_fixup_worker
      btrfs: make prealloc_file_extent_cluster take btrfs_inode
      btrfs: make btrfs_set_inode_last_trans take btrfs_inode
      btrfs: make btrfs_qgroup_check_reserved_leak take btrfs_inode
      btrfs: make get_state_failrec return failrec directly
      btrfs: streamline btrfs_get_io_failure_record logic
      btrfs: record btrfs_device directly in btrfs_io_bio
      btrfs: don't check for btrfs_device::bdev in btrfs_end_bio
      btrfs: increment device corruption error in case of checksum error
      btrfs: remove needless ASSERT check of orig_bio in end_compressed_bio_read
      btrfs: increment corrupt device counter during compressed read
      btrfs: sysfs: add bdi link to the fsid directory
      btrfs: always initialize btrfs_bio::tgtdev_map/raid_map pointers
      btrfs: raid56: remove redundant device check in rbio_add_io_page
      btrfs: raid56: assign bio in while() when using bio_list_pop
      btrfs: raid56: use in_range where applicable
      btrfs: raid56: don't opencode swap() in __raid_recover_end_io
      btrfs: remove fail label in check_compressed_csum
      btrfs: raid56: remove out label in __raid56_parity_recover
      btrfs: remove done label in writepage_delalloc

Qu Wenruo (22):
      btrfs: introduce "rescue=" mount option
      btrfs: inode: refactor the parameters of insert_reserved_file_extent()
      btrfs: inode: move qgroup reserved space release to the callers of insert_reserved_file_extent()
      btrfs: file: reserve qgroup space after the hole punch range is locked
      btrfs: change timing for qgroup reserved space for ordered extents to fix reserved space leak
      btrfs: qgroup: catch reserved space leaks at unmount time
      btrfs: allow btrfs_truncate_block() to fallback to nocow for data space reservation
      btrfs: add comments for btrfs_check_can_nocow() and can_nocow_extent()
      btrfs: refactor btrfs_check_can_nocow() into two variants
      btrfs: use __u16 for the return value of btrfs_qgroup_level()
      btrfs: qgroup: export qgroups in sysfs
      btrfs: don't allocate anonymous block device for user invisible roots
      btrfs: free anon block device right after subvolume deletion
      btrfs: preallocate anon block device at first phase of snapshot creation
      btrfs: qgroup: allow to unreserve range without releasing other ranges
      btrfs: qgroup: try to flush qgroup space when we get -EDQUOT
      btrfs: qgroup: remove ASYNC_COMMIT mechanism in favor of reserve retry-after-EDQUOT
      btrfs: qgroup: free per-trans reserved space when a subvolume gets dropped
      btrfs: relocation: allow signal to cancel balance
      btrfs: avoid possible signal interruption of btrfs_drop_snapshot() on relocation tree
      btrfs: relocation: review the call sites which can be interrupted by signal
      btrfs: add comments for btrfs_reserve_flush_enum

Tom Rix (1):
      btrfs: ref-verify: fix memory leak in add_block_entry

 fs/btrfs/block-group.c                 | 211 +++++--------
 fs/btrfs/block-group.h                 |   3 +-
 fs/btrfs/btrfs_inode.h                 |  11 +
 fs/btrfs/check-integrity.c             |  27 +-
 fs/btrfs/compression.c                 |  30 +-
 fs/btrfs/compression.h                 |   4 +-
 fs/btrfs/ctree.c                       |  17 ++
 fs/btrfs/ctree.h                       | 127 ++++++--
 fs/btrfs/delalloc-space.c              |  36 +--
 fs/btrfs/delalloc-space.h              |  10 +-
 fs/btrfs/disk-io.c                     |  92 +++++-
 fs/btrfs/disk-io.h                     |   2 +
 fs/btrfs/extent-io-tree.h              |   5 +-
 fs/btrfs/extent-tree.c                 |  17 +-
 fs/btrfs/extent_io.c                   | 241 ++++++++-------
 fs/btrfs/extent_io.h                   |   4 +-
 fs/btrfs/file-item.c                   |   4 +-
 fs/btrfs/file.c                        | 143 +++++----
 fs/btrfs/free-space-cache.c            |  23 +-
 fs/btrfs/free-space-cache.h            |   2 +-
 fs/btrfs/inode-map.c                   |   3 +-
 fs/btrfs/inode.c                       | 530 ++++++++++++++++++---------------
 fs/btrfs/ioctl.c                       |  86 +++++-
 fs/btrfs/ordered-data.c                |  63 ++--
 fs/btrfs/ordered-data.h                |  19 +-
 fs/btrfs/qgroup.c                      | 359 ++++++++++++++++------
 fs/btrfs/qgroup.h                      |  24 +-
 fs/btrfs/raid56.c                      |  65 ++--
 fs/btrfs/ref-verify.c                  |   2 +
 fs/btrfs/reflink.c                     |  26 +-
 fs/btrfs/relocation.c                  |  71 ++---
 fs/btrfs/scrub.c                       | 153 +++-------
 fs/btrfs/space-info.c                  |   2 +-
 fs/btrfs/super.c                       | 144 ++++++---
 fs/btrfs/sysfs.c                       | 163 +++++++++-
 fs/btrfs/sysfs.h                       |   7 +
 fs/btrfs/tests/free-space-tree-tests.c |   2 -
 fs/btrfs/tests/inode-tests.c           |  14 +-
 fs/btrfs/transaction.c                 |   8 +-
 fs/btrfs/transaction.h                 |  28 +-
 fs/btrfs/tree-defrag.c                 |   5 +-
 fs/btrfs/tree-log.c                    |  50 ++--
 fs/btrfs/volumes.c                     | 133 ++++++---
 fs/btrfs/volumes.h                     |   2 +-
 include/trace/events/btrfs.h           | 137 +++++----
 include/uapi/linux/btrfs.h             |  21 +-
 include/uapi/linux/btrfs_tree.h        |   4 +-
 47 files changed, 1909 insertions(+), 1221 deletions(-)

Powered by blists - more mailing lists