May 26 04:02:34 mr-fox kernel: BUG: Bad page state in process kcompactd0  pfn:15dd75
May 26 04:02:34 mr-fox kernel: page:000000003a37a5ec refcount:0 mapcount:0 mapping:00000000f0bb3562 index:0x4cd994a1 pfn:0x15dd75
May 26 04:02:34 mr-fox kernel: aops:0xffffffffa5c2a700 ino:1
May 26 04:02:34 mr-fox kernel: flags: 0x5fff0000000000c(referenced|uptodate|node=0|zone=2|lastcpupid=0x1fff)
May 26 04:02:34 mr-fox kernel: page_type: 0xffffffff()
May 26 04:02:34 mr-fox kernel: raw: 05fff0000000000c dead000000000100 dead000000000122 ffff988ac61fdd70
May 26 04:02:34 mr-fox kernel: raw: 000000004cd994a1 0000000000000000 00000000ffffffff 0000000000000000
May 26 04:02:34 mr-fox kernel: page dumped because: non-NULL mapping
May 26 04:02:34 mr-fox kernel: CPU: 12 PID: 180 Comm: kcompactd0 Tainted: G                T  6.8.11 #13
May 26 04:02:34 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 04:02:34 mr-fox kernel: Call Trace:
May 26 04:02:34 mr-fox kernel:  <TASK>
May 26 04:02:34 mr-fox kernel:  dump_stack_lvl+0x53/0x70
May 26 04:02:34 mr-fox kernel:  bad_page+0x67/0x100
May 26 04:02:34 mr-fox kernel:  free_unref_page_prepare+0x28e/0x300
May 26 04:02:34 mr-fox kernel:  free_unref_page+0x2f/0x170
May 26 04:02:34 mr-fox kernel:  btrfs_release_extent_buffer_pages+0x4a/0x70
May 26 04:02:34 mr-fox kernel:  release_extent_buffer+0x48/0xc0
May 26 04:02:34 mr-fox kernel:  btree_release_folio+0x25/0x40
May 26 04:02:34 mr-fox kernel:  btree_migrate_folio+0x31/0x70
May 26 04:02:34 mr-fox kernel:  move_to_new_folio+0x5e/0x160
May 26 04:02:34 mr-fox kernel:  migrate_pages_batch+0x84f/0xb90
May 26 04:02:34 mr-fox kernel:  ? split_map_pages+0x190/0x190
May 26 04:02:34 mr-fox kernel:  ? pick_next_task_fair+0x1c5/0x580
May 26 04:02:34 mr-fox kernel:  migrate_pages+0x9d4/0xc90
May 26 04:02:34 mr-fox kernel:  ? isolate_freepages_block+0x440/0x440
May 26 04:02:34 mr-fox kernel:  ? split_map_pages+0x190/0x190
May 26 04:02:34 mr-fox kernel:  ? __mod_memcg_lruvec_state+0xcd/0x190
May 26 04:02:34 mr-fox kernel:  compact_zone+0xb6b/0xe20
May 26 04:02:34 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 04:02:34 mr-fox kernel:  ? __schedule+0x3b4/0x17d0
May 26 04:02:34 mr-fox kernel:  proactive_compact_node+0x8a/0x100
May 26 04:02:34 mr-fox kernel:  kcompactd+0x1e7/0x410
May 26 04:02:34 mr-fox kernel:  ? swake_up_locked+0x60/0x60
May 26 04:02:34 mr-fox kernel:  ? kcompactd_do_work+0x2e0/0x2e0
May 26 04:02:34 mr-fox kernel:  kthread+0xcb/0xf0
May 26 04:02:34 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 04:02:34 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 04:02:34 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 04:02:34 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 04:02:34 mr-fox kernel:  </TASK>
May 26 04:02:34 mr-fox kernel: Disabling lock debugging due to kernel taint
May 26 04:03:00 mr-fox crond[30536]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:03:00 mr-fox CROND[30545]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:03:00 mr-fox crond[30537]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:03:00 mr-fox CROND[30550]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:03:00 mr-fox crond[30540]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:03:00 mr-fox crond[30544]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 04:03:00 mr-fox CROND[30556]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 04:03:00 mr-fox crond[30541]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:03:00 mr-fox CROND[30554]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 04:03:00 mr-fox CROND[30572]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:03:00 mr-fox CROND[30544]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 04:03:00 mr-fox CROND[30544]: pam_unix(crond:session): session closed for user tinderbox
May 26 04:03:00 mr-fox CROND[30541]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:03:00 mr-fox CROND[30541]: pam_unix(crond:session): session closed for user root
May 26 04:03:01 mr-fox CROND[30540]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 04:03:01 mr-fox CROND[30540]: pam_unix(crond:session): session closed for user root
May 26 04:03:02 mr-fox CROND[13572]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:03:02 mr-fox CROND[13572]: pam_unix(crond:session): session closed for user root
May 26 04:03:03 mr-fox CROND[13573]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:03:03 mr-fox CROND[13573]: pam_unix(crond:session): session closed for user root
May 26 04:04:00 mr-fox crond[15741]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:04:00 mr-fox crond[15739]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:04:00 mr-fox crond[15742]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:04:00 mr-fox crond[15743]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 04:04:00 mr-fox CROND[15747]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:04:00 mr-fox CROND[15748]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 04:04:00 mr-fox crond[15740]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:04:00 mr-fox CROND[15752]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 04:04:00 mr-fox CROND[15750]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:04:00 mr-fox CROND[15754]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:04:00 mr-fox CROND[15743]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 04:04:00 mr-fox CROND[15743]: pam_unix(crond:session): session closed for user tinderbox
May 26 04:04:00 mr-fox CROND[15742]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:04:00 mr-fox CROND[15742]: pam_unix(crond:session): session closed for user root
May 26 04:04:00 mr-fox CROND[15741]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 04:04:00 mr-fox CROND[15741]: pam_unix(crond:session): session closed for user root
May 26 04:04:02 mr-fox CROND[30536]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:04:02 mr-fox CROND[30536]: pam_unix(crond:session): session closed for user root
May 26 04:04:02 mr-fox CROND[30537]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:04:02 mr-fox CROND[30537]: pam_unix(crond:session): session closed for user root
May 26 04:05:00 mr-fox crond[14940]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:05:00 mr-fox crond[14937]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 04:05:00 mr-fox CROND[14950]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:05:00 mr-fox crond[14938]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:05:00 mr-fox crond[14944]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 04:05:00 mr-fox CROND[14952]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 04:05:00 mr-fox CROND[14957]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 04:05:00 mr-fox crond[14941]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:05:00 mr-fox CROND[14956]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:05:00 mr-fox CROND[14959]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 04:05:00 mr-fox crond[14949]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 04:05:00 mr-fox CROND[14966]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 04:05:01 mr-fox crond[14943]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 04:05:01 mr-fox CROND[14971]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 04:05:01 mr-fox crond[14942]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:05:01 mr-fox CROND[14979]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:05:01 mr-fox CROND[14942]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:05:01 mr-fox CROND[14942]: pam_unix(crond:session): session closed for user root
May 26 04:05:01 mr-fox CROND[14949]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 04:05:01 mr-fox CROND[14949]: pam_unix(crond:session): session closed for user tinderbox
May 26 04:05:01 mr-fox CROND[15739]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:05:01 mr-fox CROND[15739]: pam_unix(crond:session): session closed for user root
May 26 04:05:01 mr-fox CROND[14943]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 04:05:01 mr-fox CROND[14943]: pam_unix(crond:session): session closed for user tinderbox
May 26 04:05:01 mr-fox CROND[14941]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 04:05:01 mr-fox CROND[14941]: pam_unix(crond:session): session closed for user root
May 26 04:05:01 mr-fox CROND[14937]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 04:05:01 mr-fox CROND[14937]: pam_unix(crond:session): session closed for user torproject
May 26 04:05:02 mr-fox CROND[15740]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:05:02 mr-fox CROND[15740]: pam_unix(crond:session): session closed for user root
May 26 04:05:32 mr-fox CROND[14944]: (tinderbox) CMDEND (sleep 10; /opt/tb/bin/index.sh)
May 26 04:05:32 mr-fox CROND[14944]: pam_unix(crond:session): session closed for user tinderbox
May 26 04:06:00 mr-fox crond[13989]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:06:00 mr-fox crond[13990]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:06:00 mr-fox crond[13992]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:06:00 mr-fox CROND[13999]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:06:00 mr-fox crond[13996]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 04:06:00 mr-fox CROND[14002]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:06:00 mr-fox crond[13994]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:06:00 mr-fox CROND[14001]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 04:06:00 mr-fox CROND[14003]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 04:06:00 mr-fox CROND[14006]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:06:00 mr-fox CROND[13996]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 04:06:00 mr-fox CROND[13996]: pam_unix(crond:session): session closed for user tinderbox
May 26 04:06:00 mr-fox CROND[13994]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:06:00 mr-fox CROND[13994]: pam_unix(crond:session): session closed for user root
May 26 04:06:01 mr-fox CROND[13992]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 04:06:01 mr-fox CROND[13992]: pam_unix(crond:session): session closed for user root
May 26 04:06:02 mr-fox CROND[14938]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:06:02 mr-fox CROND[14938]: pam_unix(crond:session): session closed for user root
May 26 04:06:03 mr-fox CROND[14940]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:06:03 mr-fox CROND[14940]: pam_unix(crond:session): session closed for user root
May 26 04:07:00 mr-fox crond[10221]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:07:00 mr-fox crond[10225]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 04:07:00 mr-fox crond[10219]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:07:00 mr-fox crond[10222]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:07:00 mr-fox crond[10223]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:07:00 mr-fox CROND[10231]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:07:00 mr-fox CROND[10232]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 04:07:00 mr-fox CROND[10234]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 04:07:00 mr-fox CROND[10236]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:07:00 mr-fox CROND[10237]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:07:00 mr-fox CROND[10225]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 04:07:00 mr-fox CROND[10225]: pam_unix(crond:session): session closed for user tinderbox
May 26 04:07:00 mr-fox CROND[10223]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:07:00 mr-fox CROND[10223]: pam_unix(crond:session): session closed for user root
May 26 04:07:00 mr-fox CROND[10222]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 04:07:00 mr-fox CROND[10222]: pam_unix(crond:session): session closed for user root
May 26 04:07:02 mr-fox CROND[13989]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:07:02 mr-fox CROND[13989]: pam_unix(crond:session): session closed for user root
May 26 04:07:02 mr-fox CROND[13990]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:07:02 mr-fox CROND[13990]: pam_unix(crond:session): session closed for user root
May 26 04:08:00 mr-fox crond[6387]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:08:00 mr-fox crond[6391]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:08:00 mr-fox crond[6392]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 04:08:00 mr-fox CROND[6397]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:08:00 mr-fox crond[6386]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:08:00 mr-fox crond[6389]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:08:00 mr-fox CROND[6405]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:08:00 mr-fox CROND[6407]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:08:00 mr-fox CROND[6409]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 04:08:00 mr-fox CROND[6406]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 04:08:00 mr-fox CROND[6392]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 04:08:00 mr-fox CROND[6392]: pam_unix(crond:session): session closed for user tinderbox
May 26 04:08:00 mr-fox CROND[6391]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:08:00 mr-fox CROND[6391]: pam_unix(crond:session): session closed for user root
May 26 04:08:01 mr-fox CROND[10219]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:08:01 mr-fox CROND[10219]: pam_unix(crond:session): session closed for user root
May 26 04:08:02 mr-fox CROND[6389]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 04:08:02 mr-fox CROND[6389]: pam_unix(crond:session): session closed for user root
May 26 04:08:02 mr-fox CROND[10221]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:08:02 mr-fox CROND[10221]: pam_unix(crond:session): session closed for user root
May 26 04:09:00 mr-fox crond[23983]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:09:00 mr-fox crond[23982]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:09:00 mr-fox crond[23984]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:09:00 mr-fox CROND[23990]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:09:00 mr-fox CROND[23991]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:09:00 mr-fox CROND[23992]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 04:09:00 mr-fox crond[23987]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 04:09:00 mr-fox crond[23986]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:09:00 mr-fox CROND[23999]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:09:00 mr-fox CROND[23996]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 04:09:00 mr-fox CROND[23986]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 04:09:00 mr-fox CROND[23986]: pam_unix(crond:session): session closed for user root
May 26 04:09:00 mr-fox CROND[23987]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 04:09:00 mr-fox CROND[23987]: pam_unix(crond:session): session closed for user tinderbox
May 26 04:09:00 mr-fox CROND[23984]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 04:09:00 mr-fox CROND[23984]: pam_unix(crond:session): session closed for user root
May 26 04:09:04 mr-fox CROND[6386]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 04:09:04 mr-fox CROND[6386]: pam_unix(crond:session): session closed for user root
May 26 04:09:05 mr-fox CROND[6387]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:09:05 mr-fox CROND[6387]: pam_unix(crond:session): session closed for user root
May 26 04:10:00 mr-fox crond[6295]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 04:10:00 mr-fox crond[6298]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:10:00 mr-fox crond[6299]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:10:00 mr-fox CROND[6308]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 04:10:00 mr-fox crond[6296]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 04:10:00 mr-fox CROND[6310]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 04:10:00 mr-fox crond[6302]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
--
May 26 06:02:44 mr-fox kernel: RIP: 0010:xas_start+0x3d/0x120
May 26 06:02:44 mr-fox kernel: Code: 0f 84 81 00 00 00 48 81 fd 05 c0 ff ff 76 06 48 83 f8 02 74 5d 48 8b 03 48 8b 73 08 48 8b 40 08 48 89 c2 83 e2 03 48 83 fa 02 <75> 08 48 3d 00 10 00 00 77 20 48 85 f6 75 31 48 c7 43 18 00 00 00
May 26 06:02:44 mr-fox kernel: RSP: 0018:ffffa401077e3930 EFLAGS: 00000246
May 26 06:02:44 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:02:44 mr-fox kernel: RDX: 0000000000000002 RSI: 000000004cd994a1 RDI: ffffa401077e3970
May 26 06:02:44 mr-fox kernel: RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
May 26 06:02:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:02:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:02:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:02:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:02:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 000000012a228000 CR4: 0000000000f50ef0
May 26 06:02:44 mr-fox kernel: PKRU: 55555554
May 26 06:02:44 mr-fox kernel: Call Trace:
May 26 06:02:44 mr-fox kernel:  <IRQ>
May 26 06:02:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:02:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:02:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:02:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:02:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:02:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:02:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:02:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:02:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:02:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:02:44 mr-fox kernel:  </IRQ>
May 26 06:02:44 mr-fox kernel:  <TASK>
May 26 06:02:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:02:44 mr-fox kernel:  ? xas_start+0x3d/0x120
May 26 06:02:44 mr-fox kernel:  xas_load+0xe/0x60
May 26 06:02:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:02:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:02:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:02:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:02:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:02:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:02:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:02:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:02:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:02:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:02:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:02:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:02:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:02:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:02:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:02:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:02:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:02:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:02:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:02:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:02:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:02:44 mr-fox kernel:  </TASK>
May 26 06:02:47 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 15471 jiffies s: 491905 root: 0x2/.
May 26 06:02:47 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:02:47 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:02:47 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:02:47 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:02:47 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:02:47 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:02:47 mr-fox kernel: RIP: 0010:filemap_get_entry+0x6d/0x160
May 26 06:02:47 mr-fox kernel: Code: 00 00 48 c7 44 24 30 00 00 00 00 48 c7 44 24 38 00 00 00 00 48 c7 44 24 20 03 00 00 00 48 8d 7c 24 08 e8 56 70 78 00 48 89 c3 <48> 3d 02 04 00 00 74 e2 48 3d 06 04 00 00 74 da 48 85 c0 74 5a a8
May 26 06:02:47 mr-fox kernel: RSP: 0018:ffffa401077e3968 EFLAGS: 00000246
May 26 06:02:47 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 06:02:47 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:02:47 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 06:02:47 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:02:47 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:02:47 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:02:47 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:02:47 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 000000012a228000 CR4: 0000000000f50ef0
May 26 06:02:47 mr-fox kernel: PKRU: 55555554
May 26 06:02:47 mr-fox kernel: Call Trace:
May 26 06:02:47 mr-fox kernel:  <NMI>
May 26 06:02:47 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:02:47 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:02:47 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:02:47 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:02:47 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:02:47 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:02:47 mr-fox kernel:  ? filemap_get_entry+0x6d/0x160
May 26 06:02:47 mr-fox kernel:  ? filemap_get_entry+0x6d/0x160
May 26 06:02:47 mr-fox kernel:  ? filemap_get_entry+0x6d/0x160
May 26 06:02:47 mr-fox kernel:  </NMI>
May 26 06:02:47 mr-fox kernel:  <TASK>
May 26 06:02:47 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:02:47 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:02:47 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:02:47 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:02:47 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:02:47 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:02:47 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:02:47 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:02:47 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:02:47 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:02:47 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:02:47 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:02:47 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:02:47 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:02:47 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:02:47 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:02:47 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:02:47 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:02:47 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:02:47 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:02:47 mr-fox kernel:  </TASK>
May 26 06:03:00 mr-fox crond[30376]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:03:00 mr-fox crond[30378]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:03:00 mr-fox crond[30375]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:03:00 mr-fox CROND[30389]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:03:00 mr-fox CROND[30390]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:03:00 mr-fox CROND[30392]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:03:00 mr-fox crond[30380]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:03:00 mr-fox crond[30377]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:03:00 mr-fox CROND[30401]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:03:00 mr-fox CROND[30403]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:03:00 mr-fox CROND[30378]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:03:00 mr-fox CROND[30378]: pam_unix(crond:session): session closed for user root
May 26 06:03:00 mr-fox CROND[30380]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:03:00 mr-fox CROND[30380]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:03:01 mr-fox CROND[30377]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:03:01 mr-fox CROND[30377]: pam_unix(crond:session): session closed for user root
May 26 06:03:01 mr-fox CROND[2083]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:03:01 mr-fox CROND[2083]: pam_unix(crond:session): session closed for user root
May 26 06:04:00 mr-fox crond[24809]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:04:00 mr-fox crond[24811]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:04:00 mr-fox crond[24812]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:04:00 mr-fox crond[24813]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:04:00 mr-fox crond[24810]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:04:00 mr-fox CROND[24817]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:04:00 mr-fox CROND[24818]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:04:00 mr-fox CROND[24820]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:04:00 mr-fox CROND[24821]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:04:00 mr-fox CROND[24819]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:04:00 mr-fox CROND[24813]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:04:00 mr-fox CROND[24813]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:04:00 mr-fox CROND[24812]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:04:00 mr-fox CROND[24812]: pam_unix(crond:session): session closed for user root
May 26 06:04:00 mr-fox CROND[24811]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:04:00 mr-fox CROND[24811]: pam_unix(crond:session): session closed for user root
May 26 06:04:02 mr-fox CROND[30376]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:04:02 mr-fox CROND[30376]: pam_unix(crond:session): session closed for user root
May 26 06:05:00 mr-fox crond[26845]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 06:05:00 mr-fox crond[26846]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:05:00 mr-fox crond[26848]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:05:00 mr-fox crond[26852]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:05:00 mr-fox crond[26849]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:05:00 mr-fox CROND[26856]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:05:00 mr-fox crond[26851]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:05:00 mr-fox CROND[26857]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:05:00 mr-fox CROND[26859]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:05:00 mr-fox CROND[26860]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 06:05:00 mr-fox CROND[26861]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:05:00 mr-fox CROND[26862]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:05:00 mr-fox crond[26853]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:05:00 mr-fox crond[26850]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:05:00 mr-fox CROND[26865]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:05:00 mr-fox CROND[26866]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:05:00 mr-fox CROND[26853]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:05:00 mr-fox CROND[26853]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:05:00 mr-fox CROND[26850]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:05:00 mr-fox CROND[26850]: pam_unix(crond:session): session closed for user root
May 26 06:05:00 mr-fox CROND[26851]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:05:00 mr-fox CROND[26851]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:05:00 mr-fox CROND[26849]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:05:00 mr-fox CROND[26849]: pam_unix(crond:session): session closed for user root
May 26 06:05:00 mr-fox CROND[26845]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:05:00 mr-fox CROND[26845]: pam_unix(crond:session): session closed for user torproject
May 26 06:05:01 mr-fox CROND[24810]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:05:01 mr-fox CROND[24810]: pam_unix(crond:session): session closed for user root
May 26 06:05:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:05:44 mr-fox kernel: rcu: \x0921-....: (60003 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=26215
May 26 06:05:44 mr-fox kernel: rcu: \x09(t=60004 jiffies g=8794409 q=6228675 ncpus=32)
May 26 06:05:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:05:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:05:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:05:44 mr-fox kernel: RIP: 0010:xas_descend+0x13/0xd0
May 26 06:05:44 mr-fox kernel: Code: 87 04 00 e9 a3 87 04 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e <48> 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48
May 26 06:05:44 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000282
May 26 06:05:44 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: ffffa401077e3970 RCX: 000000000000000c
May 26 06:05:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988b1fdbb478 RDI: ffffa401077e3970
May 26 06:05:44 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 06:05:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 06:05:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:05:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:05:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:05:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:05:44 mr-fox kernel: PKRU: 55555554
May 26 06:05:44 mr-fox kernel: Call Trace:
May 26 06:05:44 mr-fox kernel:  <IRQ>
May 26 06:05:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:05:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:05:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:05:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:05:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:05:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:05:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:05:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:05:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:05:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:05:44 mr-fox kernel:  </IRQ>
May 26 06:05:44 mr-fox kernel:  <TASK>
May 26 06:05:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:05:44 mr-fox kernel:  ? xas_descend+0x13/0xd0
May 26 06:05:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:05:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:05:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:05:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:05:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:05:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:05:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:05:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:05:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:05:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:05:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:05:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:05:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:05:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:05:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:05:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:05:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:05:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:05:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:05:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:05:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:05:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:05:44 mr-fox kernel:  </TASK>
May 26 06:05:49 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 61039 jiffies s: 491905 root: 0x2/.
May 26 06:05:49 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:05:49 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:05:49 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:05:49 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:05:49 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:05:49 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:05:49 mr-fox kernel: RIP: 0010:tcp_ack+0x69e/0x14b0
May 26 06:05:49 mr-fox kernel: Code: c7 43 60 00 00 00 00 48 8b 74 24 38 48 89 df 48 c7 43 58 00 00 00 00 e8 30 c4 0d 00 41 8b 84 24 40 01 00 00 2b 83 d0 00 00 00 <41> 89 84 24 40 01 00 00 8b 8b b8 00 00 00 48 8b b3 c0 00 00 00 48
May 26 06:05:49 mr-fox kernel: RSP: 0018:ffffa401005f0960 EFLAGS: 00000246
May 26 06:05:49 mr-fox kernel: RAX: 0000000000000000 RBX: ffff98991a592300 RCX: 0000000000000000
May 26 06:05:49 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:05:49 mr-fox kernel: RBP: ffffa401005f09f0 R08: 0000000000000000 R09: 0000000000000000
May 26 06:05:49 mr-fox kernel: R10: 0000000000000000 R11: 000000001a179271 R12: ffff988d0c615c80
May 26 06:05:49 mr-fox kernel: R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000
May 26 06:05:49 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:05:49 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:05:49 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:05:49 mr-fox kernel: PKRU: 55555554
May 26 06:05:49 mr-fox kernel: Call Trace:
May 26 06:05:49 mr-fox kernel:  <NMI>
May 26 06:05:49 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:05:49 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:05:49 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:05:49 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:05:49 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:05:49 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:05:49 mr-fox kernel:  ? tcp_ack+0x69e/0x14b0
May 26 06:05:49 mr-fox kernel:  ? tcp_ack+0x69e/0x14b0
May 26 06:05:49 mr-fox kernel:  ? tcp_ack+0x69e/0x14b0
May 26 06:05:49 mr-fox kernel:  </NMI>
May 26 06:05:49 mr-fox kernel:  <IRQ>
May 26 06:05:49 mr-fox kernel:  tcp_rcv_established+0x146/0x6c0
May 26 06:05:49 mr-fox kernel:  ? sk_filter_trim_cap+0x40/0x220
May 26 06:05:49 mr-fox kernel:  tcp_v4_do_rcv+0x153/0x240
May 26 06:05:49 mr-fox kernel:  tcp_v4_rcv+0xe00/0xea0
May 26 06:05:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:05:49 mr-fox kernel:  ip_protocol_deliver_rcu+0x32/0x180
May 26 06:05:49 mr-fox kernel:  ip_local_deliver_finish+0x74/0xa0
May 26 06:05:49 mr-fox kernel:  ip_sublist_rcv_finish+0x7f/0x90
May 26 06:05:49 mr-fox kernel:  ip_sublist_rcv+0x176/0x1c0
May 26 06:05:49 mr-fox kernel:  ? ip_sublist_rcv+0x1c0/0x1c0
May 26 06:05:49 mr-fox kernel:  ip_list_rcv+0x138/0x170
May 26 06:05:49 mr-fox kernel:  __netif_receive_skb_list_core+0x293/0x2d0
May 26 06:05:49 mr-fox kernel:  netif_receive_skb_list_internal+0x1db/0x320
May 26 06:05:49 mr-fox kernel:  napi_gro_receive+0xcf/0x1b0
May 26 06:05:49 mr-fox kernel:  igb_poll+0x605/0x1370
May 26 06:05:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:05:49 mr-fox kernel:  ? task_tick_fair+0x85/0x470
May 26 06:05:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:05:49 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:05:49 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:05:49 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:05:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:05:49 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:05:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:05:49 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:05:49 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:05:49 mr-fox kernel:  </IRQ>
May 26 06:05:49 mr-fox kernel:  <TASK>
May 26 06:05:49 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:05:49 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 06:05:49 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 06:05:49 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 06:05:49 mr-fox kernel: RAX: ffff988d667606da RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:05:49 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:05:49 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 06:05:49 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:05:49 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:05:49 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:05:49 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:05:49 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:05:49 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:05:49 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:05:49 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:05:49 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:05:49 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:05:49 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:05:49 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:05:49 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:05:49 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:05:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:05:49 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:05:49 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:05:49 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:05:49 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:05:49 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:05:49 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:05:49 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:05:49 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:05:49 mr-fox kernel:  </TASK>
May 26 06:06:00 mr-fox crond[30958]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:06:00 mr-fox crond[30959]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:06:00 mr-fox crond[30960]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:06:00 mr-fox crond[30961]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:06:00 mr-fox CROND[30964]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:06:00 mr-fox CROND[30965]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:06:00 mr-fox crond[30956]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:06:00 mr-fox CROND[30968]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:06:00 mr-fox CROND[30969]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:06:00 mr-fox CROND[30970]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:06:00 mr-fox CROND[30961]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:06:00 mr-fox CROND[30961]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:06:00 mr-fox CROND[30960]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:06:00 mr-fox CROND[30960]: pam_unix(crond:session): session closed for user root
May 26 06:06:00 mr-fox CROND[30959]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:06:00 mr-fox CROND[30959]: pam_unix(crond:session): session closed for user root
May 26 06:06:00 mr-fox CROND[26848]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:06:00 mr-fox CROND[26848]: pam_unix(crond:session): session closed for user root
May 26 06:07:00 mr-fox CROND[30958]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:07:00 mr-fox CROND[30958]: pam_unix(crond:session): session closed for user root
May 26 06:07:00 mr-fox crond[7168]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:07:00 mr-fox crond[7170]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:07:00 mr-fox crond[7169]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:07:00 mr-fox crond[7171]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:07:00 mr-fox crond[7172]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:07:00 mr-fox CROND[7174]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:07:00 mr-fox CROND[7175]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:07:00 mr-fox CROND[7176]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:07:00 mr-fox CROND[7177]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:07:00 mr-fox CROND[7178]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:07:00 mr-fox CROND[7172]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:07:00 mr-fox CROND[7172]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:07:00 mr-fox CROND[7171]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:07:00 mr-fox CROND[7171]: pam_unix(crond:session): session closed for user root
May 26 06:07:00 mr-fox CROND[7170]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:07:00 mr-fox CROND[7170]: pam_unix(crond:session): session closed for user root
May 26 06:08:00 mr-fox crond[14400]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:08:00 mr-fox crond[14401]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:08:00 mr-fox crond[14402]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:08:00 mr-fox crond[14404]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:08:00 mr-fox CROND[14408]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:08:00 mr-fox CROND[14409]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:08:00 mr-fox crond[14405]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:08:00 mr-fox CROND[14411]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:08:00 mr-fox CROND[14413]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:08:00 mr-fox CROND[14414]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:08:00 mr-fox CROND[14405]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:08:00 mr-fox CROND[14405]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:08:00 mr-fox CROND[14404]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:08:00 mr-fox CROND[14404]: pam_unix(crond:session): session closed for user root
May 26 06:08:01 mr-fox CROND[14402]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:08:01 mr-fox CROND[14402]: pam_unix(crond:session): session closed for user root
May 26 06:08:01 mr-fox CROND[7169]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:08:01 mr-fox CROND[7169]: pam_unix(crond:session): session closed for user root
May 26 06:08:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:08:44 mr-fox kernel: rcu: \x0921-....: (105007 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=44004
May 26 06:08:44 mr-fox kernel: rcu: \x09(t=105008 jiffies g=8794409 q=8040467 ncpus=32)
May 26 06:08:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:08:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:08:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:08:44 mr-fox kernel: RIP: 0010:filemap_get_entry+0x6d/0x160
May 26 06:08:44 mr-fox kernel: Code: 00 00 48 c7 44 24 30 00 00 00 00 48 c7 44 24 38 00 00 00 00 48 c7 44 24 20 03 00 00 00 48 8d 7c 24 08 e8 56 70 78 00 48 89 c3 <48> 3d 02 04 00 00 74 e2 48 3d 06 04 00 00 74 da 48 85 c0 74 5a a8
May 26 06:08:44 mr-fox kernel: RSP: 0018:ffffa401077e3968 EFLAGS: 00000246
May 26 06:08:44 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 06:08:44 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:08:44 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 06:08:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:08:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:08:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:08:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:08:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:08:44 mr-fox kernel: PKRU: 55555554
May 26 06:08:44 mr-fox kernel: Call Trace:
May 26 06:08:44 mr-fox kernel:  <IRQ>
May 26 06:08:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:08:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:08:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:08:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:08:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:08:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:08:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:08:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:08:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:08:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:08:44 mr-fox kernel:  </IRQ>
May 26 06:08:44 mr-fox kernel:  <TASK>
May 26 06:08:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:08:44 mr-fox kernel:  ? filemap_get_entry+0x6d/0x160
May 26 06:08:44 mr-fox kernel:  ? filemap_get_entry+0x6a/0x160
May 26 06:08:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:08:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:08:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:08:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:08:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:08:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:08:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:08:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:08:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:08:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:08:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:08:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:08:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:08:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:08:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:08:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:08:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:08:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:08:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:08:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:08:44 mr-fox kernel:  </TASK>
May 26 06:08:49 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 106096 jiffies s: 491905 root: 0x2/.
May 26 06:08:49 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:08:49 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:08:49 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:08:49 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:08:49 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:08:49 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:08:49 mr-fox kernel: RIP: 0010:_raw_spin_lock_irqsave+0x13/0x40
May 26 06:08:49 mr-fox kernel: Code: 31 ff e9 3b 32 15 00 b8 01 00 00 00 31 d2 31 ff e9 2d 32 15 00 66 90 f3 0f 1e fa 53 9c 5b fa 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 0f 48 89 d8 5b 31 d2 31 f6 31 ff e9 07 32 15 00 89 c6 e8 55 03
May 26 06:08:49 mr-fox kernel: RSP: 0018:ffffa401005f0cf8 EFLAGS: 00000046
May 26 06:08:49 mr-fox kernel: RAX: 0000000000000000 RBX: 0000000000000282 RCX: 000000000000000c
May 26 06:08:49 mr-fox kernel: RDX: 0000000000000001 RSI: 00000000cec51000 RDI: ffffc400ffb51870
May 26 06:08:49 mr-fox kernel: RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
May 26 06:08:49 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000cec51
May 26 06:08:49 mr-fox kernel: R13: ffffa401005f0d68 R14: ffffa401005f0d50 R15: ffffc400ffb51870
May 26 06:08:49 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:08:49 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:08:49 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:08:49 mr-fox kernel: PKRU: 55555554
May 26 06:08:49 mr-fox kernel: Call Trace:
May 26 06:08:49 mr-fox kernel:  <NMI>
May 26 06:08:49 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:08:49 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:08:49 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:08:49 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:08:49 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:08:49 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:08:49 mr-fox kernel:  ? _raw_spin_lock_irqsave+0x13/0x40
May 26 06:08:49 mr-fox kernel:  ? _raw_spin_lock_irqsave+0x13/0x40
May 26 06:08:49 mr-fox kernel:  ? _raw_spin_lock_irqsave+0x13/0x40
May 26 06:08:49 mr-fox kernel:  </NMI>
May 26 06:08:49 mr-fox kernel:  <IRQ>
May 26 06:08:49 mr-fox kernel:  iommu_dma_free_iova+0xa1/0x220
May 26 06:08:49 mr-fox kernel:  __iommu_dma_unmap+0xe0/0x170
May 26 06:08:49 mr-fox kernel:  iommu_dma_unmap_page+0x43/0xa0
May 26 06:08:49 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 06:08:49 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 06:08:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:08:49 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 06:08:49 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:08:49 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:08:49 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:08:49 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:08:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:08:49 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:08:49 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:08:49 mr-fox kernel:  </IRQ>
May 26 06:08:49 mr-fox kernel:  <TASK>
May 26 06:08:49 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:08:49 mr-fox kernel: RIP: 0010:xas_descend+0x2/0xd0
May 26 06:08:49 mr-fox kernel: Code: 18 0f b6 4c 24 10 4c 8b 04 24 e9 60 fe ff ff e9 69 87 04 00 e9 a3 87 04 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 55 <41> 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80
May 26 06:08:49 mr-fox kernel: RSP: 0018:ffffa401077e3940 EFLAGS: 00000206
May 26 06:08:49 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:08:49 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988acf90e6c8 RDI: ffffa401077e3970
May 26 06:08:49 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:08:49 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:08:49 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:08:49 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:08:49 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:08:49 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:08:49 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:08:49 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:08:49 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:08:49 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:08:49 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:08:49 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:08:49 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:08:49 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:08:49 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:08:49 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:08:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:08:49 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:08:49 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:08:49 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:08:49 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:08:49 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:08:49 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:08:49 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:08:49 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:08:49 mr-fox kernel:  </TASK>
May 26 06:09:00 mr-fox crond[20917]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:09:00 mr-fox crond[20920]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:09:00 mr-fox crond[20921]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:09:00 mr-fox CROND[20925]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:09:00 mr-fox CROND[20926]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:09:00 mr-fox crond[20922]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:09:00 mr-fox CROND[20928]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:09:00 mr-fox crond[20919]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:09:00 mr-fox CROND[20930]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:09:00 mr-fox CROND[20931]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:09:00 mr-fox CROND[20922]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:09:00 mr-fox CROND[20922]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:09:00 mr-fox CROND[20921]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:09:00 mr-fox CROND[20921]: pam_unix(crond:session): session closed for user root
May 26 06:09:00 mr-fox CROND[20920]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:09:00 mr-fox CROND[20920]: pam_unix(crond:session): session closed for user root
May 26 06:09:01 mr-fox CROND[14401]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:09:01 mr-fox CROND[14401]: pam_unix(crond:session): session closed for user root
May 26 06:10:00 mr-fox crond[27959]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:10:00 mr-fox crond[27958]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 06:10:00 mr-fox crond[27960]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:10:00 mr-fox crond[27964]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:10:00 mr-fox crond[27961]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:10:00 mr-fox CROND[27969]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:10:00 mr-fox crond[27963]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:10:00 mr-fox crond[27965]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:10:00 mr-fox crond[27966]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:10:00 mr-fox CROND[27971]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:10:00 mr-fox CROND[27972]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:10:00 mr-fox CROND[27973]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:10:00 mr-fox CROND[27975]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:10:00 mr-fox CROND[27976]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:10:00 mr-fox CROND[27977]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:10:00 mr-fox CROND[27974]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 06:10:00 mr-fox CROND[27966]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:10:00 mr-fox CROND[27966]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:10:00 mr-fox CROND[27963]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:10:00 mr-fox CROND[27963]: pam_unix(crond:session): session closed for user root
May 26 06:10:00 mr-fox CROND[27964]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:10:00 mr-fox CROND[27964]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:10:00 mr-fox CROND[27961]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:10:00 mr-fox CROND[27961]: pam_unix(crond:session): session closed for user root
May 26 06:10:00 mr-fox CROND[27958]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:10:00 mr-fox CROND[27958]: pam_unix(crond:session): session closed for user torproject
May 26 06:10:00 mr-fox CROND[20919]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:10:00 mr-fox CROND[20919]: pam_unix(crond:session): session closed for user root
May 26 06:11:00 mr-fox crond[3902]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:11:00 mr-fox crond[3904]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:11:00 mr-fox crond[3901]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:11:00 mr-fox CROND[3908]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:11:00 mr-fox CROND[3911]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:11:00 mr-fox crond[3905]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:11:00 mr-fox crond[3903]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:11:00 mr-fox CROND[3913]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:11:00 mr-fox CROND[3914]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:11:00 mr-fox CROND[3915]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:11:00 mr-fox CROND[3905]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:11:00 mr-fox CROND[3905]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:11:00 mr-fox CROND[3904]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:11:00 mr-fox CROND[3904]: pam_unix(crond:session): session closed for user root
May 26 06:11:01 mr-fox CROND[27960]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:11:01 mr-fox CROND[27960]: pam_unix(crond:session): session closed for user root
May 26 06:11:01 mr-fox CROND[3903]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:11:01 mr-fox CROND[3903]: pam_unix(crond:session): session closed for user root
May 26 06:11:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:11:44 mr-fox kernel: rcu: \x0921-....: (150011 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=61509
May 26 06:11:44 mr-fox kernel: rcu: \x09(t=150012 jiffies g=8794409 q=9907950 ncpus=32)
May 26 06:11:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:11:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:11:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:11:44 mr-fox kernel: RIP: 0010:xas_descend+0x2/0xd0
May 26 06:11:44 mr-fox kernel: Code: 18 0f b6 4c 24 10 4c 8b 04 24 e9 60 fe ff ff e9 69 87 04 00 e9 a3 87 04 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 55 <41> 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80
May 26 06:11:44 mr-fox kernel: RSP: 0018:ffffa401077e3940 EFLAGS: 00000206
May 26 06:11:44 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:11:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988acf90e6c8 RDI: ffffa401077e3970
May 26 06:11:44 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:11:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:11:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:11:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:11:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:11:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:11:44 mr-fox kernel: PKRU: 55555554
May 26 06:11:44 mr-fox kernel: Call Trace:
May 26 06:11:44 mr-fox kernel:  <IRQ>
May 26 06:11:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:11:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:11:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:11:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:11:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:11:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:11:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:11:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:11:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:11:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:11:44 mr-fox kernel:  </IRQ>
May 26 06:11:44 mr-fox kernel:  <TASK>
May 26 06:11:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:11:44 mr-fox kernel:  ? xas_descend+0x2/0xd0
May 26 06:11:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:11:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:11:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:11:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:11:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:11:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:11:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:11:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:11:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:11:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:11:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:11:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:11:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:11:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:11:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:11:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:11:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:11:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:11:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:11:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:11:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:11:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:11:44 mr-fox kernel:  </TASK>
May 26 06:11:49 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 151152 jiffies s: 491905 root: 0x2/.
May 26 06:11:49 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:11:49 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:11:49 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:11:49 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:11:49 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:11:49 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:11:49 mr-fox kernel: RIP: 0010:iommu_dma_free_iova+0xf6/0x220
May 26 06:11:49 mr-fox kernel: Code: 2f 01 00 00 41 8b 0f 85 c9 0f 84 42 01 00 00 8d 48 01 41 23 4f 0c 41 89 4f 08 48 8d 0c 80 48 c1 e1 03 49 8d 04 0f 4c 89 60 10 <48> 89 68 18 48 8b b3 90 00 00 00 48 89 70 30 49 8b 76 18 49 39 f5
May 26 06:11:49 mr-fox kernel: RSP: 0018:ffffa401005f0d08 EFLAGS: 00000016
May 26 06:11:49 mr-fox kernel: RAX: ffffc400ffb51cd0 RBX: ffff988ac1292400 RCX: 0000000000000460
May 26 06:11:49 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:11:49 mr-fox kernel: RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
May 26 06:11:49 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000cfc53
May 26 06:11:49 mr-fox kernel: R13: ffffa401005f0d68 R14: ffffa401005f0d50 R15: ffffc400ffb51870
May 26 06:11:49 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:11:49 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:11:49 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:11:49 mr-fox kernel: PKRU: 55555554
May 26 06:11:49 mr-fox kernel: Call Trace:
May 26 06:11:49 mr-fox kernel:  <NMI>
May 26 06:11:49 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:11:49 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:11:49 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:11:49 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:11:49 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:11:49 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:11:49 mr-fox kernel:  ? iommu_dma_free_iova+0xf6/0x220
May 26 06:11:49 mr-fox kernel:  ? iommu_dma_free_iova+0xf6/0x220
May 26 06:11:49 mr-fox kernel:  ? iommu_dma_free_iova+0xf6/0x220
May 26 06:11:49 mr-fox kernel:  </NMI>
May 26 06:11:49 mr-fox kernel:  <IRQ>
May 26 06:11:49 mr-fox kernel:  __iommu_dma_unmap+0xe0/0x170
May 26 06:11:49 mr-fox kernel:  iommu_dma_unmap_page+0x43/0xa0
May 26 06:11:49 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 06:11:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:11:49 mr-fox kernel:  ? task_tick_fair+0x85/0x470
May 26 06:11:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:11:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:11:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:11:49 mr-fox kernel:  ? wq_worker_tick+0xd/0xd0
May 26 06:11:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:11:49 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:11:49 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:11:49 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:11:49 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:11:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:11:49 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:11:49 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:11:49 mr-fox kernel:  </IRQ>
May 26 06:11:49 mr-fox kernel:  <TASK>
May 26 06:11:49 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:11:49 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 06:11:49 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 06:11:49 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 06:11:49 mr-fox kernel: RAX: ffff988d667606da RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:11:49 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:11:49 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 06:11:49 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:11:49 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:11:49 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:11:49 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:11:49 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:11:49 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:11:49 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:11:49 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:11:49 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:11:49 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:11:49 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:11:49 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:11:49 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:11:49 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:11:49 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:11:49 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:11:49 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:11:49 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:11:49 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:11:49 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:11:49 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:11:49 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:11:49 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:11:49 mr-fox kernel:  </TASK>
May 26 06:12:00 mr-fox crond[10409]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:12:00 mr-fox crond[10408]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:12:00 mr-fox crond[10410]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:12:00 mr-fox crond[10411]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:12:00 mr-fox crond[10412]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:12:00 mr-fox CROND[10416]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:12:00 mr-fox CROND[10417]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:12:00 mr-fox CROND[10418]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:12:00 mr-fox CROND[10419]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:12:00 mr-fox CROND[10420]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:12:00 mr-fox CROND[10412]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:12:00 mr-fox CROND[10412]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:12:00 mr-fox CROND[10411]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:12:00 mr-fox CROND[10411]: pam_unix(crond:session): session closed for user root
May 26 06:12:00 mr-fox CROND[10410]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:12:00 mr-fox CROND[10410]: pam_unix(crond:session): session closed for user root
May 26 06:12:01 mr-fox CROND[3902]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:12:01 mr-fox CROND[3902]: pam_unix(crond:session): session closed for user root
May 26 06:13:00 mr-fox crond[19085]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:13:00 mr-fox crond[19086]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:13:00 mr-fox crond[19087]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:13:00 mr-fox crond[19089]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:13:00 mr-fox CROND[19093]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:13:00 mr-fox CROND[19095]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:13:00 mr-fox CROND[19094]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:13:00 mr-fox CROND[19096]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:13:00 mr-fox crond[19088]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:13:00 mr-fox CROND[19100]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:13:00 mr-fox CROND[19089]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:13:00 mr-fox CROND[19089]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:13:00 mr-fox CROND[19088]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:13:00 mr-fox CROND[19088]: pam_unix(crond:session): session closed for user root
May 26 06:13:00 mr-fox CROND[19087]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:13:00 mr-fox CROND[19087]: pam_unix(crond:session): session closed for user root
May 26 06:13:00 mr-fox CROND[10409]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:13:00 mr-fox CROND[10409]: pam_unix(crond:session): session closed for user root
May 26 06:14:00 mr-fox crond[25462]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:14:00 mr-fox crond[25460]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:14:00 mr-fox crond[25463]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:14:00 mr-fox crond[25465]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:14:00 mr-fox crond[25461]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:14:00 mr-fox CROND[25468]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:14:00 mr-fox CROND[25469]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:14:00 mr-fox CROND[25470]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:14:00 mr-fox CROND[25471]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:14:00 mr-fox CROND[25473]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:14:00 mr-fox CROND[25465]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:14:00 mr-fox CROND[25465]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:14:00 mr-fox CROND[25463]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:14:00 mr-fox CROND[25463]: pam_unix(crond:session): session closed for user root
May 26 06:14:00 mr-fox CROND[25462]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:14:00 mr-fox CROND[25462]: pam_unix(crond:session): session closed for user root
May 26 06:14:01 mr-fox CROND[19086]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:14:01 mr-fox CROND[19086]: pam_unix(crond:session): session closed for user root
May 26 06:14:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:14:44 mr-fox kernel: rcu: \x0921-....: (195014 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=78756
May 26 06:14:44 mr-fox kernel: rcu: \x09(t=195015 jiffies g=8794409 q=11131553 ncpus=32)
May 26 06:14:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:14:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:14:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:14:44 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 06:14:44 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 06:14:44 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 06:14:44 mr-fox kernel: RAX: ffff988ac07566ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:14:44 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:14:44 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:14:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:14:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:14:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:14:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:14:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:14:44 mr-fox kernel: PKRU: 55555554
May 26 06:14:44 mr-fox kernel: Call Trace:
May 26 06:14:44 mr-fox kernel:  <IRQ>
May 26 06:14:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:14:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:14:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:14:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:14:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:14:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:14:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:14:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:14:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:14:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:14:44 mr-fox kernel:  </IRQ>
May 26 06:14:44 mr-fox kernel:  <TASK>
May 26 06:14:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:14:44 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 06:14:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:14:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:14:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:14:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:14:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:14:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:14:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:14:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:14:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:14:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:14:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:14:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:14:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:14:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:14:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:14:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:14:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:14:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:14:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:14:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:14:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:14:44 mr-fox kernel:  </TASK>
May 26 06:14:50 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 196208 jiffies s: 491905 root: 0x2/.
May 26 06:14:50 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:14:50 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:14:50 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:14:50 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:14:50 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:14:50 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:14:50 mr-fox kernel: RIP: 0010:fetch_pte+0x78/0x170
May 26 06:14:50 mr-fox kernel: Code: 00 01 00 00 83 eb 01 44 8d 34 db 41 8d 4e 0c 83 f9 3f 0f 87 0e f9 2d 00 4c 89 e0 49 8b 95 08 01 00 00 48 d3 e8 25 ff 01 00 00 <48> 8d 04 c2 ba 01 00 00 00 48 d3 e2 48 89 55 00 85 db 0f 8e b2 00
May 26 06:14:50 mr-fox kernel: RSP: 0018:ffffa401005f0c38 EFLAGS: 00000206
May 26 06:14:50 mr-fox kernel: RAX: 0000000000000003 RBX: 0000000000000002 RCX: 000000000000001e
May 26 06:14:50 mr-fox kernel: RDX: ffff988ac41fa000 RSI: 00000000fed95000 RDI: ffff988ac2211088
May 26 06:14:50 mr-fox kernel: RBP: ffffa401005f0c80 R08: 0000000000000000 R09: 0000000000000000
May 26 06:14:50 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000fed95000
May 26 06:14:50 mr-fox kernel: R13: ffff988ac2211088 R14: 0000000000000012 R15: ffff988ac2211088
May 26 06:14:50 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:14:50 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:14:50 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:14:50 mr-fox kernel: PKRU: 55555554
May 26 06:14:50 mr-fox kernel: Call Trace:
May 26 06:14:50 mr-fox kernel:  <NMI>
May 26 06:14:50 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:14:50 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:14:50 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:14:50 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:14:50 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:14:50 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:14:50 mr-fox kernel:  ? fetch_pte+0x78/0x170
May 26 06:14:50 mr-fox kernel:  ? fetch_pte+0x78/0x170
May 26 06:14:50 mr-fox kernel:  ? fetch_pte+0x78/0x170
May 26 06:14:50 mr-fox kernel:  </NMI>
May 26 06:14:50 mr-fox kernel:  <IRQ>
May 26 06:14:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:14:50 mr-fox kernel:  iommu_v1_unmap_pages+0x81/0x140
May 26 06:14:50 mr-fox kernel:  amd_iommu_unmap_pages+0x40/0x130
May 26 06:14:50 mr-fox kernel:  __iommu_unmap+0xbf/0x120
May 26 06:14:50 mr-fox kernel:  __iommu_dma_unmap+0xb5/0x170
May 26 06:14:50 mr-fox kernel:  iommu_dma_unmap_page+0x43/0xa0
May 26 06:14:50 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 06:14:50 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 06:14:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:14:50 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 06:14:50 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:14:50 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:14:50 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:14:50 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:14:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:14:50 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:14:50 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:14:50 mr-fox kernel:  </IRQ>
May 26 06:14:50 mr-fox kernel:  <TASK>
May 26 06:14:50 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:14:50 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 06:14:50 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 06:14:50 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 06:14:50 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:14:50 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:14:50 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:14:50 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:14:50 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:14:50 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:14:50 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:14:50 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:14:50 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:14:50 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:14:50 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:14:50 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:14:50 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:14:50 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:14:50 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:14:50 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:14:50 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:14:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:14:50 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:14:50 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:14:50 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:14:50 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:14:50 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:14:50 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:14:50 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:14:50 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:14:50 mr-fox kernel:  </TASK>
May 26 06:15:00 mr-fox crond[30290]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 06:15:00 mr-fox crond[30291]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:15:00 mr-fox crond[30292]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:15:00 mr-fox crond[30294]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:15:00 mr-fox CROND[30301]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:15:00 mr-fox crond[30296]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:15:00 mr-fox crond[30295]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:15:00 mr-fox crond[30293]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:15:00 mr-fox CROND[30302]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:15:00 mr-fox CROND[30304]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:15:00 mr-fox CROND[30303]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:15:00 mr-fox crond[30298]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:15:00 mr-fox CROND[30305]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 06:15:00 mr-fox CROND[30307]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:15:00 mr-fox CROND[30308]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:15:00 mr-fox CROND[30309]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:15:00 mr-fox CROND[30298]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:15:00 mr-fox CROND[30298]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:15:00 mr-fox CROND[30294]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:15:00 mr-fox CROND[30294]: pam_unix(crond:session): session closed for user root
May 26 06:15:00 mr-fox CROND[30295]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:15:00 mr-fox CROND[30295]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:15:00 mr-fox CROND[30293]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:15:00 mr-fox CROND[30293]: pam_unix(crond:session): session closed for user root
May 26 06:15:00 mr-fox CROND[30290]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:15:00 mr-fox CROND[30290]: pam_unix(crond:session): session closed for user torproject
May 26 06:15:01 mr-fox CROND[25461]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:15:01 mr-fox CROND[25461]: pam_unix(crond:session): session closed for user root
May 26 06:16:00 mr-fox crond[5459]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:16:00 mr-fox crond[5461]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:16:00 mr-fox crond[5462]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:16:00 mr-fox crond[5464]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:16:00 mr-fox crond[5463]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:16:00 mr-fox CROND[5468]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:16:00 mr-fox CROND[5469]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:16:00 mr-fox CROND[5470]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:16:00 mr-fox CROND[5471]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:16:00 mr-fox CROND[5472]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:16:00 mr-fox CROND[5464]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:16:00 mr-fox CROND[5464]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:16:00 mr-fox CROND[5463]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:16:00 mr-fox CROND[5463]: pam_unix(crond:session): session closed for user root
May 26 06:16:00 mr-fox CROND[5462]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:16:00 mr-fox CROND[5462]: pam_unix(crond:session): session closed for user root
May 26 06:16:00 mr-fox CROND[30292]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:16:00 mr-fox CROND[30292]: pam_unix(crond:session): session closed for user root
May 26 06:17:00 mr-fox crond[14291]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:17:00 mr-fox crond[14290]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:17:00 mr-fox crond[14293]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:17:00 mr-fox CROND[14298]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:17:00 mr-fox CROND[14299]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:17:00 mr-fox crond[14292]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:17:00 mr-fox crond[14294]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:17:00 mr-fox CROND[14301]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:17:00 mr-fox CROND[14303]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:17:00 mr-fox CROND[14305]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:17:00 mr-fox CROND[14294]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:17:00 mr-fox CROND[14293]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:17:00 mr-fox CROND[14294]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:17:00 mr-fox CROND[14293]: pam_unix(crond:session): session closed for user root
May 26 06:17:00 mr-fox CROND[14292]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:17:00 mr-fox CROND[5461]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:17:00 mr-fox CROND[5461]: pam_unix(crond:session): session closed for user root
May 26 06:17:00 mr-fox CROND[14292]: pam_unix(crond:session): session closed for user root
May 26 06:17:40 mr-fox kernel: fuzz-extrainfo invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
May 26 06:17:40 mr-fox kernel: CPU: 1 PID: 20863 Comm: fuzz-extrainfo Tainted: G    B           T  6.8.11 #13
May 26 06:17:40 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:17:40 mr-fox kernel: Call Trace:
May 26 06:17:40 mr-fox kernel:  <TASK>
May 26 06:17:40 mr-fox kernel:  dump_stack_lvl+0x53/0x70
May 26 06:17:40 mr-fox kernel:  dump_header+0x40/0x1d0
May 26 06:17:40 mr-fox kernel:  oom_kill_process+0x10b/0x250
May 26 06:17:40 mr-fox kernel:  out_of_memory+0x259/0x590
May 26 06:17:40 mr-fox kernel:  mem_cgroup_out_of_memory+0x116/0x170
May 26 06:17:40 mr-fox kernel:  try_charge_memcg+0x76c/0x830
May 26 06:17:40 mr-fox kernel:  __mem_cgroup_charge+0x3f/0xc0
May 26 06:17:40 mr-fox kernel:  ? __alloc_pages+0x17d/0x300
May 26 06:17:40 mr-fox kernel:  __filemap_add_folio+0x308/0x3b0
May 26 06:17:40 mr-fox kernel:  ? scan_shadow_nodes+0x30/0x30
May 26 06:17:40 mr-fox kernel:  filemap_add_folio+0x37/0xb0
May 26 06:17:40 mr-fox kernel:  __filemap_get_folio+0x158/0x2c0
May 26 06:17:40 mr-fox kernel:  filemap_fault+0x14c/0x9e0
May 26 06:17:40 mr-fox kernel:  __do_fault+0x30/0x1a0
May 26 06:17:40 mr-fox kernel:  __handle_mm_fault+0x1312/0x1a30
May 26 06:17:40 mr-fox kernel:  handle_mm_fault+0x15f/0x370
May 26 06:17:40 mr-fox kernel:  exc_page_fault+0x16f/0x6b0
May 26 06:17:40 mr-fox kernel:  asm_exc_page_fault+0x26/0x30
May 26 06:17:40 mr-fox kernel: RIP: 0033:0x7fe55c7a09d8
May 26 06:17:40 mr-fox kernel: Code: Unable to access opcode bytes at 0x7fe55c7a09ae.
May 26 06:17:40 mr-fox kernel: RSP: 002b:00007ffe777732d8 EFLAGS: 00010206
May 26 06:17:40 mr-fox kernel: RAX: 00007fe55c7a09d8 RBX: ffffffffffffffff RCX: 0000000000000000
May 26 06:17:40 mr-fox kernel: RDX: 00007fe55c7abad0 RSI: 00007fe55c7ac000 RDI: 00007fe55c7ac008
May 26 06:17:40 mr-fox kernel: RBP: 00007fe55c95a9f0 R08: 0000000000000020 R09: 000056309c5c9820
May 26 06:17:40 mr-fox kernel: R10: 00000000e4f78060 R11: 00000000e4f78060 R12: 00007fe55c7ab3f8
May 26 06:17:40 mr-fox kernel: R13: 00007fe55c95a9f0 R14: 0000000000000007 R15: 00007ffe77773300
May 26 06:17:40 mr-fox kernel:  </TASK>
May 26 06:17:40 mr-fox kernel: memory: usage 8388608kB, limit 8388608kB, failcnt 3338
May 26 06:17:40 mr-fox kernel: swap: usage 0kB, limit 0kB, failcnt 0
May 26 06:17:40 mr-fox kernel: Memory cgroup stats for /fuzzing/tor_extrainfo_20240526-042002_7a5d94bcf842:
May 26 06:17:40 mr-fox kernel: anon 8523776
May 26 06:17:40 mr-fox kernel: file 10391552
May 26 06:17:40 mr-fox kernel: kernel 8571019264
May 26 06:17:40 mr-fox kernel: kernel_stack 32768
May 26 06:17:40 mr-fox kernel: pagetables 151552
May 26 06:17:40 mr-fox kernel: sec_pagetables 0
May 26 06:17:40 mr-fox kernel: percpu 1088
May 26 06:17:40 mr-fox kernel: sock 0
May 26 06:17:40 mr-fox kernel: vmalloc 0
May 26 06:17:40 mr-fox kernel: shmem 10346496
May 26 06:17:40 mr-fox kernel: zswap 0
May 26 06:17:40 mr-fox kernel: zswapped 0
May 26 06:17:40 mr-fox kernel: file_mapped 61440
May 26 06:17:40 mr-fox kernel: file_dirty 0
May 26 06:17:40 mr-fox kernel: file_writeback 0
May 26 06:17:40 mr-fox kernel: swapcached 0
May 26 06:17:40 mr-fox kernel: anon_thp 0
May 26 06:17:40 mr-fox kernel: file_thp 0
May 26 06:17:40 mr-fox kernel: shmem_thp 0
May 26 06:17:40 mr-fox kernel: inactive_anon 12357632
May 26 06:17:40 mr-fox kernel: active_anon 6512640
May 26 06:17:40 mr-fox kernel: inactive_file 45056
May 26 06:17:40 mr-fox kernel: active_file 0
May 26 06:17:40 mr-fox kernel: unevictable 0
May 26 06:17:40 mr-fox kernel: slab_reclaimable 9231024
May 26 06:17:40 mr-fox kernel: slab_unreclaimable 8561575768
May 26 06:17:40 mr-fox kernel: slab 8570806792
May 26 06:17:40 mr-fox kernel: workingset_refault_anon 0
May 26 06:17:40 mr-fox kernel: workingset_refault_file 510
May 26 06:17:40 mr-fox kernel: workingset_activate_anon 0
May 26 06:17:40 mr-fox kernel: workingset_activate_file 75
May 26 06:17:40 mr-fox kernel: workingset_restore_anon 0
May 26 06:17:40 mr-fox kernel: workingset_restore_file 61
May 26 06:17:40 mr-fox kernel: workingset_nodereclaim 0
May 26 06:17:40 mr-fox kernel: pgscan 5782
May 26 06:17:40 mr-fox kernel: pgsteal 1380
May 26 06:17:40 mr-fox kernel: pgscan_kswapd 87
May 26 06:17:40 mr-fox kernel: pgscan_direct 5695
May 26 06:17:40 mr-fox kernel: pgscan_khugepaged 0
May 26 06:17:40 mr-fox kernel: pgsteal_kswapd 87
May 26 06:17:40 mr-fox kernel: pgsteal_direct 1293
May 26 06:17:40 mr-fox kernel: pgsteal_khugepaged 0
May 26 06:17:40 mr-fox kernel: pgfault 1604631247
May 26 06:17:40 mr-fox kernel: pgmajfault 37
May 26 06:17:40 mr-fox kernel: pgrefill 599
May 26 06:17:40 mr-fox kernel: pgactivate 260600
May 26 06:17:40 mr-fox kernel: pgdeactivate 520
May 26 06:17:40 mr-fox kernel: pglazyfree 0
May 26 06:17:40 mr-fox kernel: pglazyfreed 0
May 26 06:17:40 mr-fox kernel: zswpin 0
May 26 06:17:40 mr-fox kernel: zswpout 0
May 26 06:17:40 mr-fox kernel: zswpwb 0
May 26 06:17:40 mr-fox kernel: thp_fault_alloc 0
May 26 06:17:40 mr-fox kernel: thp_collapse_alloc 0
May 26 06:17:40 mr-fox kernel: thp_swpout 0
May 26 06:17:40 mr-fox kernel: thp_swpout_fallback 0
May 26 06:17:40 mr-fox kernel: Tasks state (memory values in pages):
May 26 06:17:40 mr-fox kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
May 26 06:17:40 mr-fox kernel: [   4257]  1001  4257     4725     2944     2368      576         0    81920        0             0 afl-fuzz
May 26 06:17:40 mr-fox kernel: [   5677]  1001  5677     4246     2176      320     1344       512    69632        0             0 fuzz-extrainfo
May 26 06:17:40 mr-fox kernel: [  20863]  1001 20863     4246     1102      334      576       192    69632        0             0 fuzz-extrainfo
May 26 06:17:40 mr-fox kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=tor_extrainfo_20240526-042002_7a5d94bcf842,mems_allowed=0,oom_memcg=/fuzzing/tor_extrainfo_20240526-042002_7a5d94bcf842,task_memcg=/fuzzing/tor_extrainfo_20240526-042002_7a5d94bcf842,task=afl-fuzz,pid=4257,uid=1001
May 26 06:17:40 mr-fox kernel: Memory cgroup out of memory: Killed process 4257 (afl-fuzz) total-vm:18900kB, anon-rss:9472kB, file-rss:2304kB, shmem-rss:0kB, UID:1001 pgtables:80kB oom_score_adj:0
May 26 06:17:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:17:44 mr-fox kernel: rcu: \x0921-....: (240018 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=96117
May 26 06:17:44 mr-fox kernel: rcu: \x09(t=240019 jiffies g=8794409 q=12292563 ncpus=32)
May 26 06:17:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:17:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:17:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:17:44 mr-fox kernel: RIP: 0010:xas_descend+0x3c/0xd0
May 26 06:17:44 mr-fox kernel: Code: 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03 <48> 83 fa 02 75 08 48 3d fd 00 00 00 76 2f 41 88 5c 24 12 48 83 c4
May 26 06:17:44 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000202
May 26 06:17:44 mr-fox kernel: RAX: ffff988ac07566ca RBX: 0000000000000001 RCX: 000000000000001e
May 26 06:17:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac0754ff8 RDI: ffffa401077e3970
May 26 06:17:44 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:17:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 06:17:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:17:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:17:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:17:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:17:44 mr-fox kernel: PKRU: 55555554
May 26 06:17:44 mr-fox kernel: Call Trace:
May 26 06:17:44 mr-fox kernel:  <IRQ>
May 26 06:17:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:17:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:17:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:17:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:17:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:17:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:17:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:17:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:17:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:17:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:17:44 mr-fox kernel:  </IRQ>
May 26 06:17:44 mr-fox kernel:  <TASK>
May 26 06:17:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:17:44 mr-fox kernel:  ? xas_descend+0x3c/0xd0
May 26 06:17:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:17:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:17:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:17:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:17:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:17:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:17:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:17:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:17:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:17:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:17:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:17:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:17:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:17:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:17:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:17:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:17:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:17:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:17:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:17:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:17:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:17:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:17:44 mr-fox kernel:  </TASK>
May 26 06:17:50 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 241264 jiffies s: 491905 root: 0x2/.
May 26 06:17:50 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:17:50 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:17:50 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:17:50 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:17:50 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:17:50 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:17:50 mr-fox kernel: RIP: 0010:napi_consume_skb+0x27/0xc0
May 26 06:17:50 mr-fox kernel: Code: 00 66 90 f3 0f 1e fa 53 48 89 fb 85 f6 0f 84 8c 00 00 00 48 85 ff 74 7b 8b 87 d4 00 00 00 83 f8 01 75 53 48 89 df f6 43 7e 0c <75> 25 e8 62 87 ff ff 48 83 bb c0 00 00 00 00 74 0d be 01 00 00 00
May 26 06:17:50 mr-fox kernel: RSP: 0018:ffffa401005f0de8 EFLAGS: 00000202
May 26 06:17:50 mr-fox kernel: RAX: 0000000000000001 RBX: ffff989b4c351798 RCX: 0000000000000000
May 26 06:17:50 mr-fox kernel: RDX: 0000000000000001 RSI: 0000000000000040 RDI: ffff989b4c351798
May 26 06:17:50 mr-fox kernel: RBP: 00000000000043da R08: 0000000000000000 R09: 0000000000000000
May 26 06:17:50 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffa40100119040
May 26 06:17:50 mr-fox kernel: R13: ffff988ac873fa40 R14: 00000000ffffff04 R15: ffffa40100119060
May 26 06:17:50 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:17:50 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:17:50 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:17:50 mr-fox kernel: PKRU: 55555554
May 26 06:17:50 mr-fox kernel: Call Trace:
May 26 06:17:50 mr-fox kernel:  <NMI>
May 26 06:17:50 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:17:50 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:17:50 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:17:50 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:17:50 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:17:50 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:17:50 mr-fox kernel:  ? napi_consume_skb+0x27/0xc0
May 26 06:17:50 mr-fox kernel:  ? napi_consume_skb+0x27/0xc0
May 26 06:17:50 mr-fox kernel:  ? napi_consume_skb+0x27/0xc0
May 26 06:17:50 mr-fox kernel:  </NMI>
May 26 06:17:50 mr-fox kernel:  <IRQ>
May 26 06:17:50 mr-fox kernel:  igb_poll+0xea/0x1370
May 26 06:17:50 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 06:17:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:17:50 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 06:17:50 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:17:50 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:17:50 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:17:50 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:17:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:17:50 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:17:50 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:17:50 mr-fox kernel:  </IRQ>
May 26 06:17:50 mr-fox kernel:  <TASK>
May 26 06:17:50 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:17:50 mr-fox kernel: RIP: 0010:filemap_get_entry+0x60/0x160
May 26 06:17:50 mr-fox kernel: Code: 24 20 03 00 00 00 48 c7 44 24 28 00 00 00 00 48 c7 44 24 30 00 00 00 00 48 c7 44 24 38 00 00 00 00 48 c7 44 24 20 03 00 00 00 <48> 8d 7c 24 08 e8 56 70 78 00 48 89 c3 48 3d 02 04 00 00 74 e2 48
May 26 06:17:50 mr-fox kernel: RSP: 0018:ffffa401077e3968 EFLAGS: 00000246
May 26 06:17:50 mr-fox kernel: RAX: 0000000000000000 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 06:17:50 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:17:50 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 06:17:50 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:17:50 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:17:50 mr-fox kernel:  ? filemap_get_entry+0x6a/0x160
May 26 06:17:50 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:17:50 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:17:50 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:17:50 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:17:50 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:17:50 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:17:50 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:17:50 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:17:50 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:17:50 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:17:50 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:17:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:17:50 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:17:50 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:17:50 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:17:50 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:17:50 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:17:50 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:17:50 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:17:50 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:17:50 mr-fox kernel:  </TASK>
May 26 06:18:00 mr-fox crond[21217]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:18:00 mr-fox crond[21218]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:18:00 mr-fox crond[21219]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:18:00 mr-fox crond[21221]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:18:00 mr-fox CROND[21224]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:18:00 mr-fox CROND[21223]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:18:00 mr-fox crond[21220]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:18:00 mr-fox CROND[21225]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:18:00 mr-fox CROND[21226]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:18:00 mr-fox CROND[21227]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:18:00 mr-fox CROND[21221]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:18:00 mr-fox CROND[21221]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:18:00 mr-fox CROND[21220]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:18:00 mr-fox CROND[21220]: pam_unix(crond:session): session closed for user root
May 26 06:18:00 mr-fox CROND[21219]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:18:00 mr-fox CROND[21219]: pam_unix(crond:session): session closed for user root
May 26 06:18:01 mr-fox CROND[14291]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:18:01 mr-fox CROND[14291]: pam_unix(crond:session): session closed for user root
May 26 06:19:00 mr-fox crond[22782]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:19:00 mr-fox crond[22783]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:19:00 mr-fox crond[22784]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:19:00 mr-fox crond[22781]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:19:00 mr-fox crond[22785]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:19:00 mr-fox CROND[22787]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:19:00 mr-fox CROND[22788]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:19:00 mr-fox CROND[22789]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:19:00 mr-fox CROND[22790]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:19:00 mr-fox CROND[22791]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:19:00 mr-fox CROND[22785]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:19:00 mr-fox CROND[22785]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:19:00 mr-fox CROND[22784]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:19:00 mr-fox CROND[22784]: pam_unix(crond:session): session closed for user root
May 26 06:19:00 mr-fox CROND[22783]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:19:00 mr-fox CROND[22783]: pam_unix(crond:session): session closed for user root
May 26 06:19:01 mr-fox CROND[21218]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:19:01 mr-fox CROND[21218]: pam_unix(crond:session): session closed for user root
May 26 06:20:00 mr-fox crond[24340]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 06:20:00 mr-fox crond[24342]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:20:00 mr-fox crond[24341]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:20:00 mr-fox crond[24344]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:20:00 mr-fox CROND[24349]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:20:00 mr-fox CROND[24351]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:20:00 mr-fox CROND[24350]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:20:00 mr-fox crond[24343]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:20:00 mr-fox CROND[24352]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:20:00 mr-fox crond[24346]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:20:00 mr-fox crond[24345]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:20:00 mr-fox crond[24347]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:20:00 mr-fox CROND[24353]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 06:20:00 mr-fox CROND[24354]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:20:00 mr-fox CROND[24355]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:20:00 mr-fox CROND[24356]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:20:00 mr-fox CROND[24347]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:20:00 mr-fox CROND[24347]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:20:00 mr-fox CROND[24344]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:20:00 mr-fox CROND[24344]: pam_unix(crond:session): session closed for user root
May 26 06:20:00 mr-fox CROND[24345]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:20:00 mr-fox CROND[24345]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:20:00 mr-fox CROND[24343]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:20:00 mr-fox CROND[24343]: pam_unix(crond:session): session closed for user root
May 26 06:20:00 mr-fox sudo[25073]: torproject : PWD=/home/torproject ; USER=root ; COMMAND=/opt/fuzz-utils/fuzz-cgroup.sh tor_extrainfo_20240526-042002_7a5d94bcf842
May 26 06:20:00 mr-fox sudo[25073]: pam_unix(sudo:session): session opened for user root(uid=0) by torproject(uid=1001)
May 26 06:20:00 mr-fox sudo[25073]: pam_unix(sudo:session): session closed for user root
May 26 06:20:00 mr-fox CROND[22782]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:20:00 mr-fox CROND[22782]: pam_unix(crond:session): session closed for user root
May 26 06:20:01 mr-fox sudo[25160]: torproject : PWD=/tmp/torproject/fuzzing/tor_diff-apply_20240526-062001_7a5d94bcf842 ; USER=root ; COMMAND=/opt/fuzz-utils/fuzz-cgroup.sh tor_diff-apply_20240526-062001_7a5d94bcf842 25157
May 26 06:20:02 mr-fox sudo[25160]: pam_unix(sudo:session): session opened for user root(uid=0) by torproject(uid=1001)
May 26 06:20:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:20:44 mr-fox kernel: rcu: \x0921-....: (285022 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=113914
May 26 06:20:44 mr-fox kernel: rcu: \x09(t=285023 jiffies g=8794409 q=12760513 ncpus=32)
May 26 06:20:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:20:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:20:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:20:44 mr-fox kernel: RIP: 0010:xas_load+0x49/0x60
May 26 06:20:44 mr-fox kernel: Code: 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff <80> 7d 00 00 75 bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40
May 26 06:20:44 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 06:20:44 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:20:44 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:20:44 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:20:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:20:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:20:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:20:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:20:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:20:44 mr-fox kernel: PKRU: 55555554
May 26 06:20:44 mr-fox kernel: Call Trace:
May 26 06:20:44 mr-fox kernel:  <IRQ>
May 26 06:20:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:20:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:20:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:20:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:20:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:20:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:20:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:20:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:20:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:20:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:20:44 mr-fox kernel:  </IRQ>
May 26 06:20:44 mr-fox kernel:  <TASK>
May 26 06:20:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:20:44 mr-fox kernel:  ? xas_load+0x49/0x60
May 26 06:20:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:20:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:20:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:20:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:20:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:20:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:20:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:20:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:20:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:20:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:20:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:20:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:20:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:20:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:20:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:20:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:20:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:20:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:20:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:20:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:20:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:20:44 mr-fox kernel:  </TASK>
May 26 06:20:50 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 286320 jiffies s: 491905 root: 0x2/.
May 26 06:20:50 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:20:50 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:20:50 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:20:50 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:20:50 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:20:50 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:20:50 mr-fox kernel: RIP: 0010:nf_conntrack_in+0x4/0x540
May 26 06:20:50 mr-fox kernel: Code: 89 df e8 3f f6 ff ff 83 f8 01 0f 85 0b fc ff ff 48 8b 7b 68 48 83 e7 f8 e9 11 ff ff ff e8 b4 1a 18 00 0f 1f 40 00 f3 0f 1e fa <41> 57 41 56 41 55 49 89 fd 41 54 55 53 48 89 f3 48 83 ec 50 65 48
May 26 06:20:50 mr-fox kernel: RSP: 0018:ffffa401005f0b68 EFLAGS: 00000286
May 26 06:20:50 mr-fox kernel: RAX: ffffffffa5827c50 RBX: 0000000000000001 RCX: 0000000000000000
May 26 06:20:50 mr-fox kernel: RDX: ffffa401005f0c10 RSI: ffffa401005f0c10 RDI: ffff9891f62bee00
May 26 06:20:50 mr-fox kernel: RBP: ffff9891f62bee00 R08: ffffa401005f0c90 R09: 0000000000000000
May 26 06:20:50 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffa401005f0c10
May 26 06:20:50 mr-fox kernel: R13: 0000000000000001 R14: ffff988ac7fb1420 R15: ffff9891f62bee00
May 26 06:20:50 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:20:50 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:20:50 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:20:50 mr-fox kernel: PKRU: 55555554
May 26 06:20:50 mr-fox kernel: Call Trace:
May 26 06:20:50 mr-fox kernel:  <NMI>
May 26 06:20:50 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:20:50 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:20:50 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:20:50 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:20:50 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:20:50 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:20:50 mr-fox kernel:  ? nf_ct_netns_do_get+0x220/0x220
May 26 06:20:50 mr-fox kernel:  ? nf_conntrack_in+0x4/0x540
May 26 06:20:50 mr-fox kernel:  ? nf_conntrack_in+0x4/0x540
May 26 06:20:50 mr-fox kernel:  ? nf_conntrack_in+0x4/0x540
May 26 06:20:50 mr-fox kernel:  </NMI>
May 26 06:20:50 mr-fox kernel:  <IRQ>
May 26 06:20:50 mr-fox kernel:  nf_hook_slow+0x3c/0x100
May 26 06:20:50 mr-fox kernel:  nf_hook_slow_list+0x90/0x140
May 26 06:20:50 mr-fox kernel:  ip_sublist_rcv+0x71/0x1c0
May 26 06:20:50 mr-fox kernel:  ? ip_sublist_rcv+0x1c0/0x1c0
May 26 06:20:50 mr-fox kernel:  ip_list_rcv+0x138/0x170
May 26 06:20:50 mr-fox kernel:  __netif_receive_skb_list_core+0x293/0x2d0
May 26 06:20:50 mr-fox kernel:  netif_receive_skb_list_internal+0x1db/0x320
May 26 06:20:50 mr-fox kernel:  napi_gro_receive+0xcf/0x1b0
May 26 06:20:50 mr-fox kernel:  igb_poll+0x605/0x1370
May 26 06:20:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:20:50 mr-fox kernel:  ? task_tick_fair+0x85/0x470
May 26 06:20:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:20:50 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:20:50 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:20:50 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:20:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:20:50 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:20:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:20:50 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:20:50 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:20:50 mr-fox kernel:  </IRQ>
May 26 06:20:50 mr-fox kernel:  <TASK>
May 26 06:20:50 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:20:50 mr-fox kernel: RIP: 0010:xas_load+0x35/0x60
May 26 06:20:50 mr-fox kernel: Code: f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 <48> 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d
May 26 06:20:50 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000286
May 26 06:20:50 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:20:50 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:20:50 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:20:50 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:20:50 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:20:50 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:20:50 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:20:50 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:20:50 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:20:50 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:20:50 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:20:50 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:20:50 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:20:50 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:20:50 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:20:50 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:20:50 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:20:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:20:50 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:20:50 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:20:50 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:20:50 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:20:50 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:20:50 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:20:50 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:20:50 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:20:50 mr-fox kernel:  </TASK>
May 26 06:21:00 mr-fox crond[7772]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:21:00 mr-fox crond[7774]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:21:00 mr-fox crond[7770]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:21:00 mr-fox crond[7773]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:21:00 mr-fox CROND[7778]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:21:00 mr-fox CROND[7780]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:21:00 mr-fox CROND[7781]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:21:00 mr-fox CROND[7782]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:21:00 mr-fox crond[7775]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:21:00 mr-fox CROND[7783]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:21:00 mr-fox CROND[7775]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:21:00 mr-fox CROND[7775]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:21:00 mr-fox CROND[7774]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:21:00 mr-fox CROND[7774]: pam_unix(crond:session): session closed for user root
May 26 06:21:00 mr-fox CROND[24342]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:21:00 mr-fox CROND[24342]: pam_unix(crond:session): session closed for user root
May 26 06:21:00 mr-fox CROND[7773]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:21:00 mr-fox CROND[7773]: pam_unix(crond:session): session closed for user root
May 26 06:22:00 mr-fox crond[12237]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:22:00 mr-fox crond[12238]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:22:00 mr-fox crond[12239]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:22:00 mr-fox crond[12240]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:22:00 mr-fox crond[12241]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:22:00 mr-fox CROND[12245]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:22:00 mr-fox CROND[12246]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:22:00 mr-fox CROND[12247]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:22:00 mr-fox CROND[12248]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:22:00 mr-fox CROND[12249]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:22:00 mr-fox CROND[12241]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:22:00 mr-fox CROND[12241]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:22:00 mr-fox CROND[12240]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:22:00 mr-fox CROND[12240]: pam_unix(crond:session): session closed for user root
May 26 06:22:01 mr-fox CROND[12239]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:22:01 mr-fox CROND[12239]: pam_unix(crond:session): session closed for user root
May 26 06:22:01 mr-fox CROND[7772]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:22:01 mr-fox CROND[7772]: pam_unix(crond:session): session closed for user root
May 26 06:23:00 mr-fox crond[12351]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:23:00 mr-fox crond[12349]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:23:00 mr-fox crond[12350]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:23:00 mr-fox crond[12352]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:23:00 mr-fox CROND[12358]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:23:00 mr-fox CROND[12357]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:23:00 mr-fox CROND[12359]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:23:00 mr-fox crond[12354]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:23:00 mr-fox CROND[12360]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:23:00 mr-fox CROND[12362]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:23:00 mr-fox CROND[12354]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:23:00 mr-fox CROND[12354]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:23:00 mr-fox CROND[12352]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:23:00 mr-fox CROND[12352]: pam_unix(crond:session): session closed for user root
May 26 06:23:00 mr-fox CROND[12351]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:23:00 mr-fox CROND[12351]: pam_unix(crond:session): session closed for user root
May 26 06:23:01 mr-fox CROND[12238]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:23:01 mr-fox CROND[12238]: pam_unix(crond:session): session closed for user root
May 26 06:23:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:23:44 mr-fox kernel: rcu: \x0921-....: (330025 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=131149
May 26 06:23:44 mr-fox kernel: rcu: \x09(t=330026 jiffies g=8794409 q=14267849 ncpus=32)
May 26 06:23:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:23:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:23:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:23:44 mr-fox kernel: RIP: 0010:xas_descend+0x40/0xd0
May 26 06:23:44 mr-fox kernel: Code: 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 <75> 08 48 3d fd 00 00 00 76 2f 41 88 5c 24 12 48 83 c4 08 5b 5d 41
May 26 06:23:44 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000246
May 26 06:23:44 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: 0000000000000036 RCX: 0000000000000012
May 26 06:23:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988acf90e6c8 RDI: ffffa401077e3970
May 26 06:23:44 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:23:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 06:23:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:23:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:23:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:23:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:23:44 mr-fox kernel: PKRU: 55555554
May 26 06:23:44 mr-fox kernel: Call Trace:
May 26 06:23:44 mr-fox kernel:  <IRQ>
May 26 06:23:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:23:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:23:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:23:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:23:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:23:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:23:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:23:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:23:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:23:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:23:44 mr-fox kernel:  </IRQ>
May 26 06:23:44 mr-fox kernel:  <TASK>
May 26 06:23:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:23:44 mr-fox kernel:  ? xas_descend+0x40/0xd0
May 26 06:23:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:23:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:23:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:23:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:23:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:23:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:23:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:23:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:23:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:23:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:23:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:23:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:23:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:23:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:23:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:23:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:23:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:23:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:23:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:23:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:23:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:23:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:23:44 mr-fox kernel:  </TASK>
May 26 06:23:50 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 331375 jiffies s: 491905 root: 0x2/.
May 26 06:23:50 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:23:50 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:23:50 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:23:50 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:23:50 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:23:50 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:23:50 mr-fox kernel: RIP: 0010:__build_skb_around+0x9e/0x120
May 26 06:23:50 mr-fox kernel: Code: 65 8b 15 41 c0 88 5a 66 41 89 94 24 84 00 00 00 48 01 d8 48 c7 00 00 00 00 00 48 c7 40 08 00 00 00 00 48 c7 40 10 00 00 00 00 <48> c7 40 18 00 00 00 00 c7 40 20 01 00 00 00 5b 5d 41 5c 31 c0 31
May 26 06:23:50 mr-fox kernel: RSP: 0018:ffffa401005f0da0 EFLAGS: 00000286
May 26 06:23:50 mr-fox kernel: RAX: ffff989b517b1ec0 RBX: ffff989b517b1800 RCX: 00000000ffffffff
May 26 06:23:50 mr-fox kernel: RDX: 0000000000000015 RSI: ffff989b517b1800 RDI: ffff988b1b383600
May 26 06:23:50 mr-fox kernel: RBP: 0000000000000900 R08: 0000000000000000 R09: 0000000000000000
May 26 06:23:50 mr-fox kernel: R10: ffff988ac1a460c0 R11: 0000000000000000 R12: ffff988b1b383600
May 26 06:23:50 mr-fox kernel: R13: ffffa401001b9d68 R14: ffff988ac873fb40 R15: 0000000000000007
May 26 06:23:50 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:23:50 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:23:50 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:23:50 mr-fox kernel: PKRU: 55555554
May 26 06:23:50 mr-fox kernel: Call Trace:
May 26 06:23:50 mr-fox kernel:  <NMI>
May 26 06:23:50 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:23:50 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:23:50 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:23:50 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:23:50 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:23:50 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:23:50 mr-fox kernel:  ? __build_skb_around+0x9e/0x120
May 26 06:23:50 mr-fox kernel:  ? __build_skb_around+0x9e/0x120
May 26 06:23:50 mr-fox kernel:  ? __build_skb_around+0x9e/0x120
May 26 06:23:50 mr-fox kernel:  </NMI>
May 26 06:23:50 mr-fox kernel:  <IRQ>
May 26 06:23:50 mr-fox kernel:  __napi_build_skb+0x53/0x70
May 26 06:23:50 mr-fox kernel:  napi_build_skb+0x10/0xc0
May 26 06:23:50 mr-fox kernel:  igb_poll+0xa12/0x1370
May 26 06:23:50 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:23:50 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:23:50 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:23:50 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:23:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:23:50 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:23:50 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:23:50 mr-fox kernel:  </IRQ>
May 26 06:23:50 mr-fox kernel:  <TASK>
May 26 06:23:50 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:23:50 mr-fox kernel: RIP: 0010:filemap_get_entry+0x6d/0x160
May 26 06:23:50 mr-fox kernel: Code: 00 00 48 c7 44 24 30 00 00 00 00 48 c7 44 24 38 00 00 00 00 48 c7 44 24 20 03 00 00 00 48 8d 7c 24 08 e8 56 70 78 00 48 89 c3 <48> 3d 02 04 00 00 74 e2 48 3d 06 04 00 00 74 da 48 85 c0 74 5a a8
May 26 06:23:50 mr-fox kernel: RSP: 0018:ffffa401077e3968 EFLAGS: 00000246
May 26 06:23:50 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 06:23:50 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:23:50 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 06:23:50 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:23:50 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:23:50 mr-fox kernel:  ? filemap_get_entry+0x6a/0x160
May 26 06:23:50 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:23:50 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:23:50 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:23:50 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:23:50 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:23:50 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:23:50 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:23:50 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:23:50 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:23:50 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:23:50 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:23:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:23:50 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:23:50 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:23:50 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:23:50 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:23:50 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:23:50 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:23:50 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:23:50 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:23:50 mr-fox kernel:  </TASK>
May 26 06:24:00 mr-fox crond[14213]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:24:00 mr-fox crond[14214]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:24:00 mr-fox crond[14217]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:24:00 mr-fox crond[14216]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:24:00 mr-fox crond[14218]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:24:00 mr-fox CROND[14221]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:24:00 mr-fox CROND[14222]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:24:00 mr-fox CROND[14223]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:24:00 mr-fox CROND[14225]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:24:00 mr-fox CROND[14224]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:24:00 mr-fox CROND[14218]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:24:00 mr-fox CROND[14218]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:24:00 mr-fox CROND[14217]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:24:00 mr-fox CROND[14217]: pam_unix(crond:session): session closed for user root
May 26 06:24:00 mr-fox CROND[14216]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:24:00 mr-fox CROND[14216]: pam_unix(crond:session): session closed for user root
May 26 06:24:00 mr-fox CROND[12350]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:24:00 mr-fox CROND[12350]: pam_unix(crond:session): session closed for user root
May 26 06:25:00 mr-fox crond[5448]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 06:25:00 mr-fox crond[5452]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:25:00 mr-fox crond[5449]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:25:00 mr-fox crond[5451]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:25:00 mr-fox crond[5454]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:25:00 mr-fox CROND[5462]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:25:00 mr-fox crond[5456]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:25:00 mr-fox CROND[5463]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:25:00 mr-fox CROND[5464]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:25:00 mr-fox CROND[5465]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:25:00 mr-fox CROND[5466]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:25:00 mr-fox CROND[5467]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:25:00 mr-fox crond[5453]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:25:00 mr-fox CROND[5470]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:25:00 mr-fox crond[5455]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:25:00 mr-fox CROND[5473]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 06:25:00 mr-fox CROND[5448]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:25:00 mr-fox CROND[5448]: pam_unix(crond:session): session closed for user torproject
May 26 06:25:00 mr-fox CROND[5456]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:25:00 mr-fox CROND[5456]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:25:00 mr-fox CROND[5453]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:25:00 mr-fox CROND[5453]: pam_unix(crond:session): session closed for user root
May 26 06:25:01 mr-fox CROND[5454]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:25:01 mr-fox CROND[5454]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:25:01 mr-fox CROND[14214]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:25:01 mr-fox CROND[14214]: pam_unix(crond:session): session closed for user root
May 26 06:25:01 mr-fox CROND[5452]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:25:01 mr-fox CROND[5452]: pam_unix(crond:session): session closed for user root
May 26 06:26:00 mr-fox crond[5093]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:26:00 mr-fox crond[5094]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:26:00 mr-fox crond[5092]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:26:00 mr-fox crond[5095]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:26:00 mr-fox crond[5096]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:26:00 mr-fox CROND[5100]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:26:00 mr-fox CROND[5102]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:26:00 mr-fox CROND[5101]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:26:00 mr-fox CROND[5104]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:26:00 mr-fox CROND[5103]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:26:00 mr-fox CROND[5096]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:26:00 mr-fox CROND[5096]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:26:00 mr-fox CROND[5095]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:26:00 mr-fox CROND[5095]: pam_unix(crond:session): session closed for user root
May 26 06:26:00 mr-fox CROND[5094]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:26:00 mr-fox CROND[5094]: pam_unix(crond:session): session closed for user root
May 26 06:26:01 mr-fox CROND[5451]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:26:01 mr-fox CROND[5451]: pam_unix(crond:session): session closed for user root
May 26 06:26:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:26:44 mr-fox kernel: rcu: \x0921-....: (375029 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=148473
May 26 06:26:44 mr-fox kernel: rcu: \x09(t=375030 jiffies g=8794409 q=15437943 ncpus=32)
May 26 06:26:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:26:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:26:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:26:44 mr-fox kernel: RIP: 0010:xas_descend+0x2/0xd0
May 26 06:26:44 mr-fox kernel: Code: 18 0f b6 4c 24 10 4c 8b 04 24 e9 60 fe ff ff e9 69 87 04 00 e9 a3 87 04 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 55 <41> 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80
May 26 06:26:44 mr-fox kernel: RSP: 0018:ffffa401077e3940 EFLAGS: 00000206
May 26 06:26:44 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:26:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988b1fdbb478 RDI: ffffa401077e3970
May 26 06:26:44 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 06:26:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:26:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:26:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:26:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:26:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:26:44 mr-fox kernel: PKRU: 55555554
May 26 06:26:44 mr-fox kernel: Call Trace:
May 26 06:26:44 mr-fox kernel:  <IRQ>
May 26 06:26:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:26:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:26:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:26:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:26:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:26:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:26:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:26:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:26:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:26:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:26:44 mr-fox kernel:  </IRQ>
May 26 06:26:44 mr-fox kernel:  <TASK>
May 26 06:26:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:26:44 mr-fox kernel:  ? xas_descend+0x2/0xd0
May 26 06:26:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:26:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:26:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:26:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:26:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:26:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:26:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:26:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:26:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:26:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:26:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:26:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:26:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:26:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:26:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:26:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:26:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:26:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:26:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:26:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:26:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:26:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:26:44 mr-fox kernel:  </TASK>
May 26 06:26:50 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 376432 jiffies s: 491905 root: 0x2/.
May 26 06:26:50 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:26:50 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:26:50 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:26:50 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:26:50 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:26:50 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:26:50 mr-fox kernel: RIP: 0010:iommu_v1_unmap_pages+0x84/0x140
May 26 06:26:50 mr-fox kernel: Code: 00 00 00 49 bc 8f e3 38 8e e3 38 8e e3 48 d3 e3 48 85 db 0f 84 86 00 00 00 48 89 e2 4c 89 f6 4c 89 ff e8 6f fd ff ff 49 89 c0 <48> 85 c0 74 70 48 8b 34 24 49 8d 78 08 49 c7 00 00 00 00 00 f3 48
May 26 06:26:50 mr-fox kernel: RSP: 0018:ffffa401005f0c80 EFLAGS: 00000246
May 26 06:26:50 mr-fox kernel: RAX: ffff988ac7128610 RBX: 0000000000001000 RCX: 0000000000000000
May 26 06:26:50 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:26:50 mr-fox kernel: RBP: 000000000000000c R08: ffff988ac7128610 R09: 0000000000000000
May 26 06:26:50 mr-fox kernel: R10: ffff988d24f78240 R11: 0000000000000000 R12: e38e38e38e38e38f
May 26 06:26:50 mr-fox kernel: R13: 0000000000000000 R14: 00000000cf6c2000 R15: ffff988ac2211088
May 26 06:26:50 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:26:50 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:26:50 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:26:50 mr-fox kernel: PKRU: 55555554
May 26 06:26:50 mr-fox kernel: Call Trace:
May 26 06:26:50 mr-fox kernel:  <NMI>
May 26 06:26:50 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:26:50 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:26:50 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:26:50 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:26:50 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:26:50 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:26:50 mr-fox kernel:  ? iommu_v1_unmap_pages+0x84/0x140
May 26 06:26:50 mr-fox kernel:  ? iommu_v1_unmap_pages+0x84/0x140
May 26 06:26:50 mr-fox kernel:  ? iommu_v1_unmap_pages+0x84/0x140
May 26 06:26:50 mr-fox kernel:  </NMI>
May 26 06:26:50 mr-fox kernel:  <IRQ>
May 26 06:26:50 mr-fox kernel:  amd_iommu_unmap_pages+0x40/0x130
May 26 06:26:50 mr-fox kernel:  __iommu_unmap+0xbf/0x120
May 26 06:26:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:26:50 mr-fox kernel:  __iommu_dma_unmap+0xb5/0x170
May 26 06:26:50 mr-fox kernel:  iommu_dma_unmap_page+0x43/0xa0
May 26 06:26:50 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 06:26:50 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 06:26:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:26:50 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 06:26:50 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:26:50 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:26:50 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:26:50 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:26:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:26:50 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:26:50 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:26:50 mr-fox kernel:  </IRQ>
May 26 06:26:50 mr-fox kernel:  <TASK>
May 26 06:26:50 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:26:50 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 06:26:50 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 06:26:50 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 06:26:50 mr-fox kernel: RAX: ffff988d667606da RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:26:50 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:26:50 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 06:26:50 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:26:50 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:26:50 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:26:50 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:26:50 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:26:50 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:26:50 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:26:50 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:26:50 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:26:50 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:26:50 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:26:50 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:26:50 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:26:50 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:26:50 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:26:50 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:26:50 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:26:50 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:26:50 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:26:50 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:26:50 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:26:50 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:26:50 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:26:50 mr-fox kernel:  </TASK>
May 26 06:27:00 mr-fox crond[7075]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:27:00 mr-fox crond[7077]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:27:00 mr-fox crond[7076]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:27:00 mr-fox crond[7079]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:27:00 mr-fox CROND[7082]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:27:00 mr-fox crond[7080]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:27:00 mr-fox CROND[7083]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:27:00 mr-fox CROND[7084]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:27:00 mr-fox CROND[7085]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:27:00 mr-fox CROND[7087]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:27:00 mr-fox CROND[7080]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:27:00 mr-fox CROND[7080]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:27:00 mr-fox CROND[7079]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:27:00 mr-fox CROND[7079]: pam_unix(crond:session): session closed for user root
May 26 06:27:00 mr-fox CROND[5093]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:27:00 mr-fox CROND[5093]: pam_unix(crond:session): session closed for user root
May 26 06:27:00 mr-fox CROND[7077]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:27:00 mr-fox CROND[7077]: pam_unix(crond:session): session closed for user root
May 26 06:28:00 mr-fox crond[10049]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:28:00 mr-fox crond[10048]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:28:00 mr-fox crond[10051]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:28:00 mr-fox crond[10053]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:28:00 mr-fox crond[10052]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:28:00 mr-fox CROND[10057]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:28:00 mr-fox CROND[10060]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:28:00 mr-fox CROND[10061]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:28:00 mr-fox CROND[10064]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:28:00 mr-fox CROND[10063]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:28:00 mr-fox CROND[10053]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:28:00 mr-fox CROND[10053]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:28:00 mr-fox CROND[10052]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:28:00 mr-fox CROND[10052]: pam_unix(crond:session): session closed for user root
May 26 06:28:00 mr-fox CROND[10051]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:28:00 mr-fox CROND[10051]: pam_unix(crond:session): session closed for user root
May 26 06:28:01 mr-fox CROND[7076]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:28:01 mr-fox CROND[7076]: pam_unix(crond:session): session closed for user root
May 26 06:29:00 mr-fox crond[10339]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:29:00 mr-fox crond[10338]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:29:00 mr-fox crond[10340]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:29:00 mr-fox crond[10341]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:29:00 mr-fox crond[10342]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:29:00 mr-fox CROND[10347]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:29:00 mr-fox CROND[10346]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:29:00 mr-fox CROND[10348]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:29:00 mr-fox CROND[10349]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:29:00 mr-fox CROND[10350]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:29:00 mr-fox CROND[10342]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:29:00 mr-fox CROND[10342]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:29:00 mr-fox CROND[10341]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:29:00 mr-fox CROND[10341]: pam_unix(crond:session): session closed for user root
May 26 06:29:00 mr-fox CROND[10340]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:29:00 mr-fox CROND[10340]: pam_unix(crond:session): session closed for user root
May 26 06:29:01 mr-fox CROND[10049]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:29:01 mr-fox CROND[10049]: pam_unix(crond:session): session closed for user root
May 26 06:29:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:29:44 mr-fox kernel: rcu: \x0921-....: (420033 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=165880
May 26 06:29:44 mr-fox kernel: rcu: \x09(t=420034 jiffies g=8794409 q=16531831 ncpus=32)
May 26 06:29:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:29:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:29:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:29:44 mr-fox kernel: RIP: 0010:xas_load+0x3c/0x60
May 26 06:29:44 mr-fox kernel: Code: e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe <72> e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 75 bf eb d1 66
May 26 06:29:44 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 06:29:44 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:29:44 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:29:44 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:29:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:29:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:29:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:29:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:29:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:29:44 mr-fox kernel: PKRU: 55555554
May 26 06:29:44 mr-fox kernel: Call Trace:
May 26 06:29:44 mr-fox kernel:  <IRQ>
May 26 06:29:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:29:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:29:44 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 06:29:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:29:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:29:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:29:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:29:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:29:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:29:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:29:44 mr-fox kernel:  </IRQ>
May 26 06:29:44 mr-fox kernel:  <TASK>
May 26 06:29:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:29:44 mr-fox kernel:  ? xas_load+0x3c/0x60
May 26 06:29:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:29:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:29:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:29:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:29:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:29:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:29:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:29:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:29:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:29:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:29:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:29:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:29:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:29:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:29:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:29:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:29:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:29:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:29:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:29:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:29:44 mr-fox kernel:  </TASK>
May 26 06:29:51 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 421488 jiffies s: 491905 root: 0x2/.
May 26 06:29:51 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:29:51 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:29:51 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:29:51 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:29:51 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:29:51 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:29:51 mr-fox kernel: RIP: 0010:alloc_iova_fast+0xbd/0x410
May 26 06:29:51 mr-fox kernel: Code: d2 75 1c 49 8b 41 10 48 83 38 00 0f 84 e8 01 00 00 49 89 49 10 48 89 c1 49 89 41 08 4c 8b 10 41 8d 6a ff 48 63 c5 48 83 f8 7f <72> 1d e9 69 02 00 00 85 ed 0f 84 80 00 00 00 83 ed 01 48 63 c5 48
May 26 06:29:51 mr-fox kernel: RSP: 0018:ffffa401005f09b0 EFLAGS: 00000093
May 26 06:29:51 mr-fox kernel: RAX: 000000000000005e RBX: ffff988ac1292408 RCX: ffff989089626000
May 26 06:29:51 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:29:51 mr-fox kernel: RBP: 000000000000005e R08: 00000000000fffff R09: ffff98a96ed7d8c0
May 26 06:29:51 mr-fox kernel: R10: 000000000000005f R11: ffff988afab0d570 R12: 0000000000000000
May 26 06:29:51 mr-fox kernel: R13: 000000000000000c R14: 0000000000000286 R15: ffff988ac1292408
May 26 06:29:51 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:29:51 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:29:51 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:29:51 mr-fox kernel: PKRU: 55555554
May 26 06:29:51 mr-fox kernel: Call Trace:
May 26 06:29:51 mr-fox kernel:  <NMI>
May 26 06:29:51 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:29:51 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:29:51 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:29:51 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:29:51 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:29:51 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:29:51 mr-fox kernel:  ? alloc_iova_fast+0xbd/0x410
May 26 06:29:51 mr-fox kernel:  ? alloc_iova_fast+0xbd/0x410
May 26 06:29:51 mr-fox kernel:  ? alloc_iova_fast+0xbd/0x410
May 26 06:29:51 mr-fox kernel:  </NMI>
May 26 06:29:51 mr-fox kernel:  <IRQ>
May 26 06:29:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:51 mr-fox kernel:  iommu_dma_alloc_iova+0xed/0x170
May 26 06:29:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:51 mr-fox kernel:  __iommu_dma_map+0x60/0xf0
May 26 06:29:51 mr-fox kernel:  iommu_dma_map_page+0xc2/0x230
May 26 06:29:51 mr-fox kernel:  igb_xmit_frame_ring+0x91b/0xc00
May 26 06:29:51 mr-fox kernel:  ? netif_skb_features+0x93/0x2c0
May 26 06:29:51 mr-fox kernel:  dev_hard_start_xmit+0xa0/0xf0
May 26 06:29:51 mr-fox kernel:  sch_direct_xmit+0x8d/0x290
May 26 06:29:51 mr-fox kernel:  __dev_queue_xmit+0x49a/0x9a0
May 26 06:29:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:51 mr-fox kernel:  ip_finish_output2+0x258/0x500
May 26 06:29:51 mr-fox kernel:  __ip_queue_xmit+0x16b/0x480
May 26 06:29:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:51 mr-fox kernel:  __tcp_transmit_skb+0xbad/0xd30
May 26 06:29:51 mr-fox kernel:  __tcp_retransmit_skb+0x1a9/0x800
May 26 06:29:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:51 mr-fox kernel:  ? __mod_timer+0x115/0x3b0
May 26 06:29:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:51 mr-fox kernel:  ? retransmits_timed_out.part.0+0x8d/0x170
May 26 06:29:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:51 mr-fox kernel:  tcp_retransmit_skb+0x11/0xa0
May 26 06:29:51 mr-fox kernel:  tcp_retransmit_timer+0x492/0xa60
May 26 06:29:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:51 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 06:29:51 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 06:29:51 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 06:29:51 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 06:29:51 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 06:29:51 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:29:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:51 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:29:51 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:29:51 mr-fox kernel:  </IRQ>
May 26 06:29:51 mr-fox kernel:  <TASK>
May 26 06:29:51 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:29:51 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 06:29:51 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 06:29:51 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 06:29:51 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:29:51 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:29:51 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:29:51 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:29:51 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:29:51 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:29:51 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:29:51 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:29:51 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:29:51 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:29:51 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:29:51 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:29:51 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:29:51 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:29:51 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:29:51 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:29:51 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:29:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:29:51 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:29:51 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:29:51 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:29:51 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:29:51 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:29:51 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:29:51 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:29:51 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:29:51 mr-fox kernel:  </TASK>
May 26 06:30:00 mr-fox crond[12109]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 06:30:00 mr-fox crond[12110]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:30:00 mr-fox crond[12112]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:30:00 mr-fox crond[12113]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:30:00 mr-fox crond[12114]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:30:00 mr-fox crond[12117]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:30:00 mr-fox crond[12116]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:30:00 mr-fox CROND[12121]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:30:00 mr-fox CROND[12119]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:30:00 mr-fox CROND[12122]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:30:00 mr-fox crond[12115]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:30:00 mr-fox CROND[12123]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:30:00 mr-fox CROND[12124]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:30:00 mr-fox CROND[12125]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 06:30:00 mr-fox CROND[12126]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:30:00 mr-fox CROND[12127]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:30:00 mr-fox CROND[12109]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:30:00 mr-fox CROND[12109]: pam_unix(crond:session): session closed for user torproject
May 26 06:30:00 mr-fox CROND[12117]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:30:00 mr-fox CROND[12117]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:30:00 mr-fox CROND[12114]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:30:00 mr-fox CROND[12114]: pam_unix(crond:session): session closed for user root
May 26 06:30:00 mr-fox CROND[12115]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:30:00 mr-fox CROND[12115]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:30:00 mr-fox CROND[12113]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:30:00 mr-fox CROND[12113]: pam_unix(crond:session): session closed for user root
May 26 06:30:00 mr-fox CROND[10339]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:30:00 mr-fox CROND[10339]: pam_unix(crond:session): session closed for user root
May 26 06:31:00 mr-fox crond[14743]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:31:00 mr-fox crond[14745]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:31:00 mr-fox crond[14744]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:31:00 mr-fox crond[14746]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:31:00 mr-fox CROND[14749]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:31:00 mr-fox CROND[14750]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:31:00 mr-fox CROND[14751]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:31:00 mr-fox CROND[14752]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:31:00 mr-fox crond[14747]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:31:00 mr-fox CROND[14754]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:31:00 mr-fox CROND[14747]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:31:00 mr-fox CROND[14747]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:31:00 mr-fox CROND[14746]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:31:00 mr-fox CROND[14746]: pam_unix(crond:session): session closed for user root
May 26 06:31:00 mr-fox CROND[12112]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:31:00 mr-fox CROND[12112]: pam_unix(crond:session): session closed for user root
May 26 06:31:00 mr-fox CROND[14745]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:31:00 mr-fox CROND[14745]: pam_unix(crond:session): session closed for user root
May 26 06:32:00 mr-fox crond[18459]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:32:00 mr-fox crond[18460]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:32:00 mr-fox crond[18461]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:32:00 mr-fox crond[18462]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:32:00 mr-fox CROND[18467]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:32:00 mr-fox CROND[18466]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:32:00 mr-fox CROND[18468]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:32:00 mr-fox CROND[18469]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:32:00 mr-fox crond[18458]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:32:00 mr-fox CROND[18472]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:32:00 mr-fox CROND[18462]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:32:00 mr-fox CROND[18462]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:32:00 mr-fox CROND[18461]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:32:00 mr-fox CROND[18461]: pam_unix(crond:session): session closed for user root
May 26 06:32:00 mr-fox CROND[18460]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:32:00 mr-fox CROND[18460]: pam_unix(crond:session): session closed for user root
May 26 06:32:01 mr-fox CROND[14744]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:32:01 mr-fox CROND[14744]: pam_unix(crond:session): session closed for user root
May 26 06:32:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:32:44 mr-fox kernel: rcu: \x0921-....: (465037 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=183192
May 26 06:32:44 mr-fox kernel: rcu: \x09(t=465038 jiffies g=8794409 q=17647318 ncpus=32)
May 26 06:32:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:32:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:32:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:32:44 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 06:32:44 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 06:32:44 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 06:32:44 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:32:44 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:32:44 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:32:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:32:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:32:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:32:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:32:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:32:44 mr-fox kernel: PKRU: 55555554
May 26 06:32:44 mr-fox kernel: Call Trace:
May 26 06:32:44 mr-fox kernel:  <IRQ>
May 26 06:32:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:32:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:32:44 mr-fox kernel:  ? tcp_write_xmit+0xe3/0x13b0
May 26 06:32:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:32:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:32:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:32:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:32:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:32:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:32:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:32:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:32:44 mr-fox kernel:  </IRQ>
May 26 06:32:44 mr-fox kernel:  <TASK>
May 26 06:32:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:32:44 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 06:32:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:32:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:32:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:32:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:32:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:32:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:32:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:32:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:32:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:32:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:32:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:32:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:32:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:32:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:32:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:32:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:32:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:32:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:32:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:32:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:32:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:32:44 mr-fox kernel:  </TASK>
May 26 06:32:51 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 466544 jiffies s: 491905 root: 0x2/.
May 26 06:32:51 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:32:51 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:32:51 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:32:51 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:32:51 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:32:51 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:32:51 mr-fox kernel: RIP: 0010:__const_udelay+0xc/0x40
May 26 06:32:51 mr-fox kernel: Code: 1e fa 48 8b 05 ad 26 4d 00 e9 b0 56 05 00 cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 48 8b 0d 4c c8 6b 5a <48> 85 c9 48 0f 44 0d a1 01 6b 00 48 8d 04 bd 00 00 00 00 48 89 ca
May 26 06:32:51 mr-fox kernel: RSP: 0018:ffffa401005f0c38 EFLAGS: 00000002
May 26 06:32:51 mr-fox kernel: RAX: 00000000008c370e RBX: 0000000000000002 RCX: 0000000000cf1ca0
May 26 06:32:51 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000000010c7
May 26 06:32:51 mr-fox kernel: RBP: ffffa401005f0c90 R08: 0000000000000000 R09: 0000000000000000
May 26 06:32:51 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000008c370f
May 26 06:32:51 mr-fox kernel: R13: 0000000000000046 R14: ffff988ac01b3014 R15: ffff988ac01b3000
May 26 06:32:51 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:32:51 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:32:51 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:32:51 mr-fox kernel: PKRU: 55555554
May 26 06:32:51 mr-fox kernel: Call Trace:
May 26 06:32:51 mr-fox kernel:  <NMI>
May 26 06:32:51 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:32:51 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:32:51 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:32:51 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:32:51 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:32:51 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:32:51 mr-fox kernel:  ? __const_udelay+0xc/0x40
May 26 06:32:51 mr-fox kernel:  ? __const_udelay+0xc/0x40
May 26 06:32:51 mr-fox kernel:  ? __const_udelay+0xc/0x40
May 26 06:32:51 mr-fox kernel:  </NMI>
May 26 06:32:51 mr-fox kernel:  <IRQ>
May 26 06:32:51 mr-fox kernel:  iommu_completion_wait.isra.0+0xea/0x130
May 26 06:32:51 mr-fox kernel:  amd_iommu_domain_flush_pages+0x5d/0x1a0
May 26 06:32:51 mr-fox kernel:  amd_iommu_flush_iotlb_all+0x32/0x50
May 26 06:32:51 mr-fox kernel:  fq_flush_iotlb+0x20/0x40
May 26 06:32:51 mr-fox kernel:  iommu_dma_free_iova+0x207/0x220
May 26 06:32:51 mr-fox kernel:  __iommu_dma_unmap+0xe0/0x170
May 26 06:32:51 mr-fox kernel:  iommu_dma_unmap_page+0x43/0xa0
May 26 06:32:51 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 06:32:51 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 06:32:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:32:51 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 06:32:51 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:32:51 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:32:51 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:32:51 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:32:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:32:51 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:32:51 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:32:51 mr-fox kernel:  </IRQ>
May 26 06:32:51 mr-fox kernel:  <TASK>
May 26 06:32:51 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:32:51 mr-fox kernel: RIP: 0010:xas_load+0x20/0x60
May 26 06:32:51 mr-fox kernel: Code: ff eb be 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 53 48 89 fb e8 f2 f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 <77> 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48
May 26 06:32:51 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000282
May 26 06:32:51 mr-fox kernel: RAX: ffff988d667606da RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:32:51 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:32:51 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 06:32:51 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:32:51 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:32:51 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:32:51 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:32:51 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:32:51 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:32:51 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:32:51 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:32:51 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:32:51 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:32:51 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:32:51 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:32:51 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:32:51 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:32:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:32:51 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:32:51 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:32:51 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:32:51 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:32:51 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:32:51 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:32:51 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:32:51 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:32:51 mr-fox kernel:  </TASK>
May 26 06:33:00 mr-fox crond[21170]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:33:00 mr-fox crond[21167]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:33:00 mr-fox crond[21169]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:33:00 mr-fox crond[21171]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:33:00 mr-fox crond[21172]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:33:00 mr-fox CROND[21175]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:33:00 mr-fox CROND[21174]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:33:00 mr-fox CROND[21176]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:33:00 mr-fox CROND[21177]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:33:00 mr-fox CROND[21178]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:33:00 mr-fox CROND[21172]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:33:00 mr-fox CROND[21172]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:33:00 mr-fox CROND[21171]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:33:00 mr-fox CROND[21171]: pam_unix(crond:session): session closed for user root
May 26 06:33:00 mr-fox CROND[21170]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:33:00 mr-fox CROND[21170]: pam_unix(crond:session): session closed for user root
May 26 06:33:01 mr-fox CROND[18459]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:33:01 mr-fox CROND[18459]: pam_unix(crond:session): session closed for user root
May 26 06:34:00 mr-fox crond[22271]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:34:00 mr-fox crond[22272]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:34:00 mr-fox crond[22273]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:34:00 mr-fox crond[22275]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:34:00 mr-fox crond[22276]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:34:00 mr-fox CROND[22282]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:34:00 mr-fox CROND[22279]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:34:00 mr-fox CROND[22281]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:34:00 mr-fox CROND[22280]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:34:00 mr-fox CROND[22284]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:34:00 mr-fox CROND[22276]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:34:00 mr-fox CROND[22276]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:34:00 mr-fox CROND[22275]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:34:00 mr-fox CROND[22275]: pam_unix(crond:session): session closed for user root
May 26 06:34:00 mr-fox CROND[22273]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:34:00 mr-fox CROND[22273]: pam_unix(crond:session): session closed for user root
May 26 06:34:00 mr-fox CROND[21169]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:34:00 mr-fox CROND[21169]: pam_unix(crond:session): session closed for user root
May 26 06:35:00 mr-fox crond[25960]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:35:00 mr-fox crond[25959]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:35:00 mr-fox crond[25961]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:35:00 mr-fox crond[25962]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:35:00 mr-fox crond[25958]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:35:00 mr-fox crond[25957]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 06:35:00 mr-fox CROND[25966]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:35:00 mr-fox CROND[25967]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:35:00 mr-fox CROND[25968]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:35:00 mr-fox crond[25963]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:35:00 mr-fox crond[25964]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:35:00 mr-fox CROND[25969]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:35:00 mr-fox CROND[25972]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:35:00 mr-fox CROND[25970]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:35:00 mr-fox CROND[25973]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 06:35:00 mr-fox CROND[25975]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:35:00 mr-fox CROND[25957]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:35:00 mr-fox CROND[25957]: pam_unix(crond:session): session closed for user torproject
May 26 06:35:00 mr-fox CROND[25964]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:35:00 mr-fox CROND[25964]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:35:00 mr-fox CROND[25961]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:35:00 mr-fox CROND[25961]: pam_unix(crond:session): session closed for user root
May 26 06:35:00 mr-fox CROND[25962]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:35:00 mr-fox CROND[25962]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:35:00 mr-fox CROND[22272]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:35:00 mr-fox CROND[22272]: pam_unix(crond:session): session closed for user root
May 26 06:35:00 mr-fox CROND[25960]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:35:00 mr-fox CROND[25960]: pam_unix(crond:session): session closed for user root
May 26 06:35:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:35:44 mr-fox kernel: rcu: \x0921-....: (510040 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=200450
May 26 06:35:44 mr-fox kernel: rcu: \x09(t=510041 jiffies g=8794409 q=18753758 ncpus=32)
May 26 06:35:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:35:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:35:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:35:44 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 06:35:44 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 06:35:44 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 06:35:44 mr-fox kernel: RAX: ffff988ac07566ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:35:44 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:35:44 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:35:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:35:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:35:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:35:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:35:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:35:44 mr-fox kernel: PKRU: 55555554
May 26 06:35:44 mr-fox kernel: Call Trace:
May 26 06:35:44 mr-fox kernel:  <IRQ>
May 26 06:35:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:35:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:35:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:35:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:35:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:35:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:35:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:35:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:35:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:35:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:35:44 mr-fox kernel:  </IRQ>
May 26 06:35:44 mr-fox kernel:  <TASK>
May 26 06:35:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:35:44 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 06:35:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:35:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:35:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:35:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:35:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:35:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:35:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:35:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:35:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:35:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:35:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:35:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:35:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:35:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:35:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:35:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:35:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:35:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:35:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:35:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:35:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:35:44 mr-fox kernel:  </TASK>
May 26 06:35:51 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 511599 jiffies s: 491905 root: 0x2/.
May 26 06:35:51 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:35:51 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:35:51 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:35:51 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:35:51 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:35:51 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:35:51 mr-fox kernel: RIP: 0010:iommu_dma_map_page+0x93/0x230
May 26 06:35:51 mr-fox kernel: Code: 85 c0 44 0f 44 d0 45 89 d4 4c 89 f7 41 bf ff ff ff ff e8 50 e8 ff ff 48 8b 50 40 49 8b 86 08 02 00 00 48 85 c0 74 0f 4c 8b 38 <b8> ff ff ff ff 4d 85 ff 4c 0f 44 f8 49 81 7e 60 20 e7 43 a6 0f 84
May 26 06:35:51 mr-fox kernel: RSP: 0018:ffffa401005f0ab0 EFLAGS: 00000286
May 26 06:35:51 mr-fox kernel: RAX: ffff988ac1a46088 RBX: 00000004a131a6be RCX: 0000000000000042
May 26 06:35:51 mr-fox kernel: RDX: ffff988ac1292400 RSI: 000000001284c680 RDI: 0000000000000000
May 26 06:35:51 mr-fox kernel: RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000
May 26 06:35:51 mr-fox kernel: R10: ffff988c735087d8 R11: ffffa401001219e0 R12: 0000000000000005
May 26 06:35:51 mr-fox kernel: R13: 0000000000000000 R14: ffff988ac1a460c0 R15: ffffffffffffffff
May 26 06:35:51 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:35:51 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:35:51 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:35:51 mr-fox kernel: PKRU: 55555554
May 26 06:35:51 mr-fox kernel: Call Trace:
May 26 06:35:51 mr-fox kernel:  <NMI>
May 26 06:35:51 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:35:51 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:35:51 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:35:51 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:35:51 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:35:51 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:35:51 mr-fox kernel:  ? iommu_dma_map_page+0x93/0x230
May 26 06:35:51 mr-fox kernel:  ? iommu_dma_map_page+0x93/0x230
May 26 06:35:51 mr-fox kernel:  ? iommu_dma_map_page+0x93/0x230
May 26 06:35:51 mr-fox kernel:  </NMI>
May 26 06:35:51 mr-fox kernel:  <IRQ>
May 26 06:35:51 mr-fox kernel:  igb_xmit_frame_ring+0x2d7/0xc00
May 26 06:35:51 mr-fox kernel:  ? netif_skb_features+0x93/0x2c0
May 26 06:35:51 mr-fox kernel:  dev_hard_start_xmit+0xa0/0xf0
May 26 06:35:51 mr-fox kernel:  sch_direct_xmit+0x8d/0x290
May 26 06:35:51 mr-fox kernel:  __dev_queue_xmit+0x49a/0x9a0
May 26 06:35:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:35:51 mr-fox kernel:  ip_finish_output2+0x258/0x500
May 26 06:35:51 mr-fox kernel:  __ip_queue_xmit+0x16b/0x480
May 26 06:35:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:35:51 mr-fox kernel:  __tcp_transmit_skb+0xbad/0xd30
May 26 06:35:51 mr-fox kernel:  __tcp_retransmit_skb+0x1a9/0x800
May 26 06:35:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:35:51 mr-fox kernel:  ? __mod_timer+0x115/0x3b0
May 26 06:35:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:35:51 mr-fox kernel:  ? retransmits_timed_out.part.0+0x8d/0x170
May 26 06:35:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:35:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:35:51 mr-fox kernel:  tcp_retransmit_skb+0x11/0xa0
May 26 06:35:51 mr-fox kernel:  tcp_retransmit_timer+0x492/0xa60
May 26 06:35:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:35:51 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 06:35:51 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 06:35:51 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 06:35:51 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 06:35:51 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 06:35:51 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:35:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:35:51 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:35:51 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:35:51 mr-fox kernel:  </IRQ>
May 26 06:35:51 mr-fox kernel:  <TASK>
May 26 06:35:51 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:35:51 mr-fox kernel: RIP: 0010:xas_load+0xe/0x60
May 26 06:35:51 mr-fox kernel: Code: 00 00 eb 92 4c 89 ee 48 c7 c7 e0 eb 49 a6 e8 a9 58 c6 ff eb be 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 53 48 89 fb e8 f2 f7 ff ff <48> 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d
May 26 06:35:51 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 06:35:51 mr-fox kernel: RAX: ffff988ac07566ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:35:51 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:35:51 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:35:51 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:35:51 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:35:51 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:35:51 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:35:51 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:35:51 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:35:51 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:35:51 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:35:51 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:35:51 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:35:51 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:35:51 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:35:51 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:35:51 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:35:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:35:51 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:35:51 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:35:51 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:35:51 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:35:51 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:35:51 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:35:51 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:35:51 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:35:51 mr-fox kernel:  </TASK>
May 26 06:36:00 mr-fox crond[27414]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:36:00 mr-fox crond[27413]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:36:00 mr-fox crond[27412]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:36:00 mr-fox crond[27417]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:36:00 mr-fox CROND[27420]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:36:00 mr-fox CROND[27419]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:36:00 mr-fox CROND[27421]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:36:00 mr-fox crond[27415]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:36:00 mr-fox CROND[27422]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:36:00 mr-fox CROND[27424]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:36:00 mr-fox CROND[27417]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:36:00 mr-fox CROND[27417]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:36:00 mr-fox CROND[27415]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:36:00 mr-fox CROND[27415]: pam_unix(crond:session): session closed for user root
May 26 06:36:01 mr-fox CROND[27414]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:36:01 mr-fox CROND[27414]: pam_unix(crond:session): session closed for user root
May 26 06:36:01 mr-fox CROND[25959]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:36:01 mr-fox CROND[25959]: pam_unix(crond:session): session closed for user root
May 26 06:37:00 mr-fox crond[28903]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:37:00 mr-fox crond[28906]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:37:00 mr-fox crond[28904]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:37:00 mr-fox crond[28907]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:37:00 mr-fox CROND[28918]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:37:00 mr-fox CROND[28914]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:37:00 mr-fox CROND[28920]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:37:00 mr-fox CROND[28921]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:37:00 mr-fox crond[28908]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:37:00 mr-fox CROND[28925]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:37:00 mr-fox CROND[28908]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:37:00 mr-fox CROND[28908]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:37:00 mr-fox CROND[28907]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:37:00 mr-fox CROND[28907]: pam_unix(crond:session): session closed for user root
May 26 06:37:00 mr-fox CROND[28906]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:37:00 mr-fox CROND[28906]: pam_unix(crond:session): session closed for user root
May 26 06:37:01 mr-fox CROND[27413]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:37:01 mr-fox CROND[27413]: pam_unix(crond:session): session closed for user root
May 26 06:38:00 mr-fox crond[29652]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:38:00 mr-fox crond[29654]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:38:00 mr-fox CROND[29660]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:38:00 mr-fox crond[29655]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:38:00 mr-fox CROND[29661]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:38:00 mr-fox crond[29656]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:38:00 mr-fox CROND[29665]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:38:00 mr-fox crond[29657]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:38:00 mr-fox CROND[29666]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:38:00 mr-fox CROND[29667]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:38:00 mr-fox CROND[29657]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:38:00 mr-fox CROND[29657]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:38:00 mr-fox CROND[29656]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:38:00 mr-fox CROND[29656]: pam_unix(crond:session): session closed for user root
May 26 06:38:00 mr-fox CROND[29655]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:38:00 mr-fox CROND[29655]: pam_unix(crond:session): session closed for user root
May 26 06:38:00 mr-fox CROND[28904]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:38:00 mr-fox CROND[28904]: pam_unix(crond:session): session closed for user root
May 26 06:38:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:38:44 mr-fox kernel: rcu: \x0921-....: (555044 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=217679
May 26 06:38:44 mr-fox kernel: rcu: \x09(t=555045 jiffies g=8794409 q=19873872 ncpus=32)
May 26 06:38:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:38:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:38:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:38:44 mr-fox kernel: RIP: 0010:xas_load+0x35/0x60
May 26 06:38:44 mr-fox kernel: Code: f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 <48> 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d
May 26 06:38:44 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000282
May 26 06:38:44 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:38:44 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:38:44 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:38:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:38:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:38:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:38:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:38:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:38:44 mr-fox kernel: PKRU: 55555554
May 26 06:38:44 mr-fox kernel: Call Trace:
May 26 06:38:44 mr-fox kernel:  <IRQ>
May 26 06:38:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:38:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:38:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:38:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:38:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:38:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:38:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:38:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:38:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:38:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:38:44 mr-fox kernel:  </IRQ>
May 26 06:38:44 mr-fox kernel:  <TASK>
May 26 06:38:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:38:44 mr-fox kernel:  ? xas_load+0x35/0x60
May 26 06:38:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:38:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:38:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:38:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:38:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:38:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:38:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:38:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:38:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:38:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:38:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:38:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:38:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:38:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:38:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:38:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:38:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:38:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:38:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:38:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:38:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:38:44 mr-fox kernel:  </TASK>
May 26 06:38:51 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 556655 jiffies s: 491905 root: 0x2/.
May 26 06:38:51 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:38:51 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:38:51 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:38:51 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:38:51 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:38:51 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:38:51 mr-fox kernel: RIP: 0010:skb_release_head_state+0x39/0x80
May 26 06:38:51 mr-fox kernel: Code: 75 3d 48 8b 43 60 48 85 c0 74 08 48 89 df e8 6e a1 21 00 48 8b 7b 68 48 83 ff 07 76 16 48 83 e7 f8 b8 ff ff ff ff f0 0f c1 07 <83> f8 01 74 20 85 c0 7e 22 5b 31 c0 31 f6 31 ff e9 0d ca 36 00 40
May 26 06:38:51 mr-fox kernel: RSP: 0018:ffffa401005f0dd8 EFLAGS: 00000213
May 26 06:38:51 mr-fox kernel: RAX: 0000000000000002 RBX: ffff988b81979600 RCX: 0000000000000000
May 26 06:38:51 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9893aa567300
May 26 06:38:51 mr-fox kernel: RBP: 0000000000007828 R08: 0000000000000000 R09: 0000000000000000
May 26 06:38:51 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffa401001197c0
May 26 06:38:51 mr-fox kernel: R13: ffff988ac873fa40 R14: 00000000ffffff7c R15: ffffa401001197d0
May 26 06:38:51 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:38:51 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:38:51 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:38:51 mr-fox kernel: PKRU: 55555554
May 26 06:38:51 mr-fox kernel: Call Trace:
May 26 06:38:51 mr-fox kernel:  <NMI>
May 26 06:38:51 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:38:51 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:38:51 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:38:51 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:38:51 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:38:51 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:38:51 mr-fox kernel:  ? skb_release_head_state+0x39/0x80
May 26 06:38:51 mr-fox kernel:  ? skb_release_head_state+0x39/0x80
May 26 06:38:51 mr-fox kernel:  ? skb_release_head_state+0x39/0x80
May 26 06:38:51 mr-fox kernel:  </NMI>
May 26 06:38:51 mr-fox kernel:  <IRQ>
May 26 06:38:51 mr-fox kernel:  napi_consume_skb+0x2e/0xc0
May 26 06:38:51 mr-fox kernel:  igb_poll+0xea/0x1370
May 26 06:38:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:38:51 mr-fox kernel:  ? task_tick_fair+0x85/0x470
May 26 06:38:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:38:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:38:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:38:51 mr-fox kernel:  ? wq_worker_tick+0xd/0xd0
May 26 06:38:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:38:51 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:38:51 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:38:51 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:38:51 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:38:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:38:51 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:38:51 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:38:51 mr-fox kernel:  </IRQ>
May 26 06:38:51 mr-fox kernel:  <TASK>
May 26 06:38:51 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:38:51 mr-fox kernel: RIP: 0010:xas_descend+0x31/0xd0
May 26 06:38:51 mr-fox kernel: Code: 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 <49> 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d fd 00 00
May 26 06:38:51 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 06:38:51 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: 0000000000000036 RCX: 0000000000000012
May 26 06:38:51 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988acf90e6c8 RDI: ffffa401077e3970
May 26 06:38:51 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:38:51 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 06:38:51 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:38:51 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:38:51 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:38:51 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:38:51 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:38:51 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:38:51 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:38:51 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:38:51 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:38:51 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:38:51 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:38:51 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:38:51 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:38:51 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:38:51 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:38:51 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:38:51 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:38:51 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:38:51 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:38:51 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:38:51 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:38:51 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:38:51 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:38:51 mr-fox kernel:  </TASK>
May 26 06:39:00 mr-fox crond[32009]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:39:00 mr-fox crond[32011]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:39:00 mr-fox crond[32010]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:39:00 mr-fox crond[32012]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:39:00 mr-fox crond[32007]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:39:00 mr-fox CROND[32016]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:39:00 mr-fox CROND[32015]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:39:00 mr-fox CROND[32017]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:39:00 mr-fox CROND[32018]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:39:00 mr-fox CROND[32019]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:39:00 mr-fox CROND[32012]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:39:00 mr-fox CROND[32012]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:39:00 mr-fox CROND[32011]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:39:00 mr-fox CROND[32011]: pam_unix(crond:session): session closed for user root
May 26 06:39:01 mr-fox CROND[29654]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:39:01 mr-fox CROND[29654]: pam_unix(crond:session): session closed for user root
May 26 06:39:01 mr-fox CROND[32010]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:39:01 mr-fox CROND[32010]: pam_unix(crond:session): session closed for user root
May 26 06:40:00 mr-fox crond[31870]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:40:00 mr-fox crond[31869]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 06:40:00 mr-fox crond[31872]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:40:00 mr-fox crond[31871]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:40:00 mr-fox crond[31875]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:40:00 mr-fox CROND[31880]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:40:00 mr-fox CROND[31881]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:40:00 mr-fox crond[31874]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:40:00 mr-fox CROND[31882]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:40:00 mr-fox crond[31877]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:40:00 mr-fox CROND[31884]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:40:00 mr-fox crond[31876]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:40:00 mr-fox CROND[31885]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:40:00 mr-fox CROND[31886]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:40:00 mr-fox CROND[31887]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:40:00 mr-fox CROND[31888]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 06:40:00 mr-fox CROND[31869]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:40:00 mr-fox CROND[31869]: pam_unix(crond:session): session closed for user torproject
May 26 06:40:00 mr-fox CROND[31877]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:40:00 mr-fox CROND[31877]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:40:00 mr-fox CROND[31874]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:40:00 mr-fox CROND[31874]: pam_unix(crond:session): session closed for user root
May 26 06:40:00 mr-fox CROND[31875]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:40:00 mr-fox CROND[31875]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:40:00 mr-fox CROND[31872]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:40:00 mr-fox CROND[31872]: pam_unix(crond:session): session closed for user root
May 26 06:40:01 mr-fox CROND[32009]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:40:01 mr-fox CROND[32009]: pam_unix(crond:session): session closed for user root
May 26 06:41:00 mr-fox crond[2750]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:41:00 mr-fox crond[2752]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:41:00 mr-fox crond[2751]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:41:00 mr-fox crond[2749]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:41:00 mr-fox crond[2753]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:41:00 mr-fox CROND[2757]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:41:00 mr-fox CROND[2756]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:41:00 mr-fox CROND[2758]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:41:00 mr-fox CROND[2759]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:41:00 mr-fox CROND[2762]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:41:00 mr-fox CROND[2753]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:41:00 mr-fox CROND[2753]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:41:00 mr-fox CROND[2752]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:41:00 mr-fox CROND[2752]: pam_unix(crond:session): session closed for user root
May 26 06:41:00 mr-fox CROND[2751]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:41:00 mr-fox CROND[2751]: pam_unix(crond:session): session closed for user root
May 26 06:41:00 mr-fox CROND[31871]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:41:00 mr-fox CROND[31871]: pam_unix(crond:session): session closed for user root
May 26 06:41:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:41:44 mr-fox kernel: rcu: \x0921-....: (600048 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=234922
May 26 06:41:44 mr-fox kernel: rcu: \x09(t=600049 jiffies g=8794409 q=21060947 ncpus=32)
May 26 06:41:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:41:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:41:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:41:44 mr-fox kernel: RIP: 0010:xas_descend+0xc/0xd0
May 26 06:41:44 mr-fox kernel: Code: e9 60 fe ff ff e9 69 87 04 00 e9 a3 87 04 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 <48> 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3
May 26 06:41:44 mr-fox kernel: RSP: 0018:ffffa401077e3928 EFLAGS: 00000206
May 26 06:41:44 mr-fox kernel: RAX: ffff988ac07566ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:41:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac07566c8 RDI: ffffa401077e3970
May 26 06:41:44 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:41:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 06:41:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:41:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:41:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:41:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:41:44 mr-fox kernel: PKRU: 55555554
May 26 06:41:44 mr-fox kernel: Call Trace:
May 26 06:41:44 mr-fox kernel:  <IRQ>
May 26 06:41:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:41:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:41:44 mr-fox kernel:  ? tcp_write_xmit+0xe3/0x13b0
May 26 06:41:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:41:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:41:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:41:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:41:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:41:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:41:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:41:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:41:44 mr-fox kernel:  </IRQ>
May 26 06:41:44 mr-fox kernel:  <TASK>
May 26 06:41:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:41:44 mr-fox kernel:  ? xas_descend+0xc/0xd0
May 26 06:41:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:41:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:41:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:41:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:41:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:41:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:41:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:41:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:41:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:41:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:41:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:41:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:41:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:41:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:41:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:41:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:41:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:41:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:41:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:41:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:41:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:41:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:41:44 mr-fox kernel:  </TASK>
May 26 06:41:52 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 601711 jiffies s: 491905 root: 0x2/.
May 26 06:41:52 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:41:52 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:41:52 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:41:52 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:41:52 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:41:52 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:41:52 mr-fox kernel: RIP: 0010:fetch_pte+0x145/0x170
May 26 06:41:52 mr-fox kernel: Code: 74 25 48 8b 10 f6 c2 01 75 81 31 c0 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f 31 d2 31 c9 31 f6 31 ff e9 29 18 45 00 48 8b 38 <81> e7 00 0e 00 00 48 81 ff 00 0e 00 00 75 d3 48 83 c4 10 48 89 ee
May 26 06:41:52 mr-fox kernel: RSP: 0018:ffffa401005f0d58 EFLAGS: 00000246
May 26 06:41:52 mr-fox kernel: RAX: ffff988ceccfa9f0 RBX: 0000000000000000 RCX: 0000000000000003
May 26 06:41:52 mr-fox kernel: RDX: 0000000000001000 RSI: 0000000000000001 RDI: 300000011af3a001
May 26 06:41:52 mr-fox kernel: RBP: ffffa401005f0da0 R08: 0000000000000000 R09: 0000000000000000
May 26 06:41:52 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000ce93e8ea
May 26 06:41:52 mr-fox kernel: R13: 000ffffffffff000 R14: 0000000000000003 R15: 0000000000000000
May 26 06:41:52 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:41:52 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:41:52 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:41:52 mr-fox kernel: PKRU: 55555554
May 26 06:41:52 mr-fox kernel: Call Trace:
May 26 06:41:52 mr-fox kernel:  <NMI>
May 26 06:41:52 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:41:52 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:41:52 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:41:52 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:41:52 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:41:52 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:41:52 mr-fox kernel:  ? fetch_pte+0x145/0x170
May 26 06:41:52 mr-fox kernel:  ? fetch_pte+0x145/0x170
May 26 06:41:52 mr-fox kernel:  ? fetch_pte+0x145/0x170
May 26 06:41:52 mr-fox kernel:  </NMI>
May 26 06:41:52 mr-fox kernel:  <IRQ>
May 26 06:41:52 mr-fox kernel:  iommu_v1_iova_to_phys+0x2b/0xa0
May 26 06:41:52 mr-fox kernel:  iommu_dma_unmap_page+0x2d/0xa0
May 26 06:41:52 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 06:41:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:41:52 mr-fox kernel:  ? task_tick_fair+0x85/0x470
May 26 06:41:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:41:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:41:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:41:52 mr-fox kernel:  ? wq_worker_tick+0xd/0xd0
May 26 06:41:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:41:52 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:41:52 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:41:52 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:41:52 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:41:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:41:52 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:41:52 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:41:52 mr-fox kernel:  </IRQ>
May 26 06:41:52 mr-fox kernel:  <TASK>
May 26 06:41:52 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:41:52 mr-fox kernel: RIP: 0010:filemap_get_entry+0x6d/0x160
May 26 06:41:52 mr-fox kernel: Code: 00 00 48 c7 44 24 30 00 00 00 00 48 c7 44 24 38 00 00 00 00 48 c7 44 24 20 03 00 00 00 48 8d 7c 24 08 e8 56 70 78 00 48 89 c3 <48> 3d 02 04 00 00 74 e2 48 3d 06 04 00 00 74 da 48 85 c0 74 5a a8
May 26 06:41:52 mr-fox kernel: RSP: 0018:ffffa401077e3968 EFLAGS: 00000246
May 26 06:41:52 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 06:41:52 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:41:52 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 06:41:52 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:41:52 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:41:52 mr-fox kernel:  ? filemap_get_entry+0x6a/0x160
May 26 06:41:52 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:41:52 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:41:52 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:41:52 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:41:52 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:41:52 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:41:52 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:41:52 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:41:52 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:41:52 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:41:52 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:41:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:41:52 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:41:52 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:41:52 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:41:52 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:41:52 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:41:52 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:41:52 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:41:52 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:41:52 mr-fox kernel:  </TASK>
May 26 06:42:00 mr-fox crond[4517]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:42:00 mr-fox crond[4519]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:42:00 mr-fox crond[4518]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:42:00 mr-fox crond[4521]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:42:00 mr-fox crond[4520]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:42:00 mr-fox CROND[4525]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:42:00 mr-fox CROND[4524]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:42:00 mr-fox CROND[4526]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:42:00 mr-fox CROND[4527]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:42:00 mr-fox CROND[4529]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:42:00 mr-fox CROND[4521]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:42:00 mr-fox CROND[4521]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:42:00 mr-fox CROND[4520]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:42:00 mr-fox CROND[4520]: pam_unix(crond:session): session closed for user root
May 26 06:42:00 mr-fox CROND[4519]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:42:00 mr-fox CROND[4519]: pam_unix(crond:session): session closed for user root
May 26 06:42:01 mr-fox CROND[2750]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:42:01 mr-fox CROND[2750]: pam_unix(crond:session): session closed for user root
May 26 06:43:00 mr-fox crond[3635]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:43:00 mr-fox crond[3637]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:43:00 mr-fox crond[3639]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:43:00 mr-fox crond[3640]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:43:00 mr-fox CROND[3643]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:43:00 mr-fox CROND[3644]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:43:00 mr-fox CROND[3646]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:43:00 mr-fox crond[3638]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:43:00 mr-fox CROND[3649]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:43:00 mr-fox CROND[3647]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:43:00 mr-fox CROND[3640]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:43:00 mr-fox CROND[3640]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:43:00 mr-fox CROND[3639]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:43:00 mr-fox CROND[3639]: pam_unix(crond:session): session closed for user root
May 26 06:43:00 mr-fox CROND[3638]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:43:00 mr-fox CROND[3638]: pam_unix(crond:session): session closed for user root
May 26 06:43:01 mr-fox CROND[4518]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:43:01 mr-fox CROND[4518]: pam_unix(crond:session): session closed for user root
May 26 06:44:00 mr-fox crond[4346]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:44:00 mr-fox crond[4344]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:44:00 mr-fox crond[4347]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:44:00 mr-fox crond[4345]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:44:00 mr-fox crond[4348]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:44:00 mr-fox CROND[4354]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:44:00 mr-fox CROND[4353]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:44:00 mr-fox CROND[4352]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:44:00 mr-fox CROND[4356]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:44:00 mr-fox CROND[4355]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:44:00 mr-fox CROND[4348]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:44:00 mr-fox CROND[4348]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:44:00 mr-fox CROND[4347]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:44:00 mr-fox CROND[4347]: pam_unix(crond:session): session closed for user root
May 26 06:44:00 mr-fox CROND[4346]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:44:00 mr-fox CROND[4346]: pam_unix(crond:session): session closed for user root
May 26 06:44:00 mr-fox CROND[3637]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:44:00 mr-fox CROND[3637]: pam_unix(crond:session): session closed for user root
May 26 06:44:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:44:44 mr-fox kernel: rcu: \x0921-....: (645052 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=252150
May 26 06:44:44 mr-fox kernel: rcu: \x09(t=645053 jiffies g=8794409 q=22264427 ncpus=32)
May 26 06:44:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:44:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:44:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:44:44 mr-fox kernel: RIP: 0010:xas_descend+0x26/0xd0
May 26 06:44:44 mr-fox kernel: Code: 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f <89> d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03
May 26 06:44:44 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 06:44:44 mr-fox kernel: RAX: ffff988ac07566ca RBX: 000000000000000c RCX: 0000000000000018
May 26 06:44:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac07566c8 RDI: ffffa401077e3970
May 26 06:44:44 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:44:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 06:44:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:44:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:44:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:44:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:44:44 mr-fox kernel: PKRU: 55555554
May 26 06:44:44 mr-fox kernel: Call Trace:
May 26 06:44:44 mr-fox kernel:  <IRQ>
May 26 06:44:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:44:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:44:44 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 06:44:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:44:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:44:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:44:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:44:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:44:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:44:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:44:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:44:44 mr-fox kernel:  </IRQ>
May 26 06:44:44 mr-fox kernel:  <TASK>
May 26 06:44:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:44:44 mr-fox kernel:  ? xas_descend+0x26/0xd0
May 26 06:44:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:44:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:44:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:44:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:44:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:44:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:44:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:44:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:44:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:44:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:44:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:44:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:44:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:44:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:44:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:44:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:44:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:44:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:44:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:44:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:44:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:44:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:44:44 mr-fox kernel:  </TASK>
May 26 06:44:52 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 646768 jiffies s: 491905 root: 0x2/.
May 26 06:44:52 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:44:52 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:44:52 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:44:52 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:44:52 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:44:52 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:44:52 mr-fox kernel: RIP: 0010:__dev_queue_xmit+0x419/0x9a0
May 26 06:44:52 mr-fox kernel: Code: a8 04 0f 84 ac 01 00 00 49 8b 84 24 d8 00 00 00 a8 0c 0f 85 9c 01 00 00 4d 8d ac 24 44 01 00 00 4c 89 ef e8 99 9e 1f 00 85 c0 <75> 21 f0 49 0f ba ac 24 d8 00 00 00 02 0f 82 77 01 00 00 4c 89 ef
May 26 06:44:52 mr-fox kernel: RSP: 0018:ffffa401005f0ad0 EFLAGS: 00000202
May 26 06:44:52 mr-fox kernel: RAX: 0000000000000001 RBX: ffff988c73508ed8 RCX: 0000000000000000
May 26 06:44:52 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:44:52 mr-fox kernel: RBP: ffff988ac76ec3c0 R08: 0000000000000000 R09: ffff988af428b200
May 26 06:44:52 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff988ac5507400
May 26 06:44:52 mr-fox kernel: R13: ffff988ac5507544 R14: 0000000000000010 R15: ffff988ac4bb4000
May 26 06:44:52 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:44:52 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:44:52 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:44:52 mr-fox kernel: PKRU: 55555554
May 26 06:44:52 mr-fox kernel: Call Trace:
May 26 06:44:52 mr-fox kernel:  <NMI>
May 26 06:44:52 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:44:52 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:44:52 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:44:52 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:44:52 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:44:52 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:44:52 mr-fox kernel:  ? __dev_queue_xmit+0x419/0x9a0
May 26 06:44:52 mr-fox kernel:  ? __dev_queue_xmit+0x419/0x9a0
May 26 06:44:52 mr-fox kernel:  ? __dev_queue_xmit+0x419/0x9a0
May 26 06:44:52 mr-fox kernel:  </NMI>
May 26 06:44:52 mr-fox kernel:  <IRQ>
May 26 06:44:52 mr-fox kernel:  ? ip6t_do_table+0x30b/0x590
May 26 06:44:52 mr-fox kernel:  ip6_finish_output2+0x2c0/0x610
May 26 06:44:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:44:52 mr-fox kernel:  ? ip6_output+0xa7/0x290
May 26 06:44:52 mr-fox kernel:  ip6_xmit+0x3fc/0x600
May 26 06:44:52 mr-fox kernel:  ? ip6_output+0x290/0x290
May 26 06:44:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:44:52 mr-fox kernel:  ? __sk_dst_check+0x34/0xa0
May 26 06:44:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:44:52 mr-fox kernel:  ? inet6_csk_route_socket+0x132/0x210
May 26 06:44:52 mr-fox kernel:  inet6_csk_xmit+0xe9/0x160
May 26 06:44:52 mr-fox kernel:  __tcp_transmit_skb+0x5d0/0xd30
May 26 06:44:52 mr-fox kernel:  tcp_write_xmit+0x4c9/0x13b0
May 26 06:44:52 mr-fox kernel:  ? ktime_get+0x42/0xb0
May 26 06:44:52 mr-fox kernel:  tcp_send_loss_probe+0x16f/0x260
May 26 06:44:52 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 06:44:52 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 06:44:52 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 06:44:52 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 06:44:52 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 06:44:52 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:44:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:44:52 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:44:52 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:44:52 mr-fox kernel:  </IRQ>
May 26 06:44:52 mr-fox kernel:  <TASK>
May 26 06:44:52 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:44:52 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 06:44:52 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 06:44:52 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 06:44:52 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:44:52 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:44:52 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:44:52 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:44:52 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:44:52 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:44:52 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:44:52 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:44:52 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:44:52 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:44:52 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:44:52 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:44:52 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:44:52 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:44:52 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:44:52 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:44:52 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:44:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:44:52 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:44:52 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:44:52 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:44:52 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:44:52 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:44:52 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:44:52 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:44:52 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:44:52 mr-fox kernel:  </TASK>
May 26 06:45:00 mr-fox crond[7947]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 06:45:00 mr-fox crond[7950]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:45:00 mr-fox crond[7952]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:45:00 mr-fox crond[7951]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:45:00 mr-fox crond[7949]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:45:00 mr-fox CROND[7955]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:45:00 mr-fox crond[7953]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:45:00 mr-fox CROND[7956]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:45:00 mr-fox crond[7954]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:45:00 mr-fox crond[7948]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:45:00 mr-fox CROND[7957]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:45:00 mr-fox CROND[7958]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:45:00 mr-fox CROND[7959]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:45:00 mr-fox CROND[7960]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 06:45:00 mr-fox CROND[7962]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:45:00 mr-fox CROND[7961]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:45:00 mr-fox CROND[7947]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:45:00 mr-fox CROND[7947]: pam_unix(crond:session): session closed for user torproject
May 26 06:45:00 mr-fox CROND[7954]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:45:00 mr-fox CROND[7954]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:45:00 mr-fox CROND[7951]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:45:00 mr-fox CROND[7951]: pam_unix(crond:session): session closed for user root
May 26 06:45:00 mr-fox CROND[7952]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:45:00 mr-fox CROND[7952]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:45:00 mr-fox CROND[7950]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:45:00 mr-fox CROND[7950]: pam_unix(crond:session): session closed for user root
May 26 06:45:00 mr-fox CROND[4345]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:45:00 mr-fox CROND[4345]: pam_unix(crond:session): session closed for user root
May 26 06:46:00 mr-fox crond[31751]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:46:00 mr-fox crond[31754]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:46:00 mr-fox crond[31752]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:46:00 mr-fox CROND[31759]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:46:00 mr-fox crond[31753]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:46:00 mr-fox CROND[31760]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:46:00 mr-fox crond[31755]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:46:00 mr-fox CROND[31761]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:46:00 mr-fox CROND[31764]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:46:00 mr-fox CROND[31765]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:46:00 mr-fox CROND[31755]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:46:00 mr-fox CROND[31755]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:46:00 mr-fox CROND[31754]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:46:00 mr-fox CROND[31754]: pam_unix(crond:session): session closed for user root
May 26 06:46:00 mr-fox CROND[31753]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:46:00 mr-fox CROND[31753]: pam_unix(crond:session): session closed for user root
May 26 06:46:01 mr-fox CROND[7949]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:46:01 mr-fox CROND[7949]: pam_unix(crond:session): session closed for user root
May 26 06:47:00 mr-fox crond[32257]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:47:00 mr-fox crond[32258]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:47:00 mr-fox crond[32259]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:47:00 mr-fox crond[32260]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:47:00 mr-fox CROND[32265]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:47:00 mr-fox CROND[32266]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:47:00 mr-fox crond[32261]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:47:00 mr-fox CROND[32268]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:47:00 mr-fox CROND[32269]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:47:00 mr-fox CROND[32270]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:47:00 mr-fox CROND[32261]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:47:00 mr-fox CROND[32261]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:47:00 mr-fox CROND[32260]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:47:00 mr-fox CROND[32260]: pam_unix(crond:session): session closed for user root
May 26 06:47:00 mr-fox CROND[32259]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:47:00 mr-fox CROND[32259]: pam_unix(crond:session): session closed for user root
May 26 06:47:01 mr-fox CROND[31752]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:47:01 mr-fox CROND[31752]: pam_unix(crond:session): session closed for user root
May 26 06:47:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:47:44 mr-fox kernel: rcu: \x0921-....: (690056 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=269452
May 26 06:47:44 mr-fox kernel: rcu: \x09(t=690057 jiffies g=8794409 q=23557866 ncpus=32)
May 26 06:47:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:47:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:47:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:47:44 mr-fox kernel: RIP: 0010:xas_descend+0x31/0xd0
May 26 06:47:44 mr-fox kernel: Code: 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 <49> 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d fd 00 00
May 26 06:47:44 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 06:47:44 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: 0000000000000036 RCX: 0000000000000012
May 26 06:47:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988acf90e6c8 RDI: ffffa401077e3970
May 26 06:47:44 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:47:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 06:47:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:47:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:47:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:47:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:47:44 mr-fox kernel: PKRU: 55555554
May 26 06:47:44 mr-fox kernel: Call Trace:
May 26 06:47:44 mr-fox kernel:  <IRQ>
May 26 06:47:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:47:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:47:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:47:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:47:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:47:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:47:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:47:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:47:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:47:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:47:44 mr-fox kernel:  </IRQ>
May 26 06:47:44 mr-fox kernel:  <TASK>
May 26 06:47:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:47:44 mr-fox kernel:  ? xas_descend+0x31/0xd0
May 26 06:47:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:47:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:47:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:47:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:47:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:47:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:47:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:47:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:47:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:47:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:47:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:47:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:47:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:47:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:47:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:47:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:47:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:47:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:47:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:47:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:47:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:47:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:47:44 mr-fox kernel:  </TASK>
May 26 06:47:52 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 691823 jiffies s: 491905 root: 0x2/.
May 26 06:47:52 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:47:52 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:47:52 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:47:52 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:47:52 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:47:52 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:47:52 mr-fox kernel: RIP: 0010:clear_page_erms+0xd/0x20
May 26 06:47:52 mr-fox kernel: Code: 47 20 48 89 47 28 48 89 47 30 48 89 47 38 48 8d 7f 40 75 d9 90 e9 be 8a 1a 00 0f 1f 00 f3 0f 1e fa b9 00 10 00 00 31 c0 f3 aa <e9> a9 8a 1a 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 f3 0f 1e
May 26 06:47:52 mr-fox kernel: RSP: 0018:ffffa401005f0d58 EFLAGS: 00000246
May 26 06:47:52 mr-fox kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
May 26 06:47:52 mr-fox kernel: RDX: ffffcf6320570200 RSI: ffffcf6320570400 RDI: ffff98a1d5c09000
May 26 06:47:52 mr-fox kernel: RBP: ffffcf6320570200 R08: 0000000000000001 R09: 0000000000000008
May 26 06:47:52 mr-fox kernel: R10: 0000000000000004 R11: 0000000000000000 R12: 0000000000000003
May 26 06:47:52 mr-fox kernel: R13: ffffcf6320570400 R14: 0000000000000001 R15: 0000000000000008
May 26 06:47:52 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:47:52 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:47:52 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:47:52 mr-fox kernel: PKRU: 55555554
May 26 06:47:52 mr-fox kernel: Call Trace:
May 26 06:47:52 mr-fox kernel:  <NMI>
May 26 06:47:52 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:47:52 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:47:52 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:47:52 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:47:52 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:47:52 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:47:52 mr-fox kernel:  ? clear_page_erms+0xd/0x20
May 26 06:47:52 mr-fox kernel:  ? clear_page_erms+0xd/0x20
May 26 06:47:52 mr-fox kernel:  ? clear_page_erms+0xd/0x20
May 26 06:47:52 mr-fox kernel:  </NMI>
May 26 06:47:52 mr-fox kernel:  <IRQ>
May 26 06:47:52 mr-fox kernel:  free_unref_page_prepare+0x114/0x300
May 26 06:47:52 mr-fox kernel:  free_unref_page+0x2f/0x170
May 26 06:47:52 mr-fox kernel:  skb_release_data.isra.0+0xfb/0x1e0
May 26 06:47:52 mr-fox kernel:  __kfree_skb+0x24/0x30
May 26 06:47:52 mr-fox kernel:  tcp_write_queue_purge+0xed/0x310
May 26 06:47:52 mr-fox kernel:  tcp_retransmit_timer+0x28e/0xa60
May 26 06:47:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:47:52 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 06:47:52 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 06:47:52 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 06:47:52 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 06:47:52 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 06:47:52 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:47:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:47:52 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:47:52 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:47:52 mr-fox kernel:  </IRQ>
May 26 06:47:52 mr-fox kernel:  <TASK>
May 26 06:47:52 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:47:52 mr-fox kernel: RIP: 0010:xas_start+0x0/0x120
May 26 06:47:52 mr-fox kernel: Code: d3 e0 48 23 47 08 48 d3 e3 48 01 c3 48 89 5d 08 48 83 c4 08 5b 5d 41 5c 31 c0 31 d2 31 c9 31 f6 31 ff e9 2e c8 1a 00 0f 1f 00 <55> 53 48 89 fb 48 83 ec 08 48 8b 6f 18 48 89 e8 83 e0 03 0f 84 81
May 26 06:47:52 mr-fox kernel: RSP: 0018:ffffa401077e3948 EFLAGS: 00000246
May 26 06:47:52 mr-fox kernel: RAX: 0000000000000000 RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:47:52 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffa401077e3970
May 26 06:47:52 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 06:47:52 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:47:52 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:47:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:47:52 mr-fox kernel:  xas_load+0xe/0x60
May 26 06:47:52 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:47:52 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:47:52 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:47:52 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:47:52 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:47:52 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:47:52 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:47:52 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:47:52 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:47:52 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:47:52 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:47:52 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:47:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:47:52 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:47:52 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:47:52 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:47:52 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:47:52 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:47:52 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:47:52 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:47:52 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:47:52 mr-fox kernel:  </TASK>
May 26 06:48:00 mr-fox crond[4685]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:48:00 mr-fox crond[4687]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:48:00 mr-fox crond[4688]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:48:00 mr-fox crond[4689]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:48:00 mr-fox CROND[4691]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:48:00 mr-fox crond[4684]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:48:00 mr-fox CROND[4694]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:48:00 mr-fox CROND[4695]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:48:00 mr-fox CROND[4692]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:48:00 mr-fox CROND[4696]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:48:00 mr-fox CROND[4689]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:48:00 mr-fox CROND[4689]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:48:00 mr-fox CROND[4688]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:48:00 mr-fox CROND[4688]: pam_unix(crond:session): session closed for user root
May 26 06:48:00 mr-fox CROND[4687]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:48:00 mr-fox CROND[4687]: pam_unix(crond:session): session closed for user root
May 26 06:48:00 mr-fox CROND[32258]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:48:00 mr-fox CROND[32258]: pam_unix(crond:session): session closed for user root
May 26 06:49:00 mr-fox crond[8783]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:49:00 mr-fox crond[8782]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:49:00 mr-fox crond[8780]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:49:00 mr-fox crond[8784]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:49:00 mr-fox CROND[8788]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:49:00 mr-fox crond[8785]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:49:00 mr-fox CROND[8791]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:49:00 mr-fox CROND[8790]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:49:00 mr-fox CROND[8792]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:49:00 mr-fox CROND[8793]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:49:00 mr-fox CROND[8785]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:49:00 mr-fox CROND[8785]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:49:00 mr-fox CROND[8784]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:49:00 mr-fox CROND[8784]: pam_unix(crond:session): session closed for user root
May 26 06:49:00 mr-fox CROND[8783]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:49:00 mr-fox CROND[8783]: pam_unix(crond:session): session closed for user root
May 26 06:49:00 mr-fox CROND[4685]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:49:00 mr-fox CROND[4685]: pam_unix(crond:session): session closed for user root
May 26 06:50:00 mr-fox crond[11624]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 06:50:00 mr-fox crond[11626]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:50:00 mr-fox crond[11625]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:50:00 mr-fox crond[11627]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:50:00 mr-fox CROND[11635]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:50:00 mr-fox crond[11629]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:50:00 mr-fox CROND[11636]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:50:00 mr-fox CROND[11637]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:50:00 mr-fox crond[11631]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:50:00 mr-fox crond[11630]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:50:00 mr-fox crond[11632]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:50:00 mr-fox CROND[11638]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:50:00 mr-fox CROND[11640]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:50:00 mr-fox CROND[11641]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 06:50:00 mr-fox CROND[11643]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:50:00 mr-fox CROND[11642]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:50:00 mr-fox CROND[11624]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:50:00 mr-fox CROND[11624]: pam_unix(crond:session): session closed for user torproject
May 26 06:50:00 mr-fox CROND[11632]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:50:00 mr-fox CROND[11632]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:50:00 mr-fox CROND[11629]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:50:00 mr-fox CROND[11629]: pam_unix(crond:session): session closed for user root
May 26 06:50:01 mr-fox CROND[11630]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:50:01 mr-fox CROND[11630]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:50:01 mr-fox CROND[11627]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:50:01 mr-fox CROND[11627]: pam_unix(crond:session): session closed for user root
May 26 06:50:01 mr-fox CROND[8782]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:50:01 mr-fox CROND[8782]: pam_unix(crond:session): session closed for user root
May 26 06:50:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:50:44 mr-fox kernel: rcu: \x0921-....: (735060 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=286664
May 26 06:50:44 mr-fox kernel: rcu: \x09(t=735061 jiffies g=8794409 q=24738565 ncpus=32)
May 26 06:50:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:50:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:50:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:50:44 mr-fox kernel: RIP: 0010:xas_descend+0x40/0xd0
May 26 06:50:44 mr-fox kernel: Code: 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 <75> 08 48 3d fd 00 00 00 76 2f 41 88 5c 24 12 48 83 c4 08 5b 5d 41
May 26 06:50:44 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000246
May 26 06:50:44 mr-fox kernel: RAX: ffff988d667606da RBX: 0000000000000019 RCX: 000000000000000c
May 26 06:50:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988b1fdbb478 RDI: ffffa401077e3970
May 26 06:50:44 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 06:50:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 06:50:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:50:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:50:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:50:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:50:44 mr-fox kernel: PKRU: 55555554
May 26 06:50:44 mr-fox kernel: Call Trace:
May 26 06:50:44 mr-fox kernel:  <IRQ>
May 26 06:50:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:50:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:50:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:50:44 mr-fox kernel:  ? tcp_delack_timer+0xb5/0xf0
May 26 06:50:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:50:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:50:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:50:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:50:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:50:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:50:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:50:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:50:44 mr-fox kernel:  </IRQ>
May 26 06:50:44 mr-fox kernel:  <TASK>
May 26 06:50:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:50:44 mr-fox kernel:  ? xas_descend+0x40/0xd0
May 26 06:50:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:50:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:50:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:50:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:50:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:50:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:50:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:50:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:50:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:50:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:50:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:50:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:50:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:50:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:50:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:50:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:50:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:50:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:50:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:50:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:50:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:50:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:50:44 mr-fox kernel:  </TASK>
May 26 06:50:52 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 736880 jiffies s: 491905 root: 0x2/.
May 26 06:50:52 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:50:52 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:50:52 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:50:52 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:50:52 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:50:52 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:50:52 mr-fox kernel: RIP: 0010:napi_consume_skb+0x1b/0xc0
May 26 06:50:52 mr-fox kernel: Code: 20 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 53 48 89 fb 85 f6 0f 84 8c 00 00 00 48 85 ff 74 7b 8b 87 d4 00 00 00 <83> f8 01 75 53 48 89 df f6 43 7e 0c 75 25 e8 62 87 ff ff 48 83 bb
May 26 06:50:52 mr-fox kernel: RSP: 0018:ffffa401005f0de8 EFLAGS: 00000282
May 26 06:50:52 mr-fox kernel: RAX: 0000000000000001 RBX: ffff988ad987c458 RCX: 0000000000000000
May 26 06:50:52 mr-fox kernel: RDX: 0000000000000001 RSI: 0000000000000040 RDI: ffff988ad987c458
May 26 06:50:52 mr-fox kernel: RBP: 0000000000009f0a R08: 0000000000000000 R09: 0000000000000000
May 26 06:50:52 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffa40100119f90
May 26 06:50:52 mr-fox kernel: R13: ffff988ac873fa40 R14: 00000000fffffff9 R15: ffffa40100119fb0
May 26 06:50:52 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:50:52 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:50:52 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:50:52 mr-fox kernel: PKRU: 55555554
May 26 06:50:52 mr-fox kernel: Call Trace:
May 26 06:50:52 mr-fox kernel:  <NMI>
May 26 06:50:52 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:50:52 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:50:52 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:50:52 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:50:52 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:50:52 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:50:52 mr-fox kernel:  ? napi_consume_skb+0x1b/0xc0
May 26 06:50:52 mr-fox kernel:  ? napi_consume_skb+0x1b/0xc0
May 26 06:50:52 mr-fox kernel:  ? napi_consume_skb+0x1b/0xc0
May 26 06:50:52 mr-fox kernel:  </NMI>
May 26 06:50:52 mr-fox kernel:  <IRQ>
May 26 06:50:52 mr-fox kernel:  igb_poll+0xea/0x1370
May 26 06:50:52 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 06:50:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:50:52 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 06:50:52 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:50:52 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:50:52 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:50:52 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:50:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:50:52 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:50:52 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:50:52 mr-fox kernel:  </IRQ>
May 26 06:50:52 mr-fox kernel:  <TASK>
May 26 06:50:52 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:50:52 mr-fox kernel: RIP: 0010:filemap_get_entry+0x6d/0x160
May 26 06:50:52 mr-fox kernel: Code: 00 00 48 c7 44 24 30 00 00 00 00 48 c7 44 24 38 00 00 00 00 48 c7 44 24 20 03 00 00 00 48 8d 7c 24 08 e8 56 70 78 00 48 89 c3 <48> 3d 02 04 00 00 74 e2 48 3d 06 04 00 00 74 da 48 85 c0 74 5a a8
May 26 06:50:52 mr-fox kernel: RSP: 0018:ffffa401077e3968 EFLAGS: 00000246
May 26 06:50:52 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 06:50:52 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:50:52 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 06:50:52 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:50:52 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:50:52 mr-fox kernel:  ? filemap_get_entry+0x6a/0x160
May 26 06:50:52 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:50:52 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:50:52 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:50:52 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:50:52 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:50:52 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:50:52 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:50:52 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:50:52 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:50:52 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:50:52 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:50:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:50:52 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:50:52 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:50:52 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:50:52 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:50:52 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:50:52 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:50:52 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:50:52 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:50:52 mr-fox kernel:  </TASK>
May 26 06:51:00 mr-fox crond[12429]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:51:00 mr-fox crond[12430]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:51:00 mr-fox crond[12431]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:51:00 mr-fox crond[12433]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:51:00 mr-fox CROND[12436]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:51:00 mr-fox CROND[12438]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:51:00 mr-fox CROND[12437]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:51:00 mr-fox CROND[12439]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:51:00 mr-fox crond[12432]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:51:00 mr-fox CROND[12441]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:51:00 mr-fox CROND[12433]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:51:00 mr-fox CROND[12433]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:51:00 mr-fox CROND[12432]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:51:00 mr-fox CROND[12432]: pam_unix(crond:session): session closed for user root
May 26 06:51:00 mr-fox CROND[12431]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:51:00 mr-fox CROND[12431]: pam_unix(crond:session): session closed for user root
May 26 06:51:01 mr-fox CROND[11626]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:51:01 mr-fox CROND[11626]: pam_unix(crond:session): session closed for user root
May 26 06:52:00 mr-fox crond[14931]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:52:00 mr-fox crond[14932]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:52:00 mr-fox crond[14933]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:52:00 mr-fox crond[14935]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:52:00 mr-fox crond[14930]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:52:00 mr-fox CROND[14938]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:52:00 mr-fox CROND[14940]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:52:00 mr-fox CROND[14939]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:52:00 mr-fox CROND[14941]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:52:00 mr-fox CROND[14942]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:52:00 mr-fox CROND[14935]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:52:00 mr-fox CROND[14935]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:52:00 mr-fox CROND[14933]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:52:00 mr-fox CROND[14933]: pam_unix(crond:session): session closed for user root
May 26 06:52:00 mr-fox CROND[14932]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:52:00 mr-fox CROND[14932]: pam_unix(crond:session): session closed for user root
May 26 06:52:00 mr-fox CROND[12430]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:52:00 mr-fox CROND[12430]: pam_unix(crond:session): session closed for user root
May 26 06:53:00 mr-fox crond[18691]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:53:00 mr-fox crond[18689]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:53:00 mr-fox crond[18692]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:53:00 mr-fox crond[18693]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:53:00 mr-fox crond[18694]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:53:00 mr-fox CROND[18697]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:53:00 mr-fox CROND[18698]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:53:00 mr-fox CROND[18700]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:53:00 mr-fox CROND[18701]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:53:00 mr-fox CROND[18702]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:53:00 mr-fox CROND[18694]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:53:00 mr-fox CROND[18694]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:53:00 mr-fox CROND[18693]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:53:00 mr-fox CROND[18693]: pam_unix(crond:session): session closed for user root
May 26 06:53:01 mr-fox CROND[18692]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:53:01 mr-fox CROND[18692]: pam_unix(crond:session): session closed for user root
May 26 06:53:01 mr-fox CROND[14931]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:53:01 mr-fox CROND[14931]: pam_unix(crond:session): session closed for user root
May 26 06:53:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:53:44 mr-fox kernel: rcu: \x0921-....: (780064 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=303960
May 26 06:53:44 mr-fox kernel: rcu: \x09(t=780065 jiffies g=8794409 q=25927955 ncpus=32)
May 26 06:53:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:53:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:53:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:53:44 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 06:53:44 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 06:53:44 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 06:53:44 mr-fox kernel: RAX: ffff988d667606da RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:53:44 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:53:44 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 06:53:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:53:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:53:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:53:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:53:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:53:44 mr-fox kernel: PKRU: 55555554
May 26 06:53:44 mr-fox kernel: Call Trace:
May 26 06:53:44 mr-fox kernel:  <IRQ>
May 26 06:53:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:53:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:53:44 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 06:53:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:53:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:53:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:53:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:53:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:53:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:53:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:53:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:53:44 mr-fox kernel:  </IRQ>
May 26 06:53:44 mr-fox kernel:  <TASK>
May 26 06:53:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:53:44 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 06:53:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:53:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:53:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:53:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:53:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:53:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:53:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:53:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:53:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:53:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:53:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:53:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:53:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:53:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:53:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:53:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:53:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:53:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:53:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:53:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:53:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:53:44 mr-fox kernel:  </TASK>
May 26 06:53:52 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 781936 jiffies s: 491905 root: 0x2/.
May 26 06:53:52 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:53:52 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:53:52 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:53:52 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:53:52 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:53:52 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:53:52 mr-fox kernel: RIP: 0010:__nf_conntrack_find_get.isra.0+0x8d/0x280
May 26 06:53:52 mr-fox kernel: Code: 00 49 8b 07 49 8b 4f 08 48 33 43 10 48 33 4b 18 48 09 c8 75 09 8b 43 20 41 39 47 10 74 7f 48 8b 1b f6 c3 01 75 5b 0f b6 4b 37 <4c> 89 ed 48 8d 04 cd 00 00 00 00 48 29 c8 48 c1 e0 03 48 29 c5 48
May 26 06:53:52 mr-fox kernel: RSP: 0018:ffffa401005f0aa8 EFLAGS: 00000246
May 26 06:53:52 mr-fox kernel: RAX: ffff988acb39f330 RBX: ffff988ac653e048 RCX: 0000000000000001
May 26 06:53:52 mr-fox kernel: RDX: 00000000cf99b3a3 RSI: ffff988acb200000 RDI: ffffffffa6ad7f40
May 26 06:53:52 mr-fox kernel: RBP: ffffa401005f0b08 R08: 0000000000000000 R09: 0000000000000000
May 26 06:53:52 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000033e66
May 26 06:53:52 mr-fox kernel: R13: fffffffffffffff0 R14: ffffffffa6ad7f40 R15: ffffa401005f0b08
May 26 06:53:52 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:53:52 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:53:52 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:53:52 mr-fox kernel: PKRU: 55555554
May 26 06:53:52 mr-fox kernel: Call Trace:
May 26 06:53:52 mr-fox kernel:  <NMI>
May 26 06:53:52 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:53:52 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:53:52 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:53:52 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:53:52 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:53:52 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:53:52 mr-fox kernel:  ? __nf_conntrack_find_get.isra.0+0x8d/0x280
May 26 06:53:52 mr-fox kernel:  ? __nf_conntrack_find_get.isra.0+0x8d/0x280
May 26 06:53:52 mr-fox kernel:  ? __nf_conntrack_find_get.isra.0+0x8d/0x280
May 26 06:53:52 mr-fox kernel:  </NMI>
May 26 06:53:52 mr-fox kernel:  <IRQ>
May 26 06:53:52 mr-fox kernel:  nf_conntrack_in+0xdc/0x540
May 26 06:53:52 mr-fox kernel:  nf_hook_slow+0x3c/0x100
May 26 06:53:52 mr-fox kernel:  nf_hook_slow_list+0x90/0x140
May 26 06:53:52 mr-fox kernel:  ip_sublist_rcv+0x71/0x1c0
May 26 06:53:52 mr-fox kernel:  ? ip_sublist_rcv+0x1c0/0x1c0
May 26 06:53:52 mr-fox kernel:  ip_list_rcv+0x138/0x170
May 26 06:53:52 mr-fox kernel:  __netif_receive_skb_list_core+0x2a5/0x2d0
May 26 06:53:52 mr-fox kernel:  netif_receive_skb_list_internal+0x1db/0x320
May 26 06:53:52 mr-fox kernel:  napi_gro_receive+0xcf/0x1b0
May 26 06:53:52 mr-fox kernel:  igb_poll+0x605/0x1370
May 26 06:53:52 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 06:53:52 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 06:53:52 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 06:53:52 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:53:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:53:52 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:53:52 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:53:52 mr-fox kernel:  </IRQ>
May 26 06:53:52 mr-fox kernel:  <TASK>
May 26 06:53:52 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:53:52 mr-fox kernel: RIP: 0010:xas_load+0x35/0x60
May 26 06:53:52 mr-fox kernel: Code: f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 <48> 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d
May 26 06:53:52 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000282
May 26 06:53:52 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 06:53:52 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:53:52 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:53:52 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:53:52 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:53:52 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:53:52 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:53:52 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:53:52 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:53:52 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:53:52 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:53:52 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:53:52 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:53:52 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:53:52 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:53:52 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:53:52 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:53:52 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:53:52 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:53:52 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:53:52 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:53:52 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:53:52 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:53:52 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:53:52 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:53:52 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:53:52 mr-fox kernel:  </TASK>
May 26 06:54:00 mr-fox crond[19580]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:54:00 mr-fox crond[19582]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:54:00 mr-fox crond[19581]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:54:00 mr-fox crond[19583]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:54:00 mr-fox crond[19585]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:54:00 mr-fox CROND[19589]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:54:00 mr-fox CROND[19587]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:54:00 mr-fox CROND[19590]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:54:00 mr-fox CROND[19591]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:54:00 mr-fox CROND[19588]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:54:00 mr-fox CROND[19585]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:54:00 mr-fox CROND[19585]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:54:00 mr-fox CROND[19583]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:54:00 mr-fox CROND[19583]: pam_unix(crond:session): session closed for user root
May 26 06:54:00 mr-fox CROND[19582]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:54:00 mr-fox CROND[19582]: pam_unix(crond:session): session closed for user root
May 26 06:54:01 mr-fox CROND[18691]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:54:01 mr-fox CROND[18691]: pam_unix(crond:session): session closed for user root
May 26 06:55:00 mr-fox crond[22083]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:55:00 mr-fox crond[22084]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:55:00 mr-fox crond[22082]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 06:55:00 mr-fox crond[22087]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:55:00 mr-fox crond[22086]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:55:00 mr-fox CROND[22093]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:55:00 mr-fox crond[22085]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:55:00 mr-fox CROND[22096]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:55:00 mr-fox CROND[22097]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:55:00 mr-fox CROND[22098]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:55:00 mr-fox crond[22089]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:55:00 mr-fox CROND[22099]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:55:00 mr-fox crond[22090]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:55:00 mr-fox CROND[22101]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:55:00 mr-fox CROND[22102]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 06:55:00 mr-fox CROND[22103]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:55:00 mr-fox CROND[22082]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 06:55:00 mr-fox CROND[22082]: pam_unix(crond:session): session closed for user torproject
May 26 06:55:00 mr-fox CROND[22090]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:55:00 mr-fox CROND[22090]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:55:00 mr-fox CROND[22086]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:55:00 mr-fox CROND[22086]: pam_unix(crond:session): session closed for user root
May 26 06:55:00 mr-fox CROND[22087]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 06:55:00 mr-fox CROND[22087]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:55:00 mr-fox CROND[22085]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:55:00 mr-fox CROND[22085]: pam_unix(crond:session): session closed for user root
May 26 06:55:00 mr-fox CROND[19581]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:55:00 mr-fox CROND[19581]: pam_unix(crond:session): session closed for user root
May 26 06:56:00 mr-fox crond[24422]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:56:00 mr-fox crond[24424]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:56:00 mr-fox crond[24425]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:56:00 mr-fox CROND[24430]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:56:00 mr-fox crond[24426]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:56:00 mr-fox CROND[24431]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:56:00 mr-fox CROND[24432]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:56:00 mr-fox crond[24427]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:56:00 mr-fox CROND[24435]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:56:00 mr-fox CROND[24436]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:56:00 mr-fox CROND[24427]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:56:00 mr-fox CROND[24427]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:56:00 mr-fox CROND[24426]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:56:00 mr-fox CROND[24426]: pam_unix(crond:session): session closed for user root
May 26 06:56:00 mr-fox CROND[24425]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:56:00 mr-fox CROND[24425]: pam_unix(crond:session): session closed for user root
May 26 06:56:00 mr-fox CROND[22084]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:56:00 mr-fox CROND[22084]: pam_unix(crond:session): session closed for user root
May 26 06:56:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:56:44 mr-fox kernel: rcu: \x0921-....: (825068 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=321402
May 26 06:56:44 mr-fox kernel: rcu: \x09(t=825069 jiffies g=8794409 q=27245568 ncpus=32)
May 26 06:56:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:56:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:56:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:56:44 mr-fox kernel: RIP: 0010:xas_descend+0x31/0xd0
May 26 06:56:44 mr-fox kernel: Code: 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 <49> 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d fd 00 00
May 26 06:56:44 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 06:56:44 mr-fox kernel: RAX: ffff988ac07566ca RBX: 0000000000000001 RCX: 000000000000001e
May 26 06:56:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac0754ff8 RDI: ffffa401077e3970
May 26 06:56:44 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:56:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 06:56:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:56:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:56:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:56:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:56:44 mr-fox kernel: PKRU: 55555554
May 26 06:56:44 mr-fox kernel: Call Trace:
May 26 06:56:44 mr-fox kernel:  <IRQ>
May 26 06:56:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:56:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:56:44 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 06:56:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:56:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:56:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:56:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:56:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:56:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:56:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:56:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:56:44 mr-fox kernel:  </IRQ>
May 26 06:56:44 mr-fox kernel:  <TASK>
May 26 06:56:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:56:44 mr-fox kernel:  ? xas_descend+0x31/0xd0
May 26 06:56:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:56:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:56:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:56:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:56:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:56:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:56:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:56:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:56:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:56:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:56:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:56:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:56:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:56:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:56:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:56:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:56:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:56:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:56:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:56:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:56:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:56:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:56:44 mr-fox kernel:  </TASK>
May 26 06:56:53 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 826992 jiffies s: 491905 root: 0x2/.
May 26 06:56:53 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:56:53 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:56:53 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:56:53 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:56:53 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:56:53 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:56:53 mr-fox kernel: RIP: 0010:_raw_spin_lock+0xf/0x30
May 26 06:56:53 mr-fox kernel: Code: 31 ff e9 af 2f 15 00 e9 0f 03 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 0d 31 c0 31 d2 31 f6 31 ff e9 7d 2f 15 00 89 c6 e9 cb 00 00 00
May 26 06:56:53 mr-fox kernel: RSP: 0018:ffffa401005f0e88 EFLAGS: 00000246
May 26 06:56:53 mr-fox kernel: RAX: 0000000000000000 RBX: ffff9892c7273248 RCX: 0000000000000000
May 26 06:56:53 mr-fox kernel: RDX: 0000000000000001 RSI: ffffffffa587d6d0 RDI: ffff9892c7272ed8
May 26 06:56:53 mr-fox kernel: RBP: ffff9892c7272e40 R08: 0000000000000000 R09: 0000000000000000
May 26 06:56:53 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffa401005f0ee8
May 26 06:56:53 mr-fox kernel: R13: ffff98a96ed5bbb0 R14: ffffa401005f0ef0 R15: ffff98a96ed5bb40
May 26 06:56:53 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:56:53 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:56:53 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:56:53 mr-fox kernel: PKRU: 55555554
May 26 06:56:53 mr-fox kernel: Call Trace:
May 26 06:56:53 mr-fox kernel:  <NMI>
May 26 06:56:53 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:56:53 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:56:53 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:56:53 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:56:53 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:56:53 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:56:53 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 06:56:53 mr-fox kernel:  ? _raw_spin_lock+0xf/0x30
May 26 06:56:53 mr-fox kernel:  ? _raw_spin_lock+0xf/0x30
May 26 06:56:53 mr-fox kernel:  ? _raw_spin_lock+0xf/0x30
May 26 06:56:53 mr-fox kernel:  </NMI>
May 26 06:56:53 mr-fox kernel:  <IRQ>
May 26 06:56:53 mr-fox kernel:  tcp_write_timer+0x1e/0xd0
May 26 06:56:53 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 06:56:53 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 06:56:53 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 06:56:53 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 06:56:53 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:56:53 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:56:53 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:56:53 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:56:53 mr-fox kernel:  </IRQ>
May 26 06:56:53 mr-fox kernel:  <TASK>
May 26 06:56:53 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:56:53 mr-fox kernel: RIP: 0010:xas_descend+0x26/0xd0
May 26 06:56:53 mr-fox kernel: Code: 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f <89> d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03
May 26 06:56:53 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 06:56:53 mr-fox kernel: RAX: ffff988d55f18ffa RBX: 0000000000000021 RCX: 0000000000000000
May 26 06:56:53 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988d55f18ff8 RDI: ffffa401077e3970
May 26 06:56:53 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:56:53 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 06:56:53 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:56:53 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:56:53 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:56:53 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:56:53 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:56:53 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:56:53 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:56:53 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:56:53 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:56:53 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:56:53 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:56:53 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:56:53 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:56:53 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:56:53 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:56:53 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:56:53 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:56:53 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:56:53 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:56:53 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:56:53 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:56:53 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:56:53 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:56:53 mr-fox kernel:  </TASK>
May 26 06:57:00 mr-fox crond[28177]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:57:00 mr-fox crond[28178]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:57:00 mr-fox crond[28176]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:57:00 mr-fox crond[28179]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:57:00 mr-fox crond[28180]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:57:00 mr-fox CROND[28182]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:57:00 mr-fox CROND[28183]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:57:00 mr-fox CROND[28185]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:57:00 mr-fox CROND[28184]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:57:00 mr-fox CROND[28186]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:57:01 mr-fox CROND[28180]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:57:01 mr-fox CROND[28180]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:57:01 mr-fox CROND[28179]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:57:01 mr-fox CROND[28179]: pam_unix(crond:session): session closed for user root
May 26 06:57:01 mr-fox CROND[28178]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:57:01 mr-fox CROND[28178]: pam_unix(crond:session): session closed for user root
May 26 06:57:01 mr-fox CROND[24424]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:57:01 mr-fox CROND[24424]: pam_unix(crond:session): session closed for user root
May 26 06:58:00 mr-fox crond[29085]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:58:00 mr-fox crond[29084]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:58:00 mr-fox crond[29086]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:58:00 mr-fox crond[29083]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:58:00 mr-fox CROND[29092]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:58:00 mr-fox CROND[29091]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:58:00 mr-fox CROND[29093]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:58:00 mr-fox CROND[29094]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:58:00 mr-fox crond[29088]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:58:00 mr-fox CROND[29098]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:58:00 mr-fox CROND[29088]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:58:00 mr-fox CROND[29088]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:58:00 mr-fox CROND[29086]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:58:00 mr-fox CROND[29086]: pam_unix(crond:session): session closed for user root
May 26 06:58:00 mr-fox CROND[29085]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:58:00 mr-fox CROND[29085]: pam_unix(crond:session): session closed for user root
May 26 06:58:01 mr-fox CROND[28177]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:58:01 mr-fox CROND[28177]: pam_unix(crond:session): session closed for user root
May 26 06:59:00 mr-fox crond[32011]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:59:00 mr-fox crond[32010]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:59:00 mr-fox crond[32014]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:59:00 mr-fox crond[32012]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 06:59:00 mr-fox crond[32015]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 06:59:00 mr-fox CROND[32018]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:59:00 mr-fox CROND[32020]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 06:59:00 mr-fox CROND[32021]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:59:00 mr-fox CROND[32023]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 06:59:00 mr-fox CROND[32024]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 06:59:00 mr-fox CROND[32015]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 06:59:00 mr-fox CROND[32015]: pam_unix(crond:session): session closed for user tinderbox
May 26 06:59:00 mr-fox CROND[32014]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 06:59:00 mr-fox CROND[32014]: pam_unix(crond:session): session closed for user root
May 26 06:59:00 mr-fox CROND[32012]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 06:59:00 mr-fox CROND[32012]: pam_unix(crond:session): session closed for user root
May 26 06:59:00 mr-fox CROND[29084]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 06:59:00 mr-fox CROND[29084]: pam_unix(crond:session): session closed for user root
May 26 06:59:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 06:59:44 mr-fox kernel: rcu: \x0921-....: (870072 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=338803
May 26 06:59:44 mr-fox kernel: rcu: \x09(t=870073 jiffies g=8794409 q=28418353 ncpus=32)
May 26 06:59:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:59:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:59:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:59:44 mr-fox kernel: RIP: 0010:xas_descend+0x31/0xd0
May 26 06:59:44 mr-fox kernel: Code: 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 <49> 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d fd 00 00
May 26 06:59:44 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 06:59:44 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: 0000000000000036 RCX: 0000000000000012
May 26 06:59:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988acf90e6c8 RDI: ffffa401077e3970
May 26 06:59:44 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 06:59:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 06:59:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:59:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:59:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:59:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:59:44 mr-fox kernel: PKRU: 55555554
May 26 06:59:44 mr-fox kernel: Call Trace:
May 26 06:59:44 mr-fox kernel:  <IRQ>
May 26 06:59:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 06:59:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 06:59:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 06:59:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 06:59:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 06:59:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 06:59:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:59:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 06:59:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 06:59:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 06:59:44 mr-fox kernel:  </IRQ>
May 26 06:59:44 mr-fox kernel:  <TASK>
May 26 06:59:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:59:44 mr-fox kernel:  ? xas_descend+0x31/0xd0
May 26 06:59:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 06:59:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 06:59:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:59:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:59:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:59:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:59:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:59:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:59:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:59:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:59:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:59:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:59:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:59:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:59:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:59:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:59:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:59:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:59:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:59:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:59:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:59:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:59:44 mr-fox kernel:  </TASK>
May 26 06:59:53 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 872048 jiffies s: 491905 root: 0x2/.
May 26 06:59:53 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 06:59:53 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 06:59:53 mr-fox kernel: NMI backtrace for cpu 21
May 26 06:59:53 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 06:59:53 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 06:59:53 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 06:59:53 mr-fox kernel: RIP: 0010:__nf_conntrack_find_get.isra.0+0x8d/0x280
May 26 06:59:53 mr-fox kernel: Code: 00 49 8b 07 49 8b 4f 08 48 33 43 10 48 33 4b 18 48 09 c8 75 09 8b 43 20 41 39 47 10 74 7f 48 8b 1b f6 c3 01 75 5b 0f b6 4b 37 <4c> 89 ed 48 8d 04 cd 00 00 00 00 48 29 c8 48 c1 e0 03 48 29 c5 48
May 26 06:59:53 mr-fox kernel: RSP: 0018:ffffa401005f0c28 EFLAGS: 00000246
May 26 06:59:53 mr-fox kernel: RAX: ffff988acb271210 RBX: ffff988d66b3fe10 RCX: 0000000000000000
May 26 06:59:53 mr-fox kernel: RDX: 00000000389089f1 RSI: ffff988acb200000 RDI: ffffffffa6ad7f40
May 26 06:59:53 mr-fox kernel: RBP: ffffa401005f0c88 R08: 0000000000000000 R09: 0000000000000000
May 26 06:59:53 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000e242
May 26 06:59:53 mr-fox kernel: R13: fffffffffffffff0 R14: ffffffffa6ad7f40 R15: ffffa401005f0c88
May 26 06:59:53 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 06:59:53 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 06:59:53 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 06:59:53 mr-fox kernel: PKRU: 55555554
May 26 06:59:53 mr-fox kernel: Call Trace:
May 26 06:59:53 mr-fox kernel:  <NMI>
May 26 06:59:53 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 06:59:53 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 06:59:53 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 06:59:53 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 06:59:53 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 06:59:53 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 06:59:53 mr-fox kernel:  ? __nf_conntrack_find_get.isra.0+0x8d/0x280
May 26 06:59:53 mr-fox kernel:  ? __nf_conntrack_find_get.isra.0+0x8d/0x280
May 26 06:59:53 mr-fox kernel:  ? __nf_conntrack_find_get.isra.0+0x8d/0x280
May 26 06:59:53 mr-fox kernel:  </NMI>
May 26 06:59:53 mr-fox kernel:  <IRQ>
May 26 06:59:53 mr-fox kernel:  nf_conntrack_in+0xdc/0x540
May 26 06:59:53 mr-fox kernel:  nf_hook_slow+0x3c/0x100
May 26 06:59:53 mr-fox kernel:  __ip_local_out+0xc6/0x100
May 26 06:59:53 mr-fox kernel:  ? ip_output+0x100/0x100
May 26 06:59:53 mr-fox kernel:  ip_local_out+0x16/0x70
May 26 06:59:53 mr-fox kernel:  __ip_queue_xmit+0x16b/0x480
May 26 06:59:53 mr-fox kernel:  __tcp_transmit_skb+0xbad/0xd30
May 26 06:59:53 mr-fox kernel:  tcp_delack_timer_handler+0xa9/0x110
May 26 06:59:53 mr-fox kernel:  tcp_delack_timer+0xb5/0xf0
May 26 06:59:53 mr-fox kernel:  ? tcp_delack_timer_handler+0x110/0x110
May 26 06:59:53 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 06:59:53 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 06:59:53 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 06:59:53 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 06:59:53 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:59:53 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 06:59:53 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 06:59:53 mr-fox kernel:  </IRQ>
May 26 06:59:53 mr-fox kernel:  <TASK>
May 26 06:59:53 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 06:59:53 mr-fox kernel: RIP: 0010:filemap_get_entry+0x6d/0x160
May 26 06:59:53 mr-fox kernel: Code: 00 00 48 c7 44 24 30 00 00 00 00 48 c7 44 24 38 00 00 00 00 48 c7 44 24 20 03 00 00 00 48 8d 7c 24 08 e8 56 70 78 00 48 89 c3 <48> 3d 02 04 00 00 74 e2 48 3d 06 04 00 00 74 da 48 85 c0 74 5a a8
May 26 06:59:53 mr-fox kernel: RSP: 0018:ffffa401077e3968 EFLAGS: 00000246
May 26 06:59:53 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 06:59:53 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 06:59:53 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 06:59:53 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 06:59:53 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 06:59:53 mr-fox kernel:  ? filemap_get_entry+0x6a/0x160
May 26 06:59:53 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 06:59:53 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 06:59:53 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 06:59:53 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 06:59:53 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 06:59:53 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 06:59:53 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 06:59:53 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 06:59:53 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 06:59:53 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 06:59:53 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 06:59:53 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 06:59:53 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 06:59:53 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 06:59:53 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 06:59:53 mr-fox kernel:  kthread+0xcb/0xf0
May 26 06:59:53 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:59:53 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 06:59:53 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 06:59:53 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 06:59:53 mr-fox kernel:  </TASK>
May 26 07:00:00 mr-fox crond[2524]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 07:00:00 mr-fox crond[2525]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:00:00 mr-fox crond[2527]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:00:00 mr-fox crond[2526]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:00:00 mr-fox crond[2531]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:00:00 mr-fox CROND[2538]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:00:00 mr-fox crond[2528]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:00:00 mr-fox CROND[2540]: (root) CMD (/etc/conf.d/ipv4-rules.sh update; /etc/conf.d/ipv6-rules.sh update)
May 26 07:00:00 mr-fox CROND[2541]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:00:00 mr-fox CROND[2539]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:00:00 mr-fox crond[2530]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:00:00 mr-fox crond[2533]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:00:00 mr-fox crond[2532]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:00:00 mr-fox crond[2534]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:00:00 mr-fox CROND[2543]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:00:00 mr-fox crond[2535]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:00:00 mr-fox crond[2537]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:00:00 mr-fox CROND[2544]: (root) CMD (/opt/torutils/update_tor.sh)
May 26 07:00:00 mr-fox CROND[2545]: (tinderbox) CMD (sudo /opt/tb/bin/collect_data.sh 2>/dev/null)
May 26 07:00:00 mr-fox CROND[2553]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:00:00 mr-fox CROND[2548]: (tinderbox) CMD (sudo /opt/tb/bin/house_keeping.sh >/dev/null)
May 26 07:00:00 mr-fox CROND[2547]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:00:00 mr-fox CROND[2546]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:00:00 mr-fox CROND[2554]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 07:00:00 mr-fox CROND[2524]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:00:00 mr-fox CROND[2524]: pam_unix(crond:session): session closed for user torproject
May 26 07:00:00 mr-fox CROND[2537]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:00:00 mr-fox CROND[2537]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:00:00 mr-fox CROND[2531]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:00:00 mr-fox CROND[2531]: pam_unix(crond:session): session closed for user root
May 26 07:00:00 mr-fox sudo[2561]: tinderbox : PWD=/home/tinderbox ; USER=root ; COMMAND=/opt/tb/bin/house_keeping.sh
May 26 07:00:00 mr-fox sudo[2560]: tinderbox : PWD=/home/tinderbox ; USER=root ; COMMAND=/opt/tb/bin/collect_data.sh
May 26 07:00:00 mr-fox sudo[2560]: pam_unix(sudo:session): session opened for user root(uid=0) by tinderbox(uid=1003)
May 26 07:00:00 mr-fox sudo[2561]: pam_unix(sudo:session): session opened for user root(uid=0) by tinderbox(uid=1003)
May 26 07:00:00 mr-fox CROND[2533]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:00:00 mr-fox CROND[2533]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:00:00 mr-fox CROND[2530]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:00:00 mr-fox CROND[2530]: pam_unix(crond:session): session closed for user root
May 26 07:00:01 mr-fox CROND[32011]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:00:01 mr-fox CROND[32011]: pam_unix(crond:session): session closed for user root
May 26 07:00:02 mr-fox CROND[2528]: (root) CMDEND (/opt/torutils/update_tor.sh)
May 26 07:00:02 mr-fox CROND[2528]: pam_unix(crond:session): session closed for user root
May 26 07:00:03 mr-fox CROND[2527]: (root) CMDEND (/etc/conf.d/ipv4-rules.sh update; /etc/conf.d/ipv6-rules.sh update)
May 26 07:00:03 mr-fox CROND[2527]: pam_unix(crond:session): session closed for user root
May 26 07:01:00 mr-fox CROND[2345]: (root) CMD (run-parts /etc/cron.hourly)
May 26 07:01:00 mr-fox crond[2341]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:01:00 mr-fox crond[2340]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:01:00 mr-fox crond[2344]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:01:00 mr-fox crond[2343]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:01:00 mr-fox crond[2346]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:01:00 mr-fox CROND[2350]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:01:00 mr-fox CROND[2352]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:01:00 mr-fox CROND[2353]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:01:00 mr-fox CROND[2355]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:01:00 mr-fox CROND[2351]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:01:00 mr-fox CROND[2339]: (root) CMDEND (run-parts /etc/cron.hourly)
May 26 07:01:00 mr-fox CROND[2346]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:01:00 mr-fox CROND[2346]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:01:00 mr-fox CROND[2344]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:01:00 mr-fox CROND[2344]: pam_unix(crond:session): session closed for user root
May 26 07:01:00 mr-fox CROND[2343]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:01:00 mr-fox CROND[2343]: pam_unix(crond:session): session closed for user root
May 26 07:01:01 mr-fox CROND[2526]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:01:01 mr-fox CROND[2526]: pam_unix(crond:session): session closed for user root
May 26 07:02:00 mr-fox crond[4484]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:02:00 mr-fox crond[4486]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:02:00 mr-fox crond[4487]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:02:00 mr-fox crond[4488]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:02:00 mr-fox CROND[4492]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:02:00 mr-fox crond[4483]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:02:00 mr-fox CROND[4493]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:02:00 mr-fox CROND[4494]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:02:00 mr-fox CROND[4495]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:02:00 mr-fox CROND[4496]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:02:00 mr-fox CROND[4488]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:02:00 mr-fox CROND[4488]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:02:00 mr-fox CROND[4487]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:02:00 mr-fox CROND[4487]: pam_unix(crond:session): session closed for user root
May 26 07:02:00 mr-fox CROND[4486]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:02:00 mr-fox CROND[4486]: pam_unix(crond:session): session closed for user root
May 26 07:02:00 mr-fox CROND[2341]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:02:00 mr-fox CROND[2341]: pam_unix(crond:session): session closed for user root
May 26 07:02:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:02:44 mr-fox kernel: rcu: \x0921-....: (915076 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=356210
May 26 07:02:44 mr-fox kernel: rcu: \x09(t=915077 jiffies g=8794409 q=29608275 ncpus=32)
May 26 07:02:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:02:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:02:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:02:44 mr-fox kernel: RIP: 0010:xas_start+0x2b/0x120
May 26 07:02:44 mr-fox kernel: Code: 53 48 89 fb 48 83 ec 08 48 8b 6f 18 48 89 e8 83 e0 03 0f 84 81 00 00 00 48 81 fd 05 c0 ff ff 76 06 48 83 f8 02 74 5d 48 8b 03 <48> 8b 73 08 48 8b 40 08 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d
May 26 07:02:44 mr-fox kernel: RSP: 0018:ffffa401077e3930 EFLAGS: 00000213
May 26 07:02:44 mr-fox kernel: RAX: ffff988ac61fdda0 RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:02:44 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffa401077e3970
May 26 07:02:44 mr-fox kernel: RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
May 26 07:02:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:02:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:02:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:02:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:02:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:02:44 mr-fox kernel: PKRU: 55555554
May 26 07:02:44 mr-fox kernel: Call Trace:
May 26 07:02:44 mr-fox kernel:  <IRQ>
May 26 07:02:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:02:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:02:44 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 07:02:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:02:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:02:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:02:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:02:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:02:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:02:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:02:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:02:44 mr-fox kernel:  </IRQ>
May 26 07:02:44 mr-fox kernel:  <TASK>
May 26 07:02:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:02:44 mr-fox kernel:  ? xas_start+0x2b/0x120
May 26 07:02:44 mr-fox kernel:  xas_load+0xe/0x60
May 26 07:02:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:02:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:02:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:02:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:02:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:02:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:02:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:02:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:02:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:02:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:02:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:02:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:02:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:02:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:02:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:02:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:02:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:02:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:02:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:02:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:02:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:02:44 mr-fox kernel:  </TASK>
May 26 07:02:53 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 917103 jiffies s: 491905 root: 0x2/.
May 26 07:02:53 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:02:53 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:02:53 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:02:53 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:02:53 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:02:53 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:02:53 mr-fox kernel: RIP: 0010:iommu_dma_free_iova+0xb3/0x220
May 26 07:02:53 mr-fox kernel: Code: 93 88 00 00 00 65 48 03 15 0a f1 95 5a 49 89 d7 4c 89 ff e8 bf 75 2f 00 4c 89 fe 48 89 df 48 89 04 24 e8 80 f4 ff ff 41 8b 07 <85> c0 0f 84 60 01 00 00 41 8b 47 08 8d 48 01 41 23 4f 0c 41 3b 4f
May 26 07:02:53 mr-fox kernel: RSP: 0018:ffffa401005f0d08 EFLAGS: 00000046
May 26 07:02:53 mr-fox kernel: RAX: 0000000000000001 RBX: ffff988ac1292400 RCX: 000000000000000c
May 26 07:02:53 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:02:53 mr-fox kernel: RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
May 26 07:02:53 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000c9614
May 26 07:02:53 mr-fox kernel: R13: ffffa401005f0d68 R14: ffffa401005f0d50 R15: ffffc400ffb51870
May 26 07:02:53 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:02:53 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:02:53 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:02:53 mr-fox kernel: PKRU: 55555554
May 26 07:02:53 mr-fox kernel: Call Trace:
May 26 07:02:53 mr-fox kernel:  <NMI>
May 26 07:02:53 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:02:53 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:02:53 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:02:53 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:02:53 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:02:53 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:02:53 mr-fox kernel:  ? iommu_dma_free_iova+0xb3/0x220
May 26 07:02:53 mr-fox kernel:  ? iommu_dma_free_iova+0xb3/0x220
May 26 07:02:53 mr-fox kernel:  ? iommu_dma_free_iova+0xb3/0x220
May 26 07:02:53 mr-fox kernel:  </NMI>
May 26 07:02:53 mr-fox kernel:  <IRQ>
May 26 07:02:53 mr-fox kernel:  __iommu_dma_unmap+0xe0/0x170
May 26 07:02:53 mr-fox kernel:  iommu_dma_unmap_page+0x43/0xa0
May 26 07:02:53 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 07:02:53 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 07:02:53 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:02:53 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 07:02:53 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 07:02:53 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 07:02:53 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:02:53 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:02:53 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:02:53 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:02:53 mr-fox kernel:  </IRQ>
May 26 07:02:53 mr-fox kernel:  <TASK>
May 26 07:02:53 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:02:53 mr-fox kernel: RIP: 0010:xas_load+0x35/0x60
May 26 07:02:53 mr-fox kernel: Code: f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 <48> 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d
May 26 07:02:53 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000286
May 26 07:02:53 mr-fox kernel: RAX: ffff988ac07566ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:02:53 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:02:53 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:02:53 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:02:53 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:02:53 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:02:53 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:02:53 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:02:53 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:02:53 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:02:53 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:02:53 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:02:53 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:02:53 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:02:53 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:02:53 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:02:53 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:02:53 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:02:53 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:02:53 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:02:53 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:02:53 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:02:53 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:02:53 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:02:53 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:02:53 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:02:53 mr-fox kernel:  </TASK>
May 26 07:03:00 mr-fox crond[6492]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:03:00 mr-fox crond[6493]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:03:00 mr-fox crond[6495]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:03:00 mr-fox crond[6496]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:03:00 mr-fox CROND[6499]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:03:00 mr-fox CROND[6501]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:03:00 mr-fox crond[6494]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:03:00 mr-fox CROND[6502]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:03:00 mr-fox CROND[6503]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:03:00 mr-fox CROND[6504]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:03:00 mr-fox CROND[6496]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:03:00 mr-fox CROND[6496]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:03:00 mr-fox CROND[6495]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:03:00 mr-fox CROND[6495]: pam_unix(crond:session): session closed for user root
May 26 07:03:00 mr-fox CROND[6494]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:03:00 mr-fox CROND[6494]: pam_unix(crond:session): session closed for user root
May 26 07:03:00 mr-fox CROND[4484]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:03:00 mr-fox CROND[4484]: pam_unix(crond:session): session closed for user root
May 26 07:04:00 mr-fox crond[10549]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:04:00 mr-fox crond[10550]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:04:00 mr-fox crond[10551]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:04:00 mr-fox crond[10548]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:04:00 mr-fox crond[10552]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:04:00 mr-fox CROND[10556]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:04:00 mr-fox CROND[10557]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:04:00 mr-fox CROND[10558]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:04:00 mr-fox CROND[10559]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:04:00 mr-fox CROND[10560]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:04:00 mr-fox CROND[10552]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:04:00 mr-fox CROND[10552]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:04:00 mr-fox CROND[10551]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:04:00 mr-fox CROND[10551]: pam_unix(crond:session): session closed for user root
May 26 07:04:01 mr-fox CROND[10550]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:04:01 mr-fox CROND[10550]: pam_unix(crond:session): session closed for user root
May 26 07:04:01 mr-fox CROND[6493]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:04:01 mr-fox CROND[6493]: pam_unix(crond:session): session closed for user root
May 26 07:05:00 mr-fox crond[13200]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:05:00 mr-fox crond[13199]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 07:05:00 mr-fox crond[13201]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:05:00 mr-fox crond[13205]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:05:00 mr-fox crond[13204]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:05:00 mr-fox crond[13206]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:05:00 mr-fox crond[13203]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:05:00 mr-fox CROND[13209]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:05:00 mr-fox CROND[13210]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:05:00 mr-fox crond[13207]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:05:00 mr-fox CROND[13211]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:05:00 mr-fox CROND[13212]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 07:05:00 mr-fox CROND[13213]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:05:00 mr-fox CROND[13214]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:05:00 mr-fox CROND[13215]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:05:00 mr-fox CROND[13216]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:05:00 mr-fox CROND[13199]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:05:00 mr-fox CROND[13199]: pam_unix(crond:session): session closed for user torproject
May 26 07:05:00 mr-fox CROND[13207]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:05:00 mr-fox CROND[13207]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:05:00 mr-fox CROND[13204]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:05:00 mr-fox CROND[13204]: pam_unix(crond:session): session closed for user root
May 26 07:05:00 mr-fox CROND[13205]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:05:00 mr-fox CROND[13205]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:05:00 mr-fox CROND[13203]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:05:00 mr-fox CROND[13203]: pam_unix(crond:session): session closed for user root
May 26 07:05:01 mr-fox CROND[10549]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:05:01 mr-fox CROND[10549]: pam_unix(crond:session): session closed for user root
May 26 07:05:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:05:44 mr-fox kernel: rcu: \x0921-....: (960080 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=374122
May 26 07:05:44 mr-fox kernel: rcu: \x09(t=960081 jiffies g=8794409 q=30803666 ncpus=32)
May 26 07:05:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:05:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:05:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:05:44 mr-fox kernel: RIP: 0010:xas_load+0x35/0x60
May 26 07:05:44 mr-fox kernel: Code: f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 <48> 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d
May 26 07:05:44 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000286
May 26 07:05:44 mr-fox kernel: RAX: ffff988ac07566ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:05:44 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:05:44 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:05:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:05:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:05:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:05:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:05:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:05:44 mr-fox kernel: PKRU: 55555554
May 26 07:05:44 mr-fox kernel: Call Trace:
May 26 07:05:44 mr-fox kernel:  <IRQ>
May 26 07:05:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:05:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:05:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:05:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:05:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:05:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:05:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:05:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:05:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:05:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:05:44 mr-fox kernel:  </IRQ>
May 26 07:05:44 mr-fox kernel:  <TASK>
May 26 07:05:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:05:44 mr-fox kernel:  ? xas_load+0x35/0x60
May 26 07:05:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:05:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:05:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:05:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:05:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:05:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:05:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:05:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:05:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:05:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:05:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:05:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:05:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:05:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:05:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:05:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:05:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:05:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:05:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:05:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:05:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:05:44 mr-fox kernel:  </TASK>
May 26 07:05:53 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 962159 jiffies s: 491905 root: 0x2/.
May 26 07:05:53 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:05:53 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:05:53 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:05:53 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:05:53 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:05:53 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:05:53 mr-fox kernel: RIP: 0010:skb_clone+0x21/0xd0
May 26 07:05:53 mr-fox kernel: Code: 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 53 48 89 fb 48 83 ec 08 48 85 ff 74 1c 8b 87 b8 00 00 00 48 03 87 c0 00 00 00 0f b6 10 <f6> c2 01 74 07 48 83 78 28 00 75 71 0f b6 43 7e 89 c2 83 e2 0c 80
May 26 07:05:53 mr-fox kernel: RSP: 0018:ffffa401005f0d50 EFLAGS: 00000282
May 26 07:05:53 mr-fox kernel: RAX: ffff98998f234e80 RBX: ffff988dc55d41c0 RCX: 0000000000000801
May 26 07:05:53 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000820 RDI: ffff988dc55d41c0
May 26 07:05:53 mr-fox kernel: RBP: ffff988dc55d41c0 R08: 0000000088737df5 R09: 0000000000000000
May 26 07:05:53 mr-fox kernel: R10: 0000000000000001 R11: 0000000000000000 R12: ffff988dc55d4218
May 26 07:05:53 mr-fox kernel: R13: 0000000088737df5 R14: 00002154c5d8553e R15: 00000000000005a8
May 26 07:05:53 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:05:53 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:05:53 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:05:53 mr-fox kernel: PKRU: 55555554
May 26 07:05:53 mr-fox kernel: Call Trace:
May 26 07:05:53 mr-fox kernel:  <NMI>
May 26 07:05:53 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:05:53 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:05:53 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:05:53 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:05:53 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:05:53 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:05:53 mr-fox kernel:  ? skb_clone+0x21/0xd0
May 26 07:05:53 mr-fox kernel:  ? skb_clone+0x21/0xd0
May 26 07:05:53 mr-fox kernel:  ? skb_clone+0x21/0xd0
May 26 07:05:53 mr-fox kernel:  </NMI>
May 26 07:05:53 mr-fox kernel:  <IRQ>
May 26 07:05:53 mr-fox kernel:  __tcp_transmit_skb+0x68c/0xd30
May 26 07:05:53 mr-fox kernel:  tcp_write_xmit+0x4c9/0x13b0
May 26 07:05:53 mr-fox kernel:  ? ktime_get+0x42/0xb0
May 26 07:05:53 mr-fox kernel:  tcp_send_loss_probe+0x16f/0x260
May 26 07:05:53 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 07:05:53 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 07:05:53 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 07:05:53 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 07:05:53 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 07:05:53 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:05:53 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:05:53 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:05:53 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:05:53 mr-fox kernel:  </IRQ>
May 26 07:05:53 mr-fox kernel:  <TASK>
May 26 07:05:53 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:05:53 mr-fox kernel: RIP: 0010:xas_load+0x22/0x60
May 26 07:05:53 mr-fox kernel: Code: be 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 53 48 89 fb e8 f2 f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f <5b> 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68
May 26 07:05:53 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 07:05:53 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:05:53 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:05:53 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:05:53 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:05:53 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:05:53 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:05:53 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:05:53 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:05:53 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:05:53 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:05:53 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:05:53 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:05:53 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:05:53 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:05:53 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:05:53 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:05:53 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:05:53 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:05:53 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:05:53 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:05:53 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:05:53 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:05:53 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:05:53 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:05:53 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:05:53 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:05:53 mr-fox kernel:  </TASK>
May 26 07:06:00 mr-fox crond[15611]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:06:00 mr-fox crond[15610]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:06:00 mr-fox crond[15612]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:06:00 mr-fox crond[15613]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:06:00 mr-fox crond[15614]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:06:00 mr-fox CROND[15617]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:06:00 mr-fox CROND[15618]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:06:00 mr-fox CROND[15620]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:06:00 mr-fox CROND[15622]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:06:00 mr-fox CROND[15623]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:06:00 mr-fox CROND[15614]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:06:00 mr-fox CROND[15614]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:06:00 mr-fox CROND[15613]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:06:00 mr-fox CROND[15613]: pam_unix(crond:session): session closed for user root
May 26 07:06:00 mr-fox CROND[15612]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:06:00 mr-fox CROND[15612]: pam_unix(crond:session): session closed for user root
May 26 07:06:00 mr-fox CROND[13201]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:06:00 mr-fox CROND[13201]: pam_unix(crond:session): session closed for user root
May 26 07:07:00 mr-fox crond[19516]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:07:00 mr-fox crond[19519]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:07:00 mr-fox crond[19520]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:07:00 mr-fox crond[19521]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:07:00 mr-fox crond[19518]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:07:00 mr-fox CROND[19525]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:07:00 mr-fox CROND[19526]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:07:00 mr-fox CROND[19527]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:07:00 mr-fox CROND[19528]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:07:00 mr-fox CROND[19529]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:07:00 mr-fox CROND[19521]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:07:00 mr-fox CROND[19521]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:07:00 mr-fox CROND[19520]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:07:00 mr-fox CROND[19520]: pam_unix(crond:session): session closed for user root
May 26 07:07:00 mr-fox CROND[19519]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:07:00 mr-fox CROND[19519]: pam_unix(crond:session): session closed for user root
May 26 07:07:00 mr-fox CROND[15611]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:07:00 mr-fox CROND[15611]: pam_unix(crond:session): session closed for user root
May 26 07:08:00 mr-fox crond[13553]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:08:00 mr-fox crond[13554]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:08:00 mr-fox crond[13555]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:08:00 mr-fox crond[13556]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:08:00 mr-fox CROND[13561]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:08:00 mr-fox crond[13558]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:08:00 mr-fox CROND[13562]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:08:00 mr-fox CROND[13565]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:08:00 mr-fox CROND[13566]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:08:00 mr-fox CROND[13568]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:08:01 mr-fox CROND[13558]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:08:01 mr-fox CROND[13558]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:08:01 mr-fox CROND[13556]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:08:01 mr-fox CROND[13556]: pam_unix(crond:session): session closed for user root
May 26 07:08:01 mr-fox CROND[13555]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:08:01 mr-fox CROND[13555]: pam_unix(crond:session): session closed for user root
May 26 07:08:01 mr-fox CROND[19518]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:08:01 mr-fox CROND[19518]: pam_unix(crond:session): session closed for user root
May 26 07:08:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:08:44 mr-fox kernel: rcu: \x0921-....: (1005084 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=391805
May 26 07:08:44 mr-fox kernel: rcu: \x09(t=1005085 jiffies g=8794409 q=32038023 ncpus=32)
May 26 07:08:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:08:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:08:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:08:44 mr-fox kernel: RIP: 0010:filemap_get_entry+0x6d/0x160
May 26 07:08:44 mr-fox kernel: Code: 00 00 48 c7 44 24 30 00 00 00 00 48 c7 44 24 38 00 00 00 00 48 c7 44 24 20 03 00 00 00 48 8d 7c 24 08 e8 56 70 78 00 48 89 c3 <48> 3d 02 04 00 00 74 e2 48 3d 06 04 00 00 74 da 48 85 c0 74 5a a8
May 26 07:08:44 mr-fox kernel: RSP: 0018:ffffa401077e3968 EFLAGS: 00000246
May 26 07:08:44 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 07:08:44 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:08:44 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 07:08:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:08:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:08:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:08:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:08:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:08:44 mr-fox kernel: PKRU: 55555554
May 26 07:08:44 mr-fox kernel: Call Trace:
May 26 07:08:44 mr-fox kernel:  <IRQ>
May 26 07:08:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:08:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:08:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:08:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:08:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:08:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:08:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:08:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:08:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:08:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:08:44 mr-fox kernel:  </IRQ>
May 26 07:08:44 mr-fox kernel:  <TASK>
May 26 07:08:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:08:44 mr-fox kernel:  ? filemap_get_entry+0x6d/0x160
May 26 07:08:44 mr-fox kernel:  ? filemap_get_entry+0x6a/0x160
May 26 07:08:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:08:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:08:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:08:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:08:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:08:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:08:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:08:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:08:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:08:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:08:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:08:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:08:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:08:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:08:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:08:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:08:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:08:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:08:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:08:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:08:44 mr-fox kernel:  </TASK>
May 26 07:08:54 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1007216 jiffies s: 491905 root: 0x2/.
May 26 07:08:54 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:08:54 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:08:54 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:08:54 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:08:54 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:08:54 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:08:54 mr-fox kernel: RIP: 0010:kmem_cache_free_bulk+0x2fa/0x310
May 26 07:08:54 mr-fox kernel: Code: fe ff ff 4c 8b 4c 24 08 44 8b 44 24 14 4c 89 d1 48 89 da 48 8b 74 24 48 4c 89 f7 4c 89 5c 24 40 e8 6b b2 ff ff 4c 8b 5c 24 40 <eb> 85 0f 0b e8 cd 1a 76 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90
May 26 07:08:54 mr-fox kernel: RSP: 0018:ffffa401005f0d10 EFLAGS: 00000246
May 26 07:08:54 mr-fox kernel: RAX: 0000000000000000 RBX: ffff988b2447cb00 RCX: 0000000000000000
May 26 07:08:54 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:08:54 mr-fox kernel: RBP: ffffa401005f0dc0 R08: 0000000000000000 R09: 0000000000000000
May 26 07:08:54 mr-fox kernel: R10: 0000000000000000 R11: 000000000000000e R12: ffff988b2447cb00
May 26 07:08:54 mr-fox kernel: R13: ffff988ac0137b00 R14: ffff988ac0137b00 R15: ffff98a96ed60970
May 26 07:08:54 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:08:54 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:08:54 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:08:54 mr-fox kernel: PKRU: 55555554
May 26 07:08:54 mr-fox kernel: Call Trace:
May 26 07:08:54 mr-fox kernel:  <NMI>
May 26 07:08:54 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:08:54 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:08:54 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:08:54 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:08:54 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:08:54 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:08:54 mr-fox kernel:  ? kmem_cache_free_bulk+0x2fa/0x310
May 26 07:08:54 mr-fox kernel:  ? kmem_cache_free_bulk+0x2fa/0x310
May 26 07:08:54 mr-fox kernel:  ? kmem_cache_free_bulk+0x2fa/0x310
May 26 07:08:54 mr-fox kernel:  </NMI>
May 26 07:08:54 mr-fox kernel:  <IRQ>
May 26 07:08:54 mr-fox kernel:  ? napi_skb_cache_put+0x80/0xc0
May 26 07:08:54 mr-fox kernel:  napi_skb_cache_put+0x80/0xc0
May 26 07:08:54 mr-fox kernel:  igb_poll+0xea/0x1370
May 26 07:08:54 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 07:08:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:08:54 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 07:08:54 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 07:08:54 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 07:08:54 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 07:08:54 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:08:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:08:54 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:08:54 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:08:54 mr-fox kernel:  </IRQ>
May 26 07:08:54 mr-fox kernel:  <TASK>
May 26 07:08:54 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:08:54 mr-fox kernel: RIP: 0010:xas_descend+0x1a/0xd0
May 26 07:08:54 mr-fox kernel: Code: 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f <0f> 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5
May 26 07:08:54 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000293
May 26 07:08:54 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: 000000004cd994a1 RCX: 000000000000000c
May 26 07:08:54 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988b1fdbb478 RDI: ffffa401077e3970
May 26 07:08:54 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 07:08:54 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 07:08:54 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:08:54 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:08:54 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:08:54 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:08:54 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:08:54 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:08:54 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:08:54 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:08:54 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:08:54 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:08:54 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:08:54 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:08:54 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:08:54 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:08:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:08:54 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:08:54 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:08:54 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:08:54 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:08:54 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:08:54 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:08:54 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:08:54 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:08:54 mr-fox kernel:  </TASK>
May 26 07:09:00 mr-fox crond[15081]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:09:00 mr-fox crond[15082]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:09:00 mr-fox crond[15084]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:09:00 mr-fox crond[15085]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:09:00 mr-fox crond[15086]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:09:00 mr-fox CROND[15089]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:09:00 mr-fox CROND[15090]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:09:00 mr-fox CROND[15091]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:09:00 mr-fox CROND[15093]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:09:00 mr-fox CROND[15092]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:09:00 mr-fox CROND[15086]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:09:00 mr-fox CROND[15086]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:09:00 mr-fox CROND[15085]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:09:00 mr-fox CROND[15085]: pam_unix(crond:session): session closed for user root
May 26 07:09:00 mr-fox CROND[15084]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:09:00 mr-fox CROND[15084]: pam_unix(crond:session): session closed for user root
May 26 07:09:01 mr-fox CROND[13554]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:09:01 mr-fox CROND[13554]: pam_unix(crond:session): session closed for user root
May 26 07:10:00 mr-fox crond[18098]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 07:10:00 mr-fox crond[18104]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:10:00 mr-fox crond[18101]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:10:00 mr-fox crond[18100]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:10:00 mr-fox crond[18102]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:10:00 mr-fox crond[18103]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:10:00 mr-fox crond[18105]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:10:00 mr-fox crond[18106]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:10:00 mr-fox CROND[18112]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:10:00 mr-fox CROND[18113]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:10:00 mr-fox CROND[18115]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:10:00 mr-fox CROND[18114]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:10:00 mr-fox CROND[18116]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:10:00 mr-fox CROND[18117]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 07:10:00 mr-fox CROND[18118]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:10:00 mr-fox CROND[18119]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:10:00 mr-fox CROND[18098]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:10:00 mr-fox CROND[18098]: pam_unix(crond:session): session closed for user torproject
May 26 07:10:00 mr-fox CROND[18106]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:10:00 mr-fox CROND[18106]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:10:00 mr-fox CROND[18103]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:10:00 mr-fox CROND[18103]: pam_unix(crond:session): session closed for user root
May 26 07:10:00 mr-fox CROND[18104]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:10:00 mr-fox CROND[18104]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:10:00 mr-fox CROND[18102]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:10:00 mr-fox CROND[18102]: pam_unix(crond:session): session closed for user root
May 26 07:10:00 mr-fox CROND[15082]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:10:00 mr-fox CROND[15082]: pam_unix(crond:session): session closed for user root
May 26 07:11:00 mr-fox crond[21417]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:11:00 mr-fox crond[21416]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:11:00 mr-fox crond[21418]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:11:00 mr-fox crond[21419]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:11:00 mr-fox CROND[21425]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:11:00 mr-fox CROND[21426]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:11:00 mr-fox CROND[21428]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:11:00 mr-fox CROND[21427]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:11:00 mr-fox crond[21421]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:11:00 mr-fox CROND[21431]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:11:00 mr-fox CROND[21421]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:11:00 mr-fox CROND[21421]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:11:00 mr-fox CROND[21419]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:11:00 mr-fox CROND[21419]: pam_unix(crond:session): session closed for user root
May 26 07:11:00 mr-fox CROND[21418]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:11:00 mr-fox CROND[21418]: pam_unix(crond:session): session closed for user root
May 26 07:11:01 mr-fox CROND[18101]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:11:01 mr-fox CROND[18101]: pam_unix(crond:session): session closed for user root
May 26 07:11:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:11:44 mr-fox kernel: rcu: \x0921-....: (1050088 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=409781
May 26 07:11:44 mr-fox kernel: rcu: \x09(t=1050089 jiffies g=8794409 q=33148757 ncpus=32)
May 26 07:11:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:11:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:11:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:11:44 mr-fox kernel: RIP: 0010:xas_load+0xe/0x60
May 26 07:11:44 mr-fox kernel: Code: 00 00 eb 92 4c 89 ee 48 c7 c7 e0 eb 49 a6 e8 a9 58 c6 ff eb be 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 53 48 89 fb e8 f2 f7 ff ff <48> 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d
May 26 07:11:44 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 07:11:44 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:11:44 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:11:44 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 07:11:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:11:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:11:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:11:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:11:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:11:44 mr-fox kernel: PKRU: 55555554
May 26 07:11:44 mr-fox kernel: Call Trace:
May 26 07:11:44 mr-fox kernel:  <IRQ>
May 26 07:11:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:11:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:11:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:11:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:11:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:11:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:11:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:11:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:11:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:11:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:11:44 mr-fox kernel:  </IRQ>
May 26 07:11:44 mr-fox kernel:  <TASK>
May 26 07:11:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:11:44 mr-fox kernel:  ? xas_load+0xe/0x60
May 26 07:11:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:11:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:11:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:11:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:11:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:11:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:11:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:11:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:11:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:11:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:11:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:11:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:11:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:11:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:11:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:11:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:11:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:11:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:11:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:11:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:11:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:11:44 mr-fox kernel:  </TASK>
May 26 07:11:54 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1052272 jiffies s: 491905 root: 0x2/.
May 26 07:11:54 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:11:54 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:11:54 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:11:54 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:11:54 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:11:54 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:11:54 mr-fox kernel: RIP: 0010:iommu_v1_map_pages+0x629/0x8f0
May 26 07:11:54 mr-fox kernel: Code: b4 24 00 01 00 00 eb 1e 8d 42 fe 83 f8 04 77 0d 48 8d b4 24 b0 00 00 00 e8 24 f3 ff ff 49 83 c4 08 4d 39 ec 74 53 49 8b 14 24 <31> f6 48 89 d0 f0 49 0f b1 34 24 0f 85 9a 01 00 00 f6 c2 01 74 dd
May 26 07:11:54 mr-fox kernel: RSP: 0018:ffffa401005f08b0 EFLAGS: 00000286
May 26 07:11:54 mr-fox kernel: RAX: ffff988acd41f000 RBX: 6000000000000001 RCX: 000000000000000c
May 26 07:11:54 mr-fox kernel: RDX: 0000000000000000 RSI: 000ffffffffff000 RDI: 0000000000000000
May 26 07:11:54 mr-fox kernel: RBP: 000ffffffffff000 R08: ffff988acd41fa30 R09: 0000000000000001
May 26 07:11:54 mr-fox kernel: R10: 0000000000001000 R11: ffffa401005f09f0 R12: ffff988acd41fa30
May 26 07:11:54 mr-fox kernel: R13: ffff988acd41fa38 R14: 0000000000000820 R15: ffff988acd41fa30
May 26 07:11:54 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:11:54 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:11:54 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:11:54 mr-fox kernel: PKRU: 55555554
May 26 07:11:54 mr-fox kernel: Call Trace:
May 26 07:11:54 mr-fox kernel:  <NMI>
May 26 07:11:54 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:11:54 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:11:54 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:11:54 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:11:54 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:11:54 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:11:54 mr-fox kernel:  ? iommu_v1_map_pages+0x629/0x8f0
May 26 07:11:54 mr-fox kernel:  ? iommu_v1_map_pages+0x629/0x8f0
May 26 07:11:54 mr-fox kernel:  ? iommu_v1_map_pages+0x629/0x8f0
May 26 07:11:54 mr-fox kernel:  </NMI>
May 26 07:11:54 mr-fox kernel:  <IRQ>
May 26 07:11:54 mr-fox kernel:  __iommu_map+0x106/0x1b0
May 26 07:11:54 mr-fox kernel:  iommu_map+0x27/0xa0
May 26 07:11:54 mr-fox kernel:  __iommu_dma_map+0x87/0xf0
May 26 07:11:54 mr-fox kernel:  iommu_dma_map_page+0xc2/0x230
May 26 07:11:54 mr-fox kernel:  igb_xmit_frame_ring+0x91b/0xc00
May 26 07:11:54 mr-fox kernel:  ? netif_skb_features+0x93/0x2c0
May 26 07:11:54 mr-fox kernel:  dev_hard_start_xmit+0xa0/0xf0
May 26 07:11:54 mr-fox kernel:  sch_direct_xmit+0x8d/0x290
May 26 07:11:54 mr-fox kernel:  __dev_queue_xmit+0x49a/0x9a0
May 26 07:11:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:11:54 mr-fox kernel:  ip_finish_output2+0x258/0x500
May 26 07:11:54 mr-fox kernel:  __ip_queue_xmit+0x16b/0x480
May 26 07:11:54 mr-fox kernel:  __tcp_transmit_skb+0xbad/0xd30
May 26 07:11:54 mr-fox kernel:  __tcp_retransmit_skb+0x1a9/0x800
May 26 07:11:54 mr-fox kernel:  ? lock_timer_base+0x2f/0xc0
May 26 07:11:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:11:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:11:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:11:54 mr-fox kernel:  tcp_retransmit_skb+0x11/0xa0
May 26 07:11:54 mr-fox kernel:  tcp_retransmit_timer+0x492/0xa60
May 26 07:11:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:11:54 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 07:11:54 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 07:11:54 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 07:11:54 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 07:11:54 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 07:11:54 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:11:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:11:54 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:11:54 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:11:54 mr-fox kernel:  </IRQ>
May 26 07:11:54 mr-fox kernel:  <TASK>
May 26 07:11:54 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:11:54 mr-fox kernel: RIP: 0010:xas_load+0x35/0x60
May 26 07:11:54 mr-fox kernel: Code: f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 <48> 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d
May 26 07:11:54 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000286
May 26 07:11:54 mr-fox kernel: RAX: ffff988d55f18ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:11:54 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:11:54 mr-fox kernel: RBP: ffff988d667606d8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:11:54 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:11:54 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:11:54 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:11:54 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:11:54 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:11:54 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:11:54 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:11:54 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:11:54 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:11:54 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:11:54 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:11:54 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:11:54 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:11:54 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:11:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:11:54 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:11:54 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:11:54 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:11:54 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:11:54 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:11:54 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:11:54 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:11:54 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:11:54 mr-fox kernel:  </TASK>
May 26 07:12:00 mr-fox crond[25328]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:12:00 mr-fox crond[25329]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:12:00 mr-fox crond[25327]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:12:00 mr-fox crond[25330]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:12:00 mr-fox CROND[25333]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:12:00 mr-fox CROND[25334]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:12:00 mr-fox crond[25325]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:12:00 mr-fox CROND[25335]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:12:00 mr-fox CROND[25336]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:12:00 mr-fox CROND[25337]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:12:00 mr-fox CROND[25330]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:12:00 mr-fox CROND[25330]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:12:00 mr-fox CROND[25329]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:12:00 mr-fox CROND[25329]: pam_unix(crond:session): session closed for user root
May 26 07:12:00 mr-fox CROND[25328]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:12:00 mr-fox CROND[25328]: pam_unix(crond:session): session closed for user root
May 26 07:12:01 mr-fox CROND[21417]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:12:01 mr-fox CROND[21417]: pam_unix(crond:session): session closed for user root
May 26 07:13:00 mr-fox crond[27159]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:13:00 mr-fox crond[27161]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:13:00 mr-fox crond[27160]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:13:00 mr-fox crond[27162]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:13:00 mr-fox CROND[27165]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:13:00 mr-fox CROND[27166]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:13:00 mr-fox CROND[27168]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:13:00 mr-fox CROND[27167]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:13:00 mr-fox crond[27163]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:13:00 mr-fox CROND[27173]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:13:00 mr-fox CROND[27163]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:13:00 mr-fox CROND[27163]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:13:00 mr-fox CROND[27162]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:13:00 mr-fox CROND[27162]: pam_unix(crond:session): session closed for user root
May 26 07:13:00 mr-fox CROND[27161]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:13:00 mr-fox CROND[27161]: pam_unix(crond:session): session closed for user root
May 26 07:13:00 mr-fox CROND[25327]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:13:00 mr-fox CROND[25327]: pam_unix(crond:session): session closed for user root
May 26 07:14:00 mr-fox crond[29486]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:14:00 mr-fox crond[29487]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:14:00 mr-fox crond[29489]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:14:00 mr-fox crond[29490]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:14:00 mr-fox crond[29488]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:14:00 mr-fox CROND[29494]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:14:00 mr-fox CROND[29496]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:14:00 mr-fox CROND[29495]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:14:00 mr-fox CROND[29497]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:14:00 mr-fox CROND[29498]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:14:00 mr-fox CROND[29490]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:14:00 mr-fox CROND[29490]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:14:00 mr-fox CROND[29489]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:14:00 mr-fox CROND[29489]: pam_unix(crond:session): session closed for user root
May 26 07:14:00 mr-fox CROND[29488]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:14:00 mr-fox CROND[29488]: pam_unix(crond:session): session closed for user root
May 26 07:14:00 mr-fox CROND[27160]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:14:00 mr-fox CROND[27160]: pam_unix(crond:session): session closed for user root
May 26 07:14:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:14:44 mr-fox kernel: rcu: \x0921-....: (1095092 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=427902
May 26 07:14:44 mr-fox kernel: rcu: \x09(t=1095093 jiffies g=8794409 q=34188666 ncpus=32)
May 26 07:14:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:14:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:14:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:14:44 mr-fox kernel: RIP: 0010:xas_descend+0x26/0xd0
May 26 07:14:44 mr-fox kernel: Code: 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f <89> d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03
May 26 07:14:44 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 07:14:44 mr-fox kernel: RAX: ffff988acf90e6ca RBX: 0000000000000036 RCX: 0000000000000012
May 26 07:14:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988acf90e6c8 RDI: ffffa401077e3970
May 26 07:14:44 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:14:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 07:14:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:14:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:14:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:14:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:14:44 mr-fox kernel: PKRU: 55555554
May 26 07:14:44 mr-fox kernel: Call Trace:
May 26 07:14:44 mr-fox kernel:  <IRQ>
May 26 07:14:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:14:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:14:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:14:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:14:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:14:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:14:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:14:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:14:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:14:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:14:44 mr-fox kernel:  </IRQ>
May 26 07:14:44 mr-fox kernel:  <TASK>
May 26 07:14:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:14:44 mr-fox kernel:  ? xas_descend+0x26/0xd0
May 26 07:14:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:14:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:14:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:14:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:14:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:14:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:14:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:14:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:14:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:14:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:14:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:14:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:14:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:14:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:14:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:14:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:14:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:14:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:14:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:14:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:14:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:14:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:14:44 mr-fox kernel:  </TASK>
May 26 07:14:54 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1097328 jiffies s: 491905 root: 0x2/.
May 26 07:14:54 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:14:54 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:14:54 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:14:54 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:14:54 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:14:54 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:14:54 mr-fox kernel: RIP: 0010:clear_page_erms+0xb/0x20
May 26 07:14:54 mr-fox kernel: Code: 48 89 47 20 48 89 47 28 48 89 47 30 48 89 47 38 48 8d 7f 40 75 d9 90 e9 be 8a 1a 00 0f 1f 00 f3 0f 1e fa b9 00 10 00 00 31 c0 <f3> aa e9 a9 8a 1a 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 f3
May 26 07:14:54 mr-fox kernel: RSP: 0018:ffffa401005f0d58 EFLAGS: 00000246
May 26 07:14:54 mr-fox kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000920
May 26 07:14:54 mr-fox kernel: RDX: ffffcf62f4cf5740 RSI: ffffcf62f4cf5800 RDI: ffff9896f3d5d6e0
May 26 07:14:54 mr-fox kernel: RBP: ffffcf62f4cf5600 R08: 0000000000000001 R09: 0000000000000008
May 26 07:14:54 mr-fox kernel: R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000003
May 26 07:14:54 mr-fox kernel: R13: ffffcf62f4cf5800 R14: 0000000000000001 R15: 0000000000000008
May 26 07:14:54 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:14:54 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:14:54 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:14:54 mr-fox kernel: PKRU: 55555554
May 26 07:14:54 mr-fox kernel: Call Trace:
May 26 07:14:54 mr-fox kernel:  <NMI>
May 26 07:14:54 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:14:54 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:14:54 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:14:54 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:14:54 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:14:54 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:14:54 mr-fox kernel:  ? clear_page_erms+0xb/0x20
May 26 07:14:54 mr-fox kernel:  ? clear_page_erms+0xb/0x20
May 26 07:14:54 mr-fox kernel:  ? clear_page_erms+0xb/0x20
May 26 07:14:54 mr-fox kernel:  </NMI>
May 26 07:14:54 mr-fox kernel:  <IRQ>
May 26 07:14:54 mr-fox kernel:  free_unref_page_prepare+0x114/0x300
May 26 07:14:54 mr-fox kernel:  free_unref_page+0x2f/0x170
May 26 07:14:54 mr-fox kernel:  skb_release_data.isra.0+0xfb/0x1e0
May 26 07:14:54 mr-fox kernel:  __kfree_skb+0x24/0x30
May 26 07:14:54 mr-fox kernel:  tcp_write_queue_purge+0xed/0x310
May 26 07:14:54 mr-fox kernel:  tcp_retransmit_timer+0x28e/0xa60
May 26 07:14:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:14:54 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 07:14:54 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 07:14:54 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 07:14:54 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 07:14:54 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 07:14:54 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:14:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:14:54 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:14:54 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:14:54 mr-fox kernel:  </IRQ>
May 26 07:14:54 mr-fox kernel:  <TASK>
May 26 07:14:54 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:14:54 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 07:14:54 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 07:14:54 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 07:14:54 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:14:54 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:14:54 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:14:54 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:14:54 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:14:54 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:14:54 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:14:54 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:14:54 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:14:54 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:14:54 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:14:54 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:14:54 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:14:54 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:14:54 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:14:54 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:14:54 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:14:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:14:54 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:14:54 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:14:54 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:14:54 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:14:54 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:14:54 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:14:54 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:14:54 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:14:54 mr-fox kernel:  </TASK>
May 26 07:15:00 mr-fox crond[32184]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:15:00 mr-fox crond[32183]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:15:00 mr-fox crond[32180]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 07:15:00 mr-fox crond[32181]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:15:00 mr-fox crond[32185]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:15:00 mr-fox crond[32186]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:15:00 mr-fox CROND[32191]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:15:00 mr-fox CROND[32193]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:15:00 mr-fox CROND[32194]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:15:00 mr-fox CROND[32192]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:15:00 mr-fox crond[32189]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:15:00 mr-fox crond[32187]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:15:00 mr-fox CROND[32195]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:15:00 mr-fox CROND[32196]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:15:00 mr-fox CROND[32197]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:15:00 mr-fox CROND[32199]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 07:15:00 mr-fox CROND[32180]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:15:00 mr-fox CROND[32180]: pam_unix(crond:session): session closed for user torproject
May 26 07:15:00 mr-fox CROND[32189]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:15:00 mr-fox CROND[32189]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:15:00 mr-fox CROND[32185]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:15:00 mr-fox CROND[32185]: pam_unix(crond:session): session closed for user root
May 26 07:15:01 mr-fox CROND[32186]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:15:01 mr-fox CROND[32186]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:15:01 mr-fox CROND[32184]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:15:01 mr-fox CROND[32184]: pam_unix(crond:session): session closed for user root
May 26 07:15:01 mr-fox CROND[29487]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:15:01 mr-fox CROND[29487]: pam_unix(crond:session): session closed for user root
May 26 07:16:00 mr-fox crond[3442]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:16:00 mr-fox crond[3446]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:16:00 mr-fox crond[3443]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:16:00 mr-fox crond[3447]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:16:00 mr-fox crond[3444]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:16:00 mr-fox CROND[3450]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:16:00 mr-fox CROND[3451]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:16:00 mr-fox CROND[3452]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:16:00 mr-fox CROND[3456]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:16:00 mr-fox CROND[3457]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:16:00 mr-fox CROND[3447]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:16:00 mr-fox CROND[3447]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:16:00 mr-fox CROND[3446]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:16:00 mr-fox CROND[3446]: pam_unix(crond:session): session closed for user root
May 26 07:16:00 mr-fox CROND[3444]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:16:00 mr-fox CROND[3444]: pam_unix(crond:session): session closed for user root
May 26 07:16:01 mr-fox CROND[32183]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:16:01 mr-fox CROND[32183]: pam_unix(crond:session): session closed for user root
May 26 07:17:00 mr-fox crond[7744]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:17:00 mr-fox crond[7743]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:17:00 mr-fox crond[7747]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:17:00 mr-fox crond[7746]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:17:00 mr-fox crond[7745]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:17:00 mr-fox CROND[7751]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:17:00 mr-fox CROND[7752]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:17:00 mr-fox CROND[7753]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:17:00 mr-fox CROND[7754]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:17:00 mr-fox CROND[7755]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:17:00 mr-fox CROND[7747]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:17:00 mr-fox CROND[7747]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:17:00 mr-fox CROND[7746]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:17:00 mr-fox CROND[7746]: pam_unix(crond:session): session closed for user root
May 26 07:17:00 mr-fox CROND[7745]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:17:00 mr-fox CROND[7745]: pam_unix(crond:session): session closed for user root
May 26 07:17:00 mr-fox CROND[3443]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:17:00 mr-fox CROND[3443]: pam_unix(crond:session): session closed for user root
May 26 07:17:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:17:44 mr-fox kernel: rcu: \x0921-....: (1140096 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=445939
May 26 07:17:44 mr-fox kernel: rcu: \x09(t=1140097 jiffies g=8794409 q=35247467 ncpus=32)
May 26 07:17:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:17:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:17:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:17:44 mr-fox kernel: RIP: 0010:xas_load+0x11/0x60
May 26 07:17:44 mr-fox kernel: Code: 92 4c 89 ee 48 c7 c7 e0 eb 49 a6 e8 a9 58 c6 ff eb be 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 53 48 89 fb e8 f2 f7 ff ff 48 89 c2 <83> e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31
May 26 07:17:44 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 07:17:44 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:17:44 mr-fox kernel: RDX: ffff988ac0754ffa RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:17:44 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 07:17:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:17:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:17:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:17:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:17:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:17:44 mr-fox kernel: PKRU: 55555554
May 26 07:17:44 mr-fox kernel: Call Trace:
May 26 07:17:44 mr-fox kernel:  <IRQ>
May 26 07:17:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:17:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:17:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:17:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:17:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:17:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:17:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:17:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:17:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:17:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:17:44 mr-fox kernel:  </IRQ>
May 26 07:17:44 mr-fox kernel:  <TASK>
May 26 07:17:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:17:44 mr-fox kernel:  ? xas_load+0x11/0x60
May 26 07:17:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:17:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:17:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:17:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:17:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:17:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:17:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:17:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:17:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:17:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:17:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:17:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:17:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:17:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:17:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:17:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:17:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:17:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:17:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:17:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:17:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:17:44 mr-fox kernel:  </TASK>
May 26 07:17:54 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1142384 jiffies s: 491905 root: 0x2/.
May 26 07:17:54 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:17:54 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:17:54 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:17:54 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:17:54 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:17:54 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:17:54 mr-fox kernel: RIP: 0010:tcp4_gro_complete+0x2f/0x90
May 26 07:17:54 mr-fox kernel: Code: 57 70 48 8b 8f c0 00 00 00 31 c0 44 0f b7 87 b0 00 00 00 29 f2 83 c2 06 c1 e2 08 42 03 44 01 10 42 13 44 01 0c 11 d0 83 d0 00 <89> c2 66 31 c0 c1 e2 10 01 d0 15 ff ff 00 00 f7 d0 0f b7 97 ae 00
May 26 07:17:54 mr-fox kernel: RSP: 0018:ffffa401005f0d20 EFLAGS: 00000286
May 26 07:17:54 mr-fox kernel: RAX: 00000000d6b90b74 RBX: ffff988a7360d300 RCX: ffff989c858ef000
May 26 07:17:54 mr-fox kernel: RDX: 0000000000084400 RSI: 0000000000000014 RDI: ffff988a7360d300
May 26 07:17:54 mr-fox kernel: RBP: ffffffffa64a74c0 R08: 00000000000000ce R09: 000000000000dc05
May 26 07:17:54 mr-fox kernel: R10: 00000000ffff23fa R11: 0000000000000000 R12: ffff988ac873f850
May 26 07:17:54 mr-fox kernel: R13: 0000000000000081 R14: ffff988ac873f850 R15: 0000000000000002
May 26 07:17:54 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:17:54 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:17:54 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:17:54 mr-fox kernel: PKRU: 55555554
May 26 07:17:54 mr-fox kernel: Call Trace:
May 26 07:17:54 mr-fox kernel:  <NMI>
May 26 07:17:54 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:17:54 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:17:54 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:17:54 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:17:54 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:17:54 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:17:54 mr-fox kernel:  ? tcp4_gro_complete+0x2f/0x90
May 26 07:17:54 mr-fox kernel:  ? tcp4_gro_complete+0x2f/0x90
May 26 07:17:54 mr-fox kernel:  ? tcp4_gro_complete+0x2f/0x90
May 26 07:17:54 mr-fox kernel:  </NMI>
May 26 07:17:54 mr-fox kernel:  <IRQ>
May 26 07:17:54 mr-fox kernel:  napi_gro_complete.constprop.0+0x17c/0x190
May 26 07:17:54 mr-fox kernel:  dev_gro_receive+0x43a/0x7b0
May 26 07:17:54 mr-fox kernel:  napi_gro_receive+0x67/0x1b0
May 26 07:17:54 mr-fox kernel:  igb_poll+0x605/0x1370
May 26 07:17:54 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 07:17:54 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 07:17:54 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 07:17:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:17:54 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:17:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:17:54 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:17:54 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:17:54 mr-fox kernel:  </IRQ>
May 26 07:17:54 mr-fox kernel:  <TASK>
May 26 07:17:54 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:17:54 mr-fox kernel: RIP: 0010:xas_descend+0x31/0xd0
May 26 07:17:54 mr-fox kernel: Code: 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 <49> 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d fd 00 00
May 26 07:17:54 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 07:17:54 mr-fox kernel: RAX: ffff988ac07566ca RBX: 0000000000000001 RCX: 000000000000001e
May 26 07:17:54 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac0754ff8 RDI: ffffa401077e3970
May 26 07:17:54 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:17:54 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 07:17:54 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:17:54 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:17:54 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:17:54 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:17:54 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:17:54 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:17:54 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:17:54 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:17:54 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:17:54 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:17:54 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:17:54 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:17:54 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:17:54 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:17:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:17:54 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:17:54 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:17:54 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:17:54 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:17:54 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:17:54 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:17:54 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:17:54 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:17:54 mr-fox kernel:  </TASK>
May 26 07:18:00 mr-fox crond[10804]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:18:00 mr-fox crond[10803]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:18:00 mr-fox crond[10802]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:18:00 mr-fox crond[10801]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:18:00 mr-fox crond[10805]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:18:00 mr-fox CROND[10806]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:18:00 mr-fox CROND[10808]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:18:00 mr-fox CROND[10809]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:18:00 mr-fox CROND[10807]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:18:00 mr-fox CROND[10810]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:18:00 mr-fox CROND[10805]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:18:00 mr-fox CROND[10805]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:18:00 mr-fox CROND[10804]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:18:00 mr-fox CROND[10804]: pam_unix(crond:session): session closed for user root
May 26 07:18:01 mr-fox CROND[10803]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:18:01 mr-fox CROND[10803]: pam_unix(crond:session): session closed for user root
May 26 07:18:01 mr-fox CROND[7744]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:18:01 mr-fox CROND[7744]: pam_unix(crond:session): session closed for user root
May 26 07:19:00 mr-fox crond[15386]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:19:00 mr-fox crond[15388]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:19:00 mr-fox crond[15387]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:19:00 mr-fox crond[15390]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:19:00 mr-fox crond[15389]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:19:00 mr-fox CROND[15394]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:19:00 mr-fox CROND[15395]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:19:00 mr-fox CROND[15397]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:19:00 mr-fox CROND[15398]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:19:00 mr-fox CROND[15396]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:19:00 mr-fox CROND[15390]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:19:00 mr-fox CROND[15390]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:19:00 mr-fox CROND[15389]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:19:00 mr-fox CROND[15389]: pam_unix(crond:session): session closed for user root
May 26 07:19:00 mr-fox CROND[15388]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:19:00 mr-fox CROND[15388]: pam_unix(crond:session): session closed for user root
May 26 07:19:01 mr-fox CROND[10802]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:19:01 mr-fox CROND[10802]: pam_unix(crond:session): session closed for user root
May 26 07:20:00 mr-fox crond[20108]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 07:20:00 mr-fox crond[20110]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:20:00 mr-fox crond[20111]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:20:00 mr-fox crond[20112]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:20:00 mr-fox crond[20109]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:20:00 mr-fox CROND[20118]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:20:00 mr-fox crond[20114]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:20:00 mr-fox CROND[20119]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:20:00 mr-fox crond[20113]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:20:00 mr-fox CROND[20121]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:20:00 mr-fox CROND[20122]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:20:00 mr-fox CROND[20123]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:20:00 mr-fox CROND[20124]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 07:20:00 mr-fox crond[20116]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:20:00 mr-fox CROND[20125]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:20:00 mr-fox CROND[20127]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:20:00 mr-fox CROND[20108]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:20:00 mr-fox CROND[20108]: pam_unix(crond:session): session closed for user torproject
May 26 07:20:00 mr-fox CROND[20116]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:20:00 mr-fox CROND[20116]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:20:00 mr-fox CROND[20112]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:20:00 mr-fox CROND[20112]: pam_unix(crond:session): session closed for user root
May 26 07:20:00 mr-fox CROND[20113]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:20:00 mr-fox CROND[20113]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:20:00 mr-fox CROND[20111]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:20:00 mr-fox CROND[20111]: pam_unix(crond:session): session closed for user root
May 26 07:20:00 mr-fox CROND[15387]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:20:00 mr-fox CROND[15387]: pam_unix(crond:session): session closed for user root
May 26 07:20:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:20:44 mr-fox kernel: rcu: \x0921-....: (1185100 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=464292
May 26 07:20:44 mr-fox kernel: rcu: \x09(t=1185101 jiffies g=8794409 q=36298079 ncpus=32)
May 26 07:20:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:20:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:20:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:20:44 mr-fox kernel: RIP: 0010:tcp_mt+0x121/0x260
May 26 07:20:44 mr-fox kernel: Code: 66 41 39 54 24 06 0f 93 c2 21 ca 40 38 d7 0f 84 49 ff ff ff 0f b6 50 0d 41 22 54 24 09 41 3a 54 24 0a 89 f2 0f 95 c1 c0 ea 02 <83> e2 01 38 d1 0f 85 28 ff ff ff 45 0f b6 6c 24 08 ba 01 00 00 00
May 26 07:20:44 mr-fox kernel: RSP: 0018:ffffa401005f0ac8 EFLAGS: 00000212
May 26 07:20:44 mr-fox kernel: RAX: ffff988f7e3160e2 RBX: ffffa401005f0b80 RCX: 0000000000000001
May 26 07:20:44 mr-fox kernel: RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
May 26 07:20:44 mr-fox kernel: RBP: ffff988b4b85ed00 R08: ffff988f7e3160ce R09: ffff988ac4bb4110
May 26 07:20:44 mr-fox kernel: R10: ffff988af2b74040 R11: 0000000000000000 R12: ffff988af2b74288
May 26 07:20:44 mr-fox kernel: R13: ffffffffa5cd0238 R14: ffff988af2b741f8 R15: ffff988f7e3160ce
May 26 07:20:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:20:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:20:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:20:44 mr-fox kernel: PKRU: 55555554
May 26 07:20:44 mr-fox kernel: Call Trace:
May 26 07:20:44 mr-fox kernel:  <IRQ>
May 26 07:20:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:20:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:20:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:20:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:20:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:20:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:20:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:20:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:20:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x31/0x80
May 26 07:20:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:20:44 mr-fox kernel:  ? tcp_mt+0x121/0x260
May 26 07:20:44 mr-fox kernel:  ? __mod_timer+0x115/0x3b0
May 26 07:20:44 mr-fox kernel:  ipt_do_table+0x290/0x500
May 26 07:20:44 mr-fox kernel:  nf_hook_slow+0x3c/0x100
May 26 07:20:44 mr-fox kernel:  ip_local_deliver+0x9b/0x100
May 26 07:20:44 mr-fox kernel:  ? ip_protocol_deliver_rcu+0x180/0x180
May 26 07:20:44 mr-fox kernel:  ip_sublist_rcv_finish+0x7f/0x90
May 26 07:20:44 mr-fox kernel:  ip_sublist_rcv+0x176/0x1c0
May 26 07:20:44 mr-fox kernel:  ? ip_sublist_rcv+0x1c0/0x1c0
May 26 07:20:44 mr-fox kernel:  ip_list_rcv+0x138/0x170
May 26 07:20:44 mr-fox kernel:  __netif_receive_skb_list_core+0x293/0x2d0
May 26 07:20:44 mr-fox kernel:  ? iommu_dma_map_page+0xc2/0x230
May 26 07:20:44 mr-fox kernel:  netif_receive_skb_list_internal+0x1db/0x320
May 26 07:20:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:20:44 mr-fox kernel:  napi_gro_complete.constprop.0+0x13b/0x190
May 26 07:20:44 mr-fox kernel:  napi_gro_flush+0xc3/0x170
May 26 07:20:44 mr-fox kernel:  __napi_poll+0xd8/0x1a0
May 26 07:20:44 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 07:20:44 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:20:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:20:44 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:20:44 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:20:44 mr-fox kernel:  </IRQ>
May 26 07:20:44 mr-fox kernel:  <TASK>
May 26 07:20:44 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:20:44 mr-fox kernel: RIP: 0010:xas_load+0x49/0x60
May 26 07:20:44 mr-fox kernel: Code: 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff <80> 7d 00 00 75 bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40
May 26 07:20:44 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 07:20:44 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:20:44 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:20:44 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:20:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:20:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:20:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:20:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:20:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:20:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:20:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:20:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:20:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:20:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:20:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:20:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:20:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:20:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:20:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:20:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:20:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:20:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:20:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:20:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:20:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:20:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:20:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:20:44 mr-fox kernel:  </TASK>
May 26 07:20:54 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1187440 jiffies s: 491905 root: 0x2/.
May 26 07:20:54 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:20:54 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:20:54 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:20:54 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:20:54 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:20:54 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:20:54 mr-fox kernel: RIP: 0010:ip6_xmit+0x27e/0x600
May 26 07:20:54 mr-fox kernel: Code: 0f 85 e0 02 00 00 e8 21 1d 01 00 3b 43 70 0f 82 e7 00 00 00 48 8b 44 24 20 48 85 c0 74 0c 48 8b 80 98 03 00 00 65 48 ff 40 28 <48> 8b 4c 24 28 48 8b 81 a0 01 00 00 65 48 ff 40 28 48 85 db 0f 84
May 26 07:20:54 mr-fox kernel: RSP: 0018:ffffa401005f0c30 EFLAGS: 00000202
May 26 07:20:54 mr-fox kernel: RAX: 00002b5790e21310 RBX: ffff98a8b979e000 RCX: 0000000000000040
May 26 07:20:54 mr-fox kernel: RDX: 0000000000000000 RSI: 000c64027a24022a RDI: 0000000000000000
May 26 07:20:54 mr-fox kernel: RBP: ffffa401005f0d20 R08: 0000000000000040 R09: 0000000000000000
May 26 07:20:54 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000020 R12: 0000000000002000
May 26 07:20:54 mr-fox kernel: R13: ffff988cf4b8b200 R14: ffffa401005f0d38 R15: ffff988aeb41bf00
May 26 07:20:54 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:20:54 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:20:54 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:20:54 mr-fox kernel: PKRU: 55555554
May 26 07:20:54 mr-fox kernel: Call Trace:
May 26 07:20:54 mr-fox kernel:  <NMI>
May 26 07:20:54 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:20:54 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:20:54 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:20:54 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:20:54 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:20:54 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:20:54 mr-fox kernel:  ? ip6_xmit+0x27e/0x600
May 26 07:20:54 mr-fox kernel:  ? ip6_xmit+0x27e/0x600
May 26 07:20:54 mr-fox kernel:  ? ip6_xmit+0x27e/0x600
May 26 07:20:54 mr-fox kernel:  </NMI>
May 26 07:20:54 mr-fox kernel:  <IRQ>
May 26 07:20:54 mr-fox kernel:  ? ip6_output+0x290/0x290
May 26 07:20:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:20:54 mr-fox kernel:  ? __sk_dst_check+0x34/0xa0
May 26 07:20:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:20:54 mr-fox kernel:  ? inet6_csk_route_socket+0x132/0x210
May 26 07:20:54 mr-fox kernel:  inet6_csk_xmit+0xe9/0x160
May 26 07:20:54 mr-fox kernel:  __tcp_transmit_skb+0x5d0/0xd30
May 26 07:20:54 mr-fox kernel:  tcp_delack_timer_handler+0xa9/0x110
May 26 07:20:54 mr-fox kernel:  tcp_delack_timer+0xb5/0xf0
May 26 07:20:54 mr-fox kernel:  ? tcp_delack_timer_handler+0x110/0x110
May 26 07:20:54 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 07:20:54 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 07:20:54 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 07:20:54 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:20:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:20:54 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:20:54 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:20:54 mr-fox kernel:  </IRQ>
May 26 07:20:54 mr-fox kernel:  <TASK>
May 26 07:20:54 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:20:54 mr-fox kernel: RIP: 0010:xas_descend+0x2/0xd0
May 26 07:20:54 mr-fox kernel: Code: 18 0f b6 4c 24 10 4c 8b 04 24 e9 60 fe ff ff e9 69 87 04 00 e9 a3 87 04 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 55 <41> 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80
May 26 07:20:54 mr-fox kernel: RSP: 0018:ffffa401077e3940 EFLAGS: 00000206
May 26 07:20:54 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:20:54 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac0754ff8 RDI: ffffa401077e3970
May 26 07:20:54 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:20:54 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:20:54 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:20:54 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:20:54 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:20:54 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:20:54 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:20:54 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:20:54 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:20:54 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:20:54 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:20:54 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:20:54 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:20:54 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:20:54 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:20:54 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:20:54 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:20:54 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:20:54 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:20:54 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:20:54 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:20:54 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:20:54 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:20:54 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:20:54 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:20:54 mr-fox kernel:  </TASK>
May 26 07:21:00 mr-fox crond[25560]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:21:00 mr-fox crond[25558]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:21:00 mr-fox crond[25562]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:21:00 mr-fox crond[25563]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:21:00 mr-fox crond[25559]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:21:00 mr-fox CROND[25565]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:21:00 mr-fox CROND[25566]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:21:00 mr-fox CROND[25567]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:21:00 mr-fox CROND[25568]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:21:00 mr-fox CROND[25569]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:21:00 mr-fox CROND[25563]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:21:00 mr-fox CROND[25563]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:21:00 mr-fox CROND[25562]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:21:00 mr-fox CROND[25562]: pam_unix(crond:session): session closed for user root
May 26 07:21:00 mr-fox CROND[25560]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:21:00 mr-fox CROND[25560]: pam_unix(crond:session): session closed for user root
May 26 07:21:01 mr-fox CROND[20110]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:21:01 mr-fox CROND[20110]: pam_unix(crond:session): session closed for user root
May 26 07:22:00 mr-fox crond[25248]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:22:00 mr-fox crond[25247]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:22:00 mr-fox crond[25250]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:22:00 mr-fox crond[25249]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:22:00 mr-fox CROND[25255]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:22:00 mr-fox CROND[25256]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:22:00 mr-fox CROND[25257]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:22:00 mr-fox CROND[25259]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:22:00 mr-fox crond[25251]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:22:00 mr-fox CROND[25261]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:22:00 mr-fox CROND[25251]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:22:00 mr-fox CROND[25251]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:22:00 mr-fox CROND[25250]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:22:00 mr-fox CROND[25250]: pam_unix(crond:session): session closed for user root
May 26 07:22:00 mr-fox CROND[25249]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:22:00 mr-fox CROND[25249]: pam_unix(crond:session): session closed for user root
May 26 07:22:01 mr-fox CROND[25559]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:22:01 mr-fox CROND[25559]: pam_unix(crond:session): session closed for user root
May 26 07:23:00 mr-fox crond[26989]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:23:00 mr-fox crond[26991]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:23:00 mr-fox crond[26994]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:23:00 mr-fox crond[26993]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:23:00 mr-fox crond[26992]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:23:00 mr-fox CROND[26998]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:23:00 mr-fox CROND[26999]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:23:00 mr-fox CROND[27000]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:23:00 mr-fox CROND[27001]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:23:00 mr-fox CROND[27002]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:23:00 mr-fox CROND[26994]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:23:00 mr-fox CROND[26994]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:23:00 mr-fox CROND[26993]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:23:00 mr-fox CROND[26993]: pam_unix(crond:session): session closed for user root
May 26 07:23:00 mr-fox CROND[26992]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:23:00 mr-fox CROND[26992]: pam_unix(crond:session): session closed for user root
May 26 07:23:00 mr-fox CROND[25248]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:23:00 mr-fox CROND[25248]: pam_unix(crond:session): session closed for user root
May 26 07:23:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:23:44 mr-fox kernel: rcu: \x0921-....: (1230104 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=482666
May 26 07:23:44 mr-fox kernel: rcu: \x09(t=1230105 jiffies g=8794409 q=37360230 ncpus=32)
May 26 07:23:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:23:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:23:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:23:44 mr-fox kernel: RIP: 0010:xas_descend+0x26/0xd0
May 26 07:23:44 mr-fox kernel: Code: 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f <89> d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03
May 26 07:23:44 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 07:23:44 mr-fox kernel: RAX: ffff988ac07566ca RBX: 000000000000000c RCX: 0000000000000018
May 26 07:23:44 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac07566c8 RDI: ffffa401077e3970
May 26 07:23:44 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:23:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 07:23:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:23:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:23:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:23:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:23:44 mr-fox kernel: PKRU: 55555554
May 26 07:23:44 mr-fox kernel: Call Trace:
May 26 07:23:44 mr-fox kernel:  <IRQ>
May 26 07:23:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:23:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:23:44 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 07:23:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:23:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:23:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:23:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:23:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:23:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:23:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:23:44 mr-fox kernel:  </IRQ>
May 26 07:23:44 mr-fox kernel:  <TASK>
May 26 07:23:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:23:44 mr-fox kernel:  ? xas_descend+0x26/0xd0
May 26 07:23:44 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:23:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:23:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:23:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:23:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:23:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:23:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:23:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:23:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:23:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:23:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:23:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:23:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:23:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:23:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:23:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:23:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:23:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:23:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:23:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:23:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:23:44 mr-fox kernel:  </TASK>
May 26 07:23:55 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1232496 jiffies s: 491905 root: 0x2/.
May 26 07:23:55 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:23:55 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:23:55 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:23:55 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:23:55 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:23:55 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:23:55 mr-fox kernel: RIP: 0010:iommu_map+0x29/0xa0
May 26 07:23:55 mr-fox kernel: Code: 90 f3 0f 1e fa 41 56 41 55 41 54 55 53 4c 8b 77 08 41 f7 c1 07 00 04 00 75 5e 48 89 fd 49 89 f4 49 89 cd e8 29 fe ff ff 89 c3 <85> c0 75 1d 49 8b 46 20 48 85 c0 74 14 4c 89 ea 4c 89 e6 48 89 ef
May 26 07:23:55 mr-fox kernel: RSP: 0018:ffffa401005f08b0 EFLAGS: 00000246
May 26 07:23:55 mr-fox kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
May 26 07:23:55 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:23:55 mr-fox kernel: RBP: ffff988ac2211010 R08: 0000000000000000 R09: 0000000000000000
May 26 07:23:55 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000cf742000
May 26 07:23:55 mr-fox kernel: R13: 0000000000001000 R14: ffffffffa5ca5540 R15: ffff988ac1292400
May 26 07:23:55 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:23:55 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:23:55 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:23:55 mr-fox kernel: PKRU: 55555554
May 26 07:23:55 mr-fox kernel: Call Trace:
May 26 07:23:55 mr-fox kernel:  <NMI>
May 26 07:23:55 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:23:55 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:23:55 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:23:55 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:23:55 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:23:55 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:23:55 mr-fox kernel:  ? iommu_map+0x29/0xa0
May 26 07:23:55 mr-fox kernel:  ? iommu_map+0x29/0xa0
May 26 07:23:55 mr-fox kernel:  ? iommu_map+0x29/0xa0
May 26 07:23:55 mr-fox kernel:  </NMI>
May 26 07:23:55 mr-fox kernel:  <IRQ>
May 26 07:23:55 mr-fox kernel:  __iommu_dma_map+0x87/0xf0
May 26 07:23:55 mr-fox kernel:  iommu_dma_map_page+0xc2/0x230
May 26 07:23:55 mr-fox kernel:  igb_xmit_frame_ring+0x91b/0xc00
May 26 07:23:55 mr-fox kernel:  ? netif_skb_features+0x93/0x2c0
May 26 07:23:55 mr-fox kernel:  dev_hard_start_xmit+0xa0/0xf0
May 26 07:23:55 mr-fox kernel:  sch_direct_xmit+0x8d/0x290
May 26 07:23:55 mr-fox kernel:  __dev_queue_xmit+0x49a/0x9a0
May 26 07:23:55 mr-fox kernel:  ? ip6t_do_table+0x30b/0x590
May 26 07:23:55 mr-fox kernel:  ip6_finish_output2+0x2c0/0x610
May 26 07:23:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:55 mr-fox kernel:  ? ip6_output+0xa7/0x290
May 26 07:23:55 mr-fox kernel:  ip6_xmit+0x3fc/0x600
May 26 07:23:55 mr-fox kernel:  ? ip6_output+0x290/0x290
May 26 07:23:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:55 mr-fox kernel:  ? __sk_dst_check+0x34/0xa0
May 26 07:23:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:55 mr-fox kernel:  ? inet6_csk_route_socket+0x132/0x210
May 26 07:23:55 mr-fox kernel:  inet6_csk_xmit+0xe9/0x160
May 26 07:23:55 mr-fox kernel:  __tcp_transmit_skb+0x5d0/0xd30
May 26 07:23:55 mr-fox kernel:  __tcp_retransmit_skb+0x1a9/0x800
May 26 07:23:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:55 mr-fox kernel:  ? __mod_timer+0x115/0x3b0
May 26 07:23:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:55 mr-fox kernel:  ? retransmits_timed_out.part.0+0x8d/0x170
May 26 07:23:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:55 mr-fox kernel:  tcp_retransmit_skb+0x11/0xa0
May 26 07:23:55 mr-fox kernel:  tcp_retransmit_timer+0x492/0xa60
May 26 07:23:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:55 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 07:23:55 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 07:23:55 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 07:23:55 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 07:23:55 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 07:23:55 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:23:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:55 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:23:55 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:23:55 mr-fox kernel:  </IRQ>
May 26 07:23:55 mr-fox kernel:  <TASK>
May 26 07:23:55 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:23:55 mr-fox kernel: RIP: 0010:xas_load+0x49/0x60
May 26 07:23:55 mr-fox kernel: Code: 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff <80> 7d 00 00 75 bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40
May 26 07:23:55 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 07:23:55 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:23:55 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:23:55 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:23:55 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:23:55 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:23:55 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:23:55 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:23:55 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:23:55 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:23:55 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:23:55 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:23:55 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:23:55 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:23:55 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:23:55 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:23:55 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:23:55 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:23:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:23:55 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:23:55 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:23:55 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:23:55 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:23:55 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:23:55 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:23:55 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:23:55 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:23:55 mr-fox kernel:  </TASK>
May 26 07:24:00 mr-fox crond[29060]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:24:00 mr-fox crond[29059]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:24:00 mr-fox crond[29061]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:24:00 mr-fox crond[29062]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:24:00 mr-fox CROND[29066]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:24:00 mr-fox CROND[29067]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:24:00 mr-fox crond[29063]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:24:00 mr-fox CROND[29069]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:24:00 mr-fox CROND[29068]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:24:00 mr-fox CROND[29071]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:24:00 mr-fox CROND[29063]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:24:00 mr-fox CROND[29063]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:24:00 mr-fox CROND[29062]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:24:00 mr-fox CROND[29062]: pam_unix(crond:session): session closed for user root
May 26 07:24:00 mr-fox CROND[29061]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:24:00 mr-fox CROND[29061]: pam_unix(crond:session): session closed for user root
May 26 07:24:00 mr-fox CROND[26991]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:24:00 mr-fox CROND[26991]: pam_unix(crond:session): session closed for user root
May 26 07:25:00 mr-fox crond[30521]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 07:25:00 mr-fox crond[30522]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:25:00 mr-fox crond[30523]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:25:00 mr-fox crond[30524]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:25:00 mr-fox CROND[30533]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:25:00 mr-fox CROND[30534]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:25:00 mr-fox CROND[30535]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:25:00 mr-fox CROND[30536]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:25:00 mr-fox crond[30525]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:25:00 mr-fox crond[30527]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:25:00 mr-fox crond[30529]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:25:00 mr-fox crond[30528]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:25:00 mr-fox CROND[30540]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:25:00 mr-fox CROND[30541]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:25:00 mr-fox CROND[30542]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:25:00 mr-fox CROND[30543]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 07:25:00 mr-fox CROND[30521]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:25:00 mr-fox CROND[30521]: pam_unix(crond:session): session closed for user torproject
May 26 07:25:00 mr-fox CROND[30529]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:25:00 mr-fox CROND[30529]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:25:00 mr-fox CROND[30525]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:25:00 mr-fox CROND[30525]: pam_unix(crond:session): session closed for user root
May 26 07:25:00 mr-fox CROND[30527]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:25:00 mr-fox CROND[30527]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:25:00 mr-fox CROND[30524]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:25:00 mr-fox CROND[30524]: pam_unix(crond:session): session closed for user root
May 26 07:25:01 mr-fox CROND[29060]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:25:01 mr-fox CROND[29060]: pam_unix(crond:session): session closed for user root
May 26 07:26:00 mr-fox crond[31551]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:26:00 mr-fox crond[31550]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:26:00 mr-fox crond[31549]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:26:00 mr-fox crond[31552]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:26:00 mr-fox crond[31553]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:26:00 mr-fox CROND[31557]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:26:00 mr-fox CROND[31558]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:26:00 mr-fox CROND[31559]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:26:00 mr-fox CROND[31560]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:26:00 mr-fox CROND[31562]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:26:00 mr-fox CROND[31553]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:26:00 mr-fox CROND[31553]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:26:00 mr-fox CROND[31552]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:26:00 mr-fox CROND[31552]: pam_unix(crond:session): session closed for user root
May 26 07:26:00 mr-fox CROND[31551]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:26:00 mr-fox CROND[31551]: pam_unix(crond:session): session closed for user root
May 26 07:26:01 mr-fox CROND[30523]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:26:01 mr-fox CROND[30523]: pam_unix(crond:session): session closed for user root
May 26 07:26:44 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:26:44 mr-fox kernel: rcu: \x0921-....: (1275108 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=501198
May 26 07:26:44 mr-fox kernel: rcu: \x09(t=1275109 jiffies g=8794409 q=38408927 ncpus=32)
May 26 07:26:44 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:26:44 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:26:44 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:26:44 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 07:26:44 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 07:26:44 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 07:26:44 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:26:44 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:26:44 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:26:44 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:26:44 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:26:44 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:26:44 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:26:44 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:26:44 mr-fox kernel: PKRU: 55555554
May 26 07:26:44 mr-fox kernel: Call Trace:
May 26 07:26:44 mr-fox kernel:  <IRQ>
May 26 07:26:44 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:26:44 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:26:44 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 07:26:44 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:26:44 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:26:44 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:26:44 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:26:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:26:44 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:26:44 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:26:44 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:26:44 mr-fox kernel:  </IRQ>
May 26 07:26:44 mr-fox kernel:  <TASK>
May 26 07:26:44 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:26:44 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 07:26:44 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:26:44 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:26:44 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:26:44 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:26:44 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:26:44 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:26:44 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:26:44 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:26:44 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:26:44 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:26:44 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:26:44 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:26:44 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:26:44 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:26:44 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:26:44 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:26:44 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:26:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:26:44 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:26:44 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:26:44 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:26:44 mr-fox kernel:  </TASK>
May 26 07:26:55 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1277552 jiffies s: 491905 root: 0x2/.
May 26 07:26:55 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:26:55 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:26:55 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:26:55 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:26:55 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:26:55 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:26:55 mr-fox kernel: RIP: 0010:_raw_spin_lock_irqsave+0x7/0x40
May 26 07:26:55 mr-fox kernel: Code: 00 00 f0 0f b1 17 74 0b 31 c0 31 d2 31 ff e9 3b 32 15 00 b8 01 00 00 00 31 d2 31 ff e9 2d 32 15 00 66 90 f3 0f 1e fa 53 9c 5b <fa> 31 c0 ba 01 00 00 00 f0 0f b1 17 75 0f 48 89 d8 5b 31 d2 31 f6
May 26 07:26:55 mr-fox kernel: RSP: 0018:ffffa401005f0c88 EFLAGS: 00000086
May 26 07:26:55 mr-fox kernel: RAX: 0000000000000000 RBX: 0000000000000086 RCX: 0000000000000000
May 26 07:26:55 mr-fox kernel: RDX: 0000000000000001 RSI: 00000000000cf1e7 RDI: ffff98a96ed7d8c0
May 26 07:26:55 mr-fox kernel: RBP: ffff988ac4405000 R08: 0000000000000000 R09: 0000000000000000
May 26 07:26:55 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff988ac1292408
May 26 07:26:55 mr-fox kernel: R13: 00000000008f7ba1 R14: 00000000000cf1e7 R15: ffffc400ffb51870
May 26 07:26:55 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:26:55 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:26:55 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:26:55 mr-fox kernel: PKRU: 55555554
May 26 07:26:55 mr-fox kernel: Call Trace:
May 26 07:26:55 mr-fox kernel:  <NMI>
May 26 07:26:55 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:26:55 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:26:55 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:26:55 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:26:55 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:26:55 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:26:55 mr-fox kernel:  ? _raw_spin_lock_irqsave+0x7/0x40
May 26 07:26:55 mr-fox kernel:  ? _raw_spin_lock_irqsave+0x7/0x40
May 26 07:26:55 mr-fox kernel:  ? _raw_spin_lock_irqsave+0x7/0x40
May 26 07:26:55 mr-fox kernel:  </NMI>
May 26 07:26:55 mr-fox kernel:  <IRQ>
May 26 07:26:55 mr-fox kernel:  free_iova_fast+0x60/0x1b0
May 26 07:26:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:26:55 mr-fox kernel:  fq_ring_free_locked+0x43/0x90
May 26 07:26:55 mr-fox kernel:  iommu_dma_free_iova+0x212/0x220
May 26 07:26:55 mr-fox kernel:  __iommu_dma_unmap+0xe0/0x170
May 26 07:26:55 mr-fox kernel:  iommu_dma_unmap_page+0x43/0xa0
May 26 07:26:55 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 07:26:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:26:55 mr-fox kernel:  ? task_tick_fair+0x85/0x470
May 26 07:26:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:26:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:26:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:26:55 mr-fox kernel:  ? wq_worker_tick+0xd/0xd0
May 26 07:26:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:26:55 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 07:26:55 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 07:26:55 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 07:26:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:26:55 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:26:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:26:55 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:26:55 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:26:55 mr-fox kernel:  </IRQ>
May 26 07:26:55 mr-fox kernel:  <TASK>
May 26 07:26:55 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:26:55 mr-fox kernel: RIP: 0010:xas_load+0x3c/0x60
May 26 07:26:55 mr-fox kernel: Code: e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe <72> e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 75 bf eb d1 66
May 26 07:26:55 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 07:26:55 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:26:55 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:26:55 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 07:26:55 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:26:55 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:26:55 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:26:55 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:26:55 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:26:55 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:26:55 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:26:55 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:26:55 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:26:55 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:26:55 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:26:55 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:26:55 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:26:55 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:26:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:26:55 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:26:55 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:26:55 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:26:55 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:26:55 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:26:55 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:26:55 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:26:55 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:26:55 mr-fox kernel:  </TASK>
May 26 07:27:00 mr-fox crond[31959]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:27:00 mr-fox crond[31958]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:27:00 mr-fox crond[31960]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:27:00 mr-fox crond[31961]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:27:00 mr-fox CROND[31965]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:27:00 mr-fox CROND[31966]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:27:00 mr-fox CROND[31967]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:27:00 mr-fox crond[31963]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:27:00 mr-fox CROND[31968]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:27:00 mr-fox CROND[31970]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:27:00 mr-fox CROND[31963]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:27:00 mr-fox CROND[31963]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:27:00 mr-fox CROND[31961]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:27:00 mr-fox CROND[31961]: pam_unix(crond:session): session closed for user root
May 26 07:27:00 mr-fox CROND[31960]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:27:00 mr-fox CROND[31960]: pam_unix(crond:session): session closed for user root
May 26 07:27:00 mr-fox CROND[31550]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:27:00 mr-fox CROND[31550]: pam_unix(crond:session): session closed for user root
May 26 07:28:00 mr-fox crond[2630]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:28:00 mr-fox crond[2627]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:28:00 mr-fox crond[2631]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:28:00 mr-fox crond[2628]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:28:00 mr-fox crond[2632]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:28:00 mr-fox CROND[2636]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:28:00 mr-fox CROND[2638]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:28:00 mr-fox CROND[2637]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:28:00 mr-fox CROND[2639]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:28:00 mr-fox CROND[2640]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:28:00 mr-fox CROND[2632]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:28:00 mr-fox CROND[2632]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:28:00 mr-fox CROND[2631]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:28:00 mr-fox CROND[2631]: pam_unix(crond:session): session closed for user root
May 26 07:28:00 mr-fox CROND[2630]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:28:00 mr-fox CROND[2630]: pam_unix(crond:session): session closed for user root
May 26 07:28:00 mr-fox CROND[31959]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:28:00 mr-fox CROND[31959]: pam_unix(crond:session): session closed for user root
May 26 07:29:00 mr-fox crond[4110]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:29:00 mr-fox crond[4112]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:29:00 mr-fox crond[4109]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:29:00 mr-fox crond[4113]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:29:00 mr-fox crond[4111]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:29:00 mr-fox CROND[4117]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:29:00 mr-fox CROND[4118]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:29:00 mr-fox CROND[4119]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:29:00 mr-fox CROND[4120]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:29:00 mr-fox CROND[4121]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:29:00 mr-fox CROND[4113]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:29:00 mr-fox CROND[4113]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:29:00 mr-fox CROND[4112]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:29:00 mr-fox CROND[4112]: pam_unix(crond:session): session closed for user root
May 26 07:29:01 mr-fox CROND[4111]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:29:01 mr-fox CROND[4111]: pam_unix(crond:session): session closed for user root
May 26 07:29:01 mr-fox CROND[2628]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:29:01 mr-fox CROND[2628]: pam_unix(crond:session): session closed for user root
May 26 07:29:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:29:45 mr-fox kernel: rcu: \x0921-....: (1320112 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=519936
May 26 07:29:45 mr-fox kernel: rcu: \x09(t=1320113 jiffies g=8794409 q=39445448 ncpus=32)
May 26 07:29:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:29:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:29:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:29:45 mr-fox kernel: RIP: 0010:xas_load+0x20/0x60
May 26 07:29:45 mr-fox kernel: Code: ff eb be 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 53 48 89 fb e8 f2 f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 <77> 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48
May 26 07:29:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000286
May 26 07:29:45 mr-fox kernel: RAX: ffff988d55f18ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:29:45 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:29:45 mr-fox kernel: RBP: ffff988d667606d8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:29:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:29:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:29:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:29:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:29:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:29:45 mr-fox kernel: PKRU: 55555554
May 26 07:29:45 mr-fox kernel: Call Trace:
May 26 07:29:45 mr-fox kernel:  <IRQ>
May 26 07:29:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:29:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:29:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:29:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:29:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:29:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:29:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:29:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:29:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:29:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:29:45 mr-fox kernel:  </IRQ>
May 26 07:29:45 mr-fox kernel:  <TASK>
May 26 07:29:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:29:45 mr-fox kernel:  ? xas_load+0x20/0x60
May 26 07:29:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:29:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:29:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:29:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:29:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:29:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:29:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:29:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:29:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:29:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:29:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:29:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:29:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:29:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:29:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:29:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:29:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:29:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:29:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:29:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:29:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:29:45 mr-fox kernel:  </TASK>
May 26 07:29:55 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1322607 jiffies s: 491905 root: 0x2/.
May 26 07:29:55 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:29:55 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:29:55 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:29:55 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:29:55 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:29:55 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:29:55 mr-fox kernel: RIP: 0010:fq_ring_free_locked+0x33/0x90
May 26 07:29:55 mr-fox kernel: Code: 8b af 98 00 00 00 8b 06 85 c0 74 6c 8b 6e 04 48 89 f3 39 6e 08 74 4c 4c 8d 67 08 eb 32 48 8d 7c 03 20 83 c5 01 e8 3d 43 b2 ff <49> 8b 56 18 49 8b 76 10 4c 89 e7 e8 4d 3a 00 00 8b 43 04 8b 53 0c
May 26 07:29:55 mr-fox kernel: RSP: 0018:ffffa401005f0cd8 EFLAGS: 00000046
May 26 07:29:55 mr-fox kernel: RAX: 0000000000000000 RBX: ffffc400ffb51870 RCX: 0000000000000000
May 26 07:29:55 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:29:55 mr-fox kernel: RBP: 000000000000006b R08: 0000000000000000 R09: 0000000000000000
May 26 07:29:55 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff988ac1292408
May 26 07:29:55 mr-fox kernel: R13: 0000000000900279 R14: ffffc400ffb52900 R15: ffffc400ffb51870
May 26 07:29:55 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:29:55 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:29:55 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:29:55 mr-fox kernel: PKRU: 55555554
May 26 07:29:55 mr-fox kernel: Call Trace:
May 26 07:29:55 mr-fox kernel:  <NMI>
May 26 07:29:55 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:29:55 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:29:55 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:29:55 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:29:55 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:29:55 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:29:55 mr-fox kernel:  ? fq_ring_free_locked+0x33/0x90
May 26 07:29:55 mr-fox kernel:  ? fq_ring_free_locked+0x33/0x90
May 26 07:29:55 mr-fox kernel:  ? fq_ring_free_locked+0x33/0x90
May 26 07:29:55 mr-fox kernel:  </NMI>
May 26 07:29:55 mr-fox kernel:  <IRQ>
May 26 07:29:55 mr-fox kernel:  iommu_dma_free_iova+0x212/0x220
May 26 07:29:55 mr-fox kernel:  __iommu_dma_unmap+0xe0/0x170
May 26 07:29:55 mr-fox kernel:  iommu_dma_unmap_page+0x43/0xa0
May 26 07:29:55 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 07:29:55 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 07:29:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:29:55 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 07:29:55 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 07:29:55 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 07:29:55 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 07:29:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:29:55 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:29:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:29:55 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:29:55 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:29:55 mr-fox kernel:  </IRQ>
May 26 07:29:55 mr-fox kernel:  <TASK>
May 26 07:29:55 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:29:55 mr-fox kernel: RIP: 0010:xas_load+0x11/0x60
May 26 07:29:55 mr-fox kernel: Code: 92 4c 89 ee 48 c7 c7 e0 eb 49 a6 e8 a9 58 c6 ff eb be 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 53 48 89 fb e8 f2 f7 ff ff 48 89 c2 <83> e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31
May 26 07:29:55 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 07:29:55 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:29:55 mr-fox kernel: RDX: ffff988ac0754ffa RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:29:55 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 07:29:55 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:29:55 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:29:55 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:29:55 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:29:55 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:29:55 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:29:55 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:29:55 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:29:55 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:29:55 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:29:55 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:29:55 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:29:55 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:29:55 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:29:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:29:55 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:29:55 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:29:55 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:29:55 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:29:55 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:29:55 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:29:55 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:29:55 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:29:55 mr-fox kernel:  </TASK>
May 26 07:30:00 mr-fox crond[4677]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:30:00 mr-fox crond[4676]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:30:00 mr-fox crond[4675]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 07:30:00 mr-fox crond[4679]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:30:00 mr-fox crond[4680]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:30:00 mr-fox crond[4678]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:30:00 mr-fox CROND[4685]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:30:00 mr-fox CROND[4683]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:30:00 mr-fox crond[4681]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:30:00 mr-fox CROND[4686]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:30:00 mr-fox CROND[4687]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:30:00 mr-fox crond[4682]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:30:00 mr-fox CROND[4688]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:30:00 mr-fox CROND[4690]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 07:30:00 mr-fox CROND[4689]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:30:00 mr-fox CROND[4691]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:30:00 mr-fox CROND[4675]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:30:00 mr-fox CROND[4682]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:30:00 mr-fox CROND[4675]: pam_unix(crond:session): session closed for user torproject
May 26 07:30:00 mr-fox CROND[4682]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:30:00 mr-fox CROND[4679]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:30:00 mr-fox CROND[4679]: pam_unix(crond:session): session closed for user root
May 26 07:30:00 mr-fox CROND[4680]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:30:00 mr-fox CROND[4680]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:30:00 mr-fox CROND[4678]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:30:00 mr-fox CROND[4678]: pam_unix(crond:session): session closed for user root
May 26 07:30:01 mr-fox CROND[4110]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:30:01 mr-fox CROND[4110]: pam_unix(crond:session): session closed for user root
May 26 07:31:00 mr-fox crond[7753]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:31:00 mr-fox crond[7750]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:31:00 mr-fox crond[7754]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:31:00 mr-fox crond[7751]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:31:00 mr-fox crond[7755]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:31:00 mr-fox CROND[7760]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:31:00 mr-fox CROND[7761]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:31:00 mr-fox CROND[7762]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:31:00 mr-fox CROND[7763]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:31:00 mr-fox CROND[7764]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:31:00 mr-fox CROND[7755]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:31:00 mr-fox CROND[7755]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:31:00 mr-fox CROND[7754]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:31:00 mr-fox CROND[7754]: pam_unix(crond:session): session closed for user root
May 26 07:31:00 mr-fox CROND[7753]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:31:00 mr-fox CROND[7753]: pam_unix(crond:session): session closed for user root
May 26 07:31:00 mr-fox CROND[4677]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:31:00 mr-fox CROND[4677]: pam_unix(crond:session): session closed for user root
May 26 07:32:00 mr-fox crond[11526]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:32:00 mr-fox crond[11525]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:32:00 mr-fox crond[11524]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:32:00 mr-fox crond[11528]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:32:00 mr-fox CROND[11532]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:32:00 mr-fox crond[11529]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:32:00 mr-fox CROND[11533]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:32:00 mr-fox CROND[11535]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:32:00 mr-fox CROND[11536]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:32:00 mr-fox CROND[11537]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:32:00 mr-fox CROND[11529]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:32:00 mr-fox CROND[11529]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:32:00 mr-fox CROND[11528]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:32:00 mr-fox CROND[11528]: pam_unix(crond:session): session closed for user root
May 26 07:32:00 mr-fox CROND[11526]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:32:00 mr-fox CROND[11526]: pam_unix(crond:session): session closed for user root
May 26 07:32:01 mr-fox CROND[7751]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:32:01 mr-fox CROND[7751]: pam_unix(crond:session): session closed for user root
May 26 07:32:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:32:45 mr-fox kernel: rcu: \x0921-....: (1365116 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=538444
May 26 07:32:45 mr-fox kernel: rcu: \x09(t=1365117 jiffies g=8794409 q=40490189 ncpus=32)
May 26 07:32:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:32:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:32:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:32:45 mr-fox kernel: RIP: 0010:xas_start+0x7b/0x120
May 26 07:32:45 mr-fox kernel: Code: 00 00 00 48 83 c4 08 5b 5d 31 d2 31 c9 31 f6 31 ff e9 c4 c7 1a 00 0f b6 48 fe 80 f9 3f 0f 87 a3 00 00 00 48 d3 ee 48 83 fe 3f <76> cf 48 c7 43 18 01 00 00 00 48 83 c4 08 31 c0 5b 5d 31 d2 31 c9
May 26 07:32:45 mr-fox kernel: RSP: 0018:ffffa401077e3930 EFLAGS: 00000293
May 26 07:32:45 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 000000000000001e
May 26 07:32:45 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000001 RDI: ffffa401077e3970
May 26 07:32:45 mr-fox kernel: RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
May 26 07:32:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:32:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:32:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:32:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:32:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:32:45 mr-fox kernel: PKRU: 55555554
May 26 07:32:45 mr-fox kernel: Call Trace:
May 26 07:32:45 mr-fox kernel:  <IRQ>
May 26 07:32:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:32:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:32:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:32:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:32:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:32:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:32:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:32:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:32:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:32:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:32:45 mr-fox kernel:  </IRQ>
May 26 07:32:45 mr-fox kernel:  <TASK>
May 26 07:32:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:32:45 mr-fox kernel:  ? xas_start+0x7b/0x120
May 26 07:32:45 mr-fox kernel:  xas_load+0xe/0x60
May 26 07:32:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:32:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:32:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:32:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:32:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:32:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:32:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:32:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:32:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:32:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:32:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:32:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:32:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:32:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:32:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:32:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:32:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:32:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:32:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:32:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:32:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:32:45 mr-fox kernel:  </TASK>
May 26 07:32:55 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1367663 jiffies s: 491905 root: 0x2/.
May 26 07:32:55 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:32:55 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:32:55 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:32:55 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:32:55 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:32:55 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:32:55 mr-fox kernel: RIP: 0010:__tcp_transmit_skb+0x289/0xd30
May 26 07:32:55 mr-fox kernel: Code: 0f b6 85 80 00 00 00 4c 8b bd c8 00 00 00 83 e2 01 83 e0 ef c1 e2 04 09 d0 88 85 80 00 00 00 0f b7 83 de 02 00 00 66 41 89 07 <0f> b7 43 0c 66 41 89 47 02 8b 45 28 0f c8 41 89 47 04 44 89 e8 0f
May 26 07:32:55 mr-fox kernel: RSP: 0018:ffffa401005f0d10 EFLAGS: 00000202
May 26 07:32:55 mr-fox kernel: RAX: 000000000000e324 RBX: ffff98949c718000 RCX: ffffffffa578da10
May 26 07:32:55 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff98949c718144
May 26 07:32:55 mr-fox kernel: RBP: ffff98992457cb58 R08: 00000000e8ecf2a7 R09: 0000000000000000
May 26 07:32:55 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000020
May 26 07:32:55 mr-fox kernel: R13: 00000000e8ecf2a7 R14: 000022b819abe335 R15: ffff988b4cced2a0
May 26 07:32:55 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:32:55 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:32:55 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:32:55 mr-fox kernel: PKRU: 55555554
May 26 07:32:55 mr-fox kernel: Call Trace:
May 26 07:32:55 mr-fox kernel:  <NMI>
May 26 07:32:55 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:32:55 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:32:55 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:32:55 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:32:55 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:32:55 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:32:55 mr-fox kernel:  ? sock_efree+0x70/0x70
May 26 07:32:55 mr-fox kernel:  ? __tcp_transmit_skb+0x289/0xd30
May 26 07:32:55 mr-fox kernel:  ? __tcp_transmit_skb+0x289/0xd30
May 26 07:32:55 mr-fox kernel:  ? __tcp_transmit_skb+0x289/0xd30
May 26 07:32:55 mr-fox kernel:  </NMI>
May 26 07:32:55 mr-fox kernel:  <IRQ>
May 26 07:32:55 mr-fox kernel:  __tcp_retransmit_skb+0x1a9/0x800
May 26 07:32:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:32:55 mr-fox kernel:  ? __mod_timer+0x115/0x3b0
May 26 07:32:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:32:55 mr-fox kernel:  ? retransmits_timed_out.part.0+0x8d/0x170
May 26 07:32:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:32:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:32:55 mr-fox kernel:  tcp_retransmit_skb+0x11/0xa0
May 26 07:32:55 mr-fox kernel:  tcp_retransmit_timer+0x492/0xa60
May 26 07:32:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:32:55 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 07:32:55 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 07:32:55 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 07:32:55 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 07:32:55 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 07:32:55 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:32:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:32:55 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:32:55 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:32:55 mr-fox kernel:  </IRQ>
May 26 07:32:55 mr-fox kernel:  <TASK>
May 26 07:32:55 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:32:55 mr-fox kernel: RIP: 0010:xas_load+0x49/0x60
May 26 07:32:55 mr-fox kernel: Code: 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff <80> 7d 00 00 75 bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40
May 26 07:32:55 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 07:32:55 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:32:55 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:32:55 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:32:55 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:32:55 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:32:55 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:32:55 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:32:55 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:32:55 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:32:55 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:32:55 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:32:55 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:32:55 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:32:55 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:32:55 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:32:55 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:32:55 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:32:55 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:32:55 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:32:55 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:32:55 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:32:55 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:32:55 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:32:55 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:32:55 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:32:55 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:32:55 mr-fox kernel:  </TASK>
May 26 07:33:00 mr-fox crond[11048]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:33:00 mr-fox crond[11047]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:33:00 mr-fox crond[11049]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:33:00 mr-fox crond[11052]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:33:00 mr-fox crond[11051]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:33:00 mr-fox CROND[11055]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:33:00 mr-fox CROND[11056]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:33:00 mr-fox CROND[11054]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:33:00 mr-fox CROND[11058]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:33:00 mr-fox CROND[11057]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:33:00 mr-fox CROND[11052]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:33:00 mr-fox CROND[11052]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:33:00 mr-fox CROND[11051]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:33:00 mr-fox CROND[11051]: pam_unix(crond:session): session closed for user root
May 26 07:33:00 mr-fox CROND[11049]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:33:00 mr-fox CROND[11049]: pam_unix(crond:session): session closed for user root
May 26 07:33:01 mr-fox CROND[11525]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:33:01 mr-fox CROND[11525]: pam_unix(crond:session): session closed for user root
May 26 07:34:00 mr-fox crond[14356]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:34:00 mr-fox crond[14355]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:34:00 mr-fox crond[14357]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:34:00 mr-fox crond[14358]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:34:00 mr-fox CROND[14362]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:34:00 mr-fox CROND[14363]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:34:00 mr-fox CROND[14364]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:34:00 mr-fox CROND[14365]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:34:00 mr-fox crond[14359]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:34:00 mr-fox CROND[14368]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:34:00 mr-fox CROND[14359]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:34:00 mr-fox CROND[14359]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:34:00 mr-fox CROND[14358]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:34:00 mr-fox CROND[14358]: pam_unix(crond:session): session closed for user root
May 26 07:34:00 mr-fox CROND[14357]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:34:00 mr-fox CROND[14357]: pam_unix(crond:session): session closed for user root
May 26 07:34:00 mr-fox CROND[11048]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:34:00 mr-fox CROND[11048]: pam_unix(crond:session): session closed for user root
May 26 07:35:00 mr-fox crond[15677]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:35:00 mr-fox crond[15676]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:35:00 mr-fox crond[15678]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:35:00 mr-fox crond[15679]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:35:00 mr-fox crond[15681]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:35:00 mr-fox crond[15675]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 07:35:00 mr-fox CROND[15684]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:35:00 mr-fox CROND[15685]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:35:00 mr-fox CROND[15686]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:35:00 mr-fox crond[15680]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:35:00 mr-fox CROND[15687]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:35:00 mr-fox crond[15682]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:35:00 mr-fox CROND[15688]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 07:35:00 mr-fox CROND[15689]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:35:00 mr-fox CROND[15690]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:35:00 mr-fox CROND[15692]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:35:00 mr-fox CROND[15675]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:35:00 mr-fox CROND[15675]: pam_unix(crond:session): session closed for user torproject
May 26 07:35:00 mr-fox CROND[15682]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:35:00 mr-fox CROND[15682]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:35:00 mr-fox CROND[15679]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:35:00 mr-fox CROND[15679]: pam_unix(crond:session): session closed for user root
May 26 07:35:00 mr-fox CROND[15680]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:35:00 mr-fox CROND[15680]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:35:00 mr-fox CROND[15678]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:35:00 mr-fox CROND[15678]: pam_unix(crond:session): session closed for user root
May 26 07:35:00 mr-fox CROND[14356]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:35:00 mr-fox CROND[14356]: pam_unix(crond:session): session closed for user root
May 26 07:35:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:35:45 mr-fox kernel: rcu: \x0921-....: (1410120 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=557083
May 26 07:35:45 mr-fox kernel: rcu: \x09(t=1410121 jiffies g=8794409 q=41521604 ncpus=32)
May 26 07:35:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:35:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:35:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:35:45 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 07:35:45 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 07:35:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 07:35:45 mr-fox kernel: RAX: ffff988d55f18ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:35:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:35:45 mr-fox kernel: RBP: ffff988d667606d8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:35:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:35:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:35:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:35:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:35:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:35:45 mr-fox kernel: PKRU: 55555554
May 26 07:35:45 mr-fox kernel: Call Trace:
May 26 07:35:45 mr-fox kernel:  <IRQ>
May 26 07:35:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:35:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:35:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:35:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:35:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:35:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:35:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:35:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:35:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:35:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:35:45 mr-fox kernel:  </IRQ>
May 26 07:35:45 mr-fox kernel:  <TASK>
May 26 07:35:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:35:45 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 07:35:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:35:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:35:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:35:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:35:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:35:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:35:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:35:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:35:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:35:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:35:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:35:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:35:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:35:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:35:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:35:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:35:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:35:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:35:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:35:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:35:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:35:45 mr-fox kernel:  </TASK>
May 26 07:35:56 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1412720 jiffies s: 491905 root: 0x2/.
May 26 07:35:56 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:35:56 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:35:56 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:35:56 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:35:56 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:35:56 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:35:56 mr-fox kernel: RIP: 0010:raw_local_deliver+0x7e/0x1d0
May 26 07:35:56 mr-fox kernel: Code: 83 fa 01 76 10 48 83 e2 fe 44 8b b2 94 00 00 00 45 85 f6 75 07 44 8b b5 90 00 00 00 48 83 c0 08 31 c9 48 8b 1c c5 80 9f ad a6 <48> 85 db 74 1e 48 83 eb 68 74 18 49 01 f5 4c 3b 63 30 74 41 48 8b
May 26 07:35:56 mr-fox kernel: RSP: 0018:ffffa401005f0b50 EFLAGS: 00000246
May 26 07:35:56 mr-fox kernel: RAX: 0000000000000082 RBX: 0000000000000000 RCX: 0000000000000000
May 26 07:35:56 mr-fox kernel: RDX: ffff988ac3005b00 RSI: 00000000000000ce RDI: ffff988bce2d7800
May 26 07:35:56 mr-fox kernel: RBP: ffff988bce2d7800 R08: 0000000000000000 R09: 0000000000000000
May 26 07:35:56 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa6ad7f40
May 26 07:35:56 mr-fox kernel: R13: ffff989ef929e000 R14: 0000000000000002 R15: ffff988ac3005b00
May 26 07:35:56 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:35:56 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:35:56 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:35:56 mr-fox kernel: PKRU: 55555554
May 26 07:35:56 mr-fox kernel: Call Trace:
May 26 07:35:56 mr-fox kernel:  <NMI>
May 26 07:35:56 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:35:56 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:35:56 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:35:56 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:35:56 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:35:56 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:35:56 mr-fox kernel:  ? raw_local_deliver+0x7e/0x1d0
May 26 07:35:56 mr-fox kernel:  ? raw_local_deliver+0x7e/0x1d0
May 26 07:35:56 mr-fox kernel:  ? raw_local_deliver+0x7e/0x1d0
May 26 07:35:56 mr-fox kernel:  </NMI>
May 26 07:35:56 mr-fox kernel:  <IRQ>
May 26 07:35:56 mr-fox kernel:  ? nf_hook_slow+0x3c/0x100
May 26 07:35:56 mr-fox kernel:  ip_protocol_deliver_rcu+0x4b/0x180
May 26 07:35:56 mr-fox kernel:  ip_local_deliver_finish+0x74/0xa0
May 26 07:35:56 mr-fox kernel:  ip_sublist_rcv_finish+0x7f/0x90
May 26 07:35:56 mr-fox kernel:  ip_sublist_rcv+0x176/0x1c0
May 26 07:35:56 mr-fox kernel:  ? ip_sublist_rcv+0x1c0/0x1c0
May 26 07:35:56 mr-fox kernel:  ip_list_rcv+0x138/0x170
May 26 07:35:56 mr-fox kernel:  __netif_receive_skb_list_core+0x2a5/0x2d0
May 26 07:35:56 mr-fox kernel:  netif_receive_skb_list_internal+0x1db/0x320
May 26 07:35:56 mr-fox kernel:  napi_gro_receive+0xcf/0x1b0
May 26 07:35:56 mr-fox kernel:  igb_poll+0x605/0x1370
May 26 07:35:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:35:56 mr-fox kernel:  ? task_tick_fair+0x85/0x470
May 26 07:35:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:35:56 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 07:35:56 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 07:35:56 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 07:35:56 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:35:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:35:56 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:35:56 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:35:56 mr-fox kernel:  </IRQ>
May 26 07:35:56 mr-fox kernel:  <TASK>
May 26 07:35:56 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:35:56 mr-fox kernel: RIP: 0010:filemap_get_entry+0x6a/0x160
May 26 07:35:56 mr-fox kernel: Code: 28 00 00 00 00 48 c7 44 24 30 00 00 00 00 48 c7 44 24 38 00 00 00 00 48 c7 44 24 20 03 00 00 00 48 8d 7c 24 08 e8 56 70 78 00 <48> 89 c3 48 3d 02 04 00 00 74 e2 48 3d 06 04 00 00 74 da 48 85 c0
May 26 07:35:56 mr-fox kernel: RSP: 0018:ffffa401077e3968 EFLAGS: 00000246
May 26 07:35:56 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 07:35:56 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:35:56 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 07:35:56 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:35:56 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:35:56 mr-fox kernel:  ? filemap_get_entry+0x6a/0x160
May 26 07:35:56 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:35:56 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:35:56 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:35:56 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:35:56 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:35:56 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:35:56 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:35:56 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:35:56 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:35:56 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:35:56 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:35:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:35:56 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:35:56 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:35:56 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:35:56 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:35:56 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:35:56 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:35:56 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:35:56 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:35:56 mr-fox kernel:  </TASK>
May 26 07:36:00 mr-fox crond[18040]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:36:00 mr-fox crond[18037]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:36:00 mr-fox crond[18041]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:36:00 mr-fox crond[18042]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:36:00 mr-fox CROND[18045]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:36:00 mr-fox crond[18039]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:36:00 mr-fox CROND[18047]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:36:00 mr-fox CROND[18046]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:36:00 mr-fox CROND[18048]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:36:00 mr-fox CROND[18050]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:36:00 mr-fox CROND[18042]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:36:00 mr-fox CROND[18042]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:36:00 mr-fox CROND[18041]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:36:00 mr-fox CROND[18041]: pam_unix(crond:session): session closed for user root
May 26 07:36:01 mr-fox CROND[18040]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:36:01 mr-fox CROND[18040]: pam_unix(crond:session): session closed for user root
May 26 07:36:01 mr-fox CROND[15677]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:36:01 mr-fox CROND[15677]: pam_unix(crond:session): session closed for user root
May 26 07:37:00 mr-fox crond[19724]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:37:00 mr-fox crond[19725]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:37:00 mr-fox crond[19723]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:37:00 mr-fox crond[19726]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:37:00 mr-fox CROND[19733]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:37:00 mr-fox CROND[19734]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:37:00 mr-fox CROND[19731]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:37:00 mr-fox CROND[19732]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:37:00 mr-fox crond[19728]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:37:00 mr-fox CROND[19737]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:37:00 mr-fox CROND[19728]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:37:00 mr-fox CROND[19728]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:37:00 mr-fox CROND[19726]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:37:00 mr-fox CROND[19726]: pam_unix(crond:session): session closed for user root
May 26 07:37:00 mr-fox CROND[19725]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:37:00 mr-fox CROND[19725]: pam_unix(crond:session): session closed for user root
May 26 07:37:01 mr-fox CROND[18039]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:37:01 mr-fox CROND[18039]: pam_unix(crond:session): session closed for user root
May 26 07:38:00 mr-fox crond[21717]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:38:00 mr-fox crond[21718]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:38:00 mr-fox crond[21720]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:38:00 mr-fox CROND[21724]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:38:00 mr-fox crond[21721]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:38:00 mr-fox CROND[21725]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:38:00 mr-fox crond[21719]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:38:00 mr-fox CROND[21727]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:38:00 mr-fox CROND[21728]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:38:00 mr-fox CROND[21729]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:38:00 mr-fox CROND[21721]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:38:00 mr-fox CROND[21721]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:38:00 mr-fox CROND[21720]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:38:00 mr-fox CROND[21720]: pam_unix(crond:session): session closed for user root
May 26 07:38:00 mr-fox CROND[21719]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:38:00 mr-fox CROND[21719]: pam_unix(crond:session): session closed for user root
May 26 07:38:00 mr-fox CROND[19724]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:38:00 mr-fox CROND[19724]: pam_unix(crond:session): session closed for user root
May 26 07:38:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:38:45 mr-fox kernel: rcu: \x0921-....: (1455124 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=576097
May 26 07:38:45 mr-fox kernel: rcu: \x09(t=1455125 jiffies g=8794409 q=42553763 ncpus=32)
May 26 07:38:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:38:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:38:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:38:45 mr-fox kernel: RIP: 0010:xas_descend+0x31/0xd0
May 26 07:38:45 mr-fox kernel: Code: 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 <49> 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d fd 00 00
May 26 07:38:45 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 07:38:45 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: 0000000000000036 RCX: 0000000000000012
May 26 07:38:45 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988acf90e6c8 RDI: ffffa401077e3970
May 26 07:38:45 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:38:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 07:38:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:38:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:38:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:38:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:38:45 mr-fox kernel: PKRU: 55555554
May 26 07:38:45 mr-fox kernel: Call Trace:
May 26 07:38:45 mr-fox kernel:  <IRQ>
May 26 07:38:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:38:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:38:45 mr-fox kernel:  ? tcp_write_xmit+0xe3/0x13b0
May 26 07:38:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:38:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:38:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:38:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:38:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:38:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:38:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:38:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:38:45 mr-fox kernel:  </IRQ>
May 26 07:38:45 mr-fox kernel:  <TASK>
May 26 07:38:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:38:45 mr-fox kernel:  ? xas_descend+0x31/0xd0
May 26 07:38:45 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:38:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:38:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:38:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:38:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:38:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:38:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:38:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:38:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:38:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:38:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:38:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:38:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:38:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:38:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:38:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:38:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:38:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:38:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:38:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:38:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:38:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:38:45 mr-fox kernel:  </TASK>
May 26 07:38:56 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1457775 jiffies s: 491905 root: 0x2/.
May 26 07:38:56 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:38:56 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:38:56 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:38:56 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:38:56 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:38:56 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:38:56 mr-fox kernel: RIP: 0010:__inet_lookup_established+0x60/0x140
May 26 07:38:56 mr-fox kernel: Code: e1 10 45 09 c1 44 89 cd e8 fd fd ff ff 41 89 c7 41 89 c4 48 8b 03 44 23 7b 10 4a 8d 0c f8 48 8b 19 f6 c3 01 75 0e 44 39 63 a0 <74> 33 48 8b 1b f6 c3 01 74 f2 48 d1 eb 49 39 df 75 e2 31 c0 48 83
May 26 07:38:56 mr-fox kernel: RSP: 0018:ffffa401005f0a98 EFLAGS: 00000246
May 26 07:38:56 mr-fox kernel: RAX: ffff988ac2400000 RBX: ffff988df40aca68 RCX: ffff988ac242ce10
May 26 07:38:56 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:38:56 mr-fox kernel: RBP: 00000000e6a02923 R08: 0000000000000000 R09: 00000000e6a02923
May 26 07:38:56 mr-fox kernel: R10: ffff98926e6be8ce R11: 000000000d5e1541 R12: 00000000841059c2
May 26 07:38:56 mr-fox kernel: R13: 0d5e15410e20d85f R14: ffffffffa6ad7f40 R15: 00000000000059c2
May 26 07:38:56 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:38:56 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:38:56 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:38:56 mr-fox kernel: PKRU: 55555554
May 26 07:38:56 mr-fox kernel: Call Trace:
May 26 07:38:56 mr-fox kernel:  <NMI>
May 26 07:38:56 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:38:56 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:38:56 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:38:56 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:38:56 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:38:56 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:38:56 mr-fox kernel:  ? __inet_lookup_established+0x60/0x140
May 26 07:38:56 mr-fox kernel:  ? __inet_lookup_established+0x60/0x140
May 26 07:38:56 mr-fox kernel:  ? __inet_lookup_established+0x60/0x140
May 26 07:38:56 mr-fox kernel:  </NMI>
May 26 07:38:56 mr-fox kernel:  <IRQ>
May 26 07:38:56 mr-fox kernel:  tcp_v4_rcv+0x3c8/0xea0
May 26 07:38:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:38:56 mr-fox kernel:  ip_protocol_deliver_rcu+0x32/0x180
May 26 07:38:56 mr-fox kernel:  ip_local_deliver_finish+0x74/0xa0
May 26 07:38:56 mr-fox kernel:  ip_sublist_rcv_finish+0x7f/0x90
May 26 07:38:56 mr-fox kernel:  ip_sublist_rcv+0x176/0x1c0
May 26 07:38:56 mr-fox kernel:  ? ip_sublist_rcv+0x1c0/0x1c0
May 26 07:38:56 mr-fox kernel:  ip_list_rcv+0x138/0x170
May 26 07:38:56 mr-fox kernel:  __netif_receive_skb_list_core+0x2a5/0x2d0
May 26 07:38:56 mr-fox kernel:  ? tcp_gro_receive+0x1d7/0x330
May 26 07:38:56 mr-fox kernel:  netif_receive_skb_list_internal+0x1db/0x320
May 26 07:38:56 mr-fox kernel:  napi_gro_receive+0xcf/0x1b0
May 26 07:38:56 mr-fox kernel:  igb_poll+0x605/0x1370
May 26 07:38:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:38:56 mr-fox kernel:  ? task_tick_fair+0x85/0x470
May 26 07:38:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:38:56 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 07:38:56 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 07:38:56 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 07:38:56 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:38:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:38:56 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:38:56 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:38:56 mr-fox kernel:  </IRQ>
May 26 07:38:56 mr-fox kernel:  <TASK>
May 26 07:38:56 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:38:56 mr-fox kernel: RIP: 0010:xas_descend+0x2/0xd0
May 26 07:38:56 mr-fox kernel: Code: 18 0f b6 4c 24 10 4c 8b 04 24 e9 60 fe ff ff e9 69 87 04 00 e9 a3 87 04 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 55 <41> 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80
May 26 07:38:56 mr-fox kernel: RSP: 0018:ffffa401077e3940 EFLAGS: 00000246
May 26 07:38:56 mr-fox kernel: RAX: ffff988d55f18ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:38:56 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988d55f18ff8 RDI: ffffa401077e3970
May 26 07:38:56 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:38:56 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:38:56 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:38:56 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:38:56 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:38:56 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:38:56 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:38:56 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:38:56 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:38:56 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:38:56 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:38:56 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:38:56 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:38:56 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:38:56 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:38:56 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:38:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:38:56 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:38:56 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:38:56 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:38:56 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:38:56 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:38:56 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:38:56 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:38:56 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:38:56 mr-fox kernel:  </TASK>
May 26 07:39:00 mr-fox crond[23323]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:39:00 mr-fox crond[23324]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:39:00 mr-fox crond[23326]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:39:00 mr-fox crond[23325]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:39:00 mr-fox crond[23327]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:39:00 mr-fox CROND[23330]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:39:00 mr-fox CROND[23331]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:39:00 mr-fox CROND[23332]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:39:00 mr-fox CROND[23333]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:39:00 mr-fox CROND[23334]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:39:00 mr-fox CROND[23327]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:39:00 mr-fox CROND[23327]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:39:00 mr-fox CROND[23326]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:39:00 mr-fox CROND[23326]: pam_unix(crond:session): session closed for user root
May 26 07:39:00 mr-fox CROND[23325]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:39:00 mr-fox CROND[23325]: pam_unix(crond:session): session closed for user root
May 26 07:39:01 mr-fox CROND[21718]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:39:01 mr-fox CROND[21718]: pam_unix(crond:session): session closed for user root
May 26 07:40:00 mr-fox crond[24748]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:40:00 mr-fox crond[24750]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:40:00 mr-fox crond[24749]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:40:00 mr-fox crond[24747]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:40:00 mr-fox crond[24746]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 07:40:00 mr-fox crond[24751]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:40:00 mr-fox CROND[24757]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:40:00 mr-fox crond[24753]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:40:00 mr-fox CROND[24759]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:40:00 mr-fox CROND[24760]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:40:00 mr-fox crond[24754]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:40:00 mr-fox CROND[24758]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:40:00 mr-fox CROND[24762]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 07:40:00 mr-fox CROND[24763]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:40:00 mr-fox CROND[24764]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:40:00 mr-fox CROND[24765]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:40:00 mr-fox CROND[24746]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:40:00 mr-fox CROND[24746]: pam_unix(crond:session): session closed for user torproject
May 26 07:40:00 mr-fox CROND[24754]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:40:00 mr-fox CROND[24754]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:40:00 mr-fox CROND[24750]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:40:00 mr-fox CROND[24750]: pam_unix(crond:session): session closed for user root
May 26 07:40:00 mr-fox CROND[24751]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:40:00 mr-fox CROND[24751]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:40:00 mr-fox CROND[24749]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:40:00 mr-fox CROND[24749]: pam_unix(crond:session): session closed for user root
May 26 07:40:01 mr-fox CROND[23324]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:40:01 mr-fox CROND[23324]: pam_unix(crond:session): session closed for user root
May 26 07:41:00 mr-fox crond[26598]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:41:00 mr-fox crond[26600]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:41:00 mr-fox crond[26599]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:41:00 mr-fox crond[26602]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:41:00 mr-fox crond[26601]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:41:00 mr-fox CROND[26607]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:41:00 mr-fox CROND[26606]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:41:00 mr-fox CROND[26608]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:41:00 mr-fox CROND[26609]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:41:00 mr-fox CROND[26610]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:41:00 mr-fox CROND[26602]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:41:00 mr-fox CROND[26602]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:41:00 mr-fox CROND[26601]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:41:00 mr-fox CROND[26601]: pam_unix(crond:session): session closed for user root
May 26 07:41:00 mr-fox CROND[26600]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:41:00 mr-fox CROND[26600]: pam_unix(crond:session): session closed for user root
May 26 07:41:00 mr-fox CROND[24748]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:41:00 mr-fox CROND[24748]: pam_unix(crond:session): session closed for user root
May 26 07:41:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:41:45 mr-fox kernel: rcu: \x0921-....: (1500128 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=595308
May 26 07:41:45 mr-fox kernel: rcu: \x09(t=1500129 jiffies g=8794409 q=43595347 ncpus=32)
May 26 07:41:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:41:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:41:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:41:45 mr-fox kernel: RIP: 0010:xas_descend+0x1a/0xd0
May 26 07:41:45 mr-fox kernel: Code: 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f <0f> 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5
May 26 07:41:45 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000293
May 26 07:41:45 mr-fox kernel: RAX: ffff988d55f18ffa RBX: 000000004cd994a1 RCX: 0000000000000000
May 26 07:41:45 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988d55f18ff8 RDI: ffffa401077e3970
May 26 07:41:45 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:41:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 07:41:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:41:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:41:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:41:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:41:45 mr-fox kernel: PKRU: 55555554
May 26 07:41:45 mr-fox kernel: Call Trace:
May 26 07:41:45 mr-fox kernel:  <IRQ>
May 26 07:41:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:41:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:41:45 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 07:41:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:41:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:41:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:41:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:41:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:41:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:41:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:41:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:41:45 mr-fox kernel:  </IRQ>
May 26 07:41:45 mr-fox kernel:  <TASK>
May 26 07:41:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:41:45 mr-fox kernel:  ? xas_descend+0x1a/0xd0
May 26 07:41:45 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:41:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:41:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:41:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:41:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:41:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:41:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:41:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:41:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:41:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:41:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:41:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:41:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:41:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:41:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:41:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:41:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:41:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:41:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:41:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:41:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:41:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:41:45 mr-fox kernel:  </TASK>
May 26 07:41:56 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1502831 jiffies s: 491905 root: 0x2/.
May 26 07:41:56 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:41:56 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:41:56 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:41:56 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:41:56 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:41:56 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:41:56 mr-fox kernel: RIP: 0010:rb_next+0x1c/0x60
May 26 07:41:56 mr-fox kernel: Code: e7 48 8b 14 24 4c 8b 65 10 e9 17 ff ff ff f3 0f 1e fa 48 8b 0f 48 39 cf 74 39 48 8b 57 08 48 85 d2 74 23 48 89 d0 48 8b 52 10 <48> 85 d2 75 f4 31 d2 31 c9 31 ff e9 0f 61 1b 00 48 3b 78 08 75 15
May 26 07:41:56 mr-fox kernel: RSP: 0018:ffffa401005f0e18 EFLAGS: 00000286
May 26 07:41:56 mr-fox kernel: RAX: ffff988d57d564c0 RBX: ffff989969200e00 RCX: ffff988d17cee141
May 26 07:41:56 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff989969200e00
May 26 07:41:56 mr-fox kernel: RBP: ffff988b47c8f100 R08: 0000000000000000 R09: 0000000000080300
May 26 07:41:56 mr-fox kernel: R10: 0000000000000026 R11: 0000000000000000 R12: 0000000000000000
May 26 07:41:56 mr-fox kernel: R13: 0000000000000004 R14: ffffffffa6ad7f40 R15: ffff98938f4e4a00
May 26 07:41:56 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:41:56 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:41:56 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:41:56 mr-fox kernel: PKRU: 55555554
May 26 07:41:56 mr-fox kernel: Call Trace:
May 26 07:41:56 mr-fox kernel:  <NMI>
May 26 07:41:56 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:41:56 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:41:56 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:41:56 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:41:56 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:41:56 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:41:56 mr-fox kernel:  ? rb_next+0x1c/0x60
May 26 07:41:56 mr-fox kernel:  ? rb_next+0x1c/0x60
May 26 07:41:56 mr-fox kernel:  ? rb_next+0x1c/0x60
May 26 07:41:56 mr-fox kernel:  </NMI>
May 26 07:41:56 mr-fox kernel:  <IRQ>
May 26 07:41:56 mr-fox kernel:  tcp_enter_loss+0x8b/0x370
May 26 07:41:56 mr-fox kernel:  tcp_retransmit_timer+0x434/0xa60
May 26 07:41:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:41:56 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 07:41:56 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 07:41:56 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 07:41:56 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 07:41:56 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 07:41:56 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:41:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:41:56 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:41:56 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:41:56 mr-fox kernel:  </IRQ>
May 26 07:41:56 mr-fox kernel:  <TASK>
May 26 07:41:56 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:41:56 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 07:41:56 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 07:41:56 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 07:41:56 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:41:56 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:41:56 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:41:56 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:41:56 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:41:56 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:41:56 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:41:56 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:41:56 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:41:56 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:41:56 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:41:56 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:41:56 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:41:56 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:41:56 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:41:56 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:41:56 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:41:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:41:56 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:41:56 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:41:56 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:41:56 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:41:56 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:41:56 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:41:56 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:41:56 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:41:56 mr-fox kernel:  </TASK>
May 26 07:42:00 mr-fox crond[29761]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:42:00 mr-fox crond[29762]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:42:00 mr-fox crond[29760]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:42:00 mr-fox CROND[29767]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:42:00 mr-fox crond[29763]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:42:00 mr-fox crond[29764]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:42:00 mr-fox CROND[29768]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:42:00 mr-fox CROND[29769]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:42:00 mr-fox CROND[29770]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:42:00 mr-fox CROND[29771]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:42:00 mr-fox CROND[29764]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:42:00 mr-fox CROND[29764]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:42:00 mr-fox CROND[29763]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:42:00 mr-fox CROND[29763]: pam_unix(crond:session): session closed for user root
May 26 07:42:00 mr-fox CROND[29762]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:42:00 mr-fox CROND[29762]: pam_unix(crond:session): session closed for user root
May 26 07:42:00 mr-fox CROND[26599]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:42:00 mr-fox CROND[26599]: pam_unix(crond:session): session closed for user root
May 26 07:43:00 mr-fox crond[31688]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:43:00 mr-fox crond[31689]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:43:00 mr-fox crond[31686]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:43:00 mr-fox crond[31690]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:43:00 mr-fox crond[31687]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:43:00 mr-fox CROND[31695]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:43:00 mr-fox CROND[31696]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:43:00 mr-fox CROND[31697]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:43:00 mr-fox CROND[31694]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:43:00 mr-fox CROND[31698]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:43:00 mr-fox CROND[31690]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:43:00 mr-fox CROND[31690]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:43:00 mr-fox CROND[31689]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:43:00 mr-fox CROND[31689]: pam_unix(crond:session): session closed for user root
May 26 07:43:00 mr-fox CROND[31688]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:43:00 mr-fox CROND[31688]: pam_unix(crond:session): session closed for user root
May 26 07:43:01 mr-fox CROND[29761]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:43:01 mr-fox CROND[29761]: pam_unix(crond:session): session closed for user root
May 26 07:44:00 mr-fox crond[31641]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:44:00 mr-fox crond[31643]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:44:00 mr-fox crond[31642]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:44:00 mr-fox crond[31645]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:44:00 mr-fox CROND[31649]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:44:00 mr-fox CROND[31651]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:44:00 mr-fox crond[31644]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:44:00 mr-fox CROND[31652]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:44:00 mr-fox CROND[31653]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:44:00 mr-fox CROND[31654]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:44:00 mr-fox CROND[31645]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:44:00 mr-fox CROND[31645]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:44:00 mr-fox CROND[31644]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:44:00 mr-fox CROND[31644]: pam_unix(crond:session): session closed for user root
May 26 07:44:00 mr-fox CROND[31643]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:44:00 mr-fox CROND[31643]: pam_unix(crond:session): session closed for user root
May 26 07:44:01 mr-fox CROND[31687]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:44:01 mr-fox CROND[31687]: pam_unix(crond:session): session closed for user root
May 26 07:44:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:44:45 mr-fox kernel: rcu: \x0921-....: (1545132 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=614498
May 26 07:44:45 mr-fox kernel: rcu: \x09(t=1545133 jiffies g=8794409 q=44664468 ncpus=32)
May 26 07:44:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:44:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:44:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:44:45 mr-fox kernel: RIP: 0010:xas_load+0x11/0x60
May 26 07:44:45 mr-fox kernel: Code: 92 4c 89 ee 48 c7 c7 e0 eb 49 a6 e8 a9 58 c6 ff eb be 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 53 48 89 fb e8 f2 f7 ff ff 48 89 c2 <83> e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31
May 26 07:44:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 07:44:45 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:44:45 mr-fox kernel: RDX: ffff988ac0754ffa RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:44:45 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 07:44:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:44:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:44:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:44:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:44:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:44:45 mr-fox kernel: PKRU: 55555554
May 26 07:44:45 mr-fox kernel: Call Trace:
May 26 07:44:45 mr-fox kernel:  <IRQ>
May 26 07:44:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:44:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:44:45 mr-fox kernel:  ? tcp_write_xmit+0xe3/0x13b0
May 26 07:44:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:44:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:44:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:44:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:44:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:44:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:44:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:44:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:44:45 mr-fox kernel:  </IRQ>
May 26 07:44:45 mr-fox kernel:  <TASK>
May 26 07:44:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:44:45 mr-fox kernel:  ? xas_load+0x11/0x60
May 26 07:44:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:44:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:44:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:44:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:44:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:44:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:44:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:44:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:44:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:44:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:44:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:44:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:44:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:44:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:44:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:44:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:44:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:44:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:44:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:44:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:44:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:44:45 mr-fox kernel:  </TASK>
May 26 07:44:56 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1547888 jiffies s: 491905 root: 0x2/.
May 26 07:44:56 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:44:56 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:44:56 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:44:56 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:44:56 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:44:56 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:44:56 mr-fox kernel: RIP: 0010:igb_poll+0x153/0x1370
May 26 07:44:56 mr-fox kernel: Code: c3 38 4d 39 e7 75 e2 49 83 c4 10 41 83 c6 01 0f 84 be 0a 00 00 41 0f 18 0c 24 83 6c 24 18 01 0f 85 5c ff ff ff 41 0f b6 45 4e <49> 8b 55 08 4c 8b 5c 24 10 4c 8d 3c 80 49 c1 e7 06 4c 03 7a 18 85
May 26 07:44:56 mr-fox kernel: RSP: 0018:ffffa401005f0df8 EFLAGS: 00000246
May 26 07:44:56 mr-fox kernel: RAX: 0000000000000002 RBX: ffffa40107bf0298 RCX: 0000000000000000
May 26 07:44:56 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:44:56 mr-fox kernel: RBP: 000000000000db2c R08: 0000000000000000 R09: 0000000000000000
May 26 07:44:56 mr-fox kernel: R10: ffff98a279748240 R11: 0000000000000000 R12: ffffa40100119550
May 26 07:44:56 mr-fox kernel: R13: ffff988ac873fa40 R14: 00000000ffffff55 R15: 0000000000000000
May 26 07:44:56 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:44:56 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:44:56 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:44:56 mr-fox kernel: PKRU: 55555554
May 26 07:44:56 mr-fox kernel: Call Trace:
May 26 07:44:56 mr-fox kernel:  <NMI>
May 26 07:44:56 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:44:56 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:44:56 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:44:56 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:44:56 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:44:56 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:44:56 mr-fox kernel:  ? igb_poll+0x153/0x1370
May 26 07:44:56 mr-fox kernel:  ? igb_poll+0x153/0x1370
May 26 07:44:56 mr-fox kernel:  ? igb_poll+0x153/0x1370
May 26 07:44:56 mr-fox kernel:  </NMI>
May 26 07:44:56 mr-fox kernel:  <IRQ>
May 26 07:44:56 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 07:44:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:44:56 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 07:44:56 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 07:44:56 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 07:44:56 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 07:44:56 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:44:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:44:56 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:44:56 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:44:56 mr-fox kernel:  </IRQ>
May 26 07:44:56 mr-fox kernel:  <TASK>
May 26 07:44:56 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:44:56 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 07:44:56 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 07:44:56 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 07:44:56 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:44:56 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:44:56 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:44:56 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:44:56 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:44:56 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:44:56 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:44:56 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:44:56 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:44:56 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:44:56 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:44:56 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:44:56 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:44:56 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:44:56 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:44:56 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:44:56 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:44:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:44:56 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:44:56 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:44:56 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:44:56 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:44:56 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:44:56 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:44:56 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:44:56 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:44:56 mr-fox kernel:  </TASK>
May 26 07:45:00 mr-fox crond[3101]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:45:00 mr-fox crond[3097]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 07:45:00 mr-fox crond[3099]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:45:00 mr-fox crond[3098]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:45:00 mr-fox crond[3102]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:45:00 mr-fox CROND[3106]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:45:00 mr-fox CROND[3107]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:45:00 mr-fox CROND[3110]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:45:00 mr-fox crond[3105]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:45:00 mr-fox crond[3104]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:45:00 mr-fox CROND[3115]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:45:00 mr-fox crond[3103]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:45:00 mr-fox CROND[3117]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:45:00 mr-fox CROND[3118]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:45:00 mr-fox CROND[3119]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 07:45:00 mr-fox CROND[3120]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:45:00 mr-fox CROND[3097]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:45:00 mr-fox CROND[3097]: pam_unix(crond:session): session closed for user torproject
May 26 07:45:00 mr-fox CROND[3105]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:45:00 mr-fox CROND[3105]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:45:00 mr-fox CROND[3102]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:45:00 mr-fox CROND[3102]: pam_unix(crond:session): session closed for user root
May 26 07:45:00 mr-fox CROND[3103]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:45:00 mr-fox CROND[3103]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:45:00 mr-fox CROND[3101]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:45:00 mr-fox CROND[3101]: pam_unix(crond:session): session closed for user root
May 26 07:45:00 mr-fox CROND[31642]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:45:00 mr-fox CROND[31642]: pam_unix(crond:session): session closed for user root
May 26 07:46:00 mr-fox crond[25209]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:46:00 mr-fox crond[25211]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:46:00 mr-fox crond[25212]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:46:00 mr-fox crond[25218]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:46:00 mr-fox CROND[25220]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:46:00 mr-fox crond[25210]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:46:00 mr-fox CROND[25221]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:46:00 mr-fox CROND[25222]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:46:00 mr-fox CROND[25223]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:46:00 mr-fox CROND[25225]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:46:00 mr-fox CROND[25218]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:46:00 mr-fox CROND[25218]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:46:00 mr-fox CROND[25212]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:46:00 mr-fox CROND[25212]: pam_unix(crond:session): session closed for user root
May 26 07:46:00 mr-fox CROND[3099]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:46:00 mr-fox CROND[3099]: pam_unix(crond:session): session closed for user root
May 26 07:46:00 mr-fox CROND[25211]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:46:00 mr-fox CROND[25211]: pam_unix(crond:session): session closed for user root
May 26 07:47:00 mr-fox crond[27690]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:47:00 mr-fox crond[27693]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:47:00 mr-fox crond[27692]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:47:00 mr-fox crond[27695]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:47:00 mr-fox crond[27691]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:47:00 mr-fox CROND[27698]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:47:00 mr-fox CROND[27699]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:47:00 mr-fox CROND[27701]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:47:00 mr-fox CROND[27702]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:47:00 mr-fox CROND[27703]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:47:00 mr-fox CROND[27695]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:47:00 mr-fox CROND[27695]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:47:00 mr-fox CROND[27693]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:47:00 mr-fox CROND[27693]: pam_unix(crond:session): session closed for user root
May 26 07:47:01 mr-fox CROND[27692]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:47:01 mr-fox CROND[27692]: pam_unix(crond:session): session closed for user root
May 26 07:47:01 mr-fox CROND[25210]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:47:01 mr-fox CROND[25210]: pam_unix(crond:session): session closed for user root
May 26 07:47:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:47:45 mr-fox kernel: rcu: \x0921-....: (1590136 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=633716
May 26 07:47:45 mr-fox kernel: rcu: \x09(t=1590137 jiffies g=8794409 q=46116498 ncpus=32)
May 26 07:47:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:47:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:47:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:47:45 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 07:47:45 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 07:47:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 07:47:45 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:47:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:47:45 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:47:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:47:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:47:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:47:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:47:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:47:45 mr-fox kernel: PKRU: 55555554
May 26 07:47:45 mr-fox kernel: Call Trace:
May 26 07:47:45 mr-fox kernel:  <IRQ>
May 26 07:47:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:47:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:47:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:47:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:47:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:47:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:47:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:47:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:47:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:47:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:47:45 mr-fox kernel:  </IRQ>
May 26 07:47:45 mr-fox kernel:  <TASK>
May 26 07:47:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:47:45 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 07:47:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:47:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:47:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:47:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:47:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:47:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:47:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:47:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:47:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:47:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:47:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:47:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:47:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:47:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:47:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:47:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:47:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:47:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:47:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:47:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:47:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:47:45 mr-fox kernel:  </TASK>
May 26 07:47:56 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1592944 jiffies s: 491905 root: 0x2/.
May 26 07:47:56 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:47:56 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:47:56 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:47:56 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:47:56 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:47:56 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:47:56 mr-fox kernel: RIP: 0010:ep_poll_callback+0x20/0x280
May 26 07:47:56 mr-fox kernel: Code: ff e9 3f 97 80 00 0f 1f 40 00 f3 0f 1e fa 41 57 41 56 41 55 41 54 49 89 fc 55 48 89 cd 53 48 83 ec 18 48 8b 5f f8 4c 8b 7b 48 <4d> 8d 77 60 4c 89 f7 e8 d4 65 6b 00 48 89 44 24 08 8b 05 49 8c 1a
May 26 07:47:56 mr-fox kernel: RSP: 0018:ffffa401005f0930 EFLAGS: 00000086
May 26 07:47:56 mr-fox kernel: RAX: ffffffffa52fa9d0 RBX: ffff988dc9583d80 RCX: 00000000000000c3
May 26 07:47:56 mr-fox kernel: RDX: 0000000000000010 RSI: 0000000000000001 RDI: ffff988dc9582f10
May 26 07:47:56 mr-fox kernel: RBP: 00000000000000c3 R08: ffff988dc3fe98c8 R09: 0000000000000000
May 26 07:47:56 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff988dc9582f10
May 26 07:47:56 mr-fox kernel: R13: 0000000000000001 R14: 0000000000000010 R15: ffff988ad0a32480
May 26 07:47:56 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:47:56 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:47:56 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:47:56 mr-fox kernel: PKRU: 55555554
May 26 07:47:56 mr-fox kernel: Call Trace:
May 26 07:47:56 mr-fox kernel:  <NMI>
May 26 07:47:56 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:47:56 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:47:56 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:47:56 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:47:56 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:47:56 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:47:56 mr-fox kernel:  ? ep_autoremove_wake_function+0x50/0x50
May 26 07:47:56 mr-fox kernel:  ? ep_poll_callback+0x20/0x280
May 26 07:47:56 mr-fox kernel:  ? ep_poll_callback+0x20/0x280
May 26 07:47:56 mr-fox kernel:  ? ep_poll_callback+0x20/0x280
May 26 07:47:56 mr-fox kernel:  </NMI>
May 26 07:47:56 mr-fox kernel:  <IRQ>
May 26 07:47:56 mr-fox kernel:  __wake_up_common+0x73/0xa0
May 26 07:47:56 mr-fox kernel:  __wake_up_sync_key+0x36/0x60
May 26 07:47:56 mr-fox kernel:  sock_def_readable+0x36/0x80
May 26 07:47:56 mr-fox kernel:  tcp_data_queue+0x8ce/0x1090
May 26 07:47:56 mr-fox kernel:  tcp_rcv_established+0x1f2/0x6c0
May 26 07:47:56 mr-fox kernel:  ? sk_filter_trim_cap+0x40/0x220
May 26 07:47:56 mr-fox kernel:  tcp_v4_do_rcv+0x153/0x240
May 26 07:47:56 mr-fox kernel:  tcp_v4_rcv+0xe00/0xea0
May 26 07:47:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:47:56 mr-fox kernel:  ip_protocol_deliver_rcu+0x32/0x180
May 26 07:47:56 mr-fox kernel:  ip_local_deliver_finish+0x74/0xa0
May 26 07:47:56 mr-fox kernel:  ip_sublist_rcv_finish+0x7f/0x90
May 26 07:47:56 mr-fox kernel:  ip_sublist_rcv+0x176/0x1c0
May 26 07:47:56 mr-fox kernel:  ? ip_sublist_rcv+0x1c0/0x1c0
May 26 07:47:56 mr-fox kernel:  ip_list_rcv+0x138/0x170
May 26 07:47:56 mr-fox kernel:  __netif_receive_skb_list_core+0x293/0x2d0
May 26 07:47:56 mr-fox kernel:  netif_receive_skb_list_internal+0x1db/0x320
May 26 07:47:56 mr-fox kernel:  napi_gro_receive+0xcf/0x1b0
May 26 07:47:56 mr-fox kernel:  igb_poll+0x605/0x1370
May 26 07:47:56 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 07:47:56 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 07:47:56 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:47:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:47:56 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:47:56 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:47:56 mr-fox kernel:  </IRQ>
May 26 07:47:56 mr-fox kernel:  <TASK>
May 26 07:47:56 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:47:56 mr-fox kernel: RIP: 0010:xas_descend+0x26/0xd0
May 26 07:47:56 mr-fox kernel: Code: 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f <89> d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03
May 26 07:47:56 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 07:47:56 mr-fox kernel: RAX: ffff988ac07566ca RBX: 000000000000000c RCX: 0000000000000018
May 26 07:47:56 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac07566c8 RDI: ffffa401077e3970
May 26 07:47:56 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:47:56 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 07:47:56 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:47:56 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:47:56 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:47:56 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:47:56 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:47:56 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:47:56 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:47:56 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:47:56 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:47:56 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:47:56 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:47:56 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:47:56 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:47:56 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:47:56 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:47:56 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:47:56 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:47:56 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:47:56 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:47:56 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:47:56 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:47:56 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:47:56 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:47:56 mr-fox kernel:  </TASK>
May 26 07:48:00 mr-fox crond[19993]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:48:00 mr-fox crond[19994]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:48:00 mr-fox crond[19991]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:48:00 mr-fox crond[19996]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:48:00 mr-fox CROND[20001]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:48:00 mr-fox CROND[20000]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:48:00 mr-fox CROND[19999]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:48:00 mr-fox crond[19995]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:48:00 mr-fox CROND[20002]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:48:00 mr-fox CROND[20003]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:48:00 mr-fox CROND[19996]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:48:00 mr-fox CROND[19996]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:48:00 mr-fox CROND[19995]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:48:00 mr-fox CROND[19995]: pam_unix(crond:session): session closed for user root
May 26 07:48:00 mr-fox CROND[19994]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:48:00 mr-fox CROND[19994]: pam_unix(crond:session): session closed for user root
May 26 07:48:01 mr-fox CROND[27691]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:48:01 mr-fox CROND[27691]: pam_unix(crond:session): session closed for user root
May 26 07:49:00 mr-fox crond[23097]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:49:00 mr-fox crond[23099]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:49:00 mr-fox crond[23101]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:49:00 mr-fox crond[23102]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:49:00 mr-fox CROND[23106]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:49:00 mr-fox crond[23100]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:49:00 mr-fox CROND[23107]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:49:00 mr-fox CROND[23109]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:49:00 mr-fox CROND[23108]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:49:00 mr-fox CROND[23111]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:49:00 mr-fox CROND[23102]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:49:00 mr-fox CROND[23102]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:49:00 mr-fox CROND[23101]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:49:00 mr-fox CROND[23101]: pam_unix(crond:session): session closed for user root
May 26 07:49:00 mr-fox CROND[23100]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:49:00 mr-fox CROND[23100]: pam_unix(crond:session): session closed for user root
May 26 07:49:00 mr-fox CROND[19993]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:49:00 mr-fox CROND[19993]: pam_unix(crond:session): session closed for user root
May 26 07:50:00 mr-fox crond[26184]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:50:00 mr-fox crond[26183]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 07:50:00 mr-fox crond[26187]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:50:00 mr-fox crond[26186]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:50:00 mr-fox crond[26188]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:50:00 mr-fox CROND[26195]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:50:00 mr-fox CROND[26196]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:50:00 mr-fox CROND[26197]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:50:00 mr-fox CROND[26198]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:50:00 mr-fox crond[26190]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:50:00 mr-fox crond[26189]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:50:00 mr-fox crond[26191]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:50:00 mr-fox CROND[26199]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:50:00 mr-fox CROND[26202]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:50:00 mr-fox CROND[26203]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:50:00 mr-fox CROND[26204]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 07:50:00 mr-fox CROND[26183]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:50:00 mr-fox CROND[26183]: pam_unix(crond:session): session closed for user torproject
May 26 07:50:00 mr-fox CROND[26191]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:50:00 mr-fox CROND[26191]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:50:00 mr-fox CROND[26188]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:50:00 mr-fox CROND[26188]: pam_unix(crond:session): session closed for user root
May 26 07:50:01 mr-fox CROND[26189]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:50:01 mr-fox CROND[26189]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:50:01 mr-fox CROND[23099]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:50:01 mr-fox CROND[23099]: pam_unix(crond:session): session closed for user root
May 26 07:50:01 mr-fox CROND[26187]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:50:01 mr-fox CROND[26187]: pam_unix(crond:session): session closed for user root
May 26 07:50:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:50:45 mr-fox kernel: rcu: \x0921-....: (1635139 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=652834
May 26 07:50:45 mr-fox kernel: rcu: \x09(t=1635140 jiffies g=8794409 q=47155035 ncpus=32)
May 26 07:50:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:50:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:50:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:50:45 mr-fox kernel: RIP: 0010:xas_descend+0x2/0xd0
May 26 07:50:45 mr-fox kernel: Code: 18 0f b6 4c 24 10 4c 8b 04 24 e9 60 fe ff ff e9 69 87 04 00 e9 a3 87 04 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 55 <41> 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80
May 26 07:50:45 mr-fox kernel: RSP: 0018:ffffa401077e3940 EFLAGS: 00000206
May 26 07:50:45 mr-fox kernel: RAX: ffff988ac07566ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:50:45 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac07566c8 RDI: ffffa401077e3970
May 26 07:50:45 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:50:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:50:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:50:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:50:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:50:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:50:45 mr-fox kernel: PKRU: 55555554
May 26 07:50:45 mr-fox kernel: Call Trace:
May 26 07:50:45 mr-fox kernel:  <IRQ>
May 26 07:50:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:50:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:50:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:50:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:50:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:50:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:50:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:50:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:50:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:50:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:50:45 mr-fox kernel:  </IRQ>
May 26 07:50:45 mr-fox kernel:  <TASK>
May 26 07:50:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:50:45 mr-fox kernel:  ? xas_descend+0x2/0xd0
May 26 07:50:45 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:50:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:50:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:50:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:50:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:50:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:50:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:50:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:50:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:50:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:50:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:50:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:50:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:50:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:50:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:50:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:50:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:50:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:50:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:50:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:50:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:50:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:50:45 mr-fox kernel:  </TASK>
May 26 07:50:57 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1637999 jiffies s: 491905 root: 0x2/.
May 26 07:50:57 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:50:57 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:50:57 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:50:57 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:50:57 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:50:57 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:50:57 mr-fox kernel: RIP: 0010:__nf_conntrack_find_get.isra.0+0x8d/0x280
May 26 07:50:57 mr-fox kernel: Code: 00 49 8b 07 49 8b 4f 08 48 33 43 10 48 33 4b 18 48 09 c8 75 09 8b 43 20 41 39 47 10 74 7f 48 8b 1b f6 c3 01 75 5b 0f b6 4b 37 <4c> 89 ed 48 8d 04 cd 00 00 00 00 48 29 c8 48 c1 e0 03 48 29 c5 48
May 26 07:50:57 mr-fox kernel: RSP: 0018:ffffa401005f0c28 EFLAGS: 00000246
May 26 07:50:57 mr-fox kernel: RAX: ffff988acb3dcc20 RBX: ffff988d66be0810 RCX: 0000000000000000
May 26 07:50:57 mr-fox kernel: RDX: 00000000ee612205 RSI: ffff988acb200000 RDI: ffffffffa6ad7f40
May 26 07:50:57 mr-fox kernel: RBP: ffffa401005f0c88 R08: 0000000000000000 R09: 0000000000000000
May 26 07:50:57 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000003b984
May 26 07:50:57 mr-fox kernel: R13: fffffffffffffff0 R14: ffffffffa6ad7f40 R15: ffffa401005f0c88
May 26 07:50:57 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:50:57 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:50:57 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:50:57 mr-fox kernel: PKRU: 55555554
May 26 07:50:57 mr-fox kernel: Call Trace:
May 26 07:50:57 mr-fox kernel:  <NMI>
May 26 07:50:57 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:50:57 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:50:57 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:50:57 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:50:57 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:50:57 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:50:57 mr-fox kernel:  ? __nf_conntrack_find_get.isra.0+0x8d/0x280
May 26 07:50:57 mr-fox kernel:  ? __nf_conntrack_find_get.isra.0+0x8d/0x280
May 26 07:50:57 mr-fox kernel:  ? __nf_conntrack_find_get.isra.0+0x8d/0x280
May 26 07:50:57 mr-fox kernel:  </NMI>
May 26 07:50:57 mr-fox kernel:  <IRQ>
May 26 07:50:57 mr-fox kernel:  nf_conntrack_in+0xdc/0x540
May 26 07:50:57 mr-fox kernel:  nf_hook_slow+0x3c/0x100
May 26 07:50:57 mr-fox kernel:  __ip_local_out+0xc6/0x100
May 26 07:50:57 mr-fox kernel:  ? ip_output+0x100/0x100
May 26 07:50:57 mr-fox kernel:  ip_local_out+0x16/0x70
May 26 07:50:57 mr-fox kernel:  __ip_queue_xmit+0x16b/0x480
May 26 07:50:57 mr-fox kernel:  __tcp_transmit_skb+0xbad/0xd30
May 26 07:50:57 mr-fox kernel:  tcp_delack_timer_handler+0xa9/0x110
May 26 07:50:57 mr-fox kernel:  tcp_delack_timer+0xb5/0xf0
May 26 07:50:57 mr-fox kernel:  ? tcp_delack_timer_handler+0x110/0x110
May 26 07:50:57 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 07:50:57 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 07:50:57 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 07:50:57 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:50:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:50:57 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:50:57 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:50:57 mr-fox kernel:  </IRQ>
May 26 07:50:57 mr-fox kernel:  <TASK>
May 26 07:50:57 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:50:57 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 07:50:57 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 07:50:57 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 07:50:57 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:50:57 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:50:57 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:50:57 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:50:57 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:50:57 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:50:57 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:50:57 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:50:57 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:50:57 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:50:57 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:50:57 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:50:57 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:50:57 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:50:57 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:50:57 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:50:57 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:50:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:50:57 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:50:57 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:50:57 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:50:57 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:50:57 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:50:57 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:50:57 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:50:57 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:50:57 mr-fox kernel:  </TASK>
May 26 07:51:00 mr-fox crond[26497]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:51:00 mr-fox crond[26496]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:51:00 mr-fox crond[26498]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:51:00 mr-fox crond[26500]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:51:00 mr-fox crond[26501]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:51:00 mr-fox CROND[26503]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:51:00 mr-fox CROND[26504]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:51:00 mr-fox CROND[26505]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:51:00 mr-fox CROND[26506]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:51:00 mr-fox CROND[26507]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:51:00 mr-fox CROND[26501]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:51:00 mr-fox CROND[26501]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:51:00 mr-fox CROND[26500]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:51:00 mr-fox CROND[26500]: pam_unix(crond:session): session closed for user root
May 26 07:51:00 mr-fox CROND[26498]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:51:00 mr-fox CROND[26498]: pam_unix(crond:session): session closed for user root
May 26 07:51:01 mr-fox CROND[26186]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:51:01 mr-fox CROND[26186]: pam_unix(crond:session): session closed for user root
May 26 07:52:00 mr-fox crond[30638]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:52:00 mr-fox crond[30637]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:52:00 mr-fox crond[30639]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:52:00 mr-fox crond[30640]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:52:00 mr-fox CROND[30645]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:52:00 mr-fox CROND[30646]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:52:00 mr-fox CROND[30648]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:52:00 mr-fox CROND[30647]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:52:00 mr-fox crond[30641]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:52:00 mr-fox CROND[30650]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:52:00 mr-fox CROND[30641]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:52:00 mr-fox CROND[30641]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:52:00 mr-fox CROND[30640]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:52:00 mr-fox CROND[30640]: pam_unix(crond:session): session closed for user root
May 26 07:52:00 mr-fox CROND[26497]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:52:00 mr-fox CROND[26497]: pam_unix(crond:session): session closed for user root
May 26 07:52:00 mr-fox CROND[30639]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:52:00 mr-fox CROND[30639]: pam_unix(crond:session): session closed for user root
May 26 07:53:00 mr-fox crond[32141]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:53:00 mr-fox crond[32139]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:53:00 mr-fox crond[32136]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:53:00 mr-fox crond[32137]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:53:00 mr-fox crond[32142]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:53:00 mr-fox CROND[32145]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:53:00 mr-fox CROND[32146]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:53:00 mr-fox CROND[32147]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:53:00 mr-fox CROND[32149]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:53:00 mr-fox CROND[32151]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:53:00 mr-fox CROND[32142]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:53:00 mr-fox CROND[32142]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:53:00 mr-fox CROND[32141]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:53:00 mr-fox CROND[32141]: pam_unix(crond:session): session closed for user root
May 26 07:53:00 mr-fox CROND[32139]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:53:00 mr-fox CROND[32139]: pam_unix(crond:session): session closed for user root
May 26 07:53:01 mr-fox CROND[30638]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:53:01 mr-fox CROND[30638]: pam_unix(crond:session): session closed for user root
May 26 07:53:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:53:45 mr-fox kernel: rcu: \x0921-....: (1680143 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=672038
May 26 07:53:45 mr-fox kernel: rcu: \x09(t=1680144 jiffies g=8794409 q=48172632 ncpus=32)
May 26 07:53:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:53:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:53:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:53:45 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 07:53:45 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 07:53:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 07:53:45 mr-fox kernel: RAX: ffff988ac07566ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:53:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:53:45 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:53:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:53:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:53:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:53:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:53:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:53:45 mr-fox kernel: PKRU: 55555554
May 26 07:53:45 mr-fox kernel: Call Trace:
May 26 07:53:45 mr-fox kernel:  <IRQ>
May 26 07:53:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:53:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:53:45 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 07:53:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:53:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:53:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:53:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:53:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:53:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:53:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:53:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:53:45 mr-fox kernel:  </IRQ>
May 26 07:53:45 mr-fox kernel:  <TASK>
May 26 07:53:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:53:45 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 07:53:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:53:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:53:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:53:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:53:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:53:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:53:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:53:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:53:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:53:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:53:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:53:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:53:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:53:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:53:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:53:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:53:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:53:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:53:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:53:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:53:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:53:45 mr-fox kernel:  </TASK>
May 26 07:53:57 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1683055 jiffies s: 491905 root: 0x2/.
May 26 07:53:57 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:53:57 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:53:57 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:53:57 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:53:57 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:53:57 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:53:57 mr-fox kernel: RIP: 0010:__dev_queue_xmit+0x3ff/0x9a0
May 26 07:53:57 mr-fox kernel: Code: 69 fd ff ff c7 85 84 00 00 00 ff ff ff ff c6 85 80 00 00 00 00 e9 17 fe ff ff a8 04 0f 84 ac 01 00 00 49 8b 84 24 d8 00 00 00 <a8> 0c 0f 85 9c 01 00 00 4d 8d ac 24 44 01 00 00 4c 89 ef e8 99 9e
May 26 07:53:57 mr-fox kernel: RSP: 0018:ffffa401005f0b40 EFLAGS: 00000202
May 26 07:53:57 mr-fox kernel: RAX: 0000000000000000 RBX: ffff988b22535f00 RCX: 0000000000000000
May 26 07:53:57 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:53:57 mr-fox kernel: RBP: ffff988ac76ec140 R08: 0000000000000000 R09: ffff988e055ee400
May 26 07:53:57 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff988ac5503800
May 26 07:53:57 mr-fox kernel: R13: 0000000000000000 R14: 0000000000000010 R15: ffff988ac4bb4000
May 26 07:53:57 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:53:57 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:53:57 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:53:57 mr-fox kernel: PKRU: 55555554
May 26 07:53:57 mr-fox kernel: Call Trace:
May 26 07:53:57 mr-fox kernel:  <NMI>
May 26 07:53:57 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:53:57 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:53:57 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:53:57 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:53:57 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:53:57 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:53:57 mr-fox kernel:  ? __dev_queue_xmit+0x3ff/0x9a0
May 26 07:53:57 mr-fox kernel:  ? __dev_queue_xmit+0x3ff/0x9a0
May 26 07:53:57 mr-fox kernel:  ? __dev_queue_xmit+0x3ff/0x9a0
May 26 07:53:57 mr-fox kernel:  </NMI>
May 26 07:53:57 mr-fox kernel:  <IRQ>
May 26 07:53:57 mr-fox kernel:  ? ip6t_do_table+0x30b/0x590
May 26 07:53:57 mr-fox kernel:  ip6_finish_output2+0x2c0/0x610
May 26 07:53:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:53:57 mr-fox kernel:  ? ip6_output+0xa7/0x290
May 26 07:53:57 mr-fox kernel:  ip6_xmit+0x3fc/0x600
May 26 07:53:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:53:57 mr-fox kernel:  ? ip6_output+0x290/0x290
May 26 07:53:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:53:57 mr-fox kernel:  ? __sk_dst_check+0x34/0xa0
May 26 07:53:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:53:57 mr-fox kernel:  ? inet6_csk_route_socket+0x132/0x210
May 26 07:53:57 mr-fox kernel:  inet6_csk_xmit+0xe9/0x160
May 26 07:53:57 mr-fox kernel:  __tcp_transmit_skb+0x5d0/0xd30
May 26 07:53:57 mr-fox kernel:  tcp_delack_timer_handler+0xa9/0x110
May 26 07:53:57 mr-fox kernel:  tcp_delack_timer+0xb5/0xf0
May 26 07:53:57 mr-fox kernel:  ? tcp_delack_timer_handler+0x110/0x110
May 26 07:53:57 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 07:53:57 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 07:53:57 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 07:53:57 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:53:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:53:57 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:53:57 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:53:57 mr-fox kernel:  </IRQ>
May 26 07:53:57 mr-fox kernel:  <TASK>
May 26 07:53:57 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:53:57 mr-fox kernel: RIP: 0010:xas_descend+0xc/0xd0
May 26 07:53:57 mr-fox kernel: Code: e9 60 fe ff ff e9 69 87 04 00 e9 a3 87 04 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 <48> 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3
May 26 07:53:57 mr-fox kernel: RSP: 0018:ffffa401077e3928 EFLAGS: 00000246
May 26 07:53:57 mr-fox kernel: RAX: ffff988d55f18ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 07:53:57 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988d55f18ff8 RDI: ffffa401077e3970
May 26 07:53:57 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:53:57 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 07:53:57 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:53:57 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:53:57 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:53:57 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:53:57 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:53:57 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:53:57 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:53:57 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:53:57 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:53:57 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:53:57 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:53:57 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:53:57 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:53:57 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:53:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:53:57 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:53:57 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:53:57 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:53:57 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:53:57 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:53:57 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:53:57 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:53:57 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:53:57 mr-fox kernel:  </TASK>
May 26 07:54:00 mr-fox crond[379]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:54:00 mr-fox crond[380]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:54:00 mr-fox crond[382]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:54:00 mr-fox crond[381]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:54:00 mr-fox crond[384]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:54:00 mr-fox CROND[386]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:54:00 mr-fox CROND[388]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:54:00 mr-fox CROND[387]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:54:00 mr-fox CROND[389]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:54:00 mr-fox CROND[390]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:54:00 mr-fox CROND[384]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:54:00 mr-fox CROND[384]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:54:00 mr-fox CROND[382]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:54:00 mr-fox CROND[382]: pam_unix(crond:session): session closed for user root
May 26 07:54:00 mr-fox CROND[381]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:54:00 mr-fox CROND[381]: pam_unix(crond:session): session closed for user root
May 26 07:54:01 mr-fox CROND[32137]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:54:01 mr-fox CROND[32137]: pam_unix(crond:session): session closed for user root
May 26 07:55:00 mr-fox crond[2083]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 07:55:00 mr-fox crond[2085]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:55:00 mr-fox crond[2084]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:55:00 mr-fox crond[2086]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:55:00 mr-fox crond[2087]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:55:00 mr-fox crond[2088]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:55:00 mr-fox crond[2089]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:55:00 mr-fox CROND[2094]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:55:00 mr-fox CROND[2095]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:55:00 mr-fox CROND[2096]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:55:00 mr-fox CROND[2098]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:55:00 mr-fox crond[2091]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:55:00 mr-fox CROND[2112]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:55:00 mr-fox CROND[2113]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:55:00 mr-fox CROND[2116]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 07:55:00 mr-fox CROND[2118]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:55:00 mr-fox CROND[2083]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 07:55:00 mr-fox CROND[2083]: pam_unix(crond:session): session closed for user torproject
May 26 07:55:00 mr-fox CROND[2091]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:55:00 mr-fox CROND[2091]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:55:00 mr-fox CROND[2087]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:55:00 mr-fox CROND[2087]: pam_unix(crond:session): session closed for user root
May 26 07:55:00 mr-fox CROND[2088]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 07:55:00 mr-fox CROND[2088]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:55:00 mr-fox CROND[2086]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:55:00 mr-fox CROND[2086]: pam_unix(crond:session): session closed for user root
May 26 07:55:00 mr-fox CROND[380]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:55:00 mr-fox CROND[380]: pam_unix(crond:session): session closed for user root
May 26 07:56:00 mr-fox crond[5062]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:56:00 mr-fox crond[5063]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:56:00 mr-fox crond[5064]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:56:00 mr-fox crond[5065]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:56:00 mr-fox crond[5061]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:56:00 mr-fox CROND[5069]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:56:00 mr-fox CROND[5072]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:56:00 mr-fox CROND[5073]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:56:00 mr-fox CROND[5074]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:56:00 mr-fox CROND[5075]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:56:00 mr-fox CROND[5065]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:56:00 mr-fox CROND[5065]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:56:00 mr-fox CROND[5064]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:56:00 mr-fox CROND[5064]: pam_unix(crond:session): session closed for user root
May 26 07:56:00 mr-fox CROND[5063]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:56:00 mr-fox CROND[5063]: pam_unix(crond:session): session closed for user root
May 26 07:56:00 mr-fox CROND[2085]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:56:00 mr-fox CROND[2085]: pam_unix(crond:session): session closed for user root
May 26 07:56:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:56:45 mr-fox kernel: rcu: \x0921-....: (1725147 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=691167
May 26 07:56:45 mr-fox kernel: rcu: \x09(t=1725148 jiffies g=8794409 q=49192845 ncpus=32)
May 26 07:56:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:56:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:56:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:56:45 mr-fox kernel: RIP: 0010:xas_load+0x23/0x60
May 26 07:56:45 mr-fox kernel: Code: 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 53 48 89 fb e8 f2 f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b <5d> 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe
May 26 07:56:45 mr-fox kernel: RSP: 0018:ffffa401077e3958 EFLAGS: 00000246
May 26 07:56:45 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 07:56:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 07:56:45 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:56:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 07:56:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:56:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:56:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:56:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:56:45 mr-fox kernel: PKRU: 55555554
May 26 07:56:45 mr-fox kernel: Call Trace:
May 26 07:56:45 mr-fox kernel:  <IRQ>
May 26 07:56:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:56:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:56:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:56:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:56:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:56:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:56:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:56:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:56:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:56:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:56:45 mr-fox kernel:  </IRQ>
May 26 07:56:45 mr-fox kernel:  <TASK>
May 26 07:56:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:56:45 mr-fox kernel:  ? xas_load+0x23/0x60
May 26 07:56:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:56:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:56:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:56:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:56:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:56:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:56:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:56:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:56:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:56:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:56:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:56:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:56:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:56:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:56:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:56:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:56:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:56:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:56:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:56:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:56:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:56:45 mr-fox kernel:  </TASK>
May 26 07:56:57 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1728111 jiffies s: 491905 root: 0x2/.
May 26 07:56:57 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:56:57 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:56:57 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:56:57 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:56:57 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:56:57 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:56:57 mr-fox kernel: RIP: 0010:__sock_wfree+0x24/0x60
May 26 07:56:57 mr-fox kernel: Code: f4 ff ff 0f 1f 00 f3 0f 1e fa 48 89 f8 8b 90 d0 00 00 00 48 8b 7f 18 89 d0 4c 8d 87 44 01 00 00 f7 d8 f0 0f c1 87 44 01 00 00 <39> c2 74 1a 89 c1 29 d1 09 c1 78 17 31 c0 31 d2 31 c9 31 f6 31 ff
May 26 07:56:57 mr-fox kernel: RSP: 0018:ffffa401005f0dd0 EFLAGS: 00000213
May 26 07:56:57 mr-fox kernel: RAX: 0000000000000003 RBX: ffff989b3a303100 RCX: 0000000000000000
May 26 07:56:57 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000040 RDI: ffff98938d66d340
May 26 07:56:57 mr-fox kernel: RBP: 000000000000163c R08: ffff98938d66d484 R09: 0000000000000000
May 26 07:56:57 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffa40100119ad0
May 26 07:56:57 mr-fox kernel: R13: ffff988ac873fa40 R14: 00000000ffffffad R15: ffffa40100119ae0
May 26 07:56:57 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:56:57 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:56:57 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:56:57 mr-fox kernel: PKRU: 55555554
May 26 07:56:57 mr-fox kernel: Call Trace:
May 26 07:56:57 mr-fox kernel:  <NMI>
May 26 07:56:57 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:56:57 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:56:57 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:56:57 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:56:57 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:56:57 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:56:57 mr-fox kernel:  ? __sock_wfree+0x24/0x60
May 26 07:56:57 mr-fox kernel:  ? __sock_wfree+0x24/0x60
May 26 07:56:57 mr-fox kernel:  ? __sock_wfree+0x24/0x60
May 26 07:56:57 mr-fox kernel:  </NMI>
May 26 07:56:57 mr-fox kernel:  <IRQ>
May 26 07:56:57 mr-fox kernel:  skb_release_head_state+0x22/0x80
May 26 07:56:57 mr-fox kernel:  napi_consume_skb+0x2e/0xc0
May 26 07:56:57 mr-fox kernel:  igb_poll+0xea/0x1370
May 26 07:56:57 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 07:56:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:56:57 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 07:56:57 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 07:56:57 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 07:56:57 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:56:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:56:57 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:56:57 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:56:57 mr-fox kernel:  </IRQ>
May 26 07:56:57 mr-fox kernel:  <TASK>
May 26 07:56:57 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:56:57 mr-fox kernel: RIP: 0010:xas_descend+0x31/0xd0
May 26 07:56:57 mr-fox kernel: Code: 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 <49> 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d fd 00 00
May 26 07:56:57 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 07:56:57 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: 0000000000000036 RCX: 0000000000000012
May 26 07:56:57 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988acf90e6c8 RDI: ffffa401077e3970
May 26 07:56:57 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:56:57 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 07:56:57 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:56:57 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:56:57 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:56:57 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:56:57 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:56:57 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:56:57 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:56:57 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:56:57 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:56:57 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:56:57 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:56:57 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:56:57 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:56:57 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:56:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:56:57 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:56:57 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:56:57 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:56:57 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:56:57 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:56:57 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:56:57 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:56:57 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:56:57 mr-fox kernel:  </TASK>
May 26 07:57:00 mr-fox crond[5612]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:57:00 mr-fox crond[5610]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:57:00 mr-fox crond[5611]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:57:00 mr-fox crond[5614]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:57:00 mr-fox crond[5615]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:57:00 mr-fox CROND[5619]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:57:00 mr-fox CROND[5618]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:57:00 mr-fox CROND[5617]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:57:00 mr-fox CROND[5621]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:57:00 mr-fox CROND[5622]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:57:00 mr-fox CROND[5615]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:57:00 mr-fox CROND[5615]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:57:00 mr-fox CROND[5614]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:57:00 mr-fox CROND[5614]: pam_unix(crond:session): session closed for user root
May 26 07:57:01 mr-fox CROND[5612]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:57:01 mr-fox CROND[5612]: pam_unix(crond:session): session closed for user root
May 26 07:57:01 mr-fox CROND[5062]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:57:01 mr-fox CROND[5062]: pam_unix(crond:session): session closed for user root
May 26 07:58:00 mr-fox crond[7582]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:58:00 mr-fox crond[7584]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:58:00 mr-fox crond[7583]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:58:00 mr-fox crond[7585]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:58:00 mr-fox crond[7586]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:58:00 mr-fox CROND[7591]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:58:00 mr-fox CROND[7590]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:58:00 mr-fox CROND[7592]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:58:00 mr-fox CROND[7593]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:58:00 mr-fox CROND[7594]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:58:00 mr-fox CROND[7586]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:58:00 mr-fox CROND[7586]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:58:00 mr-fox CROND[7585]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:58:00 mr-fox CROND[7585]: pam_unix(crond:session): session closed for user root
May 26 07:58:00 mr-fox CROND[7584]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:58:00 mr-fox CROND[7584]: pam_unix(crond:session): session closed for user root
May 26 07:58:01 mr-fox CROND[5611]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:58:01 mr-fox CROND[5611]: pam_unix(crond:session): session closed for user root
May 26 07:59:00 mr-fox crond[9494]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:59:00 mr-fox crond[9496]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:59:00 mr-fox crond[9495]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:59:00 mr-fox CROND[9501]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 07:59:00 mr-fox CROND[9502]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 07:59:00 mr-fox crond[9498]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 07:59:00 mr-fox crond[9497]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 07:59:00 mr-fox CROND[9506]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:59:00 mr-fox CROND[9507]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 07:59:00 mr-fox CROND[9508]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:59:00 mr-fox CROND[9498]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 07:59:00 mr-fox CROND[9498]: pam_unix(crond:session): session closed for user tinderbox
May 26 07:59:00 mr-fox CROND[9497]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 07:59:00 mr-fox CROND[9497]: pam_unix(crond:session): session closed for user root
May 26 07:59:00 mr-fox CROND[9496]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 07:59:00 mr-fox CROND[9496]: pam_unix(crond:session): session closed for user root
May 26 07:59:00 mr-fox CROND[7583]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 07:59:00 mr-fox CROND[7583]: pam_unix(crond:session): session closed for user root
May 26 07:59:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 07:59:45 mr-fox kernel: rcu: \x0921-....: (1770151 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=710169
May 26 07:59:45 mr-fox kernel: rcu: \x09(t=1770152 jiffies g=8794409 q=50199720 ncpus=32)
May 26 07:59:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:59:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:59:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:59:45 mr-fox kernel: RIP: 0010:xas_descend+0x31/0xd0
May 26 07:59:45 mr-fox kernel: Code: 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 <49> 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d fd 00 00
May 26 07:59:45 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000202
May 26 07:59:45 mr-fox kernel: RAX: ffff988d55f18ffa RBX: 0000000000000012 RCX: 0000000000000006
May 26 07:59:45 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988d667606d8 RDI: ffffa401077e3970
May 26 07:59:45 mr-fox kernel: RBP: ffff988d667606d8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:59:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 07:59:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:59:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:59:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:59:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:59:45 mr-fox kernel: PKRU: 55555554
May 26 07:59:45 mr-fox kernel: Call Trace:
May 26 07:59:45 mr-fox kernel:  <IRQ>
May 26 07:59:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 07:59:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 07:59:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 07:59:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 07:59:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 07:59:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 07:59:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:59:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 07:59:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 07:59:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 07:59:45 mr-fox kernel:  </IRQ>
May 26 07:59:45 mr-fox kernel:  <TASK>
May 26 07:59:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:59:45 mr-fox kernel:  ? xas_descend+0x31/0xd0
May 26 07:59:45 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:59:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:59:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:59:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:59:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:59:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:59:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:59:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:59:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:59:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:59:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:59:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:59:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:59:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:59:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:59:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:59:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:59:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:59:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:59:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:59:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:59:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:59:45 mr-fox kernel:  </TASK>
May 26 07:59:57 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1773168 jiffies s: 491905 root: 0x2/.
May 26 07:59:57 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 07:59:57 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 07:59:57 mr-fox kernel: NMI backtrace for cpu 21
May 26 07:59:57 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 07:59:57 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 07:59:57 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 07:59:57 mr-fox kernel: RIP: 0010:napi_consume_skb+0x1b/0xc0
May 26 07:59:57 mr-fox kernel: Code: 20 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 53 48 89 fb 85 f6 0f 84 8c 00 00 00 48 85 ff 74 7b 8b 87 d4 00 00 00 <83> f8 01 75 53 48 89 df f6 43 7e 0c 75 25 e8 62 87 ff ff 48 83 bb
May 26 07:59:57 mr-fox kernel: RSP: 0018:ffffa401005f0de8 EFLAGS: 00000286
May 26 07:59:57 mr-fox kernel: RAX: 0000000000000001 RBX: ffff989969201418 RCX: 0000000000000000
May 26 07:59:57 mr-fox kernel: RDX: 0000000000000001 RSI: 0000000000000040 RDI: ffff989969201418
May 26 07:59:57 mr-fox kernel: RBP: 0000000000004ec0 R08: 0000000000000000 R09: 0000000000000000
May 26 07:59:57 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffa40100119740
May 26 07:59:57 mr-fox kernel: R13: ffff988ac873fa40 R14: 00000000ffffff74 R15: ffffa40100119760
May 26 07:59:57 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 07:59:57 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 07:59:57 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 07:59:57 mr-fox kernel: PKRU: 55555554
May 26 07:59:57 mr-fox kernel: Call Trace:
May 26 07:59:57 mr-fox kernel:  <NMI>
May 26 07:59:57 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 07:59:57 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 07:59:57 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 07:59:57 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 07:59:57 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 07:59:57 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 07:59:57 mr-fox kernel:  ? napi_consume_skb+0x1b/0xc0
May 26 07:59:57 mr-fox kernel:  ? napi_consume_skb+0x1b/0xc0
May 26 07:59:57 mr-fox kernel:  ? napi_consume_skb+0x1b/0xc0
May 26 07:59:57 mr-fox kernel:  </NMI>
May 26 07:59:57 mr-fox kernel:  <IRQ>
May 26 07:59:57 mr-fox kernel:  igb_poll+0xea/0x1370
May 26 07:59:57 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 07:59:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:59:57 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 07:59:57 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 07:59:57 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 07:59:57 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 07:59:57 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 07:59:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:59:57 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 07:59:57 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 07:59:57 mr-fox kernel:  </IRQ>
May 26 07:59:57 mr-fox kernel:  <TASK>
May 26 07:59:57 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 07:59:57 mr-fox kernel: RIP: 0010:xas_descend+0x31/0xd0
May 26 07:59:57 mr-fox kernel: Code: 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 <49> 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d fd 00 00
May 26 07:59:57 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000212
May 26 07:59:57 mr-fox kernel: RAX: ffff988acf90e6ca RBX: 000000000000000c RCX: 0000000000000018
May 26 07:59:57 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac07566c8 RDI: ffffa401077e3970
May 26 07:59:57 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 07:59:57 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 07:59:57 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 07:59:57 mr-fox kernel:  xas_load+0x49/0x60
May 26 07:59:57 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 07:59:57 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 07:59:57 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 07:59:57 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 07:59:57 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 07:59:57 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 07:59:57 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 07:59:57 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 07:59:57 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 07:59:57 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 07:59:57 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 07:59:57 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 07:59:57 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 07:59:57 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 07:59:57 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 07:59:57 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 07:59:57 mr-fox kernel:  kthread+0xcb/0xf0
May 26 07:59:57 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:59:57 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 07:59:57 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 07:59:57 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 07:59:57 mr-fox kernel:  </TASK>
May 26 08:00:00 mr-fox crond[12148]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:00:00 mr-fox crond[12150]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:00:00 mr-fox crond[12149]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:00:00 mr-fox crond[12147]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 08:00:00 mr-fox crond[12153]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:00:00 mr-fox CROND[12161]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:00:00 mr-fox crond[12154]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:00:00 mr-fox CROND[12162]: (root) CMD (/etc/conf.d/ipv4-rules.sh update; /etc/conf.d/ipv6-rules.sh update)
May 26 08:00:00 mr-fox crond[12151]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:00:00 mr-fox CROND[12164]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:00:00 mr-fox CROND[12165]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:00:00 mr-fox crond[12155]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:00:00 mr-fox CROND[12163]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:00:00 mr-fox CROND[12167]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:00:00 mr-fox CROND[12168]: (tinderbox) CMD (sudo /opt/tb/bin/collect_data.sh 2>/dev/null)
May 26 08:00:00 mr-fox crond[12156]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:00:00 mr-fox crond[12159]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:00:00 mr-fox CROND[12169]: (root) CMD (/opt/torutils/update_tor.sh)
May 26 08:00:00 mr-fox crond[12160]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:00:00 mr-fox crond[12157]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:00:00 mr-fox CROND[12172]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:00:00 mr-fox CROND[12173]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:00:00 mr-fox CROND[12174]: (tinderbox) CMD (sudo /opt/tb/bin/house_keeping.sh >/dev/null)
May 26 08:00:00 mr-fox CROND[12170]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 08:00:00 mr-fox sudo[12179]: tinderbox : PWD=/home/tinderbox ; USER=root ; COMMAND=/opt/tb/bin/collect_data.sh
May 26 08:00:00 mr-fox sudo[12179]: pam_unix(sudo:session): session opened for user root(uid=0) by tinderbox(uid=1003)
May 26 08:00:00 mr-fox CROND[12147]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:00:00 mr-fox CROND[12147]: pam_unix(crond:session): session closed for user torproject
May 26 08:00:00 mr-fox sudo[12181]: tinderbox : PWD=/home/tinderbox ; USER=root ; COMMAND=/opt/tb/bin/house_keeping.sh
May 26 08:00:00 mr-fox sudo[12181]: pam_unix(sudo:session): session opened for user root(uid=0) by tinderbox(uid=1003)
May 26 08:00:00 mr-fox CROND[12160]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:00:00 mr-fox CROND[12160]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:00:00 mr-fox CROND[12154]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:00:00 mr-fox CROND[12154]: pam_unix(crond:session): session closed for user root
May 26 08:00:00 mr-fox CROND[12156]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:00:00 mr-fox CROND[12156]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:00:00 mr-fox CROND[9495]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:00:00 mr-fox CROND[9495]: pam_unix(crond:session): session closed for user root
May 26 08:00:00 mr-fox CROND[12153]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:00:00 mr-fox CROND[12153]: pam_unix(crond:session): session closed for user root
May 26 08:00:04 mr-fox CROND[12151]: (root) CMDEND (/opt/torutils/update_tor.sh)
May 26 08:00:04 mr-fox CROND[12151]: pam_unix(crond:session): session closed for user root
May 26 08:00:08 mr-fox CROND[12150]: (root) CMDEND (/etc/conf.d/ipv4-rules.sh update; /etc/conf.d/ipv6-rules.sh update)
May 26 08:00:08 mr-fox CROND[12150]: pam_unix(crond:session): session closed for user root
May 26 08:01:00 mr-fox CROND[13072]: (root) CMD (run-parts /etc/cron.hourly)
May 26 08:01:00 mr-fox crond[13071]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:01:00 mr-fox crond[13069]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:01:00 mr-fox crond[13068]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:01:00 mr-fox CROND[13080]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:01:00 mr-fox CROND[13079]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:01:00 mr-fox crond[13075]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:01:00 mr-fox CROND[13081]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:01:00 mr-fox CROND[13082]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:01:00 mr-fox crond[13073]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:01:00 mr-fox CROND[13085]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:01:00 mr-fox CROND[13067]: (root) CMDEND (run-parts /etc/cron.hourly)
May 26 08:01:00 mr-fox CROND[13075]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:01:00 mr-fox CROND[13075]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:01:00 mr-fox CROND[13073]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:01:00 mr-fox CROND[13073]: pam_unix(crond:session): session closed for user root
May 26 08:01:00 mr-fox CROND[13071]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:01:00 mr-fox CROND[13071]: pam_unix(crond:session): session closed for user root
May 26 08:01:01 mr-fox CROND[12149]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:01:01 mr-fox CROND[12149]: pam_unix(crond:session): session closed for user root
May 26 08:02:00 mr-fox crond[16112]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:02:00 mr-fox crond[16113]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:02:00 mr-fox crond[16114]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:02:00 mr-fox crond[16116]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:02:00 mr-fox crond[16115]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:02:00 mr-fox CROND[16120]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:02:00 mr-fox CROND[16121]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:02:00 mr-fox CROND[16125]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:02:00 mr-fox CROND[16124]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:02:00 mr-fox CROND[16126]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:02:00 mr-fox CROND[16116]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:02:00 mr-fox CROND[16116]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:02:00 mr-fox CROND[16115]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:02:00 mr-fox CROND[16115]: pam_unix(crond:session): session closed for user root
May 26 08:02:00 mr-fox CROND[16114]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:02:00 mr-fox CROND[16114]: pam_unix(crond:session): session closed for user root
May 26 08:02:00 mr-fox CROND[13069]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:02:00 mr-fox CROND[13069]: pam_unix(crond:session): session closed for user root
May 26 08:02:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:02:45 mr-fox kernel: rcu: \x0921-....: (1815155 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=729107
May 26 08:02:45 mr-fox kernel: rcu: \x09(t=1815156 jiffies g=8794409 q=51244979 ncpus=32)
May 26 08:02:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:02:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:02:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:02:45 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:02:45 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:02:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:02:45 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:02:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:02:45 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:02:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:02:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:02:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:02:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:02:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:02:45 mr-fox kernel: PKRU: 55555554
May 26 08:02:45 mr-fox kernel: Call Trace:
May 26 08:02:45 mr-fox kernel:  <IRQ>
May 26 08:02:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:02:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:02:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:02:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:02:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:02:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:02:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:02:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:02:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:02:45 mr-fox kernel:  </IRQ>
May 26 08:02:45 mr-fox kernel:  <TASK>
May 26 08:02:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:02:45 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 08:02:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:02:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:02:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:02:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:02:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:02:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:02:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:02:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:02:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:02:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:02:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:02:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:02:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:02:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:02:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:02:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:02:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:02:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:02:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:02:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:02:45 mr-fox kernel:  </TASK>
May 26 08:02:58 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1818223 jiffies s: 491905 root: 0x2/.
May 26 08:02:58 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:02:58 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:02:58 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:02:58 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:02:58 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:02:58 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:02:58 mr-fox kernel: RIP: 0010:__ip_dev_find+0x57/0x100
May 26 08:02:58 mr-fox kernel: Code: 47 86 c8 61 c1 e8 18 48 8b 14 c5 20 a8 ad a6 48 85 d2 75 0a eb 58 48 8b 12 48 85 d2 74 50 3b 72 30 75 f3 48 8b 42 18 48 8b 00 <4c> 3b 80 08 01 00 00 75 e3 48 85 c0 74 0e 84 db 74 0a 48 8b 90 50
May 26 08:02:58 mr-fox kernel: RSP: 0018:ffffa401005f0c30 EFLAGS: 00000246
May 26 08:02:58 mr-fox kernel: RAX: ffff988ac4bb4000 RBX: 0000000000000000 RCX: 0000000000000000
May 26 08:02:58 mr-fox kernel: RDX: ffff988ac4b50580 RSI: 000000000d5e1541 RDI: ffffffffa6ad7f40
May 26 08:02:58 mr-fox kernel: RBP: ffffffffa6ad7f40 R08: ffffffffa6ad7f40 R09: 000000000000fa80
May 26 08:02:58 mr-fox kernel: R10: 0000000000000001 R11: 0000000000000000 R12: ffffa401005f0d10
May 26 08:02:58 mr-fox kernel: R13: 00000000e324dcbd R14: 0000000000000000 R15: 0000000000000000
May 26 08:02:58 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:02:58 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:02:58 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:02:58 mr-fox kernel: PKRU: 55555554
May 26 08:02:58 mr-fox kernel: Call Trace:
May 26 08:02:58 mr-fox kernel:  <NMI>
May 26 08:02:58 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:02:58 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:02:58 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:02:58 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:02:58 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:02:58 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:02:58 mr-fox kernel:  ? __ip_dev_find+0x57/0x100
May 26 08:02:58 mr-fox kernel:  ? __ip_dev_find+0x57/0x100
May 26 08:02:58 mr-fox kernel:  ? __ip_dev_find+0x57/0x100
May 26 08:02:58 mr-fox kernel:  </NMI>
May 26 08:02:58 mr-fox kernel:  <IRQ>
May 26 08:02:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:58 mr-fox kernel:  ? __sk_dst_check+0x34/0xa0
May 26 08:02:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:58 mr-fox kernel:  ? inet6_csk_route_socket+0x132/0x210
May 26 08:02:58 mr-fox kernel:  ip_route_output_key_hash_rcu+0x4e2/0x780
May 26 08:02:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:58 mr-fox kernel:  ? inet6_csk_xmit+0xe9/0x160
May 26 08:02:58 mr-fox kernel:  ip_route_output_flow+0x55/0xa0
May 26 08:02:58 mr-fox kernel:  inet_sk_rebuild_header+0x185/0x410
May 26 08:02:58 mr-fox kernel:  __tcp_retransmit_skb+0x9e/0x800
May 26 08:02:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:58 mr-fox kernel:  ? lock_timer_base+0x2f/0xc0
May 26 08:02:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:58 mr-fox kernel:  ? retransmits_timed_out.part.0+0x8d/0x170
May 26 08:02:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:58 mr-fox kernel:  tcp_retransmit_skb+0x11/0xa0
May 26 08:02:58 mr-fox kernel:  tcp_retransmit_timer+0x492/0xa60
May 26 08:02:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:58 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 08:02:58 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 08:02:58 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 08:02:58 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 08:02:58 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 08:02:58 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:02:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:58 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:02:58 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:02:58 mr-fox kernel:  </IRQ>
May 26 08:02:58 mr-fox kernel:  <TASK>
May 26 08:02:58 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:02:58 mr-fox kernel: RIP: 0010:filemap_get_entry+0x89/0x160
May 26 08:02:58 mr-fox kernel: Code: 00 48 8d 7c 24 08 e8 56 70 78 00 48 89 c3 48 3d 02 04 00 00 74 e2 48 3d 06 04 00 00 74 da 48 85 c0 74 5a a8 01 75 56 8b 40 34 <85> c0 74 ca 8d 50 01 f0 0f b1 53 34 75 f2 4c 8b 64 24 20 4d 85 e4
May 26 08:02:58 mr-fox kernel: RSP: 0018:ffffa401077e3968 EFLAGS: 00000246
May 26 08:02:58 mr-fox kernel: RAX: 0000000000000000 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 08:02:58 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:02:58 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 08:02:58 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:02:58 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:02:58 mr-fox kernel:  ? filemap_get_entry+0x6a/0x160
May 26 08:02:58 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:02:58 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:02:58 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:02:58 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:02:58 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:02:58 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:02:58 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:02:58 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:02:58 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:02:58 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:02:58 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:02:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:02:58 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:02:58 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:02:58 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:02:58 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:02:58 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:02:58 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:02:58 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:02:58 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:02:58 mr-fox kernel:  </TASK>
May 26 08:03:00 mr-fox crond[18365]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:03:00 mr-fox crond[18367]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:03:00 mr-fox crond[18370]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:03:00 mr-fox crond[18366]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:03:00 mr-fox crond[18372]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:03:00 mr-fox CROND[18375]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:03:00 mr-fox CROND[18374]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:03:00 mr-fox CROND[18376]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:03:00 mr-fox CROND[18377]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:03:00 mr-fox CROND[18378]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:03:00 mr-fox CROND[18372]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:03:00 mr-fox CROND[18372]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:03:00 mr-fox CROND[18370]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:03:00 mr-fox CROND[18370]: pam_unix(crond:session): session closed for user root
May 26 08:03:00 mr-fox CROND[18367]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:03:00 mr-fox CROND[18367]: pam_unix(crond:session): session closed for user root
May 26 08:03:00 mr-fox CROND[16113]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:03:00 mr-fox CROND[16113]: pam_unix(crond:session): session closed for user root
May 26 08:04:00 mr-fox crond[22180]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:04:00 mr-fox crond[22182]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:04:00 mr-fox crond[22183]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:04:00 mr-fox crond[22181]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:04:00 mr-fox crond[22184]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:04:00 mr-fox CROND[22188]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:04:00 mr-fox CROND[22189]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:04:00 mr-fox CROND[22190]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:04:00 mr-fox CROND[22191]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:04:00 mr-fox CROND[22192]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:04:01 mr-fox CROND[22184]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:04:01 mr-fox CROND[22184]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:04:01 mr-fox CROND[22183]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:04:01 mr-fox CROND[22183]: pam_unix(crond:session): session closed for user root
May 26 08:04:01 mr-fox CROND[22182]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:04:01 mr-fox CROND[22182]: pam_unix(crond:session): session closed for user root
May 26 08:04:01 mr-fox CROND[18366]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:04:01 mr-fox CROND[18366]: pam_unix(crond:session): session closed for user root
May 26 08:05:00 mr-fox crond[23607]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:05:00 mr-fox crond[23606]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 08:05:00 mr-fox crond[23608]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:05:00 mr-fox crond[23609]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:05:00 mr-fox crond[23610]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:05:00 mr-fox crond[23612]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:05:00 mr-fox CROND[23619]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:05:00 mr-fox CROND[23618]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:05:00 mr-fox crond[23613]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:05:00 mr-fox CROND[23617]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:05:00 mr-fox crond[23614]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:05:00 mr-fox CROND[23620]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:05:00 mr-fox CROND[23622]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:05:00 mr-fox CROND[23624]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 08:05:00 mr-fox CROND[23623]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:05:00 mr-fox CROND[23625]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:05:00 mr-fox CROND[23606]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:05:00 mr-fox CROND[23606]: pam_unix(crond:session): session closed for user torproject
May 26 08:05:00 mr-fox CROND[23614]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:05:00 mr-fox CROND[23614]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:05:00 mr-fox CROND[23610]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:05:00 mr-fox CROND[23610]: pam_unix(crond:session): session closed for user root
May 26 08:05:00 mr-fox CROND[23612]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:05:00 mr-fox CROND[23612]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:05:00 mr-fox CROND[23609]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:05:00 mr-fox CROND[23609]: pam_unix(crond:session): session closed for user root
May 26 08:05:01 mr-fox CROND[22181]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:05:01 mr-fox CROND[22181]: pam_unix(crond:session): session closed for user root
May 26 08:05:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:05:45 mr-fox kernel: rcu: \x0921-....: (1860159 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=748064
May 26 08:05:45 mr-fox kernel: rcu: \x09(t=1860160 jiffies g=8794409 q=52270626 ncpus=32)
May 26 08:05:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:05:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:05:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:05:45 mr-fox kernel: RIP: 0010:xas_descend+0x40/0xd0
May 26 08:05:45 mr-fox kernel: Code: 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 <75> 08 48 3d fd 00 00 00 76 2f 41 88 5c 24 12 48 83 c4 08 5b 5d 41
May 26 08:05:45 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000293
May 26 08:05:45 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: 0000000000000021 RCX: 0000000000000000
May 26 08:05:45 mr-fox kernel: RDX: 0000000000000000 RSI: ffff988d55f18ff8 RDI: ffffa401077e3970
May 26 08:05:45 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:05:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 08:05:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:05:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:05:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:05:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:05:45 mr-fox kernel: PKRU: 55555554
May 26 08:05:45 mr-fox kernel: Call Trace:
May 26 08:05:45 mr-fox kernel:  <IRQ>
May 26 08:05:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:05:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:05:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:05:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:05:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:05:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:05:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:05:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:05:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:05:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:05:45 mr-fox kernel:  </IRQ>
May 26 08:05:45 mr-fox kernel:  <TASK>
May 26 08:05:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:05:45 mr-fox kernel:  ? xas_descend+0x40/0xd0
May 26 08:05:45 mr-fox kernel:  xas_load+0x49/0x60
May 26 08:05:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:05:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:05:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:05:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:05:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:05:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:05:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:05:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:05:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:05:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:05:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:05:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:05:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:05:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:05:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:05:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:05:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:05:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:05:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:05:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:05:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:05:45 mr-fox kernel:  </TASK>
May 26 08:05:58 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1863280 jiffies s: 491905 root: 0x2/.
May 26 08:05:58 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:05:58 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:05:58 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:05:58 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:05:58 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:05:58 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:05:58 mr-fox kernel: RIP: 0010:fetch_pte+0x78/0x170
May 26 08:05:58 mr-fox kernel: Code: 00 01 00 00 83 eb 01 44 8d 34 db 41 8d 4e 0c 83 f9 3f 0f 87 0e f9 2d 00 4c 89 e0 49 8b 95 08 01 00 00 48 d3 e8 25 ff 01 00 00 <48> 8d 04 c2 ba 01 00 00 00 48 d3 e2 48 89 55 00 85 db 0f 8e b2 00
May 26 08:05:58 mr-fox kernel: RSP: 0018:ffffa401005f0c38 EFLAGS: 00000206
May 26 08:05:58 mr-fox kernel: RAX: 0000000000000003 RBX: 0000000000000002 RCX: 000000000000001e
May 26 08:05:58 mr-fox kernel: RDX: ffff988ac41fa000 RSI: 00000000cfefd000 RDI: ffff988ac2211088
May 26 08:05:58 mr-fox kernel: RBP: ffffa401005f0c80 R08: 0000000000000000 R09: 0000000000000000
May 26 08:05:58 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000cfefd000
May 26 08:05:58 mr-fox kernel: R13: ffff988ac2211088 R14: 0000000000000012 R15: ffff988ac2211088
May 26 08:05:58 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:05:58 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:05:58 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:05:58 mr-fox kernel: PKRU: 55555554
May 26 08:05:58 mr-fox kernel: Call Trace:
May 26 08:05:58 mr-fox kernel:  <NMI>
May 26 08:05:58 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:05:58 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:05:58 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:05:58 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:05:58 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:05:58 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:05:58 mr-fox kernel:  ? fetch_pte+0x78/0x170
May 26 08:05:58 mr-fox kernel:  ? fetch_pte+0x78/0x170
May 26 08:05:58 mr-fox kernel:  ? fetch_pte+0x78/0x170
May 26 08:05:58 mr-fox kernel:  </NMI>
May 26 08:05:58 mr-fox kernel:  <IRQ>
May 26 08:05:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:05:58 mr-fox kernel:  iommu_v1_unmap_pages+0x81/0x140
May 26 08:05:58 mr-fox kernel:  amd_iommu_unmap_pages+0x40/0x130
May 26 08:05:58 mr-fox kernel:  __iommu_unmap+0xbf/0x120
May 26 08:05:58 mr-fox kernel:  __iommu_dma_unmap+0xb5/0x170
May 26 08:05:58 mr-fox kernel:  iommu_dma_unmap_page+0x43/0xa0
May 26 08:05:58 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 08:05:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:05:58 mr-fox kernel:  ? tcp_delack_timer_handler+0xa9/0x110
May 26 08:05:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:05:58 mr-fox kernel:  ? tcp_delack_timer+0xb5/0xf0
May 26 08:05:58 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 08:05:58 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 08:05:58 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:05:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:05:58 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:05:58 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:05:58 mr-fox kernel:  </IRQ>
May 26 08:05:58 mr-fox kernel:  <TASK>
May 26 08:05:58 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:05:58 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:05:58 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:05:58 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:05:58 mr-fox kernel: RAX: ffff988d667606da RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:05:58 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:05:58 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 08:05:58 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:05:58 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:05:58 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:05:58 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:05:58 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:05:58 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:05:58 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:05:58 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:05:58 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:05:58 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:05:58 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:05:58 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:05:58 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:05:58 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:05:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:05:58 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:05:58 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:05:58 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:05:58 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:05:58 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:05:58 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:05:58 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:05:58 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:05:58 mr-fox kernel:  </TASK>
May 26 08:06:00 mr-fox crond[29160]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:06:00 mr-fox crond[29162]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:06:00 mr-fox crond[29161]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:06:00 mr-fox crond[29164]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:06:00 mr-fox CROND[29167]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:06:00 mr-fox crond[29165]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:06:00 mr-fox CROND[29168]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:06:00 mr-fox CROND[29170]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:06:00 mr-fox CROND[29171]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:06:00 mr-fox CROND[29172]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:06:00 mr-fox CROND[29165]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:06:00 mr-fox CROND[29165]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:06:00 mr-fox CROND[29164]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:06:00 mr-fox CROND[29164]: pam_unix(crond:session): session closed for user root
May 26 08:06:00 mr-fox CROND[29162]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:06:00 mr-fox CROND[29162]: pam_unix(crond:session): session closed for user root
May 26 08:06:00 mr-fox CROND[23608]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:06:00 mr-fox CROND[23608]: pam_unix(crond:session): session closed for user root
May 26 08:07:00 mr-fox crond[498]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:07:00 mr-fox crond[499]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:07:00 mr-fox crond[501]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:07:00 mr-fox crond[500]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:07:00 mr-fox crond[502]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:07:00 mr-fox CROND[505]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:07:00 mr-fox CROND[506]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:07:00 mr-fox CROND[507]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:07:00 mr-fox CROND[509]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:07:00 mr-fox CROND[510]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:07:00 mr-fox CROND[502]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:07:00 mr-fox CROND[502]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:07:00 mr-fox CROND[501]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:07:00 mr-fox CROND[501]: pam_unix(crond:session): session closed for user root
May 26 08:07:00 mr-fox CROND[500]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:07:00 mr-fox CROND[500]: pam_unix(crond:session): session closed for user root
May 26 08:07:01 mr-fox CROND[29161]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:07:01 mr-fox CROND[29161]: pam_unix(crond:session): session closed for user root
May 26 08:08:00 mr-fox crond[3977]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:08:00 mr-fox crond[3976]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:08:00 mr-fox crond[3979]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:08:00 mr-fox crond[3978]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:08:00 mr-fox crond[3980]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:08:00 mr-fox CROND[3985]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:08:00 mr-fox CROND[3986]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:08:00 mr-fox CROND[3987]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:08:00 mr-fox CROND[3988]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:08:00 mr-fox CROND[3984]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:08:00 mr-fox CROND[3980]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:08:00 mr-fox CROND[3980]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:08:00 mr-fox CROND[3979]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:08:00 mr-fox CROND[3979]: pam_unix(crond:session): session closed for user root
May 26 08:08:00 mr-fox CROND[3978]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:08:00 mr-fox CROND[3978]: pam_unix(crond:session): session closed for user root
May 26 08:08:01 mr-fox CROND[499]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:08:01 mr-fox CROND[499]: pam_unix(crond:session): session closed for user root
May 26 08:08:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:08:45 mr-fox kernel: rcu: \x0921-....: (1905163 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=767107
May 26 08:08:45 mr-fox kernel: rcu: \x09(t=1905164 jiffies g=8794409 q=53285729 ncpus=32)
May 26 08:08:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:08:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:08:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:08:45 mr-fox kernel: RIP: 0010:xas_descend+0x26/0xd0
May 26 08:08:45 mr-fox kernel: Code: 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f <89> d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03
May 26 08:08:45 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 08:08:45 mr-fox kernel: RAX: ffff988acf90e6ca RBX: 0000000000000036 RCX: 0000000000000012
May 26 08:08:45 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988acf90e6c8 RDI: ffffa401077e3970
May 26 08:08:45 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:08:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 08:08:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:08:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:08:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:08:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:08:45 mr-fox kernel: PKRU: 55555554
May 26 08:08:45 mr-fox kernel: Call Trace:
May 26 08:08:45 mr-fox kernel:  <IRQ>
May 26 08:08:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:08:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:08:45 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 08:08:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:08:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:08:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:08:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:08:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:08:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:08:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:08:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:08:45 mr-fox kernel:  </IRQ>
May 26 08:08:45 mr-fox kernel:  <TASK>
May 26 08:08:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:08:45 mr-fox kernel:  ? xas_descend+0x26/0xd0
May 26 08:08:45 mr-fox kernel:  xas_load+0x49/0x60
May 26 08:08:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:08:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:08:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:08:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:08:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:08:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:08:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:08:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:08:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:08:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:08:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:08:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:08:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:08:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:08:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:08:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:08:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:08:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:08:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:08:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:08:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:08:45 mr-fox kernel:  </TASK>
May 26 08:08:58 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1908336 jiffies s: 491905 root: 0x2/.
May 26 08:08:58 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:08:58 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:08:58 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:08:58 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:08:58 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:08:58 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:08:58 mr-fox kernel: RIP: 0010:ip6_xmit+0x27e/0x600
May 26 08:08:58 mr-fox kernel: Code: 0f 85 e0 02 00 00 e8 21 1d 01 00 3b 43 70 0f 82 e7 00 00 00 48 8b 44 24 20 48 85 c0 74 0c 48 8b 80 98 03 00 00 65 48 ff 40 28 <48> 8b 4c 24 28 48 8b 81 a0 01 00 00 65 48 ff 40 28 48 85 db 0f 84
May 26 08:08:58 mr-fox kernel: RSP: 0018:ffffa401005f0c30 EFLAGS: 00000202
May 26 08:08:58 mr-fox kernel: RAX: 00002b5790e21310 RBX: ffff9891f26e6700 RCX: 0000000000000040
May 26 08:08:58 mr-fox kernel: RDX: 0000000000000000 RSI: 4f244001f804012a RDI: 0000000000000000
May 26 08:08:58 mr-fox kernel: RBP: ffffa401005f0d20 R08: 0000000000000040 R09: 0000000000000000
May 26 08:08:58 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000020 R12: 0000000000002000
May 26 08:08:58 mr-fox kernel: R13: ffff988d9e786e00 R14: ffffa401005f0d38 R15: ffff988ae91ac300
May 26 08:08:58 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:08:58 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:08:58 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:08:58 mr-fox kernel: PKRU: 55555554
May 26 08:08:58 mr-fox kernel: Call Trace:
May 26 08:08:58 mr-fox kernel:  <NMI>
May 26 08:08:58 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:08:58 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:08:58 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:08:58 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:08:58 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:08:58 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:08:58 mr-fox kernel:  ? ip6_xmit+0x27e/0x600
May 26 08:08:58 mr-fox kernel:  ? ip6_xmit+0x27e/0x600
May 26 08:08:58 mr-fox kernel:  ? ip6_xmit+0x27e/0x600
May 26 08:08:58 mr-fox kernel:  </NMI>
May 26 08:08:58 mr-fox kernel:  <IRQ>
May 26 08:08:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:08:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:08:58 mr-fox kernel:  ? sch_direct_xmit+0x8d/0x290
May 26 08:08:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:08:58 mr-fox kernel:  ? __sk_dst_check+0x34/0xa0
May 26 08:08:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:08:58 mr-fox kernel:  ? inet6_csk_route_socket+0x132/0x210
May 26 08:08:58 mr-fox kernel:  inet6_csk_xmit+0xe9/0x160
May 26 08:08:58 mr-fox kernel:  __tcp_transmit_skb+0x5d0/0xd30
May 26 08:08:58 mr-fox kernel:  tcp_delack_timer_handler+0xa9/0x110
May 26 08:08:58 mr-fox kernel:  tcp_delack_timer+0xb5/0xf0
May 26 08:08:58 mr-fox kernel:  ? tcp_delack_timer_handler+0x110/0x110
May 26 08:08:58 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 08:08:58 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 08:08:58 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 08:08:58 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:08:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:08:58 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:08:58 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:08:58 mr-fox kernel:  </IRQ>
May 26 08:08:58 mr-fox kernel:  <TASK>
May 26 08:08:58 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:08:58 mr-fox kernel: RIP: 0010:xas_load+0x35/0x60
May 26 08:08:58 mr-fox kernel: Code: f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 <48> 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d
May 26 08:08:58 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000286
May 26 08:08:58 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:08:58 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:08:58 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 08:08:58 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:08:58 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:08:58 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:08:58 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:08:58 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:08:58 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:08:58 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:08:58 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:08:58 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:08:58 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:08:58 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:08:58 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:08:58 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:08:58 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:08:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:08:58 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:08:58 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:08:58 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:08:58 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:08:58 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:08:58 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:08:58 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:08:58 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:08:58 mr-fox kernel:  </TASK>
May 26 08:09:00 mr-fox crond[7565]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:09:00 mr-fox crond[7566]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:09:00 mr-fox crond[7568]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:09:00 mr-fox crond[7570]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:09:00 mr-fox crond[7569]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:09:00 mr-fox CROND[7572]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:09:00 mr-fox CROND[7574]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:09:00 mr-fox CROND[7573]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:09:00 mr-fox CROND[7576]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:09:00 mr-fox CROND[7577]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:09:00 mr-fox CROND[7570]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:09:00 mr-fox CROND[7570]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:09:00 mr-fox CROND[7569]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:09:00 mr-fox CROND[7569]: pam_unix(crond:session): session closed for user root
May 26 08:09:00 mr-fox CROND[7568]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:09:00 mr-fox CROND[7568]: pam_unix(crond:session): session closed for user root
May 26 08:09:00 mr-fox CROND[3977]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:09:00 mr-fox CROND[3977]: pam_unix(crond:session): session closed for user root
May 26 08:10:00 mr-fox crond[12547]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:10:00 mr-fox crond[12548]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:10:00 mr-fox crond[12544]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 08:10:00 mr-fox crond[12545]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:10:00 mr-fox crond[12550]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:10:00 mr-fox crond[12552]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:10:00 mr-fox crond[12553]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:10:00 mr-fox CROND[12557]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:10:00 mr-fox CROND[12558]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:10:00 mr-fox CROND[12559]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:10:00 mr-fox CROND[12560]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:10:00 mr-fox CROND[12563]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:10:00 mr-fox CROND[12564]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 08:10:00 mr-fox crond[12554]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:10:00 mr-fox CROND[12562]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:10:00 mr-fox CROND[12566]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:10:00 mr-fox CROND[12544]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:10:00 mr-fox CROND[12544]: pam_unix(crond:session): session closed for user torproject
May 26 08:10:00 mr-fox CROND[12554]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:10:00 mr-fox CROND[12554]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:10:00 mr-fox CROND[12550]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:10:00 mr-fox CROND[12550]: pam_unix(crond:session): session closed for user root
May 26 08:10:00 mr-fox CROND[12552]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:10:00 mr-fox CROND[12552]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:10:00 mr-fox CROND[12548]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:10:00 mr-fox CROND[12548]: pam_unix(crond:session): session closed for user root
May 26 08:10:00 mr-fox CROND[7566]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:10:00 mr-fox CROND[7566]: pam_unix(crond:session): session closed for user root
May 26 08:11:00 mr-fox crond[10418]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:11:00 mr-fox crond[10419]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:11:00 mr-fox crond[10422]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:11:00 mr-fox crond[10421]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:11:00 mr-fox crond[10423]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:11:00 mr-fox CROND[10427]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:11:00 mr-fox CROND[10429]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:11:00 mr-fox CROND[10430]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:11:00 mr-fox CROND[10431]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:11:00 mr-fox CROND[10432]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:11:00 mr-fox CROND[10423]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:11:00 mr-fox CROND[10423]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:11:00 mr-fox CROND[10422]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:11:00 mr-fox CROND[10422]: pam_unix(crond:session): session closed for user root
May 26 08:11:01 mr-fox CROND[10421]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:11:01 mr-fox CROND[10421]: pam_unix(crond:session): session closed for user root
May 26 08:11:01 mr-fox CROND[12547]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:11:01 mr-fox CROND[12547]: pam_unix(crond:session): session closed for user root
May 26 08:11:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:11:45 mr-fox kernel: rcu: \x0921-....: (1950167 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=786099
May 26 08:11:45 mr-fox kernel: rcu: \x09(t=1950168 jiffies g=8794409 q=54324356 ncpus=32)
May 26 08:11:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:11:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:11:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:11:45 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:11:45 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:11:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:11:45 mr-fox kernel: RAX: ffff988d667606da RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:11:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:11:45 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 08:11:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:11:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:11:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:11:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:11:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:11:45 mr-fox kernel: PKRU: 55555554
May 26 08:11:45 mr-fox kernel: Call Trace:
May 26 08:11:45 mr-fox kernel:  <IRQ>
May 26 08:11:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:11:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:11:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:11:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:11:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:11:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:11:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:11:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:11:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:11:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:11:45 mr-fox kernel:  </IRQ>
May 26 08:11:45 mr-fox kernel:  <TASK>
May 26 08:11:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:11:45 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 08:11:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:11:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:11:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:11:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:11:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:11:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:11:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:11:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:11:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:11:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:11:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:11:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:11:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:11:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:11:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:11:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:11:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:11:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:11:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:11:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:11:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:11:45 mr-fox kernel:  </TASK>
May 26 08:11:58 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1953391 jiffies s: 491905 root: 0x2/.
May 26 08:11:58 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:11:58 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:11:58 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:11:58 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:11:58 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:11:58 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:11:58 mr-fox kernel: RIP: 0010:iommu_dma_unmap_page+0x30/0xa0
May 26 08:11:58 mr-fox kernel: Code: 4d 89 c7 41 56 41 89 ce 41 55 49 89 d5 41 54 49 89 f4 55 53 48 89 fb e8 3e ec ff ff 4c 89 e6 48 89 c7 e8 13 bb ff ff 48 85 c0 <74> 46 4c 89 ea 4c 89 e6 48 89 df 48 89 c5 e8 3d fe ff ff 48 8b 83
May 26 08:11:58 mr-fox kernel: RSP: 0018:ffffa401005f0dc0 EFLAGS: 00000202
May 26 08:11:58 mr-fox kernel: RAX: 000000140662993e RBX: ffff988ac1a460c0 RCX: 0000000000000000
May 26 08:11:58 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:11:58 mr-fox kernel: RBP: 0000000000008bbf R08: 0000000000000000 R09: 0000000000000000
May 26 08:11:58 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000fec7293e
May 26 08:11:58 mr-fox kernel: R13: 0000000000000042 R14: 0000000000000001 R15: 0000000000000000
May 26 08:11:58 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:11:58 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:11:58 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:11:58 mr-fox kernel: PKRU: 55555554
May 26 08:11:58 mr-fox kernel: Call Trace:
May 26 08:11:58 mr-fox kernel:  <NMI>
May 26 08:11:58 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:11:58 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:11:58 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:11:58 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:11:58 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:11:58 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:11:58 mr-fox kernel:  ? iommu_dma_unmap_page+0x30/0xa0
May 26 08:11:58 mr-fox kernel:  ? iommu_dma_unmap_page+0x30/0xa0
May 26 08:11:58 mr-fox kernel:  ? iommu_dma_unmap_page+0x30/0xa0
May 26 08:11:58 mr-fox kernel:  </NMI>
May 26 08:11:58 mr-fox kernel:  <IRQ>
May 26 08:11:58 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 08:11:58 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 08:11:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:11:58 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 08:11:58 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 08:11:58 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 08:11:58 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:11:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:11:58 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:11:58 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:11:58 mr-fox kernel:  </IRQ>
May 26 08:11:58 mr-fox kernel:  <TASK>
May 26 08:11:58 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:11:58 mr-fox kernel: RIP: 0010:xas_descend+0x26/0xd0
May 26 08:11:58 mr-fox kernel: Code: 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f <89> d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03
May 26 08:11:58 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000202
May 26 08:11:58 mr-fox kernel: RAX: ffff988ac0754ffa RBX: 0000000000000001 RCX: 000000000000001e
May 26 08:11:58 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac0754ff8 RDI: ffffa401077e3970
May 26 08:11:58 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:11:58 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 08:11:58 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:11:58 mr-fox kernel:  xas_load+0x49/0x60
May 26 08:11:58 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:11:58 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:11:58 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:11:58 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:11:58 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:11:58 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:11:58 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:11:58 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:11:58 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:11:58 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:11:58 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:11:58 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:11:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:11:58 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:11:58 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:11:58 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:11:58 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:11:58 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:11:58 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:11:58 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:11:58 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:11:58 mr-fox kernel:  </TASK>
May 26 08:12:00 mr-fox crond[11454]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:12:00 mr-fox crond[11455]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:12:00 mr-fox crond[11453]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:12:00 mr-fox crond[11450]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:12:00 mr-fox crond[11452]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:12:00 mr-fox CROND[11458]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:12:00 mr-fox CROND[11459]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:12:00 mr-fox CROND[11461]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:12:00 mr-fox CROND[11462]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:12:00 mr-fox CROND[11460]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:12:00 mr-fox CROND[11455]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:12:00 mr-fox CROND[11455]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:12:00 mr-fox CROND[11454]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:12:00 mr-fox CROND[11454]: pam_unix(crond:session): session closed for user root
May 26 08:12:00 mr-fox CROND[11453]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:12:00 mr-fox CROND[11453]: pam_unix(crond:session): session closed for user root
May 26 08:12:01 mr-fox CROND[10419]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:12:01 mr-fox CROND[10419]: pam_unix(crond:session): session closed for user root
May 26 08:13:00 mr-fox crond[5538]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:13:00 mr-fox crond[5539]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:13:00 mr-fox crond[5541]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:13:00 mr-fox crond[5540]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:13:00 mr-fox crond[5537]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:13:00 mr-fox CROND[5545]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:13:00 mr-fox CROND[5550]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:13:00 mr-fox CROND[5551]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:13:00 mr-fox CROND[5552]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:13:00 mr-fox CROND[5554]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:13:00 mr-fox CROND[5541]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:13:00 mr-fox CROND[5541]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:13:00 mr-fox CROND[5540]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:13:00 mr-fox CROND[5540]: pam_unix(crond:session): session closed for user root
May 26 08:13:00 mr-fox CROND[5539]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:13:00 mr-fox CROND[5539]: pam_unix(crond:session): session closed for user root
May 26 08:13:00 mr-fox CROND[11452]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:13:00 mr-fox CROND[11452]: pam_unix(crond:session): session closed for user root
May 26 08:14:00 mr-fox crond[8572]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:14:00 mr-fox crond[8571]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:14:00 mr-fox crond[8570]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:14:00 mr-fox crond[8574]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:14:00 mr-fox crond[8569]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:14:00 mr-fox CROND[8577]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:14:00 mr-fox CROND[8578]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:14:00 mr-fox CROND[8579]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:14:00 mr-fox CROND[8580]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:14:00 mr-fox CROND[8582]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:14:00 mr-fox CROND[8574]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:14:00 mr-fox CROND[8574]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:14:00 mr-fox CROND[8572]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:14:00 mr-fox CROND[8572]: pam_unix(crond:session): session closed for user root
May 26 08:14:00 mr-fox CROND[8571]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:14:00 mr-fox CROND[8571]: pam_unix(crond:session): session closed for user root
May 26 08:14:01 mr-fox CROND[5538]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:14:01 mr-fox CROND[5538]: pam_unix(crond:session): session closed for user root
May 26 08:14:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:14:45 mr-fox kernel: rcu: \x0921-....: (1995171 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=805121
May 26 08:14:45 mr-fox kernel: rcu: \x09(t=1995172 jiffies g=8794409 q=55371266 ncpus=32)
May 26 08:14:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:14:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:14:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:14:45 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:14:45 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:14:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:14:45 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:14:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:14:45 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:14:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:14:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:14:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:14:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:14:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:14:45 mr-fox kernel: PKRU: 55555554
May 26 08:14:45 mr-fox kernel: Call Trace:
May 26 08:14:45 mr-fox kernel:  <IRQ>
May 26 08:14:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:14:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:14:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:14:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:14:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:14:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:14:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:14:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:14:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:14:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:14:45 mr-fox kernel:  </IRQ>
May 26 08:14:45 mr-fox kernel:  <TASK>
May 26 08:14:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:14:45 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 08:14:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:14:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:14:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:14:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:14:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:14:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:14:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:14:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:14:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:14:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:14:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:14:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:14:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:14:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:14:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:14:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:14:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:14:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:14:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:14:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:14:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:14:45 mr-fox kernel:  </TASK>
May 26 08:14:58 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 1998448 jiffies s: 491905 root: 0x2/.
May 26 08:14:58 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:14:58 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:14:58 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:14:58 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:14:58 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:14:58 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:14:58 mr-fox kernel: RIP: 0010:amd_iommu_iova_to_phys+0x4/0x20
May 26 08:14:58 mr-fox kernel: Code: b8 ea ff ff ff 31 d2 31 c9 31 f6 31 ff 45 31 c0 45 31 c9 45 31 d2 45 31 db e9 73 9e 45 00 0f 1f 84 00 00 00 00 00 f3 0f 1e fa <48> 8d 97 58 01 00 00 48 8b 87 60 01 00 00 48 89 d7 e9 86 75 30 00
May 26 08:14:58 mr-fox kernel: RSP: 0018:ffffa401005f0db8 EFLAGS: 00000202
May 26 08:14:58 mr-fox kernel: RAX: ffffffffa56aa2a0 RBX: ffff988ac1a460c0 RCX: 0000000000000001
May 26 08:14:58 mr-fox kernel: RDX: 0000000000000056 RSI: 00000000cf5d046a RDI: ffff988ac2211010
May 26 08:14:58 mr-fox kernel: RBP: 000000000000782a R08: 0000000000000000 R09: 0000000000000000
May 26 08:14:58 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000cf5d046a
May 26 08:14:58 mr-fox kernel: R13: 0000000000000056 R14: 0000000000000001 R15: 0000000000000000
May 26 08:14:58 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:14:58 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:14:58 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:14:58 mr-fox kernel: PKRU: 55555554
May 26 08:14:58 mr-fox kernel: Call Trace:
May 26 08:14:58 mr-fox kernel:  <NMI>
May 26 08:14:58 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:14:58 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:14:58 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:14:58 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:14:58 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:14:58 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:14:58 mr-fox kernel:  ? amd_iommu_map_pages+0x70/0x70
May 26 08:14:58 mr-fox kernel:  ? amd_iommu_iova_to_phys+0x4/0x20
May 26 08:14:58 mr-fox kernel:  ? amd_iommu_iova_to_phys+0x4/0x20
May 26 08:14:58 mr-fox kernel:  ? amd_iommu_iova_to_phys+0x4/0x20
May 26 08:14:58 mr-fox kernel:  </NMI>
May 26 08:14:58 mr-fox kernel:  <IRQ>
May 26 08:14:58 mr-fox kernel:  iommu_dma_unmap_page+0x2d/0xa0
May 26 08:14:58 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 08:14:58 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 08:14:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:14:58 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 08:14:58 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 08:14:58 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 08:14:58 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 08:14:58 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:14:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:14:58 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:14:58 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:14:58 mr-fox kernel:  </IRQ>
May 26 08:14:58 mr-fox kernel:  <TASK>
May 26 08:14:58 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:14:58 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:14:58 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:14:58 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:14:58 mr-fox kernel: RAX: ffff988d667606da RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:14:58 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:14:58 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 08:14:58 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:14:58 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:14:58 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:14:58 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:14:58 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:14:58 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:14:58 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:14:58 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:14:58 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:14:58 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:14:58 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:14:58 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:14:58 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:14:58 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:14:58 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:14:58 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:14:58 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:14:58 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:14:58 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:14:58 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:14:58 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:14:58 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:14:58 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:14:58 mr-fox kernel:  </TASK>
May 26 08:15:00 mr-fox crond[9428]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:15:00 mr-fox crond[9429]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:15:00 mr-fox crond[9427]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 08:15:00 mr-fox crond[9430]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:15:00 mr-fox crond[9431]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:15:00 mr-fox crond[9432]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:15:00 mr-fox CROND[9437]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:15:00 mr-fox CROND[9438]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:15:00 mr-fox CROND[9439]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:15:00 mr-fox CROND[9440]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:15:00 mr-fox CROND[9442]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:15:00 mr-fox CROND[9441]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:15:00 mr-fox crond[9433]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:15:00 mr-fox crond[9435]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:15:00 mr-fox CROND[9444]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 08:15:00 mr-fox CROND[9445]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:15:00 mr-fox CROND[9427]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:15:00 mr-fox CROND[9427]: pam_unix(crond:session): session closed for user torproject
May 26 08:15:00 mr-fox CROND[9435]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:15:00 mr-fox CROND[9435]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:15:00 mr-fox CROND[9431]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:15:00 mr-fox CROND[9431]: pam_unix(crond:session): session closed for user root
May 26 08:15:00 mr-fox CROND[9432]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:15:00 mr-fox CROND[9432]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:15:00 mr-fox CROND[9430]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:15:00 mr-fox CROND[9430]: pam_unix(crond:session): session closed for user root
May 26 08:15:01 mr-fox CROND[8570]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:15:01 mr-fox CROND[8570]: pam_unix(crond:session): session closed for user root
May 26 08:16:00 mr-fox crond[13321]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:16:00 mr-fox crond[13324]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:16:00 mr-fox crond[13325]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:16:00 mr-fox crond[13323]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:16:00 mr-fox crond[13326]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:16:00 mr-fox CROND[13331]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:16:00 mr-fox CROND[13330]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:16:00 mr-fox CROND[13332]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:16:00 mr-fox CROND[13333]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:16:00 mr-fox CROND[13334]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:16:00 mr-fox CROND[13326]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:16:00 mr-fox CROND[13326]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:16:00 mr-fox CROND[13325]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:16:00 mr-fox CROND[13325]: pam_unix(crond:session): session closed for user root
May 26 08:16:00 mr-fox CROND[13324]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:16:00 mr-fox CROND[13324]: pam_unix(crond:session): session closed for user root
May 26 08:16:00 mr-fox CROND[9429]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:16:00 mr-fox CROND[9429]: pam_unix(crond:session): session closed for user root
May 26 08:17:00 mr-fox crond[16938]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:17:00 mr-fox crond[16936]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:17:00 mr-fox crond[16937]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:17:00 mr-fox crond[16940]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:17:00 mr-fox crond[16941]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:17:00 mr-fox CROND[16948]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:17:00 mr-fox CROND[16949]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:17:00 mr-fox CROND[16947]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:17:00 mr-fox CROND[16950]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:17:00 mr-fox CROND[16951]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:17:00 mr-fox CROND[16941]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:17:00 mr-fox CROND[16941]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:17:00 mr-fox CROND[16940]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:17:00 mr-fox CROND[16940]: pam_unix(crond:session): session closed for user root
May 26 08:17:00 mr-fox CROND[16938]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:17:00 mr-fox CROND[16938]: pam_unix(crond:session): session closed for user root
May 26 08:17:00 mr-fox CROND[13323]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:17:00 mr-fox CROND[13323]: pam_unix(crond:session): session closed for user root
May 26 08:17:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:17:45 mr-fox kernel: rcu: \x0921-....: (2040175 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=824116
May 26 08:17:45 mr-fox kernel: rcu: \x09(t=2040176 jiffies g=8794409 q=56388062 ncpus=32)
May 26 08:17:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:17:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:17:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:17:45 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:17:45 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:17:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:17:45 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:17:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:17:45 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:17:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:17:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:17:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:17:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:17:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:17:45 mr-fox kernel: PKRU: 55555554
May 26 08:17:45 mr-fox kernel: Call Trace:
May 26 08:17:45 mr-fox kernel:  <IRQ>
May 26 08:17:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:17:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:17:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:17:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:17:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:17:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:17:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:17:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:17:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:17:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:17:45 mr-fox kernel:  </IRQ>
May 26 08:17:45 mr-fox kernel:  <TASK>
May 26 08:17:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:17:45 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 08:17:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:17:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:17:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:17:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:17:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:17:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:17:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:17:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:17:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:17:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:17:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:17:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:17:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:17:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:17:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:17:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:17:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:17:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:17:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:17:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:17:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:17:45 mr-fox kernel:  </TASK>
May 26 08:17:59 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 2043504 jiffies s: 491905 root: 0x2/.
May 26 08:17:59 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:17:59 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:17:59 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:17:59 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:17:59 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:17:59 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:17:59 mr-fox kernel: RIP: 0010:rb_first+0x13/0x30
May 26 08:17:59 mr-fox kernel: Code: c0 45 31 c9 45 31 d2 e9 a6 66 1b 00 66 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 8b 07 48 85 c0 74 18 48 89 c2 48 8b 40 10 <48> 85 c0 75 f4 48 89 d0 31 d2 31 ff e9 77 66 1b 00 31 d2 eb f0 0f
May 26 08:17:59 mr-fox kernel: RSP: 0018:ffffa401005f0e50 EFLAGS: 00000282
May 26 08:17:59 mr-fox kernel: RAX: ffff988d8f31cc40 RBX: ffff9891f581ae40 RCX: 0000000000000000
May 26 08:17:59 mr-fox kernel: RDX: ffff989b4c319340 RSI: 0000000000000000 RDI: ffff9891f581af90
May 26 08:17:59 mr-fox kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
May 26 08:17:59 mr-fox kernel: R10: 0000000000000005 R11: 0000000000000000 R12: ffffffffa6ad7f40
May 26 08:17:59 mr-fox kernel: R13: ffff9891f581af90 R14: ffffa401005f0f00 R15: ffff98a96ed5bb40
May 26 08:17:59 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:17:59 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:17:59 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:17:59 mr-fox kernel: PKRU: 55555554
May 26 08:17:59 mr-fox kernel: Call Trace:
May 26 08:17:59 mr-fox kernel:  <NMI>
May 26 08:17:59 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:17:59 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:17:59 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:17:59 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:17:59 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:17:59 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:17:59 mr-fox kernel:  ? rb_first+0x13/0x30
May 26 08:17:59 mr-fox kernel:  ? rb_first+0x13/0x30
May 26 08:17:59 mr-fox kernel:  ? rb_first+0x13/0x30
May 26 08:17:59 mr-fox kernel:  </NMI>
May 26 08:17:59 mr-fox kernel:  <IRQ>
May 26 08:17:59 mr-fox kernel:  tcp_retransmit_timer+0x1ad/0xa60
May 26 08:17:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:17:59 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 08:17:59 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 08:17:59 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 08:17:59 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 08:17:59 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 08:17:59 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:17:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:17:59 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:17:59 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:17:59 mr-fox kernel:  </IRQ>
May 26 08:17:59 mr-fox kernel:  <TASK>
May 26 08:17:59 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:17:59 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:17:59 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:17:59 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:17:59 mr-fox kernel: RAX: ffff988d667606da RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:17:59 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:17:59 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 08:17:59 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:17:59 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:17:59 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:17:59 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:17:59 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:17:59 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:17:59 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:17:59 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:17:59 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:17:59 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:17:59 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:17:59 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:17:59 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:17:59 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:17:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:17:59 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:17:59 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:17:59 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:17:59 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:17:59 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:17:59 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:17:59 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:17:59 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:17:59 mr-fox kernel:  </TASK>
May 26 08:18:00 mr-fox crond[20542]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:18:00 mr-fox crond[20544]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:18:00 mr-fox crond[20541]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:18:00 mr-fox crond[20546]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:18:00 mr-fox crond[20545]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:18:00 mr-fox CROND[20548]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:18:00 mr-fox CROND[20552]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:18:00 mr-fox CROND[20550]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:18:00 mr-fox CROND[20553]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:18:00 mr-fox CROND[20549]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:18:00 mr-fox CROND[20546]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:18:00 mr-fox CROND[20546]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:18:00 mr-fox CROND[20545]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:18:00 mr-fox CROND[20545]: pam_unix(crond:session): session closed for user root
May 26 08:18:01 mr-fox CROND[20544]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:18:01 mr-fox CROND[20544]: pam_unix(crond:session): session closed for user root
May 26 08:18:01 mr-fox CROND[16937]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:18:01 mr-fox CROND[16937]: pam_unix(crond:session): session closed for user root
May 26 08:19:00 mr-fox crond[22433]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:19:00 mr-fox crond[22432]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:19:00 mr-fox crond[22431]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:19:00 mr-fox crond[22436]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:19:00 mr-fox crond[22434]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:19:00 mr-fox CROND[22439]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:19:00 mr-fox CROND[22440]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:19:00 mr-fox CROND[22441]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:19:00 mr-fox CROND[22442]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:19:00 mr-fox CROND[22443]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:19:00 mr-fox CROND[22436]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:19:00 mr-fox CROND[22436]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:19:00 mr-fox CROND[22434]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:19:00 mr-fox CROND[22434]: pam_unix(crond:session): session closed for user root
May 26 08:19:00 mr-fox CROND[22433]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:19:00 mr-fox CROND[22433]: pam_unix(crond:session): session closed for user root
May 26 08:19:01 mr-fox CROND[20542]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:19:01 mr-fox CROND[20542]: pam_unix(crond:session): session closed for user root
May 26 08:20:00 mr-fox crond[28810]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:20:00 mr-fox crond[28812]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:20:00 mr-fox crond[28809]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:20:00 mr-fox crond[28811]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:20:00 mr-fox crond[28814]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:20:00 mr-fox crond[28808]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 08:20:00 mr-fox CROND[28819]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:20:00 mr-fox CROND[28820]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:20:00 mr-fox CROND[28821]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:20:00 mr-fox CROND[28822]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:20:00 mr-fox CROND[28823]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:20:00 mr-fox crond[28816]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:20:00 mr-fox CROND[28824]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:20:00 mr-fox crond[28815]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:20:00 mr-fox CROND[28827]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 08:20:00 mr-fox CROND[28828]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:20:00 mr-fox CROND[28808]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:20:00 mr-fox CROND[28808]: pam_unix(crond:session): session closed for user torproject
May 26 08:20:00 mr-fox CROND[28816]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:20:00 mr-fox CROND[28816]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:20:00 mr-fox CROND[28812]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:20:00 mr-fox CROND[28812]: pam_unix(crond:session): session closed for user root
May 26 08:20:00 mr-fox CROND[28814]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:20:00 mr-fox CROND[28814]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:20:00 mr-fox CROND[28811]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:20:00 mr-fox CROND[28811]: pam_unix(crond:session): session closed for user root
May 26 08:20:00 mr-fox CROND[22432]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:20:00 mr-fox CROND[22432]: pam_unix(crond:session): session closed for user root
May 26 08:20:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:20:45 mr-fox kernel: rcu: \x0921-....: (2085179 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=843479
May 26 08:20:45 mr-fox kernel: rcu: \x09(t=2085180 jiffies g=8794409 q=57410221 ncpus=32)
May 26 08:20:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:20:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:20:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:20:45 mr-fox kernel: RIP: 0010:xas_descend+0x31/0xd0
May 26 08:20:45 mr-fox kernel: Code: 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 <49> 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d fd 00 00
May 26 08:20:45 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 08:20:45 mr-fox kernel: RAX: ffff988ac07566ca RBX: 0000000000000001 RCX: 000000000000001e
May 26 08:20:45 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac0754ff8 RDI: ffffa401077e3970
May 26 08:20:45 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:20:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 08:20:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:20:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:20:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:20:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:20:45 mr-fox kernel: PKRU: 55555554
May 26 08:20:45 mr-fox kernel: Call Trace:
May 26 08:20:45 mr-fox kernel:  <IRQ>
May 26 08:20:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:20:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:20:45 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 08:20:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:20:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:20:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:20:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:20:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:20:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:20:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:20:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:20:45 mr-fox kernel:  </IRQ>
May 26 08:20:45 mr-fox kernel:  <TASK>
May 26 08:20:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:20:45 mr-fox kernel:  ? xas_descend+0x31/0xd0
May 26 08:20:45 mr-fox kernel:  xas_load+0x49/0x60
May 26 08:20:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:20:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:20:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:20:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:20:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:20:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:20:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:20:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:20:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:20:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:20:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:20:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:20:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:20:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:20:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:20:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:20:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:20:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:20:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:20:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:20:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:20:45 mr-fox kernel:  </TASK>
May 26 08:20:59 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 2088559 jiffies s: 491905 root: 0x2/.
May 26 08:20:59 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:20:59 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:20:59 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:20:59 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:20:59 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:20:59 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:20:59 mr-fox kernel: RIP: 0010:_raw_spin_lock+0xf/0x30
May 26 08:20:59 mr-fox kernel: Code: 31 ff e9 af 2f 15 00 e9 0f 03 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 0d 31 c0 31 d2 31 f6 31 ff e9 7d 2f 15 00 89 c6 e9 cb 00 00 00
May 26 08:20:59 mr-fox kernel: RSP: 0018:ffffa401005f0e88 EFLAGS: 00000246
May 26 08:20:59 mr-fox kernel: RAX: 0000000000000000 RBX: ffff988dd4110e08 RCX: 0000000000000000
May 26 08:20:59 mr-fox kernel: RDX: 0000000000000001 RSI: ffffffffa587d6d0 RDI: ffff988dd4110a98
May 26 08:20:59 mr-fox kernel: RBP: ffff988dd4110a00 R08: 0000000000000000 R09: 0000000000000000
May 26 08:20:59 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffa401005f0ee8
May 26 08:20:59 mr-fox kernel: R13: ffff98a96ed5bbb0 R14: ffffa401005f0ef0 R15: ffff98a96ed5bb40
May 26 08:20:59 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:20:59 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:20:59 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:20:59 mr-fox kernel: PKRU: 55555554
May 26 08:20:59 mr-fox kernel: Call Trace:
May 26 08:20:59 mr-fox kernel:  <NMI>
May 26 08:20:59 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:20:59 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:20:59 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:20:59 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:20:59 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:20:59 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:20:59 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 08:20:59 mr-fox kernel:  ? _raw_spin_lock+0xf/0x30
May 26 08:20:59 mr-fox kernel:  ? _raw_spin_lock+0xf/0x30
May 26 08:20:59 mr-fox kernel:  ? _raw_spin_lock+0xf/0x30
May 26 08:20:59 mr-fox kernel:  </NMI>
May 26 08:20:59 mr-fox kernel:  <IRQ>
May 26 08:20:59 mr-fox kernel:  tcp_write_timer+0x1e/0xd0
May 26 08:20:59 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 08:20:59 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 08:20:59 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 08:20:59 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 08:20:59 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:20:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:20:59 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:20:59 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:20:59 mr-fox kernel:  </IRQ>
May 26 08:20:59 mr-fox kernel:  <TASK>
May 26 08:20:59 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:20:59 mr-fox kernel: RIP: 0010:filemap_get_entry+0x6d/0x160
May 26 08:20:59 mr-fox kernel: Code: 00 00 48 c7 44 24 30 00 00 00 00 48 c7 44 24 38 00 00 00 00 48 c7 44 24 20 03 00 00 00 48 8d 7c 24 08 e8 56 70 78 00 48 89 c3 <48> 3d 02 04 00 00 74 e2 48 3d 06 04 00 00 74 da 48 85 c0 74 5a a8
May 26 08:20:59 mr-fox kernel: RSP: 0018:ffffa401077e3968 EFLAGS: 00000246
May 26 08:20:59 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 08:20:59 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:20:59 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 08:20:59 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:20:59 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:20:59 mr-fox kernel:  ? filemap_get_entry+0x6a/0x160
May 26 08:20:59 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:20:59 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:20:59 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:20:59 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:20:59 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:20:59 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:20:59 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:20:59 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:20:59 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:20:59 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:20:59 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:20:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:20:59 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:20:59 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:20:59 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:20:59 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:20:59 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:20:59 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:20:59 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:20:59 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:20:59 mr-fox kernel:  </TASK>
May 26 08:21:00 mr-fox crond[986]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:21:00 mr-fox crond[987]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:21:00 mr-fox crond[988]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:21:00 mr-fox crond[989]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:21:00 mr-fox crond[990]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:21:00 mr-fox CROND[993]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:21:00 mr-fox CROND[996]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:21:00 mr-fox CROND[997]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:21:00 mr-fox CROND[999]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:21:00 mr-fox CROND[1001]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:21:00 mr-fox CROND[990]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:21:00 mr-fox CROND[990]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:21:00 mr-fox CROND[989]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:21:00 mr-fox CROND[989]: pam_unix(crond:session): session closed for user root
May 26 08:21:00 mr-fox CROND[988]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:21:00 mr-fox CROND[988]: pam_unix(crond:session): session closed for user root
May 26 08:21:01 mr-fox CROND[28810]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:21:01 mr-fox CROND[28810]: pam_unix(crond:session): session closed for user root
May 26 08:22:00 mr-fox crond[1873]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:22:00 mr-fox crond[1871]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:22:00 mr-fox crond[1869]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:22:00 mr-fox crond[1870]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:22:00 mr-fox crond[1874]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:22:00 mr-fox CROND[1878]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:22:00 mr-fox CROND[1879]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:22:00 mr-fox CROND[1881]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:22:00 mr-fox CROND[1880]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:22:00 mr-fox CROND[1887]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:22:00 mr-fox CROND[1874]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:22:00 mr-fox CROND[1874]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:22:00 mr-fox CROND[1873]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:22:00 mr-fox CROND[1873]: pam_unix(crond:session): session closed for user root
May 26 08:22:00 mr-fox CROND[1871]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:22:00 mr-fox CROND[1871]: pam_unix(crond:session): session closed for user root
May 26 08:22:01 mr-fox CROND[987]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:22:01 mr-fox CROND[987]: pam_unix(crond:session): session closed for user root
May 26 08:23:00 mr-fox crond[6998]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:23:00 mr-fox crond[6999]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:23:00 mr-fox crond[7001]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:23:00 mr-fox crond[7000]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:23:00 mr-fox crond[6997]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:23:00 mr-fox CROND[7005]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:23:00 mr-fox CROND[7006]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:23:00 mr-fox CROND[7007]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:23:00 mr-fox CROND[7009]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:23:00 mr-fox CROND[7010]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:23:00 mr-fox CROND[7001]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:23:00 mr-fox CROND[7001]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:23:00 mr-fox CROND[7000]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:23:00 mr-fox CROND[7000]: pam_unix(crond:session): session closed for user root
May 26 08:23:00 mr-fox CROND[6999]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:23:00 mr-fox CROND[6999]: pam_unix(crond:session): session closed for user root
May 26 08:23:00 mr-fox CROND[1870]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:23:00 mr-fox CROND[1870]: pam_unix(crond:session): session closed for user root
May 26 08:23:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:23:45 mr-fox kernel: rcu: \x0921-....: (2130183 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=863037
May 26 08:23:45 mr-fox kernel: rcu: \x09(t=2130184 jiffies g=8794409 q=58426312 ncpus=32)
May 26 08:23:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:23:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:23:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:23:45 mr-fox kernel: RIP: 0010:xas_descend+0x26/0xd0
May 26 08:23:45 mr-fox kernel: Code: 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e 48 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f <89> d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03
May 26 08:23:45 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000206
May 26 08:23:45 mr-fox kernel: RAX: ffff988ac07566ca RBX: 000000000000000c RCX: 0000000000000018
May 26 08:23:45 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac07566c8 RDI: ffffa401077e3970
May 26 08:23:45 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:23:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 08:23:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:23:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:23:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:23:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:23:45 mr-fox kernel: PKRU: 55555554
May 26 08:23:45 mr-fox kernel: Call Trace:
May 26 08:23:45 mr-fox kernel:  <IRQ>
May 26 08:23:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:23:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:23:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:23:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:23:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:23:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:23:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:23:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:23:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:23:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:23:45 mr-fox kernel:  </IRQ>
May 26 08:23:45 mr-fox kernel:  <TASK>
May 26 08:23:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:23:45 mr-fox kernel:  ? xas_descend+0x26/0xd0
May 26 08:23:45 mr-fox kernel:  xas_load+0x49/0x60
May 26 08:23:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:23:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:23:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:23:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:23:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:23:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:23:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:23:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:23:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:23:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:23:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:23:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:23:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:23:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:23:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:23:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:23:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:23:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:23:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:23:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:23:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:23:45 mr-fox kernel:  </TASK>
May 26 08:23:59 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 2133616 jiffies s: 491905 root: 0x2/.
May 26 08:23:59 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:23:59 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:23:59 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:23:59 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:23:59 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:23:59 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:23:59 mr-fox kernel: RIP: 0010:__iommu_dma_unmap+0xb0/0x170
May 26 08:23:59 mr-fox kernel: Code: 48 c7 44 24 18 00 00 00 00 48 01 d5 48 c7 44 24 08 ff ff ff ff 48 21 c5 49 8b 85 e0 00 00 00 48 89 ea 48 85 c0 0f 95 44 24 30 <e8> db b5 ff ff 48 39 e8 0f 85 9a 00 00 00 0f b6 44 24 30 3c 01 0f
May 26 08:23:59 mr-fox kernel: RSP: 0018:ffffa401005f0d48 EFLAGS: 00000282
May 26 08:23:59 mr-fox kernel: RAX: ffff988ac2211010 RBX: 00000000cf593000 RCX: ffffa401005f0d50
May 26 08:23:59 mr-fox kernel: RDX: 0000000000001000 RSI: 00000000cf593000 RDI: ffff988ac2211010
May 26 08:23:59 mr-fox kernel: RBP: 0000000000001000 R08: 0000000000000000 R09: 0000000000000000
May 26 08:23:59 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff988ac2211010
May 26 08:23:59 mr-fox kernel: R13: ffff988ac1292400 R14: ffffa401005f0d50 R15: ffffa401005f0d68
May 26 08:23:59 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:23:59 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:23:59 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:23:59 mr-fox kernel: PKRU: 55555554
May 26 08:23:59 mr-fox kernel: Call Trace:
May 26 08:23:59 mr-fox kernel:  <NMI>
May 26 08:23:59 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:23:59 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:23:59 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:23:59 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:23:59 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:23:59 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:23:59 mr-fox kernel:  ? __iommu_dma_unmap+0xb0/0x170
May 26 08:23:59 mr-fox kernel:  ? __iommu_dma_unmap+0xb0/0x170
May 26 08:23:59 mr-fox kernel:  ? __iommu_dma_unmap+0xb0/0x170
May 26 08:23:59 mr-fox kernel:  </NMI>
May 26 08:23:59 mr-fox kernel:  <IRQ>
May 26 08:23:59 mr-fox kernel:  iommu_dma_unmap_page+0x43/0xa0
May 26 08:23:59 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 08:23:59 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 08:23:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:23:59 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 08:23:59 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 08:23:59 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 08:23:59 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 08:23:59 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:23:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:23:59 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:23:59 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:23:59 mr-fox kernel:  </IRQ>
May 26 08:23:59 mr-fox kernel:  <TASK>
May 26 08:23:59 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:23:59 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:23:59 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:23:59 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:23:59 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:23:59 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:23:59 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:23:59 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:23:59 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:23:59 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:23:59 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:23:59 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:23:59 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:23:59 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:23:59 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:23:59 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:23:59 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:23:59 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:23:59 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:23:59 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:23:59 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:23:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:23:59 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:23:59 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:23:59 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:23:59 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:23:59 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:23:59 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:23:59 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:23:59 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:23:59 mr-fox kernel:  </TASK>
May 26 08:24:00 mr-fox crond[10341]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:24:00 mr-fox crond[10343]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:24:00 mr-fox crond[10344]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:24:00 mr-fox crond[10345]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:24:00 mr-fox crond[10346]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:24:00 mr-fox CROND[10350]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:24:00 mr-fox CROND[10351]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:24:00 mr-fox CROND[10352]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:24:00 mr-fox CROND[10353]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:24:00 mr-fox CROND[10355]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:24:00 mr-fox CROND[10346]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:24:00 mr-fox CROND[10346]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:24:00 mr-fox CROND[10345]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:24:00 mr-fox CROND[10345]: pam_unix(crond:session): session closed for user root
May 26 08:24:00 mr-fox CROND[10344]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:24:00 mr-fox CROND[10344]: pam_unix(crond:session): session closed for user root
May 26 08:24:00 mr-fox CROND[6998]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:24:00 mr-fox CROND[6998]: pam_unix(crond:session): session closed for user root
May 26 08:25:00 mr-fox crond[12535]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:25:00 mr-fox crond[12537]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:25:00 mr-fox crond[12533]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 08:25:00 mr-fox crond[12534]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:25:00 mr-fox CROND[12544]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:25:00 mr-fox CROND[12547]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:25:00 mr-fox CROND[12548]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:25:00 mr-fox crond[12541]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:25:00 mr-fox crond[12540]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:25:00 mr-fox crond[12539]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:25:00 mr-fox CROND[12550]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:25:00 mr-fox crond[12538]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:25:00 mr-fox CROND[12551]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:25:00 mr-fox CROND[12552]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:25:00 mr-fox CROND[12554]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 08:25:00 mr-fox CROND[12556]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:25:00 mr-fox CROND[12533]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:25:00 mr-fox CROND[12533]: pam_unix(crond:session): session closed for user torproject
May 26 08:25:00 mr-fox CROND[12541]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:25:00 mr-fox CROND[12541]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:25:00 mr-fox CROND[12538]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:25:00 mr-fox CROND[12538]: pam_unix(crond:session): session closed for user root
May 26 08:25:01 mr-fox CROND[12539]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:25:01 mr-fox CROND[12539]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:25:01 mr-fox CROND[12537]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:25:01 mr-fox CROND[12537]: pam_unix(crond:session): session closed for user root
May 26 08:25:01 mr-fox CROND[10343]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:25:01 mr-fox CROND[10343]: pam_unix(crond:session): session closed for user root
May 26 08:26:00 mr-fox crond[13067]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:26:00 mr-fox crond[13066]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:26:00 mr-fox crond[13071]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:26:00 mr-fox crond[13069]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:26:00 mr-fox crond[13072]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:26:00 mr-fox CROND[13075]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:26:00 mr-fox CROND[13076]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:26:00 mr-fox CROND[13078]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:26:00 mr-fox CROND[13079]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:26:00 mr-fox CROND[13081]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:26:00 mr-fox CROND[13072]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:26:00 mr-fox CROND[13072]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:26:00 mr-fox CROND[13071]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:26:00 mr-fox CROND[13071]: pam_unix(crond:session): session closed for user root
May 26 08:26:00 mr-fox CROND[13069]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:26:00 mr-fox CROND[13069]: pam_unix(crond:session): session closed for user root
May 26 08:26:01 mr-fox CROND[12535]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:26:01 mr-fox CROND[12535]: pam_unix(crond:session): session closed for user root
May 26 08:26:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:26:45 mr-fox kernel: rcu: \x0921-....: (2175187 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=882654
May 26 08:26:45 mr-fox kernel: rcu: \x09(t=2175188 jiffies g=8794409 q=59457709 ncpus=32)
May 26 08:26:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:26:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:26:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:26:45 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:26:45 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:26:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 08:26:45 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:26:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:26:45 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:26:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:26:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:26:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:26:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:26:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:26:45 mr-fox kernel: PKRU: 55555554
May 26 08:26:45 mr-fox kernel: Call Trace:
May 26 08:26:45 mr-fox kernel:  <IRQ>
May 26 08:26:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:26:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:26:45 mr-fox kernel:  ? tcp_write_xmit+0xe3/0x13b0
May 26 08:26:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:26:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:26:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:26:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:26:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:26:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:26:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:26:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:26:45 mr-fox kernel:  </IRQ>
May 26 08:26:45 mr-fox kernel:  <TASK>
May 26 08:26:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:26:45 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 08:26:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:26:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:26:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:26:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:26:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:26:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:26:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:26:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:26:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:26:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:26:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:26:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:26:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:26:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:26:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:26:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:26:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:26:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:26:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:26:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:26:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:26:45 mr-fox kernel:  </TASK>
May 26 08:26:59 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 2178672 jiffies s: 491905 root: 0x2/.
May 26 08:26:59 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:26:59 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:26:59 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:26:59 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:26:59 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:26:59 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:26:59 mr-fox kernel: RIP: 0010:skb_release_data.isra.0+0x47/0x1e0
May 26 08:26:59 mr-fox kernel: Code: 00 00 00 a8 01 74 2b 83 e0 02 3c 01 89 c2 19 c0 0d ff ff fe ff 80 fa 01 19 d2 66 31 d2 81 c2 01 00 01 00 f0 41 0f c1 44 24 20 <39> c2 0f 85 eb 00 00 00 4d 85 f6 74 57 41 8b 86 b8 00 00 00 49 03
May 26 08:26:59 mr-fox kernel: RSP: 0018:ffffa401005f0db8 EFLAGS: 00000213
May 26 08:26:59 mr-fox kernel: RAX: 0000000000010002 RBX: ffff98998f71de98 RCX: 0000000000000b5f
May 26 08:26:59 mr-fox kernel: RDX: 0000000000000001 RSI: 0000000000000002 RDI: ffff98998f71de98
May 26 08:26:59 mr-fox kernel: RBP: 0000000000009184 R08: 0000000000000000 R09: 0000000000000000
May 26 08:26:59 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff988e1dc06980
May 26 08:26:59 mr-fox kernel: R13: 0000000000000002 R14: ffff98998f71de98 R15: ffffa40100119f80
May 26 08:26:59 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:26:59 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:26:59 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:26:59 mr-fox kernel: PKRU: 55555554
May 26 08:26:59 mr-fox kernel: Call Trace:
May 26 08:26:59 mr-fox kernel:  <NMI>
May 26 08:26:59 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:26:59 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:26:59 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:26:59 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:26:59 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:26:59 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:26:59 mr-fox kernel:  ? skb_release_data.isra.0+0x47/0x1e0
May 26 08:26:59 mr-fox kernel:  ? skb_release_data.isra.0+0x47/0x1e0
May 26 08:26:59 mr-fox kernel:  ? skb_release_data.isra.0+0x47/0x1e0
May 26 08:26:59 mr-fox kernel:  </NMI>
May 26 08:26:59 mr-fox kernel:  <IRQ>
May 26 08:26:59 mr-fox kernel:  napi_consume_skb+0x6a/0xc0
May 26 08:26:59 mr-fox kernel:  igb_poll+0xea/0x1370
May 26 08:26:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:26:59 mr-fox kernel:  ? task_tick_fair+0x85/0x470
May 26 08:26:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:26:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:26:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:26:59 mr-fox kernel:  ? wq_worker_tick+0xd/0xd0
May 26 08:26:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:26:59 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 08:26:59 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 08:26:59 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 08:26:59 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:26:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:26:59 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:26:59 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:26:59 mr-fox kernel:  </IRQ>
May 26 08:26:59 mr-fox kernel:  <TASK>
May 26 08:26:59 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:26:59 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:26:59 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:26:59 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:26:59 mr-fox kernel: RAX: ffff988d667606da RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:26:59 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:26:59 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 08:26:59 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:26:59 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:26:59 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:26:59 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:26:59 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:26:59 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:26:59 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:26:59 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:26:59 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:26:59 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:26:59 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:26:59 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:26:59 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:26:59 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:26:59 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:26:59 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:26:59 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:26:59 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:26:59 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:26:59 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:26:59 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:26:59 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:26:59 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:26:59 mr-fox kernel:  </TASK>
May 26 08:27:00 mr-fox crond[17016]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:27:00 mr-fox crond[17017]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:27:00 mr-fox crond[17018]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:27:00 mr-fox crond[17019]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:27:00 mr-fox CROND[17024]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:27:00 mr-fox CROND[17025]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:27:00 mr-fox crond[17020]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:27:00 mr-fox CROND[17026]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:27:00 mr-fox CROND[17027]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:27:00 mr-fox CROND[17028]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:27:00 mr-fox CROND[17020]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:27:00 mr-fox CROND[17020]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:27:00 mr-fox CROND[17019]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:27:00 mr-fox CROND[17019]: pam_unix(crond:session): session closed for user root
May 26 08:27:00 mr-fox CROND[17018]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:27:00 mr-fox CROND[17018]: pam_unix(crond:session): session closed for user root
May 26 08:27:00 mr-fox CROND[13067]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:27:00 mr-fox CROND[13067]: pam_unix(crond:session): session closed for user root
May 26 08:28:00 mr-fox crond[20473]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:28:00 mr-fox crond[20472]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:28:00 mr-fox crond[20475]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:28:00 mr-fox crond[20471]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:28:00 mr-fox CROND[20481]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:28:00 mr-fox CROND[20482]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:28:00 mr-fox CROND[20483]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:28:00 mr-fox CROND[20485]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:28:00 mr-fox crond[20476]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:28:00 mr-fox CROND[20488]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:28:00 mr-fox CROND[20476]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:28:00 mr-fox CROND[20476]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:28:00 mr-fox CROND[20475]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:28:00 mr-fox CROND[20475]: pam_unix(crond:session): session closed for user root
May 26 08:28:00 mr-fox CROND[20473]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:28:00 mr-fox CROND[20473]: pam_unix(crond:session): session closed for user root
May 26 08:28:01 mr-fox CROND[17017]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:28:01 mr-fox CROND[17017]: pam_unix(crond:session): session closed for user root
May 26 08:29:00 mr-fox crond[23306]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:29:00 mr-fox crond[23308]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:29:00 mr-fox crond[23305]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:29:00 mr-fox crond[23307]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:29:00 mr-fox crond[23309]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:29:00 mr-fox CROND[23314]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:29:00 mr-fox CROND[23315]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:29:00 mr-fox CROND[23318]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:29:00 mr-fox CROND[23317]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:29:00 mr-fox CROND[23319]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:29:00 mr-fox CROND[23309]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:29:00 mr-fox CROND[23309]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:29:00 mr-fox CROND[23308]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:29:00 mr-fox CROND[23308]: pam_unix(crond:session): session closed for user root
May 26 08:29:00 mr-fox CROND[23307]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:29:00 mr-fox CROND[23307]: pam_unix(crond:session): session closed for user root
May 26 08:29:01 mr-fox CROND[20472]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:29:01 mr-fox CROND[20472]: pam_unix(crond:session): session closed for user root
May 26 08:29:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:29:45 mr-fox kernel: rcu: \x0921-....: (2220191 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=902227
May 26 08:29:45 mr-fox kernel: rcu: \x09(t=2220192 jiffies g=8794409 q=60466260 ncpus=32)
May 26 08:29:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:29:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:29:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:29:45 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:29:45 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:29:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 08:29:45 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:29:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:29:45 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:29:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:29:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:29:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:29:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:29:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:29:45 mr-fox kernel: PKRU: 55555554
May 26 08:29:45 mr-fox kernel: Call Trace:
May 26 08:29:45 mr-fox kernel:  <IRQ>
May 26 08:29:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:29:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:29:45 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 08:29:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:29:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:29:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:29:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:29:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:29:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:29:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:29:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:29:45 mr-fox kernel:  </IRQ>
May 26 08:29:45 mr-fox kernel:  <TASK>
May 26 08:29:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:29:45 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 08:29:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:29:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:29:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:29:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:29:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:29:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:29:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:29:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:29:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:29:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:29:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:29:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:29:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:29:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:29:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:29:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:29:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:29:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:29:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:29:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:29:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:29:45 mr-fox kernel:  </TASK>
May 26 08:30:00 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 2223728 jiffies s: 491905 root: 0x2/.
May 26 08:30:00 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:30:00 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:30:00 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:30:00 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:30:00 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:30:00 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:30:00 mr-fox kernel: RIP: 0010:fetch_pte+0x145/0x170
May 26 08:30:00 mr-fox kernel: Code: 74 25 48 8b 10 f6 c2 01 75 81 31 c0 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f 31 d2 31 c9 31 f6 31 ff e9 29 18 45 00 48 8b 38 <81> e7 00 0e 00 00 48 81 ff 00 0e 00 00 75 d3 48 83 c4 10 48 89 ee
May 26 08:30:00 mr-fox kernel: RSP: 0018:ffffa401005f0d58 EFLAGS: 00000246
May 26 08:30:00 mr-fox kernel: RAX: ffff988d028ee2e0 RBX: 0000000000000000 RCX: 0000000000000003
May 26 08:30:00 mr-fox kernel: RDX: 0000000000001000 RSI: 0000000000000001 RDI: 3000000358174e01
May 26 08:30:00 mr-fox kernel: RBP: ffffa401005f0da0 R08: 0000000000000000 R09: 0000000000000000
May 26 08:30:00 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000cdc5cc46
May 26 08:30:00 mr-fox kernel: R13: 000ffffffffff000 R14: 0000000000000003 R15: 0000000000000000
May 26 08:30:00 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:30:00 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:30:00 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:30:00 mr-fox kernel: PKRU: 55555554
May 26 08:30:00 mr-fox kernel: Call Trace:
May 26 08:30:00 mr-fox kernel:  <NMI>
May 26 08:30:00 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:30:00 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:30:00 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:30:00 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:30:00 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:30:00 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:30:00 mr-fox kernel:  ? fetch_pte+0x145/0x170
May 26 08:30:00 mr-fox kernel:  ? fetch_pte+0x145/0x170
May 26 08:30:00 mr-fox kernel:  ? fetch_pte+0x145/0x170
May 26 08:30:00 mr-fox kernel:  </NMI>
May 26 08:30:00 mr-fox kernel:  <IRQ>
May 26 08:30:00 mr-fox kernel:  iommu_v1_iova_to_phys+0x2b/0xa0
May 26 08:30:00 mr-fox kernel:  iommu_dma_unmap_page+0x2d/0xa0
May 26 08:30:00 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 08:30:00 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 08:30:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:30:00 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 08:30:00 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 08:30:00 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 08:30:00 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 08:30:00 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:30:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:30:00 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:30:00 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:30:00 mr-fox kernel:  </IRQ>
May 26 08:30:00 mr-fox kernel:  <TASK>
May 26 08:30:00 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:30:00 mr-fox kernel: RIP: 0010:xas_descend+0x40/0xd0
May 26 08:30:00 mr-fox kernel: Code: 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 <75> 08 48 3d fd 00 00 00 76 2f 41 88 5c 24 12 48 83 c4 08 5b 5d 41
May 26 08:30:00 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000293
May 26 08:30:00 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: 0000000000000021 RCX: 0000000000000000
May 26 08:30:00 mr-fox kernel: RDX: 0000000000000000 RSI: ffff988d55f18ff8 RDI: ffffa401077e3970
May 26 08:30:00 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:30:00 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 08:30:00 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:30:00 mr-fox kernel:  xas_load+0x49/0x60
May 26 08:30:00 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:30:00 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:30:00 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:30:00 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:30:00 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:30:00 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:30:00 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:30:00 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:30:00 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:30:00 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:30:00 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:30:00 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:30:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:30:00 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:30:00 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:30:00 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:30:00 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:30:00 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:30:00 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:30:00 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:30:00 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:30:00 mr-fox kernel:  </TASK>
May 26 08:30:00 mr-fox crond[28641]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 08:30:00 mr-fox crond[28644]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:30:00 mr-fox crond[28643]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:30:00 mr-fox crond[28642]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:30:00 mr-fox CROND[28651]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:30:00 mr-fox crond[28646]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:30:00 mr-fox CROND[28652]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:30:00 mr-fox CROND[28653]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:30:00 mr-fox CROND[28654]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:30:00 mr-fox crond[28648]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:30:00 mr-fox crond[28645]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:30:00 mr-fox crond[28649]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:30:00 mr-fox CROND[28655]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:30:00 mr-fox CROND[28657]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 08:30:00 mr-fox CROND[28659]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:30:00 mr-fox CROND[28658]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:30:00 mr-fox CROND[28641]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:30:00 mr-fox CROND[28641]: pam_unix(crond:session): session closed for user torproject
May 26 08:30:00 mr-fox CROND[28649]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:30:00 mr-fox CROND[28649]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:30:00 mr-fox CROND[28645]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:30:00 mr-fox CROND[28645]: pam_unix(crond:session): session closed for user root
May 26 08:30:00 mr-fox CROND[28646]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:30:00 mr-fox CROND[28646]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:30:00 mr-fox CROND[28644]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:30:00 mr-fox CROND[28644]: pam_unix(crond:session): session closed for user root
May 26 08:30:00 mr-fox CROND[23306]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:30:00 mr-fox CROND[23306]: pam_unix(crond:session): session closed for user root
May 26 08:31:00 mr-fox crond[31070]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:31:00 mr-fox crond[31067]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:31:00 mr-fox crond[31068]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:31:00 mr-fox crond[31069]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:31:00 mr-fox CROND[31076]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:31:00 mr-fox CROND[31074]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:31:00 mr-fox CROND[31075]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:31:00 mr-fox crond[31071]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:31:00 mr-fox CROND[31078]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:31:00 mr-fox CROND[31080]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:31:00 mr-fox CROND[31071]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:31:00 mr-fox CROND[31071]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:31:00 mr-fox CROND[31070]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:31:00 mr-fox CROND[31070]: pam_unix(crond:session): session closed for user root
May 26 08:31:00 mr-fox CROND[31069]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:31:00 mr-fox CROND[31069]: pam_unix(crond:session): session closed for user root
May 26 08:31:00 mr-fox CROND[28643]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:31:00 mr-fox CROND[28643]: pam_unix(crond:session): session closed for user root
May 26 08:31:32 mr-fox sshd[20093]: pam_unix(sshd:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:32:00 mr-fox crond[24925]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:32:00 mr-fox crond[24924]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:32:00 mr-fox crond[24926]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:32:00 mr-fox crond[24927]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:32:00 mr-fox CROND[24931]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:32:00 mr-fox crond[24928]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:32:00 mr-fox CROND[24932]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:32:00 mr-fox CROND[24933]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:32:00 mr-fox CROND[24934]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:32:00 mr-fox CROND[24935]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:32:00 mr-fox CROND[24928]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:32:00 mr-fox CROND[24928]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:32:00 mr-fox CROND[24927]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:32:00 mr-fox CROND[24927]: pam_unix(crond:session): session closed for user root
May 26 08:32:01 mr-fox CROND[24926]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:32:01 mr-fox CROND[24926]: pam_unix(crond:session): session closed for user root
May 26 08:32:01 mr-fox CROND[31068]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:32:01 mr-fox CROND[31068]: pam_unix(crond:session): session closed for user root
May 26 08:32:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:32:45 mr-fox kernel: rcu: \x0921-....: (2265194 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=921657
May 26 08:32:45 mr-fox kernel: rcu: \x09(t=2265195 jiffies g=8794409 q=61523075 ncpus=32)
May 26 08:32:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:32:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:32:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:32:45 mr-fox kernel: RIP: 0010:xas_load+0x20/0x60
May 26 08:32:45 mr-fox kernel: Code: ff eb be 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 53 48 89 fb e8 f2 f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 <77> 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48
May 26 08:32:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000286
May 26 08:32:45 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:32:45 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:32:45 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 08:32:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:32:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:32:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:32:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:32:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:32:45 mr-fox kernel: PKRU: 55555554
May 26 08:32:45 mr-fox kernel: Call Trace:
May 26 08:32:45 mr-fox kernel:  <IRQ>
May 26 08:32:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:32:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:32:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:32:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:32:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:32:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:32:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:32:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:32:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:32:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:32:45 mr-fox kernel:  </IRQ>
May 26 08:32:45 mr-fox kernel:  <TASK>
May 26 08:32:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:32:45 mr-fox kernel:  ? xas_load+0x20/0x60
May 26 08:32:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:32:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:32:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:32:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:32:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:32:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:32:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:32:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:32:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:32:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:32:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:32:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:32:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:32:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:32:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:32:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:32:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:32:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:32:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:32:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:32:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:32:45 mr-fox kernel:  </TASK>
May 26 08:33:00 mr-fox crond[27777]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:33:00 mr-fox crond[27776]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:33:00 mr-fox crond[27774]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:33:00 mr-fox crond[27778]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:33:00 mr-fox crond[27779]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:33:00 mr-fox CROND[27783]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:33:00 mr-fox CROND[27785]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:33:00 mr-fox CROND[27784]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:33:00 mr-fox CROND[27786]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:33:00 mr-fox CROND[27787]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:33:00 mr-fox CROND[27779]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:33:00 mr-fox CROND[27779]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:33:00 mr-fox CROND[27778]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:33:00 mr-fox CROND[27778]: pam_unix(crond:session): session closed for user root
May 26 08:33:00 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 2268783 jiffies s: 491905 root: 0x2/.
May 26 08:33:00 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:33:00 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:33:00 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:33:00 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:33:00 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:33:00 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:33:00 mr-fox kernel: RIP: 0010:fetch_pte+0x145/0x170
May 26 08:33:00 mr-fox kernel: Code: 74 25 48 8b 10 f6 c2 01 75 81 31 c0 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f 31 d2 31 c9 31 f6 31 ff e9 29 18 45 00 48 8b 38 <81> e7 00 0e 00 00 48 81 ff 00 0e 00 00 75 d3 48 83 c4 10 48 89 ee
May 26 08:33:00 mr-fox kernel: RSP: 0018:ffffa401005f0d58 EFLAGS: 00000246
May 26 08:33:00 mr-fox kernel: RAX: ffff988acb51fa80 RBX: 0000000000000000 RCX: 0000000000000003
May 26 08:33:00 mr-fox kernel: RDX: 0000000000001000 RSI: 0000000000000001 RDI: 3000000176fe4001
May 26 08:33:00 mr-fox kernel: RBP: ffffa401005f0da0 R08: 0000000000000000 R09: 0000000000000000
May 26 08:33:00 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000cff5073e
May 26 08:33:00 mr-fox kernel: R13: 000ffffffffff000 R14: 0000000000000003 R15: 0000000000000000
May 26 08:33:00 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:33:00 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:33:00 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:33:00 mr-fox kernel: PKRU: 55555554
May 26 08:33:00 mr-fox kernel: Call Trace:
May 26 08:33:00 mr-fox kernel:  <NMI>
May 26 08:33:00 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:33:00 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:33:00 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:33:00 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:33:00 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:33:00 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:33:00 mr-fox kernel:  ? fetch_pte+0x145/0x170
May 26 08:33:00 mr-fox kernel:  ? fetch_pte+0x145/0x170
May 26 08:33:00 mr-fox kernel:  ? fetch_pte+0x145/0x170
May 26 08:33:00 mr-fox kernel:  </NMI>
May 26 08:33:00 mr-fox kernel:  <IRQ>
May 26 08:33:00 mr-fox kernel:  ? skb_release_data.isra.0+0x1b2/0x1e0
May 26 08:33:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:33:00 mr-fox kernel:  iommu_v1_iova_to_phys+0x2b/0xa0
May 26 08:33:00 mr-fox kernel:  iommu_dma_unmap_page+0x2d/0xa0
May 26 08:33:00 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 08:33:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:33:00 mr-fox kernel:  ? task_tick_fair+0x85/0x470
May 26 08:33:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:33:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:33:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:33:00 mr-fox kernel:  ? wq_worker_tick+0xd/0xd0
May 26 08:33:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:33:00 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 08:33:00 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 08:33:00 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 08:33:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:33:00 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:33:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:33:00 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:33:00 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:33:00 mr-fox kernel:  </IRQ>
May 26 08:33:00 mr-fox kernel:  <TASK>
May 26 08:33:00 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:33:00 mr-fox kernel: RIP: 0010:xas_load+0x35/0x60
May 26 08:33:00 mr-fox kernel: Code: f7 ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 <48> 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d
May 26 08:33:00 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000286
May 26 08:33:00 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:33:00 mr-fox kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:33:00 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:33:00 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:33:00 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:33:00 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:33:00 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:33:00 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:33:00 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:33:00 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:33:00 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:33:00 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:33:00 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:33:00 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:33:00 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:33:00 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:33:00 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:33:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:33:00 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:33:00 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:33:00 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:33:00 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:33:00 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:33:00 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:33:00 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:33:00 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:33:00 mr-fox kernel:  </TASK>
May 26 08:33:00 mr-fox CROND[27777]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:33:00 mr-fox CROND[27777]: pam_unix(crond:session): session closed for user root
May 26 08:33:01 mr-fox CROND[24925]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:33:01 mr-fox CROND[24925]: pam_unix(crond:session): session closed for user root
May 26 08:33:11 mr-fox sshd[12450]: pam_unix(sshd:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:34:00 mr-fox crond[30835]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:34:00 mr-fox crond[30837]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:34:00 mr-fox crond[30836]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:34:00 mr-fox CROND[30843]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:34:00 mr-fox crond[30838]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:34:00 mr-fox crond[30840]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:34:00 mr-fox CROND[30846]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:34:00 mr-fox CROND[30847]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:34:00 mr-fox CROND[30849]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:34:00 mr-fox CROND[30850]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:34:00 mr-fox CROND[30840]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:34:00 mr-fox CROND[30840]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:34:00 mr-fox CROND[30838]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:34:00 mr-fox CROND[30838]: pam_unix(crond:session): session closed for user root
May 26 08:34:00 mr-fox CROND[30837]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:34:00 mr-fox CROND[30837]: pam_unix(crond:session): session closed for user root
May 26 08:34:00 mr-fox CROND[27776]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:34:00 mr-fox CROND[27776]: pam_unix(crond:session): session closed for user root
May 26 08:34:30 mr-fox sshd[15895]: pam_unix(sshd:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:35:00 mr-fox crond[974]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:35:00 mr-fox crond[973]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:35:00 mr-fox crond[975]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:35:00 mr-fox crond[979]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:35:00 mr-fox crond[983]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:35:00 mr-fox crond[971]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 08:35:00 mr-fox crond[972]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:35:00 mr-fox CROND[988]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:35:00 mr-fox CROND[989]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:35:00 mr-fox CROND[990]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:35:00 mr-fox CROND[991]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:35:00 mr-fox CROND[992]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 08:35:00 mr-fox CROND[996]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:35:00 mr-fox crond[984]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:35:00 mr-fox CROND[999]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:35:00 mr-fox CROND[1001]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:35:00 mr-fox CROND[971]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:35:00 mr-fox CROND[971]: pam_unix(crond:session): session closed for user torproject
May 26 08:35:00 mr-fox CROND[984]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:35:00 mr-fox CROND[984]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:35:00 mr-fox CROND[975]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:35:00 mr-fox CROND[975]: pam_unix(crond:session): session closed for user root
May 26 08:35:00 mr-fox CROND[979]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:35:00 mr-fox CROND[979]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:35:00 mr-fox CROND[974]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:35:00 mr-fox CROND[974]: pam_unix(crond:session): session closed for user root
May 26 08:35:01 mr-fox CROND[30836]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:35:01 mr-fox CROND[30836]: pam_unix(crond:session): session closed for user root
May 26 08:35:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:35:45 mr-fox kernel: rcu: \x0921-....: (2310198 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=941259
May 26 08:35:45 mr-fox kernel: rcu: \x09(t=2310199 jiffies g=8794409 q=62543459 ncpus=32)
May 26 08:35:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:35:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:35:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:35:45 mr-fox kernel: RIP: 0010:xas_descend+0x48/0xd0
May 26 08:35:45 mr-fox kernel: Code: 04 00 48 d3 eb 83 e3 3f 89 d8 48 83 c0 04 48 8b 44 c5 08 49 89 6c 24 18 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d fd 00 00 00 <76> 2f 41 88 5c 24 12 48 83 c4 08 5b 5d 41 5c 41 5d 31 d2 31 c9 31
May 26 08:35:45 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000296
May 26 08:35:45 mr-fox kernel: RAX: ffff988b1fdbb47a RBX: 0000000000000036 RCX: 0000000000000012
May 26 08:35:45 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988acf90e6c8 RDI: ffffa401077e3970
May 26 08:35:45 mr-fox kernel: RBP: ffff988acf90e6c8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:35:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 08:35:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:35:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:35:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:35:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:35:45 mr-fox kernel: PKRU: 55555554
May 26 08:35:45 mr-fox kernel: Call Trace:
May 26 08:35:45 mr-fox kernel:  <IRQ>
May 26 08:35:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:35:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:35:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:35:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:35:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:35:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:35:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:35:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:35:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:35:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:35:45 mr-fox kernel:  </IRQ>
May 26 08:35:45 mr-fox kernel:  <TASK>
May 26 08:35:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:35:45 mr-fox kernel:  ? xas_descend+0x48/0xd0
May 26 08:35:45 mr-fox kernel:  xas_load+0x49/0x60
May 26 08:35:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:35:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:35:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:35:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:35:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:35:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:35:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:35:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:35:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:35:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:35:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:35:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:35:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:35:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:35:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:35:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:35:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:35:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:35:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:35:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:35:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:35:45 mr-fox kernel:  </TASK>
May 26 08:36:00 mr-fox crond[873]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:36:00 mr-fox crond[874]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:36:00 mr-fox crond[876]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:36:00 mr-fox crond[875]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:36:00 mr-fox crond[878]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:36:00 mr-fox CROND[881]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:36:00 mr-fox CROND[882]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:36:00 mr-fox CROND[880]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:36:00 mr-fox CROND[883]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:36:00 mr-fox CROND[884]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:36:00 mr-fox CROND[878]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:36:00 mr-fox CROND[878]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:36:00 mr-fox CROND[876]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:36:00 mr-fox CROND[876]: pam_unix(crond:session): session closed for user root
May 26 08:36:00 mr-fox CROND[875]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:36:00 mr-fox CROND[875]: pam_unix(crond:session): session closed for user root
May 26 08:36:00 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 2313839 jiffies s: 491905 root: 0x2/.
May 26 08:36:00 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:36:00 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:36:00 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:36:00 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:36:00 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:36:00 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:36:00 mr-fox kernel: RIP: 0010:iommu_dma_unmap_page+0x4a/0xa0
May 26 08:36:00 mr-fox kernel: Code: ff ff 4c 89 e6 48 89 c7 e8 13 bb ff ff 48 85 c0 74 46 4c 89 ea 4c 89 e6 48 89 df 48 89 c5 e8 3d fe ff ff 48 8b 83 40 02 00 00 <48> 85 c0 74 2b 48 3b 28 72 26 48 3b 68 08 73 20 4d 89 f8 44 89 f1
May 26 08:36:00 mr-fox kernel: RSP: 0018:ffffa401005f0dc0 EFLAGS: 00000246
May 26 08:36:00 mr-fox kernel: RAX: ffffffffa6aaa880 RBX: ffff988ac1a460c0 RCX: 0000000000000000
May 26 08:36:00 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:36:00 mr-fox kernel: RBP: 0000000f922b097e R08: 0000000000000000 R09: 0000000000000000
May 26 08:36:00 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000cf9f797e
May 26 08:36:00 mr-fox kernel: R13: 0000000000000042 R14: 0000000000000001 R15: 0000000000000000
May 26 08:36:00 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:36:00 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:36:00 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:36:00 mr-fox kernel: PKRU: 55555554
May 26 08:36:00 mr-fox kernel: Call Trace:
May 26 08:36:00 mr-fox kernel:  <NMI>
May 26 08:36:00 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:36:00 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:36:00 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:36:00 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:36:00 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:36:00 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:36:00 mr-fox kernel:  ? iommu_dma_unmap_page+0x4a/0xa0
May 26 08:36:00 mr-fox kernel:  ? iommu_dma_unmap_page+0x4a/0xa0
May 26 08:36:00 mr-fox kernel:  ? iommu_dma_unmap_page+0x4a/0xa0
May 26 08:36:00 mr-fox kernel:  </NMI>
May 26 08:36:00 mr-fox kernel:  <IRQ>
May 26 08:36:00 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 08:36:00 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 08:36:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:36:00 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 08:36:00 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 08:36:00 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 08:36:00 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 08:36:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:36:00 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:36:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:36:00 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:36:00 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:36:00 mr-fox kernel:  </IRQ>
May 26 08:36:00 mr-fox kernel:  <TASK>
May 26 08:36:00 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:36:00 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:36:00 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:36:00 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:36:00 mr-fox kernel: RAX: ffff988acf90e6ca RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:36:00 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:36:00 mr-fox kernel: RBP: ffff988ac07566c8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:36:00 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:36:00 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:36:00 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:36:00 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:36:00 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:36:00 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:36:00 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:36:00 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:36:00 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:36:00 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:36:00 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:36:00 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:36:00 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:36:00 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:36:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:36:00 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:36:00 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:36:00 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:36:00 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:36:00 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:36:00 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:36:00 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:36:00 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:36:00 mr-fox kernel:  </TASK>
May 26 08:36:01 mr-fox CROND[973]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:36:01 mr-fox CROND[973]: pam_unix(crond:session): session closed for user root
May 26 08:36:49 mr-fox sshd[12450]: pam_unix(sshd:session): session closed for user tinderbox
May 26 08:37:00 mr-fox crond[4213]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:37:00 mr-fox crond[4214]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:37:00 mr-fox crond[4212]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:37:00 mr-fox crond[4215]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:37:00 mr-fox crond[4216]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:37:00 mr-fox CROND[4221]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:37:00 mr-fox CROND[4220]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:37:00 mr-fox CROND[4223]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:37:00 mr-fox CROND[4222]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:37:00 mr-fox CROND[4224]: (root) CMD (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:37:00 mr-fox CROND[4216]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:37:00 mr-fox CROND[4216]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:37:00 mr-fox CROND[4215]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:37:00 mr-fox CROND[4215]: pam_unix(crond:session): session closed for user root
May 26 08:37:00 mr-fox CROND[4214]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:37:00 mr-fox CROND[4214]: pam_unix(crond:session): session closed for user root
May 26 08:37:00 mr-fox CROND[874]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:37:00 mr-fox CROND[874]: pam_unix(crond:session): session closed for user root
May 26 08:37:20 mr-fox crontab[5320]: (root) BEGIN EDIT (root)
May 26 08:37:45 mr-fox crontab[5320]: (root) REPLACE (root)
May 26 08:37:45 mr-fox crontab[5320]: (root) END EDIT (root)
May 26 08:37:47 mr-fox crontab[19009]: (root) LIST (root)
May 26 08:38:00 mr-fox crond[3071]: (root) RELOAD (/var/spool/cron/crontabs/root)
May 26 08:38:00 mr-fox crond[7100]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:38:00 mr-fox crond[7099]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:38:00 mr-fox crond[7101]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:38:00 mr-fox crond[7098]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:38:00 mr-fox CROND[7105]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:38:00 mr-fox CROND[7106]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:38:00 mr-fox CROND[7108]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:38:00 mr-fox CROND[7109]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:38:00 mr-fox CROND[7098]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:38:00 mr-fox CROND[7098]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:38:00 mr-fox CROND[7101]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:38:00 mr-fox CROND[7101]: pam_unix(crond:session): session closed for user root
May 26 08:38:00 mr-fox CROND[7100]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:38:00 mr-fox CROND[7100]: pam_unix(crond:session): session closed for user root
May 26 08:38:00 mr-fox CROND[4213]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:38:00 mr-fox CROND[4213]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[873]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[2340]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[8569]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[873]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[986]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[8569]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[2340]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[6492]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[379]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[4109]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[2749]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[3442]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[6492]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[7565]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[19516]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[12534]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[7565]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[19516]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[3442]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[7582]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[31549]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[16112]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[11625]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[972]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[10408]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[5610]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[23305]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[2525]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[10408]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[5610]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[5449]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[5449]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[12534]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[20541]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[12349]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[10418]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[30375]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[4483]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[21416]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[12110]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[3098]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[12110]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[5537]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[13321]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[20471]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[5537]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[31751]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[18689]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[27159]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[9494]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[14290]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[9428]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[14930]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[18100]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[30522]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[22271]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[14930]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[13066]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[22271]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[24341]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[13066]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[22180]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[18365]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[2082]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[22431]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[28642]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[25209]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[2082]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[12237]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[25460]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[28642]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[27412]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[30956]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[17016]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[7075]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[21217]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[15676]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[19580]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[17016]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[26989]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[29652]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[29652]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[4517]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[14400]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[13068]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[4517]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[7770]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[29160]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[18458]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[29160]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[10341]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[10341]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[24924]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[31641]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[11450]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[8780]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[25247]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[16936]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[8780]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[25247]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[31870]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[19723]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[31870]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[27774]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[19723]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[26846]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[11047]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[7750]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[27774]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[26846]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[14213]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[29059]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[19991]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[15610]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[27690]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[23097]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[32136]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[15610]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[20917]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[24747]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[24422]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[986]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[24747]: pam_unix(crond:session): session closed for user root
May 26 08:38:23 mr-fox CROND[28809]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[31067]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[28903]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[31686]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[25325]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
May 26 08:38:23 mr-fox CROND[26184]: (root) CMDEND (for i in {0..3}; do /opt/tb/bin/metrics.sh &>/dev/null  ; sleep 15; done)
--
May 26 08:38:45 mr-fox kernel: RIP: 0010:xas_load+0x11/0x60
May 26 08:38:45 mr-fox kernel: Code: 92 4c 89 ee 48 c7 c7 e0 eb 49 a6 e8 a9 58 c6 ff eb be 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 53 48 89 fb e8 f2 f7 ff ff 48 89 c2 <83> e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31
May 26 08:38:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 08:38:45 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:38:45 mr-fox kernel: RDX: ffff988ac0754ffa RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:38:45 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 08:38:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:38:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:38:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:38:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:38:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:38:45 mr-fox kernel: PKRU: 55555554
May 26 08:38:45 mr-fox kernel: Call Trace:
May 26 08:38:45 mr-fox kernel:  <IRQ>
May 26 08:38:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:38:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:38:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:38:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:38:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:38:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:38:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:38:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:38:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:38:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:38:45 mr-fox kernel:  </IRQ>
May 26 08:38:45 mr-fox kernel:  <TASK>
May 26 08:38:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:38:45 mr-fox kernel:  ? xas_load+0x11/0x60
May 26 08:38:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:38:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:38:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:38:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:38:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:38:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:38:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:38:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:38:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:38:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:38:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:38:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:38:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:38:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:38:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:38:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:38:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:38:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:38:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:38:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:38:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:38:45 mr-fox kernel:  </TASK>
May 26 08:39:00 mr-fox crond[8369]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:39:00 mr-fox crond[8368]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:39:00 mr-fox crond[8370]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:39:00 mr-fox crond[8371]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:39:00 mr-fox CROND[8375]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:39:00 mr-fox CROND[8376]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:39:00 mr-fox CROND[8378]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:39:00 mr-fox CROND[8377]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:39:00 mr-fox CROND[8368]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:39:00 mr-fox CROND[8368]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:39:00 mr-fox CROND[8371]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:39:00 mr-fox CROND[8371]: pam_unix(crond:session): session closed for user root
May 26 08:39:00 mr-fox CROND[8370]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:39:00 mr-fox CROND[8370]: pam_unix(crond:session): session closed for user root
May 26 08:39:00 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 2358895 jiffies s: 491905 root: 0x2/.
May 26 08:39:00 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:39:00 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:39:00 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:39:00 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:39:00 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:39:00 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:39:00 mr-fox kernel: RIP: 0010:read_tsc+0x7/0x20
May 26 08:39:00 mr-fox kernel: Code: 00 00 00 f3 0f 1e fa 8b 05 66 26 48 01 e9 6c 64 ae 00 90 f3 0f 1e fa e9 62 64 ae 00 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 31 d2 31 c9 e9 42 64 ae 00 0f 1f 80 00
May 26 08:39:00 mr-fox kernel: RSP: 0018:ffffa401005f0e28 EFLAGS: 00000246
May 26 08:39:00 mr-fox kernel: RAX: 0000000064243c8e RBX: 00000000014239de RCX: 0000000000000015
May 26 08:39:00 mr-fox kernel: RDX: 00000000000082c3 RSI: 0000000000000000 RDI: ffffffffa60201c0
May 26 08:39:00 mr-fox kernel: RBP: 000026697e5906c2 R08: 0000000000000000 R09: 0000000000000000
May 26 08:39:00 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
May 26 08:39:00 mr-fox kernel: R13: ffff98a96ed5bbb0 R14: ffffa401005f0ef0 R15: ffff98a96ed5bb40
May 26 08:39:00 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:39:00 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:39:00 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:39:00 mr-fox kernel: PKRU: 55555554
May 26 08:39:00 mr-fox kernel: Call Trace:
May 26 08:39:00 mr-fox kernel:  <NMI>
May 26 08:39:00 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:39:00 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:39:00 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:39:00 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:39:00 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:39:00 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:39:00 mr-fox kernel:  ? read_tsc+0x7/0x20
May 26 08:39:00 mr-fox kernel:  ? read_tsc+0x7/0x20
May 26 08:39:00 mr-fox kernel:  ? read_tsc+0x7/0x20
May 26 08:39:00 mr-fox kernel:  </NMI>
May 26 08:39:00 mr-fox kernel:  <IRQ>
May 26 08:39:00 mr-fox kernel:  ktime_get+0x42/0xb0
May 26 08:39:00 mr-fox kernel:  tcp_mstamp_refresh+0xd/0x40
May 26 08:39:00 mr-fox kernel:  tcp_write_timer_handler+0x5d/0x280
May 26 08:39:00 mr-fox kernel:  tcp_write_timer+0x9f/0xd0
May 26 08:39:00 mr-fox kernel:  ? tcp_write_timer_handler+0x280/0x280
May 26 08:39:00 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 08:39:00 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 08:39:00 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 08:39:00 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:39:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:39:00 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:39:00 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:39:00 mr-fox kernel:  </IRQ>
May 26 08:39:00 mr-fox kernel:  <TASK>
May 26 08:39:00 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:39:00 mr-fox kernel: RIP: 0010:xas_load+0x11/0x60
May 26 08:39:00 mr-fox kernel: Code: 92 4c 89 ee 48 c7 c7 e0 eb 49 a6 e8 a9 58 c6 ff eb be 0f 1f 80 00 00 00 00 f3 0f 1e fa 55 53 48 89 fb e8 f2 f7 ff ff 48 89 c2 <83> e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 0f 5b 5d 31 d2 31
May 26 08:39:00 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 08:39:00 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:39:00 mr-fox kernel: RDX: ffff988ac0754ffa RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:39:00 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 08:39:00 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:39:00 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:39:00 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:39:00 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:39:00 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:39:00 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:39:00 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:39:00 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:39:00 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:39:00 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:39:00 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:39:00 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:39:00 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:39:00 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:39:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:39:00 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:39:00 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:39:00 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:39:00 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:39:00 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:39:00 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:39:00 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:39:00 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:39:00 mr-fox kernel:  </TASK>
May 26 08:39:01 mr-fox CROND[7099]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:39:01 mr-fox CROND[7099]: pam_unix(crond:session): session closed for user root
May 26 08:39:16 mr-fox sshd[20093]: pam_unix(sshd:session): session closed for user tinderbox
May 26 08:39:32 mr-fox crontab[29139]: (root) BEGIN EDIT (root)
May 26 08:39:42 mr-fox crontab[29139]: (root) REPLACE (root)
May 26 08:39:42 mr-fox crontab[29139]: (root) END EDIT (root)
May 26 08:39:46 mr-fox crontab[20171]: (root) BEGIN EDIT (root)
May 26 08:39:54 mr-fox crontab[20171]: (root) REPLACE (root)
May 26 08:39:54 mr-fox crontab[20171]: (root) END EDIT (root)
May 26 08:40:00 mr-fox crond[3071]: (root) RELOAD (/var/spool/cron/crontabs/root)
May 26 08:40:00 mr-fox crond[10107]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:40:00 mr-fox crond[10104]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 08:40:00 mr-fox crond[10108]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:40:00 mr-fox crond[10109]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:40:00 mr-fox crond[10110]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:40:00 mr-fox CROND[10115]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 08:40:00 mr-fox CROND[10116]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:40:00 mr-fox CROND[10114]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:40:00 mr-fox CROND[10118]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:40:00 mr-fox crond[10111]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:40:00 mr-fox crond[10112]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:40:00 mr-fox CROND[10119]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:40:00 mr-fox CROND[10120]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:40:00 mr-fox CROND[10121]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:40:00 mr-fox CROND[10104]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:40:00 mr-fox CROND[10104]: pam_unix(crond:session): session closed for user torproject
May 26 08:40:00 mr-fox CROND[10109]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:40:00 mr-fox CROND[10109]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:40:00 mr-fox CROND[10112]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:40:00 mr-fox CROND[10112]: pam_unix(crond:session): session closed for user root
May 26 08:40:00 mr-fox CROND[10107]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:40:00 mr-fox CROND[10107]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:40:00 mr-fox CROND[10111]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:40:00 mr-fox CROND[10111]: pam_unix(crond:session): session closed for user root
May 26 08:40:00 mr-fox CROND[8369]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:40:00 mr-fox CROND[8369]: pam_unix(crond:session): session closed for user root
May 26 08:41:00 mr-fox CROND[10110]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:41:00 mr-fox CROND[10110]: pam_unix(crond:session): session closed for user root
May 26 08:41:00 mr-fox crond[12997]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:41:00 mr-fox crond[12998]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:41:00 mr-fox crond[12996]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:41:00 mr-fox CROND[13002]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:41:00 mr-fox CROND[13004]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:41:00 mr-fox CROND[13003]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:41:00 mr-fox crond[12999]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:41:00 mr-fox CROND[13006]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:41:00 mr-fox CROND[12996]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:41:00 mr-fox CROND[12996]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:41:00 mr-fox CROND[12999]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:41:00 mr-fox CROND[12999]: pam_unix(crond:session): session closed for user root
May 26 08:41:00 mr-fox CROND[12998]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:41:00 mr-fox CROND[12998]: pam_unix(crond:session): session closed for user root
May 26 08:41:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:41:45 mr-fox kernel: rcu: \x0921-....: (2400206 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=980025
May 26 08:41:45 mr-fox kernel: rcu: \x09(t=2400207 jiffies g=8794409 q=64677320 ncpus=32)
May 26 08:41:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:41:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:41:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:41:45 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:41:45 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:41:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:41:45 mr-fox kernel: RAX: ffff988d667606da RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:41:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:41:45 mr-fox kernel: RBP: ffff988b1fdbb478 R08: 0000000000000000 R09: 0000000000000000
May 26 08:41:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:41:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:41:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:41:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:41:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:41:45 mr-fox kernel: PKRU: 55555554
May 26 08:41:45 mr-fox kernel: Call Trace:
May 26 08:41:45 mr-fox kernel:  <IRQ>
May 26 08:41:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:41:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:41:45 mr-fox kernel:  ? tcp_write_xmit+0x1e7/0x13b0
May 26 08:41:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:41:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:41:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:41:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:41:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:41:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:41:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:41:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:41:45 mr-fox kernel:  </IRQ>
May 26 08:41:45 mr-fox kernel:  <TASK>
May 26 08:41:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:41:45 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 08:41:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:41:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:41:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:41:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:41:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:41:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:41:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:41:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:41:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:41:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:41:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:41:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:41:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:41:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:41:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:41:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:41:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:41:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:41:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:41:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:41:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:41:45 mr-fox kernel:  </TASK>
May 26 08:41:46 mr-fox sSMTP[21339]: Creating SSL connection to host
May 26 08:41:46 mr-fox sSMTP[21339]: SSL connection using TLS_AES_256_GCM_SHA384
May 26 08:41:51 mr-fox sSMTP[21339]: Sent mail for root@zwiebeltoralf.de (221 www325.your-server.de closing connection) uid=0 username=root outbytes=4034
May 26 08:42:00 mr-fox crond[14079]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:42:00 mr-fox crond[14080]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:42:00 mr-fox crond[14081]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:42:00 mr-fox crond[14082]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:42:00 mr-fox CROND[14085]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:42:00 mr-fox CROND[14086]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:42:00 mr-fox CROND[14087]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:42:00 mr-fox CROND[14088]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:42:00 mr-fox CROND[14079]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:42:00 mr-fox CROND[14079]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:42:00 mr-fox CROND[14082]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:42:00 mr-fox CROND[14082]: pam_unix(crond:session): session closed for user root
May 26 08:42:00 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 2403952 jiffies s: 491905 root: 0x2/.
May 26 08:42:00 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:42:00 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:42:00 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:42:00 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:42:00 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:42:00 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:42:00 mr-fox kernel: RIP: 0010:__slab_free+0xb8/0x420
May 26 08:42:00 mr-fox kernel: Code: 89 c1 66 89 74 24 58 66 85 f6 0f 84 df 00 00 00 41 8b 46 08 45 84 c9 75 09 4d 85 ed 0f 84 75 01 00 00 45 31 ff 48 8b 4c 24 58 <a9> 00 00 00 40 0f 84 85 00 00 00 4c 89 e8 f0 49 0f c7 4c 24 20 0f
May 26 08:42:00 mr-fox kernel: RSP: 0018:ffffa401005f0cc0 EFLAGS: 00000246
May 26 08:42:00 mr-fox kernel: RAX: 0000000040042000 RBX: ffff988c27bb5440 RCX: 00000000801c001b
May 26 08:42:00 mr-fox kernel: RDX: 00000000801c001c RSI: 00000000801c001b RDI: ffffa401005f0d30
May 26 08:42:00 mr-fox kernel: RBP: ffffa401005f0d60 R08: 0000000000000001 R09: 0000000000000001
May 26 08:42:00 mr-fox kernel: R10: ffff988c27bb5440 R11: 0000000000000000 R12: ffffcf62c99eed00
May 26 08:42:00 mr-fox kernel: R13: 0000000000000000 R14: ffff988ac0137700 R15: 0000000000000000
May 26 08:42:00 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:42:00 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:42:00 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:42:00 mr-fox kernel: PKRU: 55555554
May 26 08:42:00 mr-fox kernel: Call Trace:
May 26 08:42:00 mr-fox kernel:  <NMI>
May 26 08:42:00 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:42:00 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:42:00 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:42:00 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:42:00 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:42:00 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:42:00 mr-fox kernel:  ? __slab_free+0xb8/0x420
May 26 08:42:00 mr-fox kernel:  ? __slab_free+0xb8/0x420
May 26 08:42:00 mr-fox kernel:  ? __slab_free+0xb8/0x420
May 26 08:42:00 mr-fox kernel:  </NMI>
May 26 08:42:00 mr-fox kernel:  <IRQ>
May 26 08:42:00 mr-fox kernel:  ? amd_iommu_unmap_pages+0x40/0x130
May 26 08:42:00 mr-fox kernel:  ? skb_release_data.isra.0+0x1b2/0x1e0
May 26 08:42:00 mr-fox kernel:  kmem_cache_free+0x2c2/0x360
May 26 08:42:00 mr-fox kernel:  skb_release_data.isra.0+0x1b2/0x1e0
May 26 08:42:00 mr-fox kernel:  napi_consume_skb+0x45/0xc0
May 26 08:42:00 mr-fox kernel:  igb_poll+0xea/0x1370
May 26 08:42:00 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 08:42:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:42:00 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 08:42:00 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 08:42:00 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 08:42:00 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 08:42:00 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:42:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:42:00 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:42:00 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:42:00 mr-fox kernel:  </IRQ>
May 26 08:42:00 mr-fox kernel:  <TASK>
May 26 08:42:00 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:42:00 mr-fox kernel: RIP: 0010:xas_descend+0x13/0xd0
May 26 08:42:00 mr-fox kernel: Code: 87 04 00 e9 a3 87 04 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 55 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 08 0f b6 0e <48> 8b 5f 08 80 f9 3f 0f 87 8e 87 04 00 48 d3 eb 83 e3 3f 89 d8 48
May 26 08:42:00 mr-fox kernel: RSP: 0018:ffffa401077e3920 EFLAGS: 00000282
May 26 08:42:00 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 000000000000001e
May 26 08:42:00 mr-fox kernel: RDX: 0000000000000002 RSI: ffff988ac0754ff8 RDI: ffffa401077e3970
May 26 08:42:00 mr-fox kernel: RBP: ffff988ac0754ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:42:00 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: ffffa401077e3970
May 26 08:42:00 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:42:00 mr-fox kernel:  xas_load+0x49/0x60
May 26 08:42:00 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:42:00 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:42:00 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:42:00 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:42:00 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:42:00 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:42:00 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:42:00 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:42:00 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:42:00 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:42:00 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:42:00 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:42:00 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:42:00 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:42:00 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:42:00 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:42:00 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:42:00 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:42:00 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:42:00 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:42:00 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:42:00 mr-fox kernel:  </TASK>
May 26 08:42:01 mr-fox CROND[14081]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:42:01 mr-fox CROND[14081]: pam_unix(crond:session): session closed for user root
May 26 08:42:01 mr-fox CROND[12997]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:42:01 mr-fox CROND[12997]: pam_unix(crond:session): session closed for user root
May 26 08:42:37 mr-fox su[10021]: (to tinderbox) root on pts/2
May 26 08:42:37 mr-fox su[10021]: pam_unix(su-l:session): session opened for user tinderbox(uid=1003) by root(uid=0)
May 26 08:42:55 mr-fox su[10021]: pam_unix(su-l:session): session closed for user tinderbox
May 26 08:43:00 mr-fox crond[14832]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:43:00 mr-fox crond[14831]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:43:00 mr-fox crond[14834]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:43:00 mr-fox crond[14835]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:43:00 mr-fox CROND[14838]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:43:00 mr-fox CROND[14837]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:43:00 mr-fox CROND[14839]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:43:00 mr-fox CROND[14841]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:43:00 mr-fox CROND[14831]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:43:00 mr-fox CROND[14831]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:43:00 mr-fox CROND[14835]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:43:00 mr-fox CROND[14835]: pam_unix(crond:session): session closed for user root
May 26 08:43:00 mr-fox CROND[14834]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:43:00 mr-fox CROND[14834]: pam_unix(crond:session): session closed for user root
May 26 08:43:01 mr-fox CROND[14080]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:43:01 mr-fox CROND[14080]: pam_unix(crond:session): session closed for user root
May 26 08:44:00 mr-fox crond[14874]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:44:00 mr-fox crond[14872]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:44:00 mr-fox crond[14871]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:44:00 mr-fox crond[14873]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:44:00 mr-fox CROND[14878]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:44:00 mr-fox CROND[14880]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:44:00 mr-fox CROND[14879]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:44:00 mr-fox CROND[14881]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:44:00 mr-fox CROND[14871]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:44:00 mr-fox CROND[14871]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:44:00 mr-fox CROND[14874]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:44:00 mr-fox CROND[14874]: pam_unix(crond:session): session closed for user root
May 26 08:44:00 mr-fox CROND[14873]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:44:00 mr-fox CROND[14873]: pam_unix(crond:session): session closed for user root
May 26 08:44:00 mr-fox CROND[14832]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:44:00 mr-fox CROND[14832]: pam_unix(crond:session): session closed for user root
May 26 08:44:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:44:45 mr-fox kernel: rcu: \x0921-....: (2445210 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=998833
May 26 08:44:45 mr-fox kernel: rcu: \x09(t=2445211 jiffies g=8794409 q=70987250 ncpus=32)
May 26 08:44:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:44:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:44:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:44:45 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:44:45 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:44:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:44:45 mr-fox kernel: RAX: ffff988d55f18ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:44:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:44:45 mr-fox kernel: RBP: ffff988d667606d8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:44:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:44:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:44:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:44:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:44:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:44:45 mr-fox kernel: PKRU: 55555554
May 26 08:44:45 mr-fox kernel: Call Trace:
May 26 08:44:45 mr-fox kernel:  <IRQ>
May 26 08:44:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:44:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:44:45 mr-fox kernel:  ? tcp_write_xmit+0xe3/0x13b0
May 26 08:44:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:44:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:44:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:44:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:44:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:44:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:44:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:44:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:44:45 mr-fox kernel:  </IRQ>
May 26 08:44:45 mr-fox kernel:  <TASK>
May 26 08:44:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:44:45 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 08:44:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:44:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:44:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:44:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:44:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:44:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:44:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:44:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:44:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:44:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:44:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:44:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:44:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:44:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:44:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:44:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:44:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:44:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:44:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:44:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:44:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:44:45 mr-fox kernel:  </TASK>
May 26 08:44:45 mr-fox sSMTP[21376]: Creating SSL connection to host
May 26 08:44:45 mr-fox sSMTP[21376]: SSL connection using TLS_AES_256_GCM_SHA384
May 26 08:44:50 mr-fox sSMTP[21376]: Sent mail for root@zwiebeltoralf.de (221 www325.your-server.de closing connection) uid=0 username=root outbytes=4033
May 26 08:45:00 mr-fox crond[12608]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:45:00 mr-fox crond[12606]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:45:00 mr-fox crond[12602]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 08:45:00 mr-fox crond[12607]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:45:00 mr-fox crond[12604]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:45:00 mr-fox crond[12603]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:45:00 mr-fox crond[12605]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:45:00 mr-fox CROND[12620]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:45:00 mr-fox CROND[12618]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:45:00 mr-fox CROND[12617]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:45:00 mr-fox CROND[12621]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:45:00 mr-fox CROND[12623]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:45:00 mr-fox CROND[12622]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:45:00 mr-fox CROND[12619]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 08:45:00 mr-fox CROND[12602]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:45:00 mr-fox CROND[12602]: pam_unix(crond:session): session closed for user torproject
May 26 08:45:00 mr-fox CROND[12605]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:45:00 mr-fox CROND[12605]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:45:00 mr-fox CROND[12608]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:45:00 mr-fox CROND[12608]: pam_unix(crond:session): session closed for user root
May 26 08:45:01 mr-fox CROND[12603]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:45:01 mr-fox CROND[12603]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:45:01 mr-fox CROND[12607]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:45:01 mr-fox CROND[12607]: pam_unix(crond:session): session closed for user root
May 26 08:45:01 mr-fox CROND[14872]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:45:01 mr-fox CROND[14872]: pam_unix(crond:session): session closed for user root
May 26 08:45:01 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 2449007 jiffies s: 491905 root: 0x2/.
May 26 08:45:01 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:45:01 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:45:01 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:45:01 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:45:01 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:45:01 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:45:01 mr-fox kernel: RIP: 0010:tcp_ack_update_rtt+0x1b0/0x350
May 26 08:45:01 mr-fox kernel: Code: 74 14 48 8b 52 10 48 83 e2 fc f7 02 00 20 00 00 0f 85 79 01 00 00 89 c7 e8 bd 0b 8f ff 89 83 64 07 00 00 f6 83 98 08 00 00 08 <0f> 85 41 01 00 00 8b ab 90 05 00 00 e9 27 ff ff ff 48 83 f9 ff 0f
May 26 08:45:01 mr-fox kernel: RSP: 0018:ffffa401005f0940 EFLAGS: 00000246
May 26 08:45:01 mr-fox kernel: RAX: 0000000000030d40 RBX: ffff988d825e8000 RCX: 000000000001bece
May 26 08:45:01 mr-fox kernel: RDX: ffffffffa5cbcea0 RSI: fffffffffffff558 RDI: 0000000000000000
May 26 08:45:01 mr-fox kernel: RBP: 000000000007d2d8 R08: fffffffffffe4bda R09: ffffa401005f09f0
May 26 08:45:01 mr-fox kernel: R10: 0000000000000000 R11: 0000000000004502 R12: 000000000011c444
May 26 08:45:01 mr-fox kernel: R13: ffffa401005f09f0 R14: 0000000000000001 R15: ffff98a7546b4d00
May 26 08:45:01 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:45:01 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:45:01 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:45:01 mr-fox kernel: PKRU: 55555554
May 26 08:45:01 mr-fox kernel: Call Trace:
May 26 08:45:01 mr-fox kernel:  <NMI>
May 26 08:45:01 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:45:01 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:45:01 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:45:01 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:45:01 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:45:01 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:45:01 mr-fox kernel:  ? tcp_ack_update_rtt+0x1b0/0x350
May 26 08:45:01 mr-fox kernel:  ? tcp_ack_update_rtt+0x1b0/0x350
May 26 08:45:01 mr-fox kernel:  ? tcp_ack_update_rtt+0x1b0/0x350
May 26 08:45:01 mr-fox kernel:  </NMI>
May 26 08:45:01 mr-fox kernel:  <IRQ>
May 26 08:45:01 mr-fox kernel:  tcp_ack+0xe1d/0x14b0
May 26 08:45:01 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:45:01 mr-fox kernel:  tcp_rcv_established+0x146/0x6c0
May 26 08:45:01 mr-fox kernel:  ? sk_filter_trim_cap+0x40/0x220
May 26 08:45:01 mr-fox kernel:  tcp_v4_do_rcv+0x153/0x240
May 26 08:45:01 mr-fox kernel:  tcp_v4_rcv+0xe00/0xea0
May 26 08:45:01 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:45:01 mr-fox kernel:  ip_protocol_deliver_rcu+0x32/0x180
May 26 08:45:01 mr-fox kernel:  ip_local_deliver_finish+0x74/0xa0
May 26 08:45:01 mr-fox kernel:  ip_sublist_rcv_finish+0x7f/0x90
May 26 08:45:01 mr-fox kernel:  ip_sublist_rcv+0x176/0x1c0
May 26 08:45:01 mr-fox kernel:  ? ip_sublist_rcv+0x1c0/0x1c0
May 26 08:45:01 mr-fox kernel:  ip_list_rcv+0x138/0x170
May 26 08:45:01 mr-fox kernel:  __netif_receive_skb_list_core+0x2a5/0x2d0
May 26 08:45:01 mr-fox kernel:  netif_receive_skb_list_internal+0x1db/0x320
May 26 08:45:01 mr-fox kernel:  napi_gro_receive+0xcf/0x1b0
May 26 08:45:01 mr-fox kernel:  igb_poll+0x605/0x1370
May 26 08:45:01 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 08:45:01 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 08:45:01 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 08:45:01 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:45:01 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:45:01 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:45:01 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:45:01 mr-fox kernel:  </IRQ>
May 26 08:45:01 mr-fox kernel:  <TASK>
May 26 08:45:01 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:45:01 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:45:01 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:45:01 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000246
May 26 08:45:01 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:45:01 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:45:01 mr-fox kernel: RBP: ffff988d55f18ff8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:45:01 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:45:01 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:45:01 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:45:01 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:45:01 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:45:01 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:45:01 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:45:01 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:45:01 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:45:01 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:45:01 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:45:01 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:45:01 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:45:01 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:45:01 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:45:01 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:45:01 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:45:01 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:45:01 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:45:01 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:45:01 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:45:01 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:45:01 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:45:01 mr-fox kernel:  </TASK>
May 26 08:46:00 mr-fox crond[20139]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:46:00 mr-fox crond[20138]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:46:00 mr-fox crond[20140]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:46:00 mr-fox crond[20141]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:46:00 mr-fox CROND[20164]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:46:00 mr-fox CROND[20166]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:46:00 mr-fox CROND[20165]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:46:00 mr-fox CROND[20163]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:46:00 mr-fox CROND[20138]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:46:00 mr-fox CROND[20138]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:46:00 mr-fox CROND[20141]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:46:00 mr-fox CROND[20141]: pam_unix(crond:session): session closed for user root
May 26 08:46:00 mr-fox CROND[20140]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:46:00 mr-fox CROND[20140]: pam_unix(crond:session): session closed for user root
May 26 08:46:02 mr-fox CROND[12606]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:46:02 mr-fox CROND[12606]: pam_unix(crond:session): session closed for user root
May 26 08:47:00 mr-fox crond[21926]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:47:00 mr-fox crond[21928]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:47:00 mr-fox crond[21929]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:47:00 mr-fox crond[21925]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:47:00 mr-fox CROND[21939]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:47:00 mr-fox CROND[21940]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:47:00 mr-fox CROND[21941]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:47:00 mr-fox CROND[21942]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:47:00 mr-fox CROND[21925]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:47:00 mr-fox CROND[21925]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:47:00 mr-fox CROND[21929]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:47:00 mr-fox CROND[21929]: pam_unix(crond:session): session closed for user root
May 26 08:47:00 mr-fox CROND[21928]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:47:00 mr-fox CROND[21928]: pam_unix(crond:session): session closed for user root
May 26 08:47:02 mr-fox CROND[20139]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:47:02 mr-fox CROND[20139]: pam_unix(crond:session): session closed for user root
May 26 08:47:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:47:45 mr-fox kernel: rcu: \x0921-....: (2490214 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=1018371
May 26 08:47:45 mr-fox kernel: rcu: \x09(t=2490215 jiffies g=8794409 q=73773020 ncpus=32)
May 26 08:47:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:47:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:47:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:47:45 mr-fox kernel: RIP: 0010:xas_start+0x33/0x120
May 26 08:47:45 mr-fox kernel: Code: 48 8b 6f 18 48 89 e8 83 e0 03 0f 84 81 00 00 00 48 81 fd 05 c0 ff ff 76 06 48 83 f8 02 74 5d 48 8b 03 48 8b 73 08 48 8b 40 08 <48> 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 20 48 85
May 26 08:47:45 mr-fox kernel: RSP: 0018:ffffa401077e3930 EFLAGS: 00000213
May 26 08:47:45 mr-fox kernel: RAX: ffff988ac0754ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:47:45 mr-fox kernel: RDX: 0000000000000000 RSI: 000000004cd994a1 RDI: ffffa401077e3970
May 26 08:47:45 mr-fox kernel: RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
May 26 08:47:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:47:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:47:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:47:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:47:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:47:45 mr-fox kernel: PKRU: 55555554
May 26 08:47:45 mr-fox kernel: Call Trace:
May 26 08:47:45 mr-fox kernel:  <IRQ>
May 26 08:47:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:47:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:47:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:47:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:47:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:47:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:47:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:47:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:47:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:47:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:47:45 mr-fox kernel:  </IRQ>
May 26 08:47:45 mr-fox kernel:  <TASK>
May 26 08:47:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:47:45 mr-fox kernel:  ? xas_start+0x33/0x120
May 26 08:47:45 mr-fox kernel:  xas_load+0xe/0x60
May 26 08:47:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:47:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:47:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:47:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:47:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:47:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:47:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:47:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:47:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:47:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:47:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:47:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:47:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:47:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:47:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:47:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:47:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:47:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:47:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:47:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:47:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:47:45 mr-fox kernel:  </TASK>
May 26 08:47:45 mr-fox sSMTP[21706]: Creating SSL connection to host
May 26 08:47:45 mr-fox sSMTP[21706]: SSL connection using TLS_AES_256_GCM_SHA384
May 26 08:47:50 mr-fox sSMTP[21706]: Sent mail for root@zwiebeltoralf.de (221 www325.your-server.de closing connection) uid=0 username=root outbytes=4026
May 26 08:48:00 mr-fox crond[9366]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:48:00 mr-fox crond[9364]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:48:00 mr-fox crond[9365]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:48:00 mr-fox crond[9363]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:48:00 mr-fox CROND[9373]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:48:00 mr-fox CROND[9371]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:48:00 mr-fox CROND[9374]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:48:00 mr-fox CROND[9372]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:48:00 mr-fox CROND[9363]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:48:00 mr-fox CROND[9363]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:48:00 mr-fox CROND[9366]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:48:00 mr-fox CROND[9366]: pam_unix(crond:session): session closed for user root
May 26 08:48:00 mr-fox CROND[9365]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:48:00 mr-fox CROND[9365]: pam_unix(crond:session): session closed for user root
May 26 08:48:00 mr-fox CROND[21926]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:48:00 mr-fox CROND[21926]: pam_unix(crond:session): session closed for user root
May 26 08:48:01 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 2494064 jiffies s: 491905 root: 0x2/.
May 26 08:48:01 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:48:01 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:48:01 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:48:01 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:48:01 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:48:01 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:48:01 mr-fox kernel: RIP: 0010:__iommu_dma_unmap+0xc3/0x170
May 26 08:48:01 mr-fox kernel: Code: ff ff 48 21 c5 49 8b 85 e0 00 00 00 48 89 ea 48 85 c0 0f 95 44 24 30 e8 db b5 ff ff 48 39 e8 0f 85 9a 00 00 00 0f b6 44 24 30 <3c> 01 0f 87 66 8b 2d 00 a8 01 74 3e 4c 89 f1 48 89 ea 48 89 de 4c
May 26 08:48:01 mr-fox kernel: RSP: 0018:ffffa401005f0d48 EFLAGS: 00000246
May 26 08:48:01 mr-fox kernel: RAX: 0000000000000001 RBX: 00000000cf3fb000 RCX: 0000000000000000
May 26 08:48:01 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:48:01 mr-fox kernel: RBP: 0000000000001000 R08: 0000000000000000 R09: 0000000000000000
May 26 08:48:01 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff988ac2211010
May 26 08:48:01 mr-fox kernel: R13: ffff988ac1292400 R14: ffffa401005f0d50 R15: ffffa401005f0d68
May 26 08:48:01 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:48:01 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:48:01 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:48:01 mr-fox kernel: PKRU: 55555554
May 26 08:48:01 mr-fox kernel: Call Trace:
May 26 08:48:01 mr-fox kernel:  <NMI>
May 26 08:48:01 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:48:01 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:48:01 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:48:01 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:48:01 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:48:01 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:48:01 mr-fox kernel:  ? __iommu_dma_unmap+0xc3/0x170
May 26 08:48:01 mr-fox kernel:  ? __iommu_dma_unmap+0xc3/0x170
May 26 08:48:01 mr-fox kernel:  ? __iommu_dma_unmap+0xc3/0x170
May 26 08:48:01 mr-fox kernel:  </NMI>
May 26 08:48:01 mr-fox kernel:  <IRQ>
May 26 08:48:01 mr-fox kernel:  iommu_dma_unmap_page+0x43/0xa0
May 26 08:48:01 mr-fox kernel:  igb_poll+0x106/0x1370
May 26 08:48:01 mr-fox kernel:  ? free_unref_page_commit+0x8f/0x3b0
May 26 08:48:01 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:48:01 mr-fox kernel:  ? free_unref_page+0xd7/0x170
May 26 08:48:01 mr-fox kernel:  __napi_poll+0x28/0x1a0
May 26 08:48:01 mr-fox kernel:  net_rx_action+0x202/0x590
May 26 08:48:01 mr-fox kernel:  ? enqueue_hrtimer.isra.0+0x3b/0x60
May 26 08:48:01 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:48:01 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:48:01 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:48:01 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:48:01 mr-fox kernel:  </IRQ>
May 26 08:48:01 mr-fox kernel:  <TASK>
May 26 08:48:01 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:48:01 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:48:01 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:48:01 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:48:01 mr-fox kernel: RAX: ffff988d55f18ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:48:01 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:48:01 mr-fox kernel: RBP: ffff988d667606d8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:48:01 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:48:01 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:48:01 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:48:01 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:48:01 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:48:01 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:48:01 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:48:01 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:48:01 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:48:01 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:48:01 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:48:01 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:48:01 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:48:01 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:48:01 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:48:01 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:48:01 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:48:01 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:48:01 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:48:01 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:48:01 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:48:01 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:48:01 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:48:01 mr-fox kernel:  </TASK>
May 26 08:49:00 mr-fox CROND[9364]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:49:00 mr-fox CROND[9364]: pam_unix(crond:session): session closed for user root
May 26 08:49:00 mr-fox crond[23867]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:49:00 mr-fox crond[23866]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:49:00 mr-fox crond[23868]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:49:00 mr-fox CROND[23874]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:49:00 mr-fox CROND[23873]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:49:00 mr-fox crond[23869]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:49:00 mr-fox CROND[23872]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:49:00 mr-fox CROND[23875]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:49:00 mr-fox CROND[23866]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:49:00 mr-fox CROND[23866]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:49:00 mr-fox CROND[23869]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:49:00 mr-fox CROND[23869]: pam_unix(crond:session): session closed for user root
May 26 08:49:00 mr-fox CROND[23868]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:49:00 mr-fox CROND[23868]: pam_unix(crond:session): session closed for user root
May 26 08:49:28 mr-fox sshd[26503]: pam_unix(sshd:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:49:29 mr-fox sshd[26503]: pam_unix(sshd:session): session closed for user tinderbox
May 26 08:50:00 mr-fox sshd[7712]: pam_unix(sshd:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:50:00 mr-fox crond[9530]: pam_unix(crond:session): session opened for user torproject(uid=1001) by torproject(uid=0)
May 26 08:50:00 mr-fox crond[9531]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:50:00 mr-fox crond[9532]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:50:00 mr-fox CROND[9539]: (torproject) CMD (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:50:00 mr-fox crond[9533]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:50:00 mr-fox CROND[9540]: (tinderbox) CMD (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:50:00 mr-fox crond[9536]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:50:00 mr-fox crond[9537]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:50:00 mr-fox crond[9535]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:50:00 mr-fox CROND[9541]: (tinderbox) CMD (sleep 10; /opt/tb/bin/index.sh)
May 26 08:50:00 mr-fox CROND[9543]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:50:00 mr-fox CROND[9544]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:50:00 mr-fox CROND[9545]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:50:00 mr-fox CROND[9547]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:50:00 mr-fox CROND[9530]: (torproject) CMDEND (/opt/fuzz-utils/fuzz.sh -p -f -o 1 -t 1)
May 26 08:50:00 mr-fox CROND[9530]: pam_unix(crond:session): session closed for user torproject
May 26 08:50:00 mr-fox CROND[9533]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:50:00 mr-fox CROND[9533]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:50:00 mr-fox CROND[9537]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:50:00 mr-fox CROND[9537]: pam_unix(crond:session): session closed for user root
May 26 08:50:00 mr-fox CROND[9531]: (tinderbox) CMDEND (f=/tmp/replace_img.$(date +%s).$$.log; /opt/tb/bin/replace_img.sh &>$f; [[ -s $f ]] && cat $f; rm $f)
May 26 08:50:00 mr-fox CROND[9531]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:50:00 mr-fox CROND[9536]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:50:00 mr-fox CROND[9536]: pam_unix(crond:session): session closed for user root
May 26 08:50:01 mr-fox CROND[23867]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:50:01 mr-fox CROND[23867]: pam_unix(crond:session): session closed for user root
May 26 08:50:45 mr-fox kernel: rcu: INFO: rcu_sched self-detected stall on CPU
May 26 08:50:45 mr-fox kernel: rcu: \x0921-....: (2535218 ticks this GP) idle=6004/1/0x4000000000000000 softirq=11646198/11646198 fqs=1036877
May 26 08:50:45 mr-fox kernel: rcu: \x09(t=2535219 jiffies g=8794409 q=74685064 ncpus=32)
May 26 08:50:45 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:50:45 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:50:45 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:50:45 mr-fox kernel: RIP: 0010:xas_load+0x4d/0x60
May 26 08:50:45 mr-fox kernel: Code: 5d 31 d2 31 c9 31 f6 31 ff e9 fa bf 1a 00 0f b6 4b 10 48 8d 68 fe 38 48 fe 72 e4 48 89 ee 48 89 df e8 e7 fe ff ff 80 7d 00 00 <75> bf eb d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e
May 26 08:50:45 mr-fox kernel: RSP: 0018:ffffa401077e3950 EFLAGS: 00000206
May 26 08:50:45 mr-fox kernel: RAX: ffff988d55f18ffa RBX: ffffa401077e3970 RCX: 0000000000000000
May 26 08:50:45 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:50:45 mr-fox kernel: RBP: ffff988d667606d8 R08: 0000000000000000 R09: 0000000000000000
May 26 08:50:45 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:50:45 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:50:45 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:50:45 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:50:45 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:50:45 mr-fox kernel: PKRU: 55555554
May 26 08:50:45 mr-fox kernel: Call Trace:
May 26 08:50:45 mr-fox kernel:  <IRQ>
May 26 08:50:45 mr-fox kernel:  ? rcu_dump_cpu_stacks+0x107/0x1e0
May 26 08:50:45 mr-fox kernel:  ? rcu_sched_clock_irq+0x3db/0xba0
May 26 08:50:45 mr-fox kernel:  ? update_process_times+0x6f/0xb0
May 26 08:50:45 mr-fox kernel:  ? tick_nohz_highres_handler+0x94/0xb0
May 26 08:50:45 mr-fox kernel:  ? tick_sched_do_timer+0x80/0x80
May 26 08:50:45 mr-fox kernel:  ? __hrtimer_run_queues+0xe8/0x170
May 26 08:50:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:50:45 mr-fox kernel:  ? hrtimer_interrupt+0xf3/0x250
May 26 08:50:45 mr-fox kernel:  ? __sysvec_apic_timer_interrupt+0x42/0xa0
May 26 08:50:45 mr-fox kernel:  ? sysvec_apic_timer_interrupt+0x65/0x80
May 26 08:50:45 mr-fox kernel:  </IRQ>
May 26 08:50:45 mr-fox kernel:  <TASK>
May 26 08:50:45 mr-fox kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:50:45 mr-fox kernel:  ? xas_load+0x4d/0x60
May 26 08:50:45 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:50:45 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:50:45 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:50:45 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:50:45 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:50:45 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:50:45 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:50:45 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:50:45 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:50:45 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:50:45 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:50:45 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:50:45 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:50:45 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:50:45 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:50:45 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:50:45 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:50:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:50:45 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:50:45 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:50:45 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:50:45 mr-fox kernel:  </TASK>
May 26 08:50:45 mr-fox sSMTP[4770]: Creating SSL connection to host
May 26 08:50:45 mr-fox sSMTP[4770]: SSL connection using TLS_AES_256_GCM_SHA384
May 26 08:50:50 mr-fox sSMTP[4770]: Sent mail for root@zwiebeltoralf.de (221 www325.your-server.de closing connection) uid=0 username=root outbytes=4032
May 26 08:51:00 mr-fox crond[24919]: pam_unix(crond:session): session opened for user tinderbox(uid=1003) by tinderbox(uid=0)
May 26 08:51:00 mr-fox crond[24922]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:51:00 mr-fox crond[24921]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:51:00 mr-fox crond[24923]: pam_unix(crond:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:51:00 mr-fox CROND[24925]: (tinderbox) CMD (/opt/tb/bin/logcheck.sh)
May 26 08:51:00 mr-fox CROND[24926]: (root) CMD (/opt/torutils/restart_service.sh)
May 26 08:51:00 mr-fox CROND[24928]: (root) CMD (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:51:00 mr-fox CROND[24927]: (root) CMD (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:51:00 mr-fox CROND[24919]: (tinderbox) CMDEND (/opt/tb/bin/logcheck.sh)
May 26 08:51:00 mr-fox CROND[24919]: pam_unix(crond:session): session closed for user tinderbox
May 26 08:51:00 mr-fox CROND[24923]: (root) CMDEND (/usr/lib/sa/sa1 1 1 -S XALL)
May 26 08:51:00 mr-fox CROND[24923]: pam_unix(crond:session): session closed for user root
May 26 08:51:00 mr-fox CROND[24922]: (root) CMDEND (/opt/torutils/restart_service.sh)
May 26 08:51:00 mr-fox CROND[24922]: pam_unix(crond:session): session closed for user root
May 26 08:51:01 mr-fox CROND[9535]: (root) CMDEND (for i in {0..3}; do /opt/torutils/metrics.sh &>/dev/null; sleep 15; done)
May 26 08:51:01 mr-fox CROND[9535]: pam_unix(crond:session): session closed for user root
May 26 08:51:01 mr-fox kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 21-.... } 2539119 jiffies s: 491905 root: 0x2/.
May 26 08:51:01 mr-fox kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x20/.
May 26 08:51:01 mr-fox kernel: Sending NMI from CPU 27 to CPUs 21:
May 26 08:51:01 mr-fox kernel: NMI backtrace for cpu 21
May 26 08:51:01 mr-fox kernel: CPU: 21 PID: 31911 Comm: kworker/u65:20 Tainted: G    B           T  6.8.11 #13
May 26 08:51:01 mr-fox kernel: Hardware name: ASUS System Product Name/Pro WS 565-ACE, BIOS 0502 01/15/2021
May 26 08:51:01 mr-fox kernel: Workqueue: btrfs-endio-write btrfs_work_helper
May 26 08:51:01 mr-fox kernel: RIP: 0010:__ip_queue_xmit+0x14a/0x480
May 26 08:51:01 mr-fox kernel: Code: 85 c9 0f 85 a4 00 00 00 f6 46 06 40 74 0d f6 83 80 00 00 00 08 0f 84 86 00 00 00 48 8b 3c 24 e8 fc 2d ff ff 8b 85 b0 01 00 00 <48> 8b 3c 24 48 89 da 48 89 ee 89 83 8c 00 00 00 8b 85 b4 01 00 00
May 26 08:51:01 mr-fox kernel: RSP: 0018:ffffa401005f0d80 EFLAGS: 00000202
May 26 08:51:01 mr-fox kernel: RAX: 0000000000000000 RBX: ffff989a4145f000 RCX: 0000000600010000
May 26 08:51:01 mr-fox kernel: RDX: 000000000002b45e RSI: ffff988a85fe1b8c RDI: 0000000000000000
May 26 08:51:01 mr-fox kernel: RBP: ffff988dd7ea4a00 R08: 0000000000000000 R09: 0000000000000000
May 26 08:51:01 mr-fox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000045
May 26 08:51:01 mr-fox kernel: R13: 0000000000000000 R14: ffff98979531d440 R15: ffff988dd7ea4d48
May 26 08:51:01 mr-fox kernel: FS:  0000000000000000(0000) GS:ffff98a96ed40000(0000) knlGS:0000000000000000
May 26 08:51:01 mr-fox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 26 08:51:01 mr-fox kernel: CR2: 000055f569daa8f0 CR3: 0000000412016000 CR4: 0000000000f50ef0
May 26 08:51:01 mr-fox kernel: PKRU: 55555554
May 26 08:51:01 mr-fox kernel: Call Trace:
May 26 08:51:01 mr-fox kernel:  <NMI>
May 26 08:51:01 mr-fox kernel:  ? nmi_cpu_backtrace+0x9f/0x110
May 26 08:51:01 mr-fox kernel:  ? nmi_cpu_backtrace_handler+0xc/0x20
May 26 08:51:01 mr-fox kernel:  ? nmi_handle+0x56/0x100
May 26 08:51:01 mr-fox kernel:  ? default_do_nmi+0x40/0x240
May 26 08:51:01 mr-fox kernel:  ? exc_nmi+0x10c/0x190
May 26 08:51:01 mr-fox kernel:  ? end_repeat_nmi+0xf/0x4e
May 26 08:51:01 mr-fox kernel:  ? __ip_queue_xmit+0x14a/0x480
May 26 08:51:01 mr-fox kernel:  ? __ip_queue_xmit+0x14a/0x480
May 26 08:51:01 mr-fox kernel:  ? __ip_queue_xmit+0x14a/0x480
May 26 08:51:01 mr-fox kernel:  </NMI>
May 26 08:51:01 mr-fox kernel:  <IRQ>
May 26 08:51:01 mr-fox kernel:  __tcp_transmit_skb+0xbad/0xd30
May 26 08:51:01 mr-fox kernel:  tcp_delack_timer_handler+0xa9/0x110
May 26 08:51:01 mr-fox kernel:  tcp_delack_timer+0xb5/0xf0
May 26 08:51:01 mr-fox kernel:  ? tcp_delack_timer_handler+0x110/0x110
May 26 08:51:01 mr-fox kernel:  call_timer_fn.isra.0+0x13/0xa0
May 26 08:51:01 mr-fox kernel:  __run_timers+0x20a/0x240
May 26 08:51:01 mr-fox kernel:  run_timer_softirq+0x27/0x60
May 26 08:51:01 mr-fox kernel:  __do_softirq+0xd1/0x2a4
May 26 08:51:01 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:51:01 mr-fox kernel:  irq_exit_rcu+0x83/0xa0
May 26 08:51:01 mr-fox kernel:  sysvec_apic_timer_interrupt+0x6a/0x80
May 26 08:51:01 mr-fox kernel:  </IRQ>
May 26 08:51:01 mr-fox kernel:  <TASK>
May 26 08:51:01 mr-fox kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May 26 08:51:01 mr-fox kernel: RIP: 0010:srso_alias_return_thunk+0x0/0xfbef5
May 26 08:51:01 mr-fox kernel: Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 48 8d 64 24 08 c3 cc <e8> f4 ff ff ff 0f 0b cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
May 26 08:51:01 mr-fox kernel: RSP: 0018:ffffa401077e3960 EFLAGS: 00000246
May 26 08:51:01 mr-fox kernel: RAX: ffffcf62c5775d40 RBX: ffffcf62c5775d40 RCX: 0000000000000000
May 26 08:51:01 mr-fox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 26 08:51:01 mr-fox kernel: RBP: ffff988ac61fdd70 R08: 0000000000000000 R09: 0000000000000000
May 26 08:51:01 mr-fox kernel: R10: ffffcf62f511df80 R11: 0000000000000000 R12: 0000000000000001
May 26 08:51:01 mr-fox kernel: R13: 0000000000000002 R14: ffff988ac80b7000 R15: ffff988ac61fdd70
May 26 08:51:01 mr-fox kernel:  filemap_get_entry+0x6a/0x160
May 26 08:51:01 mr-fox kernel:  __filemap_get_folio+0x34/0x2c0
May 26 08:51:01 mr-fox kernel:  alloc_extent_buffer+0x264/0xa00
May 26 08:51:01 mr-fox kernel:  read_tree_block+0x17/0xa0
May 26 08:51:01 mr-fox kernel:  read_block_for_search+0x252/0x370
May 26 08:51:01 mr-fox kernel:  btrfs_search_slot+0x371/0x1030
May 26 08:51:01 mr-fox kernel:  btrfs_lookup_csum+0x6e/0x170
May 26 08:51:01 mr-fox kernel:  btrfs_csum_file_blocks+0x1af/0x770
May 26 08:51:01 mr-fox kernel:  ? join_transaction+0x1e/0x470
May 26 08:51:01 mr-fox kernel:  ? btrfs_global_root+0x30/0x70
May 26 08:51:01 mr-fox kernel:  btrfs_finish_one_ordered+0x6c3/0xa40
May 26 08:51:01 mr-fox kernel:  btrfs_work_helper+0xb1/0x200
May 26 08:51:01 mr-fox kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
May 26 08:51:01 mr-fox kernel:  process_one_work+0x16a/0x280
May 26 08:51:01 mr-fox kernel:  worker_thread+0x281/0x3a0
May 26 08:51:01 mr-fox kernel:  ? flush_delayed_work+0x40/0x40
May 26 08:51:01 mr-fox kernel:  kthread+0xcb/0xf0
May 26 08:51:01 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:51:01 mr-fox kernel:  ret_from_fork+0x2f/0x50
May 26 08:51:01 mr-fox kernel:  ? kthread_complete_and_exit+0x20/0x20
May 26 08:51:01 mr-fox kernel:  ret_from_fork_asm+0x11/0x20
May 26 08:51:01 mr-fox kernel:  </TASK>
May 26 08:51:43 mr-fox sshd[17350]: pam_unix(sshd:session): session opened for user root(uid=0) by root(uid=0)
May 26 08:51:47 mr-fox sshd[17350]: pam_unix(sshd:session): session closed for user root
May 26 08:51:52 mr-fox sshd[30002]: pam_unix(sshd:session): session opened for user root(uid=0) by root(uid=0)