linux-kernel - Is it normal for the user and sys times in /proc/stat to move backwards with NO

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <d38f7ae8-d10e-428d-b72f-a1fb490a45ab@gmail.com>
Date: Sat, 16 Aug 2025 16:44:51 +0200
From: Thomas Lindroth <thomas.lindroth@...il.com>
To: linux-fsdevel@...r.kernel.org
Cc: Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Is it normal for the user and sys times in /proc/stat to move
 backwards with NO_HZ?

I have noticed that some of the values in /proc/stat, such as user, nice, and sys time,
can decrease on subsequent reads. The file Documentation/filesystems/proc.rst states:
"The value of iowait field in /proc/stat will decrease in certain conditions".
Since the documentation specifically mentions iowait I assume that the other values
should not decrease.

A simple way to test this is by running:
while true; do grep -F "cpu1 " /proc/stat;done| awk 'NR>1{diff=$2-p;if(diff<0)print diff}{p=$2}'

I believe this issue is related to the use of NO_HZ. I use nohz_full=1-23 on an
Intel i7-13700K CPU with 24 logical cores. I have never seen decreasing values on cpu0,
which is the housekeeping CPU with ticks always enabled. I have seen decreasing user,
nice, and sys times on all other cores on kernel 6.12.38 and an older 6.1.140 kernel.

I boot the kernel with the following arguments:
nohz_full=1-23 rcu_nocbs=1-23 intel_iommu=on kvm.ignore_msrs=1 kvm-intel.nested=0
kvm.report_ignored_msrs=0 vfio-pci.disable_vga=1 split_lock_detect=off

If this is expected behaviour it would be helpful if the documentation could be updated
to better clarify this.

/Thomas Lindroth