linux-kernel - [PATCH 0/2] sched: Fix "divide error: 0000" in find_busiest

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <4E25F002.2080503@hp.com>
Date:	Tue, 19 Jul 2011 14:58:42 -0600
From:	Terry Loftin <terry.loftin@...com>
To:	linux-kernel@...r.kernel.org, Ingo Molnar <mingo@...e.hu>,
	Peter Zijlstra <peterz@...radead.org>
CC:	Bob Montgomery <bob.montgomery@...com>
Subject: [PATCH 0/2] sched: Fix "divide error: 0000" in find_busiest_group

Howdy,

The divide error occurs in inlined function update_sg_lb_stats() in
kernel/sched.c when we adjust the relative CPU power of a group by
dividing group_load by group->cpu_power:

    /* Adjust by relative CPU power of the group */ sgs->avg_load =
    (sgs->group_load * SCHED_LOAD_SCALE) / group->cpu_power;

In this case, group->cpu_power is zero.  This was set in
update_cpu_power(), which depends on scale_rt_power() among other things.
scale_rt_power() is based in part on the rq->clock and rq->age_stamp
values for the runqueue:

    total = sched_avg_period() + (rq->clock - rq->age_stamp);

The clock and age_stamp values are in nanoseconds and come from
__cycles_2_ns() which converts the CPU tsc counter to nanoseconds.
On 64-bit systems, the computation returned from __cycles_2_ns() wraps
when the nanosecond value is 54 bits or larger (about 208.5 days).

The rq->age_stamp is designed to follow the clock value but does not
account for the fact that the clock value may wrap, and it is never reset.
After rq->clock wraps, the expression (rq->clock - rq->age_stamp) leads
to large negative values which in turn lead to very large values for
scale_rt_power().

In update_cpu_power(), an unsigned long local variable, 'power', is
used to hold the intermediate result, including the return value from
scale_rt_power(), before it is placed in an unsigned int rq->cpu_power.
If the power calculated in update_cpu_power() is > 32 bits, but all
the low order bits are zero, then the value will be truncated and
rq->cpu_power will be set to zero, leading to the divide by zero error.
There is a protective check immediately before the assignment, but it
compares the full 64-bit value instead of the 32-bit portion that will
be stored in rq->cpu_power.

I have analyzed two crash dumps from systems that were up 220 and 230
days to confirm this.

-T

Signed-off-by: Terry Loftin <terry.loftin@...com>
Signed-off-by: Bob Montgomery <bob.montgomery@...com>

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/