Message-ID: <CAK-9PRC3HhM5cfWuByv5bWvzaV7w3e6Djwv_WfXK2mvs=9V6JQ@mail.gmail.com>
Date:	Mon, 7 Sep 2015 11:00:10 +0530
From:	Chinmay V S <cvs268@...il.com>
To:	linux-kernel <linux-kernel@...r.kernel.org>,
	linux-smp@...r.kernel.org, stable-rt@...r.kernel.org
Subject: RT Scheduler - BUG_ON (idx >= MAX_RT_PRIO)

Hello everyone,

TL;DR: In Linux RT scheduler, how can rt_nr_running be non-zero AND
active-bitmap NOT have any valid bit set?

Details:
Recently i encountered the following BUG() within the realtime
scheduler (sched_rt.c) on 3.1.10 kernel.
[101640.492840] kernel BUG at kernel/sched_rt.c:1126!

This turns out to be
1126         BUG_ON(idx >= MAX_RT_PRIO);

within the function pick_next_rt_entity() as shown here:
http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/kernel/sched_rt.c?h=linux-3.1.y#n1115

What this means is that the scheduler failed to find a valid bit
within the bitmap containing a prioritised list of active tasks.
However before attempting to parse the bitmap, there is a check for a
non-zero rt_nr_running.
(i.e. parsing the bitmap should find atleast 1 bit of the active
running rt task)

So how could rt_nr_running be non-zero AND active-bitmap NOT have any
valid bit set?

The issue is observed on
- a quad-core Cortex A9 SMP embedded system.
- running an userspace app with ~25 RT threads (FIFO and RR)
- typical ubuntu-core rootfs

This issue consistently reproduces within 24-48hours on continuously
running the system.

Searching the net/lkml i could not find this issue reported before,
though there are a few memory corruption bugs in scheduler.
I have already backported the patches to fix know memory corruption
issue from upstream kernel version and still encounter the above
BUG().

Is anyone aware of this issue?

Also including the kernel OOPS below.
Do you see any tell-tale signs in the register-dump/backtrace that can
point me in the right direction?

[101640.488133] ------------[ cut here ]------------
[101640.492840] kernel BUG at kernel/sched_rt.c:1126!
[101640.497621] Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP
[101640.504742] Modules linked in: misc_arz(P) audio_sta3_2(P)
audio_sta3_1(P) audio_sta3(P) lamp_tlc8116_2(P) lamp_tlc8116_1(P)
lamp_tlc8116(P) i2c_master_pcu9669(P) tegra_gpio_helper(P)
outport_timer(P) intTimer(P) nvidia(P)
[101640.524618] CPU: 0    Tainted: P             (3.1.10 #1)
[101640.530015] PC is at pick_next_task_rt+0x138/0x140
[101640.534888] LR is at __schedule+0x63c/0x858
[101640.539150] pc : [<c0042900>]    lr : [<c0435190>]    psr: 200f0093
[101640.539154] sp : e2d7fcc8  ip : e2d7fce8  fp : e2d7fce4
[101640.550785] r10: c05a4e60  r9 : e21f6d4c  r8 : c05c8ab0
[101640.556084] r7 : e2d7e000  r6 : 00000001  r5 : c186fe60  r4 : c043f928
[101640.562685] r3 : c186ff60  r2 : 00000064  r1 : fffffff0  r0 : c186fe60
[101640.569288] Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM
Segment user
[101640.576582] Control: 10c5387d  Table: a335804a  DAC: 00000015
[101640.582400]
[101640.582403] PC: 0xc0042880:
[101640.586839] 2880  eaffffeb e5932008 e3520000 0a000015 e2621000
e0012002 e16f2f12 e262205f
[101640.595143] 28a0  eaffffe3 e30034b8 e2406f5a e18320d5 e3560000
e1c625f8 0affffc9 e2854d11
[101640.603449] 28c0  e2800070 e2844008 e1a01004 eb06b771 e5953448
e1a00006 e0534004 13a04001
[101640.611753] 28e0  e58544d4 e89da878 e593200c e2621000 e0012002
e16f2f12 e262207f eaffffcc
[101640.620057] 2900  e7f001f2 e7f001f2 e1a0c00d e92ddff0 e24cb004
e24dd01c e2914038 e1a08002
[101640.628361] 2920  0a000074 e5913054 e3530000 1a000068 e30ba720
e3a09e4b e34ca05e e50ba038
[101640.636665] 2940  e2083005 e5945124 e3530001 0a000004 e1c423d0
e1c501d8 e0922000 e0a33001
[101640.644969] 2960  e1c423f0 e1a00005 ebffee88 e3a01000 e1a00005
ebfff0b7 e5943000 e3a01000
[101640.653276]
[101640.653278] LR: 0xc0435110:
[101640.657713] 5110  e1932f9f e2822001 e1831f92 e3310000 1afffffa
eaffff5d e1a00008 eb000947
[101640.666017] 5130  e51b004c eb0009bc e1a00008 eb0009ba eaffffde
e1a00008 e1a01009 e3a02001
[101640.674323] 5150  ebf032e2 eaffffdf e59f3248 e1a08007 e51ba070
e1a07006 e1a06005 e1a05004
[101640.682627] 5170  e1a04003 ea000002 e5944000 e3540000 0a000086
e5943018 e1a00005 e12fff33
[101640.690931] 5190  e3500000 0afffff7 e1a04005 e50ba070 e1a05006
e1a0a000 e1a06007 e1a07008
[101640.699235] 51b0  eafffeb5 e1a00004 eb0008fc eaffff48 e3000518
e3071ff4 e18420d0 e3a00e4b
[101640.707539] 51d0  e34c105c e591104c e14b26fc e18420d0 e1a00001
e3a01000 e14b23fc e3083080
[101640.715845] 51f0  e34c305a e50b304c e593c000 e14b26dc e1530001
01520000 e14b03dc e3a03e51
[101640.724152]
[101640.724154] SP: 0xe2d7fc48:
[101640.728589] fc48  00000000 e21f6aa0 2c5bae94 00000000 e2d7fc74
e2d7fc68 c0042904 200f0093
[101640.736893] fc68  c000e394 00000000 e2d7fce4 e2d7fc80 c000e0c8
c00081a0 c186fe60 fffffff0
[101640.745197] fc88  00000064 c186ff60 c043f928 c186fe60 00000001
e2d7e000 c05c8ab0 e21f6d4c
[101640.753501] fca8  c05a4e60 e2d7fce4 e2d7fce8 e2d7fcc8 c0435190
c0042900 200f0093 ffffffff
[101640.761807] fcc8  c00427c8 c043f928 c186fe60 e21f6aa0 e2d7fd64
e2d7fce8 c0435190 c00427d4
[101640.770111] fce8  e39de688 00000031 00000001 c05a4e60 00062904
00000000 e2d7fd54 e2d7fd10
[101640.778415] fd08  c00404f0 c003e308 c05a4e60 c05a4e60 c05a8080
00005c71 c05a40c4 c05a4e60
[101640.786721] fd28  05303bf8 00005c71 00000000 e39de520 00000001
e2d7fe44 00000000 e2d7e038
[101640.795027]
[101640.795030] IP: 0xe2d7fc68:
[101640.799465] fc68  c000e394 00000000 e2d7fce4 e2d7fc80 c000e0c8
c00081a0 c186fe60 fffffff0
[101640.807770] fc88  00000064 c186ff60 c043f928 c186fe60 00000001
e2d7e000 c05c8ab0 e21f6d4c
[101640.816076] fca8  c05a4e60 e2d7fce4 e2d7fce8 e2d7fcc8 c0435190
c0042900 200f0093 ffffffff
[101640.824382] fcc8  c00427c8 c043f928 c186fe60 e21f6aa0 e2d7fd64
e2d7fce8 c0435190 c00427d4
[101640.832686] fce8  e39de688 00000031 00000001 c05a4e60 00062904
00000000 e2d7fd54 e2d7fd10
[101640.840990] fd08  c00404f0 c003e308 c05a4e60 c05a4e60 c05a8080
00005c71 c05a40c4 c05a4e60
[101640.849296] fd28  05303bf8 00005c71 00000000 e39de520 00000001
e2d7fe44 00000000 e2d7e038
[101640.857600] fd48  e2d7fe44 c064b020 e2d7fe7c e2d7e000 e2d7fd74
e2d7fd68 c04354fc c0434b60
[101640.865904]
[101640.865907] FP: 0xe2d7fc64:
[101640.870342] fc64  200f0093 c000e394 00000000 e2d7fce4 e2d7fc80
c000e0c8 c00081a0 c186fe60
[101640.878646] fc84  fffffff0 00000064 c186ff60 c043f928 c186fe60
00000001 e2d7e000 c05c8ab0
[101640.886952] fca4  e21f6d4c c05a4e60 e2d7fce4 e2d7fce8 e2d7fcc8
c0435190 c0042900 200f0093
[101640.895256] fcc4  ffffffff c00427c8 c043f928 c186fe60 e21f6aa0
e2d7fd64 e2d7fce8 c0435190
[101640.903560] fce4  c00427d4 e39de688 00000031 00000001 c05a4e60
00062904 00000000 e2d7fd54
[101640.911866] fd04  e2d7fd10 c00404f0 c003e308 c05a4e60 c05a4e60
c05a8080 00005c71 c05a40c4
[101640.920171] fd24  c05a4e60 05303bf8 00005c71 00000000 e39de520
00000001 e2d7fe44 00000000
[101640.928475] fd44  e2d7e038 e2d7fe44 c064b020 e2d7fe7c e2d7e000
e2d7fd74 e2d7fd68 c04354fc
[101640.936781]
[101640.936784] R0: 0xc186fde0:
[101640.941219] fde0  00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[101640.949521] fe00  00000000 00000000 00000000 00000000 0001125e
0000008d 00000000 00000000
[101640.957827] fe20  00000091 00000000 00000000 0000370a 00000004
00000000 00000000 00000000
[101640.966130] fe40  00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[101640.974436] fe60  00000001 00000001 00000000 00000000 00000060
000000bd 000000a8 0000009c
[101640.982741] fe80  009aa1e9 00000000 00000000 00000000 00000000
00000000 05303bf8 00005c71
[101640.991045] fea0  00000776 1f39f5f4 00000000 00000000 009a2769
00000000 0bf2a4f2 00000000
[101640.999350] fec0  00000000 00000000 00000000 00000000 4d4a2fa0
00000093 f116833c 00000091
[101641.007657]
[101641.007659] R3: 0xc186fee0:
[101641.012094] fee0  f116833c 00000091 00000000 00000000 c186fef0
c186fef0 00000000 00000000
[101641.020397] ff00  00000000 00000000 00000000 00000288 c186fe60
00000001 c18702e8 c18702e8
[101641.028701] ff20  c0617a50 00000000 00000400 00000000 00000000
00000000 00000000 00000000
[101641.037003] ff40  00000001 00000000 00000000 00000000 4d4a2fa0
00000093 00000000 00000000
[101641.045307] ff60  00000000 00000000 00000000 00000010 c186ff70
c186ff70 c186ff78 c186ff78
[101641.053611] ff80  c186ff80 c186ff80 c186ff88 c186ff88 c186ff90
c186ff90 c186ff98 c186ff98
[101641.061915] ffa0  c186ffa0 c186ffa0 c186ffa8 c186ffa8 c186ffb0
c186ffb0 c186ffb8 c186ffb8
[101641.070219] ffc0  c186ffc0 c186ffc0 c186ffc8 c186ffc8 c186ffd0
c186ffd0 c186ffd8 c186ffd8
[101641.078525]
[101641.078527] R4: 0xc043f8a8:
[101641.082963] f8a8  0004624c 00057ff5 0006dd25 000899bc 000abe5d
000d5d21 0010c73e 0014ff97
[101641.091267] f8c8  001a3434 0020ea87 00295252 003351fe 00400000
004fec05 00640e12 007c97d9
[101641.099573] f8e8  009aee73 00c3a13e 00f0f0f1 0130d190 017d05f4
01de5d6e 0253c825 02f14990
[101641.107877] f908  03a83a84 04924924 05b05b05 071c71c7 08d3dcb0
0b21642c 0e38e38e 11111111
[101641.116183] f928  c043f728 c0040f1c c00409e8 c0042dd8 00000000
c0042df4 c00427c8 c0040e94
[101641.124486] f948  c003f9d8 c00484ac c0048878 00000000 c0048800
c0040fa8 c003fb3c c0040094
[101641.132790] f968  c003fb0c c0042e98 00000000 c0048480 c004873c
c00483ec c003c1e0 00000000
[101641.141094] f988  00000000 00000000 c050e51c c04f9d3c c04f9d48
00000000 00000063 00000063
[101641.149398]
[101641.149401] R5: 0xc186fde0:
[101641.153835] fde0  00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[101641.162140] fe00  00000000 00000000 00000000 00000000 0001125e
0000008d 00000000 00000000
[101641.170443] fe20  00000091 00000000 00000000 0000370a 00000004
00000000 00000000 00000000
[101641.178747] fe40  00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[101641.187047] fe60  00000001 00000001 00000000 00000000 00000060
000000bd 000000a8 0000009c
[101641.195352] fe80  009aa1e9 00000000 00000000 00000000 00000000
00000000 05303bf8 00005c71
[101641.203655] fea0  00000776 1f39f5f4 00000000 00000000 009a2769
00000000 0bf2a4f2 00000000
[101641.211958] fec0  00000000 00000000 00000000 00000000 4d4a2fa0
00000093 f116833c 00000091
[101641.220262]
[101641.220264] R7: 0xe2d7df80:
[101641.224697] df80  1afffd0b e1a00004 eb178742 e3500000 059f3428
05853498 eafffd05 e1a00004
[101641.233001] dfa0  eb1785c4 e3500000 1afffcfd e1a00004 eb1785d2
e3500000 1afffcf9 e1a00004
[101641.241307] dfc0  eb1785ff e3500000 1afffcf5 e1a00004 eb178616
e3500000 1afffcf1 e1a00004
[101641.249611] dfe0  eb178624 e3500000 1afffced e1a00004 eb17866c
e3500000 1afffce9 e1a00004
[101641.257915] e000  00000000 00000004 00000000 e21f6aa0 c05ebdd4
00000000 00000015 c186fe60
[101641.266220] e020  e21f6aa0 e2d7e000 c05c8ab0 e3221580 00000000
c05cb3f0 e2d7fd64 e2d7fce8
[101641.274526] e040  c0434ec0 00000000 00000000 00000000 00000000
00000000 01010000 00000000
[101641.282831] e060  48b19900 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[101641.291135]
[101641.291137] R8: 0xc05c8a30:
[101641.295572] 8a30  00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[101641.303878] 8a50  00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[101641.312184] 8a70  00000064 00008000 0000000f c188e020 00000008
00000004 c188e000 c1886000
[101641.320490] 8a90  c186f000 0000002c c1876000 c187e000 00000001
00001000 00000000 00000003
[101641.328796] 8ab0  012cb000 012d3000 012db000 012e3000 0005af00
00000001 000a3ffb 000ffeff
[101641.337102] 8ad0  00000000 00000032 0000fffa 00000001 00000000
00000001 e3800f60 e381d580
[101641.345406] 8af0  e392ec60 00000011 c1892120 0001ffff e3800960
e38009c0 00000064 00000010
[101641.353710] 8b10  c1912120 0000ffff e3800900 00100000 e381a000
00000001 e3800f00 00000001
[101641.362016]
[101641.362019] R9: 0xe21f6ccc:
[101641.366455] 6ccc  e339e1e0 e339e1e0 e21f6cd4 e21f6cd4 e21f6cdc
e21f6cdc e300f1a0 e21f6ce8
[101641.374759] 6cec  e21f6ce8 e21f6cf0 e21f6cf0 00000000 e39e3328
e39e3320 00000000 e38297ec
[101641.383065] 6d0c  e38297e0 e339e450 e39e34b0 e39e34a0 e339f8fc
e21f755c 00000000 00000000
[101641.391371] 6d2c  48b194a8 0008834d 000178b7 0008834d 000178b7
00000000 00000000 00000000
[101641.399675] 6d4c  016b5f5b 002bbe0a 00000078 37e1e0b2 00000078
37e1e0b2 0000001e 00000000
[101641.407979] 6d6c  00000000 00000000 00000000 00000000 00000000
e21f6d80 e21f6d80 e21f6d88
[101641.416285] 6d8c  e21f6d88 e21f6d90 e21f6d90 e320f200 e320f200
00000000 64616f4c 652e7265
[101641.424589] 6dac  0000666c 00000000 00000000 00000000 e31ab280
00000000 00000000 00000000
[101641.432893]
[101641.432896] R10: 0xc05a4de0:
[101641.437417] 4de0  e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0
e7fddef0 e7fddef0 e7fddef0
[101641.445723] 4e00  e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0
e7fddef0 e7fddef0 e7fddef0
[101641.454027] 4e20  e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0
e7fddef0 e7fddef0 e7fddef0
[101641.462333] 4e40  e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0
e7fddef0 e7fddef0 e7fddef0
[101641.470639] 4e60  e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0
e7fddef0 e7fddef0 e7fddef0
[101641.478945] 4e80  e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0
e7fddef0 e7fddef0 e7fddef0
[101641.487251] 4ea0  e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0
e7fddef0 e7fddef0 e7fddef0
[101641.495555] 4ec0  e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0
e7fddef0 e7fddef0 e7fddef0
[101641.503864] Process Loader.elf (pid: 571, stack limit = 0xe2d7e2f0)
[101641.510205] Stack: (0xe2d7fcc8 to 0xe2d80000)
[101641.514642] fcc0:                   c00427c8 c043f928 c186fe60
e21f6aa0 e2d7fd64 e2d7fce8
[101641.522895] fce0: c0435190 c00427d4 e39de688 00000031 00000001
c05a4e60 00062904 00000000
[101641.531146] fd00: e2d7fd54 e2d7fd10 c00404f0 c003e308 c05a4e60
c05a4e60 c05a8080 00005c71
[101641.539397] fd20: c05a40c4 c05a4e60 05303bf8 00005c71 00000000
e39de520 00000001 e2d7fe44
[101641.547648] fd40: 00000000 e2d7e038 e2d7fe44 c064b020 e2d7fe7c
e2d7e000 e2d7fd74 e2d7fd68
[101641.555899] fd60: c04354fc c0434b60 e2d7fdc4 e2d7fd78 c007dccc
c04354b8 e2d7fdc4 e2d7fd88
[101641.564150] fd80: c007e710 e2d7fe50 c003e048 00000002 00000001
00000002 e2d7fdb4 e2d7e000
[101641.572400] fda0: 00000000 00000000 e2d7fe44 00000000 e2d7fe7c
481df054 e2d7feac e2d7fdc8
[101641.580650] fdc0: c007e934 c007dbe4 e2d7fe7c c0094c4c 00000000
c186fe60 00000000 00000002
[101641.588900] fde0: c187fe60 c05c8ab0 e39de8b8 00000002 00000000
c05a4e60 e2d7fe44 e2d7fe08
[101641.597152] fe00: c004c24c c043760c c007dd90 c05a4e60 600f0013
c05a4e60 00000000 e39de520
[101641.605405] fe20: e39de528 e3191e44 c064b01c c064b028 e2d7e000
c064b020 e2d7fe74 e2d7fe48
[101641.613656] fe40: c007e57c 00000031 e2d7fe48 e2d7fe48 e2d7fe50
e2d7fe50 e21f6aa0 00000000
[101641.621906] fe60: 481df000 e3221580 00000054 00000000 00000000
00000000 ffffffff c064b020
[101641.630158] fe80: e3221580 00000000 481df054 00000002 00000002
00000000 e2d7e000 481df054
[101641.638408] fea0: e2d7ff44 e2d7feb0 c00801c8 c007e78c ffffffff
e2d7ff50 00000000 c061404c
[101641.646658] fec0: ffffffff e10a4000 ffffffff 00000000 e2d7ff04
e2d7fee0 c0072a40 c0094c4c
[101641.654910] fee0: 00000000 e21f6c78 e10a401c e21f6aa0 e2d7e000
c05c8ab0 e2d7ff1c e2d7ff08
[101641.663160] ff00: c0072a6c c0072a04 0126113e 00000000 3800038a
00000000 e2d7ff74 00000000
[101641.671412] ff20: 00000080 481df054 00000002 00000000 e2d7e000
00000000 e2d7ffa4 e2d7ff48
[101641.679662] ff40: c0080be0 c00800e8 481df054 00000000 ffffffff
6553f400 00000107 c000e604
[101641.687912] ff60: e2d7e000 481f1b8c 6553f400 00000107 c000e604
48b18d08 00000008 481df054
[101641.696162] ff80: 00000002 00000001 000000f0 c000e604 e2d7e000
00000000 00000000 e2d7ffa8
[101641.704414] ffa0: c000e480 c0080b5c 481df054 00000002 481df054
00000080 00000002 00000000
[101641.712664] ffc0: 481df054 00000002 00000001 000000f0 00000080
00000001 00000000 48b19674
[101641.720914] ffe0: 000000f0 48b18cd0 40103f53 40106386 600f0030
481df054 00000000 00000000
[101641.729161] Backtrace:
[101641.731703] [<c00427c8>] (pick_next_task_rt+0x0/0x140) from
[<c0435190>] (__schedule+0x63c/0x858)
[101641.740643]  r6:e21f6aa0 r5:c186fe60 r4:c043f928 r3:c00427c8
[101641.746409] [<c0434b54>] (__schedule+0x0/0x858) from [<c04354fc>]
(schedule+0x50/0x68)
[101641.754405] [<c04354ac>] (schedule+0x0/0x68) from [<c007dccc>]
(futex_wait_queue_me+0xf4/0x108)
[101641.763177] [<c007dbd8>] (futex_wait_queue_me+0x0/0x108) from
[<c007e934>] (futex_wait+0x1b4/0x27c)
[101641.772298] [<c007e780>] (futex_wait+0x0/0x27c) from [<c00801c8>]
(do_futex+0xec/0xa74)
[101641.780376] [<c00800dc>] (do_futex+0x0/0xa74) from [<c0080be0>]
(sys_futex+0x90/0x188)
[101641.788372] [<c0080b50>] (sys_futex+0x0/0x188) from [<c000e480>]
(ret_fast_syscall+0x0/0x30)
[101641.796883] Code: e0012002 e16f2f12 e262207f eaffffcc (e7f001f2)


regards
CVS
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists