Date:	Thu, 21 Jun 2012 20:09:11 +0530
From:	satpal parmar <systems.satpal@...il.com>
To:	netdev@...r.kernel.org
Subject: PTP + H/W time stamping + CONFIG_PREEMPT_RT_FULL == Oops in slab

Hi  All!

Recently  I successfully  integrated a certain
(http://sourceforge.net/scm/?type=svn&group_id=503885) open source
implementation of PTP protocol with Linux  3.0.1 running on ARM based
SoC. Since Phy  (Phyter DP 83640) on board  supported  H/W time
stamping support I am using for that for the same. Integration worked
smoothly  until I applied RT patch and enabled full preemption
(CONFIG_PREEMPT_RT_FULL).

After applying RT patch and enabling full preemption  I start getting
oops in slab allocator. I did some goggling  but got mixed results.
Some of the articles says slabs are not stable with RT patch while the
code comments in slab allocator says its preemption safe. Since I am
getting this oops only while running PTP client I am not certain if
its slab corruption issue (network traffic running smoothly).  I did
little debugging and I figure out that If I comment out h/w time
stamping in TX buy commenting out  skb_tx_timestamp from  network
drivers ndo_start_xmit function I am not getting any oops. Moreover if
I run ptp as high priority rt thread ,again I am not egtting nay oops.
Now I am got little confused as there are too many black boxes to
debug.  With RT patch, slabs, network driver, PTP client  I am not
sure which way to go. I am just wondering If anyone tried what I am
doing and came across similar issues. Or if anyone well verse in
either of these subsystem can help me eliminate some of the
possibilities.

I am not sure if this is the right mailing list. If anyone know better
forum please direct me to that.  Appreciate your patience and time.

-Satpal

Logs:

1. With CONFIG_PREEMPT_RT_FULL=y

tarting kernel ...

Uncompressing Linux... done, booting the kernel.
Linux version 3.0.1-rt11-svn7595 (satpal@...-Server) (gcc version
4.3.3 (Sourcery G++ Lite 2009q1-203) ) #29 PREEMPT RT Thu Jun 21
19:47:06 IST 2012
CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c53c7f
CPU: VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine: picodmb
Truncating RAM at 80000000-bfffffff to -afffffff (vmalloc region overlap).
reserved size = 0 at 0
Memory policy: ECC disabled, Data cache writeback
OMAP chip is TI8148
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 195072
Kernel command line: console=ttyO0,115200n8 mem=1024MB root=/dev/ram
rw initrd=0x82000000,20MB ramdisk_size=131072 earlyprintk
mtdparts=physmap-flash.0:512k@0(bdp0)ro,512k(bdp1)ro,128k(env0)ro,128k(env1)ro,2M(Kernel0),2M(Kernel1),10M(rootfs0),10M(rootfs1),10M(logs),10M(rw-fs),512k(Factory)ro,128k(post),1M(misc),-(FreeNOR)
run_app=no
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 768MB = 768MB total
Memory: 754044k/754044k available, 32388k reserved, 0K highmem
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
    DMA     : 0xffc00000 - 0xffe00000   (   2 MB)
    vmalloc : 0xf0800000 - 0xf8000000   ( 120 MB)
    lowmem  : 0xc0000000 - 0xf0000000   ( 768 MB)
    modules : 0xbf000000 - 0xc0000000   (  16 MB)
      .init : 0xc0008000 - 0xc0032000   ( 168 kB)
      .text : 0xc0032000 - 0xc045f000   (4276 kB)
      .data : 0xc0460000 - 0xc04984a0   ( 226 kB)
       .bss : 0xc04984c4 - 0xc04cb470   ( 204 kB)
Preemptible hierarchical RCU implementation.
        Verbose stalled-CPUs detection is disabled.
NR_IRQS:375
IRQ: Found an INTC at 0xfa200000 (revision 5.0) with 128 interrupts
Total of 128 interrupts on 1 active controller
OMAP clockevent source: GPTIMER1 at 20000000 Hz
Console: colour dummy device 80x30
Calibrating delay loop... 597.60 BogoMIPS (lpj=2988032)
pid_max: default: 32768 minimum: 301
Security Framework initialized
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
devtmpfs: initialized
omap_hwmod: _populate_mpu_rt_base found no _mpu_rt_va for l3_slow
omap_hwmod: _populate_mpu_rt_base found no _mpu_rt_va for l4_slow
print_constraints: dummy:
NET: Registered protocol family 16
GPMC revision 6.0
Trying to set irq flags for IRQ375
ahci : Failed to get AHCI clock
Registered ti81xx_fb device
NSS Crypto DMA hardware revision 1.9 @ IRQ 116
bio: create slab <bio-0> at 0
omap_i2c omap_i2c.1: bus 1 rev4.0 at 100 kHz
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti
<giometti@...ux.it>
PTP clock support registered
Switching to clocksource gp timer
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
TCP bind hash table entries: 65536 (order: 8, 1572864 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
UDP hash table entries: 512 (order: 3, 32768 bytes)
UDP-Lite hash table entries: 512 (order: 3, 32768 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Trying to unpack rootfs image as initramfs...
rootfs image is not initramfs (no cpio magic); looks like an initrd
Freeing initrd memory: 20480K
NetWinder Floating Point Emulator V0.97 (double precision)
omap-iommu omap-iommu.0: ducati registered
omap-iommu omap-iommu.1: sys registered
JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
msgmni has been set to 1512
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
omap_uart.0: ttyO0 at MMIO 0x48020000 (irq = 72) is a OMAP UART0
console [ttyO0] enabled
omap_uart.1: ttyO1 at MMIO 0x48022000 (irq = 73) is a OMAP UART1
omap_uart.2: ttyO2 at MMIO 0x48024000 (irq = 74) is a OMAP UART2
omap_uart.3: ttyO3 at MMIO 0x481a6000 (irq = 44) is a OMAP UART3
omap_uart.4: ttyO4 at MMIO 0x481a8000 (irq = 45) is a OMAP UART4
omap_uart.5: ttyO5 at MMIO 0x481aa000 (irq = 46) is a OMAP UART5
brd: module loaded
loop: module loaded
physmap platform flash device: 04000000 at 08000000
physmap-flash.0: Found 1 x16 devices at 0x0 in 16-bit bank.
Manufacturer ID 0x000089 Chip ID 0x008965
Intel/Sharp Extended Query Table at 0x010A
Intel/Sharp Extended Query Table at 0x010A
Intel/Sharp Extended Query Table at 0x010A
Intel/Sharp Extended Query Table at 0x010A
Intel/Sharp Extended Query Table at 0x010A
Using buffer write method
Using auto-unlock on power-up/resume
cfi_cmdset_0001: Erase suspend on write enabled
14 cmdlinepart partitions found on MTD device physmap-flash.0
Creating 14 MTD partitions on "physmap-flash.0":
0x000000000000-0x000000080000 : "bdp0"
0x000000080000-0x000000100000 : "bdp1"
0x000000100000-0x000000120000 : "env0"
0x000000120000-0x000000140000 : "env1"
0x000000140000-0x000000340000 : "Kernel0"
0x000000340000-0x000000540000 : "Kernel1"
0x000000540000-0x000000f40000 : "rootfs0"
0x000000f40000-0x000001940000 : "rootfs1"
0x000001940000-0x000002340000 : "logs"
0x000002340000-0x000002d40000 : "rw-fs"
0x000002d40000-0x000002dc0000 : "Factory"
0x000002dc0000-0x000002de0000 : "post"
0x000002de0000-0x000002ee0000 : "misc"
0x000002ee0000-0x000004000000 : "FreeNOR"
Generic platform RAM MTD, (c) 2004 Simtec Electronics
davinci_mdio davinci_mdio.0: davinci mdio revision 1.6
davinci_mdio davinci_mdio.0: detected phy mask fffffffd
davinci_mdio.0: probed
davinci_mdio davinci_mdio.0: phy[1]: device 0:01, driver NatSemi DP83640
mousedev: PS/2 mouse device common for all mice
lm73 1-0048: hwmon0: sensor 'lm73'
lm73 1-0049: hwmon1: sensor 'lm73'
omap_i2c omap_i2c.1: controller timed out
omap_i2c omap_i2c.1: controller timed out
OMAP Watchdog Timer Rev 0x00: initial timeout 60 sec
nss_aes_mod_init: loading NSS AES driver
nss-aes nss-aes: NSS AES hw accel rev: 3.2 (context 0 @0x41140000)
nss-aes nss-aes: NSS AES hw accel rev: 3.2 (context 1 @0x41141000)
nss-aes nss-aes: NSS AES hw accel rev: 3.2 (context 2 @0x411a0000)
nss-aes nss-aes: NSS AES hw accel rev: 3.2 (context 3 @0x411a1000)
nss_aes_probe: probe() done
nss_des_mod_init: loading NSS DES driver
nss-des nss-des: NSS DES hw accel rev: 2.2 (context 0 @0x41160000)
nss-des nss-des: NSS DES hw accel rev: 2.2 (context 1 @0x41161000)
nss_des_probe: probe() done
nss_sham_mod_init: loading NSS SHA/MD5 driver
nss-sham nss-sham: NSS SHA/MD5 hw accel rev: 4.03 (context 0 @0x41100000)
nss-sham nss-sham: NSS SHA/MD5 hw accel rev: 4.03 (context 1 @0x41101000)
nss-sham nss-sham: NSS SHA/MD5 hw accel rev: 4.03 (context 2 @0x411c0000)
nss-sham nss-sham: NSS SHA/MD5 hw accel rev: 4.03 (context 3 @0x411c1000)
nss_sham_probe: probe() done
dsp_hpi: initialized
fpga: initialized
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 10
NET: Registered protocol family 17
NET: Registered protocol family 15
sctp: Hash tables configured (established 32768 bind 43690)
Registering the dns_resolver key type
VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 3
clock: disabling unused clocks to save power
Detected MACID=3:2:1:2:e0:2

System initialization...

Mounting /sys          : [OK]
Populating /dev        : [OK]
Mounting fstab         : [OK]
Mounting devpts        : [OK]
Starting unconfigured  : [OK]
Starting Net           : ## Error: "ue" not defined
[OK]
Starting Logging       : [OK]
Starting telnet daemon : [OK]
Starting setup.sh      : [OK]
Executing setup        : [OK]
---- Warning application (/root/lm_cp_pico.elf) not started -----
Starting ptpd: [OK]

Please press Enter to activate this console. env variable run_app=no,
so supervisor will "free run", (to keep init happy)
RAMDISK: gzip image found at block 0
VFS: Mounted root (ext2 filesystem) on device 1:0.
devtmpfs: mounted
Freeing init memory: 168K

CPSW phy found : id is : 0x20005ce1
**************phy_start****************
c2cc0e00:phy_start:phydev->state:4

CPSW phyless for slave=1
ADDRCONF(NETDEV_UP): eth0: link is not ready
c2cc0e00:phy_start_aneg:phydev->state:5
PHY: 0:01 - Link is Up - 100/Full
link up on port 1, speed 1000, full duplex
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
bug:slab names_cache, slabp->inuse= -1, cachep->num:1
kernel BUG at /home/satpal/sandbox/console_p/trunk/bts/source/vendor/linux/mm/slab.c:3205!
Unable to handle kernel NULL pointer dereference at virtual address 00000000
pgd = e9e08000
[00000000] *pgd=ae31c831, *pte=00000000, *ppte=00000000
Internal error: Oops: 817 [#1] PREEMPT
Modules linked in:
CPU: 0    Not tainted  (3.0.1-rt11-svn7595 #29)
PC is at __bug+0x20/0x2c
LR is at vprintk+0x18c/0x470
pc : [<c0041ee0>]    lr : [<c005eaa0>]    psr: 60000013
sp : ef89fea8  ip : 60000013  fp : ef89feb4
r10: ef8009c0  r9 : ef811f20  r8 : ef807e00
r7 : 0000000c  r6 : 000000d0  r5 : ffffffff  r4 : c2cf92c0
r3 : 00000000  r2 : 00000000  r1 : ef89e000  r0 : 00000061
Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Control: 10c5387d  Table: a9e08019  DAC: 00000015
Process ptp-main (pid: 136, stack limit = 0xef89e2e8)
Stack: (0xef89fea8 to 0xef8a0000)
fea0:                   ef89ff04 ef89feb8 c00c3fd4 c0041ecc c04672a0 00000010
fec0: 000000d0 ef89e000 000000d0 ef811f44 ef811f30 ef811f28 ef89e000 00000000
fee0: ef89e000 c0484cdc ef8009c0 000000d0 ef89e000 00000000 ef89ff2c ef89ff08
ff00: c00c4628 c00c3d60 00120dc4 40b73490 00000002 00000000 c0462284 ef89e000
ff20: ef89ff54 ef89ff30 c00d2468 c00c4554 00000001 40b73490 00000002 ffffff9c
ff40: c003ea28 ef89e000 ef89ff64 ef89ff58 c00d2548 c00d2450 ef89ff94 ef89ff68
ff60: c00c5780 c00d2540 00000002 00000000 00000026 00000100 00000000 40b73490
ff80: 40b72fc0 00000005 ef89ffa4 ef89ff98 c00c5840 c00c56d8 00000000 ef89ffa8
ffa0: c003e880 c00c5828 00000000 40b73490 00120dc4 00000002 00fc0a28 40b72da4
ffc0: 00000000 40b73490 40b72fc0 00000005 003d0f00 00000000 4007e134 40b72d94
ffe0: 00000000 40b72d68 4007b0a7 4007c154 80000010 00120dc4 a87ff0f3 7d3f11fa
Backtrace:
[<c0041ec0>] (__bug+0x0/0x2c) from [<c00c3fd4>] (cache_alloc_refill+0x280/0x680)
[<c00c3d54>] (cache_alloc_refill+0x0/0x680) from [<c00c4628>]
(kmem_cache_alloc+0xe0/0x110)
[<c00c4548>] (kmem_cache_alloc+0x0/0x110) from [<c00d2468>]
(getname_flags+0x24/0xf0)
 r9:ef89e000 r8:c0462284 r7:00000000 r6:00000002 r5:40b73490
r4:00120dc4
[<c00d2444>] (getname_flags+0x0/0xf0) from [<c00d2548>] (getname+0x14/0x18)
 r9:ef89e000 r8:c003ea28 r7:ffffff9c r6:00000002 r5:40b73490
r4:00000001
[<c00d2534>] (getname+0x0/0x18) from [<c00c5780>] (do_sys_open+0xb4/0x13c)
[<c00c56cc>] (do_sys_open+0x0/0x13c) from [<c00c5840>] (sys_open+0x24/0x28)
 r7:00000005 r6:40b72fc0 r5:40b73490 r4:00000000
[<c00c581c>] (sys_open+0x0/0x28) from [<c003e880>] (ret_fast_syscall+0x0/0x30)
Code: e1a01000 e59f000c eb0cb576 e3a03000 (e5833000)
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists