| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <51EC07FE.4030200@andrews.lv>
Date: Sun, 21 Jul 2013 19:10:38 +0300
From: Andrejs Eigus <loco@...rews.lv>
To: linux-kernel@...r.kernel.org
Subject: PROBLEM: kernel panic on virtual guest / pvscsi / resched
Hello,
Apologies in advance, as this is my first bug report for the kernel. I would
appreciate a lot for any time spent on solving the issue.
We're running Gentoo Linux x86_64 with kernel 3.9.10 as a guest OS under VMware
host (VMware vCenter Server 5.0.0).
Problem summary:
Kernel panic occurs during nightly backup maintenance of the host system (causes
heavy I/O load on all guests -- there are around 60 virtual machines in total).
This issue happens randomly and always under stress I/O load. It does not
happen every night, just occasionally.
Attached is the call trace in the console (unfortunately, I was not able to get
this trace report in the logs as the error seems to have occurred in the
pvscsi). I hope it helps somehow.
Additional modules used: PVSCSI VMWare Paravirtual SCSI driver, VMXNET3 VMware
network driver
Complete dmesg is attached (dmesg.txt)
Kernel version: Linux version 3.9.10-gentoo-mrbyte (root@...NRWEB01) (gcc
version 4.7.3 (Gentoo 4.7.3 p1.0, pie-0.5.5) ) #1 SMP Sun Jul 14 10:30:58 CEST 2013
Environment:
MANPATH=/usr/local/share/man:/usr/share/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.7.3/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.23.1/man:/usr/lib64/php5.3/man/:/usr/lib64/php5.4/man/
HOSTNAME=BEANRWEB01
crt=
TERM=xterm
SHELL=/bin/bash
SSH_CLIENT=88.135.136.85 57765 22
SSH_TTY=/dev/pts/0
USER=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=0
1;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.pdf=00;32:*.ps=00;32:*.txt=00;32:*.patch=00;32:*.diff=00;32:*.log=00;32:*.tex=00;32:*.doc=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
GUILE_LOAD_PATH=/usr/share/guile/1.8
LESSCHARDEF=8bcccbcc18b95.b27.b99.b
PAGER=/usr/bin/less
CONFIG_PROTECT_MASK=/etc/gentoo-release /etc/sandbox.d
/etc/php/cli-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/
/etc/php/apache2-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/
/etc/php/cgi-php5.4/ext-active/ /etc/php/apache2-php5.4/ext-active/
/etc/fonts/fonts.conf /etc/terminfo /etc/ca-certificates.conf /etc/revdep-rebuild
MAIL=/var/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.7.3:/root/scripts:/etc/php/scripts
LC_COLLATE=C
PWD=/root
EDITOR=nano -w
LANG=POSIX
DEVEL_SITE=BEANRWEB01
PS1=\h:\[\033[32;1m\]\w\[\033[0m\]\$
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
LESS=-R -M --shift 5
LOGNAME=root
GCC_SPECS=
LC_CTYPE=en_US.utf8
SSH_CONNECTION=88.135.136.85 57765 192.168.99.14 22
LESSOPEN=|lesspipe %s
INFOPATH=/usr/share/info:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.7.3/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.23.1/info
CONFIG_PROTECT=/usr/share/gnupg/qualified.txt
_=/usr/bin/env
Software:
Linux BEANRWEB01 3.9.10-gentoo-mrbyte #1 SMP Sun Jul 14 10:30:58 CEST 2013
x86_64 Intel(R) Xeon(R) CPU X5650 @ 2.67GHz GenuineIntel GNU/Linux
Gnu C 4.7.3
Gnu make 3.82
binutils 2.23.1
util-linux 2.23.1
mount assert
module-init-tools 14
e2fsprogs 1.42.7
Linux C Library 2.17
Dynamic linker (ldd) 2.17
Procps 3.3.8
Net-tools 1.60_p20130513023548
Kbd 1.15.5
Sh-utils 8.21
Modules Loaded xt_multiport xt_comment nf_conntrack_ipv4 nf_defrag_ipv4
xt_conntrack vmwgfx cfbfillrect cfbimgblt cfbcopyarea fb fbdev ttm drm i2c_core
agpgart shpchp evdev libcrc32c aes_x86_64 sha256_generic iscsi_tcp libiscsi_tcp
libiscsi scsi_transport_iscsi tg3 ptp pps_core e1000 fuse nfs lockd sunrpc
reiserfs linear raid10 raid456 async_raid6_recov async_memcpy async_pq raid6_pq
async_xor xor async_tx raid1 raid0 dm_snapshot dm_crypt megaraid_sas
megaraid_mbox megaraid_mm megaraid hpsa cciss
Processor information:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
stepping : 1
microcode : 0x14
cpu MHz : 2666.761
cache size : 12288 KB
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx lm constant_tsc
arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni
ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
bogomips : 5333.52
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
stepping : 1
microcode : 0x14
cpu MHz : 2666.761
cache size : 12288 KB
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx lm constant_tsc
arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni
ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
bogomips : 5333.52
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
stepping : 1
microcode : 0x14
cpu MHz : 2666.761
cache size : 12288 KB
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx lm constant_tsc
arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni
ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
bogomips : 5333.52
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
stepping : 1
microcode : 0x14
cpu MHz : 2666.761
cache size : 12288 KB
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx lm constant_tsc
arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni
ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
bogomips : 5333.52
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
Module information:
xt_multiport 1468 6 - Live 0xffffffffa02e0000
xt_comment 851 1 - Live 0xffffffffa02dc000
nf_conntrack_ipv4 10607 16 - Live 0xffffffffa02d5000
nf_defrag_ipv4 1115 1 nf_conntrack_ipv4, Live 0xffffffffa02d1000
xt_conntrack 2705 16 - Live 0xffffffffa02cd000
vmwgfx 101892 0 - Live 0xffffffffa02aa000
cfbfillrect 2930 1 vmwgfx, Live 0xffffffffa02a6000
cfbimgblt 1879 1 vmwgfx, Live 0xffffffffa02a2000
cfbcopyarea 2918 1 vmwgfx, Live 0xffffffffa029e000
fb 27871 1 vmwgfx, Live 0xffffffffa0291000
fbdev 734 1 fb, Live 0xffffffffa028d000
ttm 51070 1 vmwgfx, Live 0xffffffffa0277000
drm 191627 2 vmwgfx,ttm, Live 0xffffffffa0235000
i2c_core 16307 1 drm, Live 0xffffffffa022c000
agpgart 22470 2 ttm,drm, Live 0xffffffffa0220000
shpchp 21383 0 - Live 0xffffffffa0215000
evdev 8114 0 - Live 0xffffffffa020f000
libcrc32c 890 0 - Live 0xffffffffa020b000
aes_x86_64 7191 0 - Live 0xffffffffa0206000
sha256_generic 9445 0 - Live 0xffffffffa0200000
iscsi_tcp 7540 0 - Live 0xffffffffa01fa000
libiscsi_tcp 10056 1 iscsi_tcp, Live 0xffffffffa01f3000
libiscsi 28649 2 iscsi_tcp,libiscsi_tcp, Live 0xffffffffa01e6000
scsi_transport_iscsi 32193 2 iscsi_tcp,libiscsi, Live 0xffffffffa01d6000
tg3 121160 0 - Live 0xffffffffa01b0000
ptp 6587 1 tg3, Live 0xffffffffa01aa000
pps_core 5695 1 ptp, Live 0xffffffffa01a4000
e1000 85452 0 - Live 0xffffffffa0187000
fuse 58664 0 - Live 0xffffffffa0170000
nfs 96292 0 - Live 0xffffffffa0149000
lockd 50497 1 nfs, Live 0xffffffffa0134000
sunrpc 138315 2 nfs,lockd, Live 0xffffffffa00ff000
reiserfs 175631 0 - Live 0xffffffffa00c9000
linear 3111 0 - Live 0xffffffffa00c5000
raid10 33349 0 - Live 0xffffffffa00b8000
raid456 46563 0 - Live 0xffffffffa00a7000
async_raid6_recov 1169 1 raid456, Live 0xffffffffa00a3000
async_memcpy 822 1 raid456, Live 0xffffffffa009f000
async_pq 1948 1 raid456, Live 0xffffffffa009b000
raid6_pq 89031 2 async_raid6_recov,async_pq, Live 0xffffffffa0081000
async_xor 1257 2 raid456,async_pq, Live 0xffffffffa007d000
xor 9737 1 async_xor, Live 0xffffffffa0077000
async_tx 1217 5 raid456,async_raid6_recov,async_memcpy,async_pq,async_xor, Live
0xffffffffa0073000
raid1 22781 0 - Live 0xffffffffa0069000
raid0 6566 0 - Live 0xffffffffa0064000
dm_snapshot 22927 0 - Live 0xffffffffa0059000
dm_crypt 12516 0 - Live 0xffffffffa0051000
megaraid_sas 58790 0 - Live 0xffffffffa003c000
megaraid_mbox 22563 0 - Live 0xffffffffa0031000
megaraid_mm 6687 1 megaraid_mbox, Live 0xffffffffa002b000
megaraid 29643 0 - Live 0xffffffffa001f000
hpsa 39492 0 - Live 0xffffffffa0010000
cciss 39106 0 - Live 0xffffffffa0000000
Loaded driver and hardware information:
0000-0cf7 : PCI Bus 0000:00
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-0060 : keyboard
0064-0064 : keyboard
0070-0071 : rtc0
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : 0000:00:07.1
01f0-01f7 : 0000:00:07.1
02f8-02ff : serial
0376-0376 : 0000:00:07.1
03c0-03df : vga+
03f6-03f6 : 0000:00:07.1
03f8-03ff : serial
0cf0-0cf1 : pnp 00:00
0cf8-0cff : PCI conf1
0d00-feff : PCI Bus 0000:00
1000-103f : 0000:00:07.3
1000-103f : pnp 00:00
1000-1003 : ACPI PM1a_EVT_BLK
1004-1005 : ACPI PM1a_CNT_BLK
1008-100b : ACPI PM_TMR
100c-100f : ACPI GPE0_BLK
1010-1015 : ACPI CPU throttle
1040-104f : 0000:00:07.3
1040-104f : pnp 00:00
1060-107f : pnp 00:0b
1080-10bf : 0000:00:07.7
1080-10bf : vmw_vmci
10c0-10cf : 0000:00:07.1
10d0-10df : 0000:00:0f.0
10d0-10df : vmwgfx probe
2000-3fff : PCI Bus 0000:02
4000-4fff : PCI Bus 0000:03
4000-400f : 0000:03:00.0
5000-5fff : PCI Bus 0000:0b
5008-500f : 0000:0b:00.0
5008-500f : vmw_pvscsi
6000-6fff : PCI Bus 0000:13
7000-7fff : PCI Bus 0000:1b
8000-8fff : PCI Bus 0000:04
9000-9fff : PCI Bus 0000:0c
a000-afff : PCI Bus 0000:14
b000-bfff : PCI Bus 0000:1c
c000-cfff : PCI Bus 0000:05
d000-dfff : PCI Bus 0000:0d
e000-efff : PCI Bus 0000:15
00000000-00000fff : reserved
00001000-0009f7ff : System RAM
0009f800-0009ffff : reserved
000a0000-000bffff : PCI Bus 0000:00
000c0000-000c7fff : Video ROM
000c8000-000c8fff : Adapter ROM
000cc000-000cffff : PCI Bus 0000:00
000d0000-000d3fff : PCI Bus 0000:00
000d4000-000d7fff : PCI Bus 0000:00
000d8000-000dbfff : PCI Bus 0000:00
000dc000-000fffff : reserved
000f0000-000fffff : System ROM
00100000-bfedffff : System RAM
01000000-01373838 : Kernel code
01373839-0166bdbf : Kernel data
0172f000-017c5fff : Kernel bss
bfee0000-bfefefff : ACPI Tables
bfeff000-bfefffff : ACPI Non-volatile Storage
bff00000-bfffffff : System RAM
c0000000-febfffff : PCI Bus 0000:00
c0000000-c0007fff : 0000:00:0f.0
d4000000-d7ffffff : 0000:00:0f.0
d4000000-d7ffffff : vmwgfx probe
d8000000-d87fffff : 0000:00:0f.0
d8000000-d87fffff : vmwgfx probe
d8800000-d89fffff : pnp 00:0b
d9100000-d9bfffff : PCI Bus 0000:02
d9c00000-d9cfffff : PCI Bus 0000:03
d9c02000-d9c03fff : 0000:03:00.0
d9c04000-d9c04fff : 0000:03:00.0
d9c04000-d9c04fff : vmxnet3
d9c05000-d9c05fff : 0000:03:00.0
d9c05000-d9c05fff : vmxnet3
d9d00000-d9dfffff : PCI Bus 0000:0b
d9d08000-d9d0ffff : 0000:0b:00.0
d9d08000-d9d0ffff : vmw_pvscsi
d9e00000-d9efffff : PCI Bus 0000:13
d9f00000-d9ffffff : PCI Bus 0000:1b
da000000-da0fffff : PCI Bus 0000:04
da100000-da1fffff : PCI Bus 0000:0c
da200000-da2fffff : PCI Bus 0000:14
da300000-da3fffff : PCI Bus 0000:1c
da400000-da4fffff : PCI Bus 0000:05
da500000-da5fffff : PCI Bus 0000:0d
da600000-da6fffff : PCI Bus 0000:15
da700000-da7fffff : PCI Bus 0000:1d
da800000-da8fffff : PCI Bus 0000:06
da900000-da9fffff : PCI Bus 0000:0e
daa00000-daafffff : PCI Bus 0000:16
dab00000-dabfffff : PCI Bus 0000:1e
dac00000-dacfffff : PCI Bus 0000:07
dad00000-dadfffff : PCI Bus 0000:0f
dae00000-daefffff : PCI Bus 0000:17
daf00000-daffffff : PCI Bus 0000:1f
db000000-db0fffff : PCI Bus 0000:08
db100000-db1fffff : PCI Bus 0000:10
db200000-db2fffff : PCI Bus 0000:18
db300000-db3fffff : PCI Bus 0000:20
db400000-db4fffff : PCI Bus 0000:09
db500000-db5fffff : PCI Bus 0000:11
db600000-db6fffff : PCI Bus 0000:19
db700000-db7fffff : PCI Bus 0000:21
db800000-db8fffff : PCI Bus 0000:0a
db900000-db9fffff : PCI Bus 0000:12
dba00000-dbafffff : PCI Bus 0000:1a
dbb00000-dbbfffff : PCI Bus 0000:22
dc000000-dc5fffff : PCI Bus 0000:02
dc600000-dc6fffff : PCI Bus 0000:03
dc600000-dc60ffff : 0000:03:00.0
dc700000-dc7fffff : PCI Bus 0000:0b
dc700000-dc70ffff : 0000:0b:00.0
dc800000-dc8fffff : PCI Bus 0000:13
dc900000-dc9fffff : PCI Bus 0000:1b
dca00000-dcafffff : PCI Bus 0000:04
dcb00000-dcbfffff : PCI Bus 0000:0c
dcc00000-dccfffff : PCI Bus 0000:14
dcd00000-dcdfffff : PCI Bus 0000:1c
dce00000-dcefffff : PCI Bus 0000:05
dcf00000-dcffffff : PCI Bus 0000:0d
dd000000-dd0fffff : PCI Bus 0000:15
dd100000-dd1fffff : PCI Bus 0000:1d
dd200000-dd2fffff : PCI Bus 0000:06
dd300000-dd3fffff : PCI Bus 0000:0e
dd400000-dd4fffff : PCI Bus 0000:16
dd500000-dd5fffff : PCI Bus 0000:1e
dd600000-dd6fffff : PCI Bus 0000:07
dd700000-dd7fffff : PCI Bus 0000:0f
dd800000-dd8fffff : PCI Bus 0000:17
dd900000-dd9fffff : PCI Bus 0000:1f
dda00000-ddafffff : PCI Bus 0000:08
ddb00000-ddbfffff : PCI Bus 0000:10
ddc00000-ddcfffff : PCI Bus 0000:18
ddd00000-dddfffff : PCI Bus 0000:20
dde00000-ddefffff : PCI Bus 0000:09
ddf00000-ddffffff : PCI Bus 0000:11
de000000-de0fffff : PCI Bus 0000:19
de100000-de1fffff : PCI Bus 0000:21
de200000-de2fffff : PCI Bus 0000:0a
de300000-de3fffff : PCI Bus 0000:12
de400000-de4fffff : PCI Bus 0000:1a
de500000-de5fffff : PCI Bus 0000:22
e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
e0000000-efffffff : reserved
e0000000-efffffff : pnp 00:0b
fec00000-fec0ffff : reserved
fec00000-fec003ff : IOAPIC 0
fee00000-fee00fff : Local APIC
fee00000-fee00fff : reserved
fffe0000-ffffffff : reserved
100000000-43fffffff : System RAM
SCSI information:
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: VMware Model: Virtual disk Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02
Other Information of potential interest:
/etc/fstab:
/dev/sda1 /boot ext4 noauto,noatime 1 2
/dev/sda2 / ext4 noatime 0 1
/dev/sda3 none swap sw 0 0
/dev/mapper/ld-usr /usr ext4 noatime
/dev/mapper/ld-portage /usr/portage ext4 noatime
/dev/mapper/ld-var /var ext4 noatime,nosuid
/dev/mapper/ld-tmp /tmp ext4 noatime,nosuid
/dev/mapper/ld-home /home ext4 noatime,nosuid
/dev/mapper/ld-opt /opt ext4 noatime,nosuid
Kernel trace log (only available in the vSphere Client console), so I made a
screen capture :
View attachment "dmesg.txt" of type "text/plain" (74890 bytes)
Powered by blists - more mailing lists