| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20160322204510.GS4287@linux.vnet.ibm.com> Date: Tue, 22 Mar 2016 13:45:10 -0700 From: "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com> To: Bart Van Assche <bart.vanassche@...disk.com> Cc: linux-kernel@...r.kernel.org Subject: Re: RCU stall On Tue, Mar 22, 2016 at 01:20:02PM -0700, Bart Van Assche wrote: > Hello Paul, > > Maybe you remember that we met briefly during the LinuxCon VIP > reception in the Chihuly Garden in Seattle (August 2015)? > > Yesterday I tried to verify a change in the ib_srpt driver on my > setup by letting the ib_srp and ib_srpt drivers communicate with > each other on the same system using the loopback functionality > provided by Mellanox ConnectX-3 HCA's. Instead of running into any > SRP related issues I hit something that looks like an RCU stall. I > can reproduce this stall with the v4.4.6 and v4.5-rc7 kernels (I > have not yet tried any other kernels). Can you have a look at the > call traces in the attached netconsole output and check whether or > not this is related to the RCU implementation in the Linux kernel? > The test I ran is as follows: > * Load and configure the ib_srpt target driver such that it exports > a RAM disk (brd) as a SCSI device. > * Load and configure the ib_srp initiator driver (which is a SCSI > initiator driver). > * Start the srp_daemon. This daemon makes the ib_srp initiator > driver log in to the ib_srpt target driver. > * Activate multipath on top of the SCSI devices created by ib_srp. > * Make fio send a workload with a high queue depth to the ib_srp > initiator driver. > * Restart the SCSI target stack (ib_srpt + the kernel modules used > by that driver). > > I can post this information on a public mailing list if you would > prefer this. No need, I added LKML on CC. Depending on what you find, you might want to add more people or more lists. You are getting a soft lockup as well as an RCU CPU stall warning, so it looks like something is taking a very long time in blk_done_softirq(). You have multiple occurrences at different times, so it looks to be a long time as opposed to an infinite time. Are you perhaps doing something that would make a huge amount of work for blk_done_softirq()? See Documentation/RCU/stallwarn.txt in the kernel source tree for more info on how to debug this sort of thing. Thanx, Paul > Thanks, > > Bart. > [10416.888346] device-mapper: multipath: Failing path 8:112. > [10416.894423] device-mapper: multipath: Failing path 8:96. > [10416.906746] device-mapper: multipath: Failing path 8:240. > [10416.907224] device-mapper: multipath: Failing path 66:80. > [10416.915126] device-mapper: multipath: Failing path 65:192. > [10416.915479] device-mapper: multipath: Failing path 66:160. > [10416.990626] device-mapper: multipath: Failing path 67:144. > [10417.961194] device-mapper: multipath: Failing path 66:48. > [10417.968416] device-mapper: multipath: Failing path 67:128. > [10417.976784] device-mapper: multipath: Failing path 67:64. > [10418.996938] device-mapper: multipath: Failing path 8:128. > [10418.997469] device-mapper: multipath: Failing path 65:80. > [10419.004872] device-mapper: multipath: Failing path 65:128. > [10419.006014] device-mapper: multipath: Failing path 8:176. > [10419.007623] device-mapper: multipath: Failing path 67:96. > [10419.011766] device-mapper: multipath: Failing path 65:64. > [10420.054544] scsi_io_completion: 22 callbacks suppressed > [10420.054632] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.054704] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10420.054771] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.054847] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10420.054917] blk_update_request: 22 callbacks suppressed > [10420.054989] blk_update_request: I/O error, dev sdj, sector 1952320 > [10420.055114] device-mapper: multipath: Failing path 8:144. > [10420.060874] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.060927] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10420.060965] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.061010] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00 > [10420.061051] blk_update_request: I/O error, dev sdz, sector 704 > [10420.061118] device-mapper: multipath: Failing path 65:144. > [10420.067122] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.067173] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10420.067224] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.067289] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10420.067351] blk_update_request: I/O error, dev sdk, sector 1952320 > [10420.067445] device-mapper: multipath: Failing path 8:160. > [10420.067825] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.067894] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] > [10420.067954] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.068017] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10420.068075] blk_update_request: I/O error, dev sdaa, sector 1952320 > [10420.068165] device-mapper: multipath: Failing path 65:160. > [10420.073375] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.073456] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10420.073520] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.073582] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 01 10 00 00 60 00 > [10420.073641] blk_update_request: I/O error, dev sdad, sector 272 > [10420.073738] device-mapper: multipath: Failing path 65:208. > [10420.075521] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.075642] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10420.075748] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.075855] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10420.075955] blk_update_request: I/O error, dev sdo, sector 608 > [10420.076204] device-mapper: multipath: Failing path 8:224. > [10420.081491] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.081597] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10420.081685] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.081778] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d ca 70 00 00 60 00 > [10420.081850] blk_update_request: I/O error, dev sdai, sector 1952368 > [10420.081939] device-mapper: multipath: Failing path 66:32. > [10420.157841] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.157922] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10420.157982] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.158023] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10420.158059] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10420.158124] device-mapper: multipath: Failing path 67:224. > [10420.184232] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.184336] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10420.184417] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.184505] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10420.184588] blk_update_request: I/O error, dev sdas, sector 1952320 > [10420.184713] device-mapper: multipath: Failing path 66:192. > [10420.190423] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.190528] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10420.190711] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.190805] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00 > [10420.190921] blk_update_request: I/O error, dev sdar, sector 704 > [10420.191092] device-mapper: multipath: Failing path 66:176. > [10421.152191] device-mapper: multipath: Failing path 65:176. > [10421.152905] device-mapper: multipath: Failing path 66:208. > [10421.158332] device-mapper: multipath: Failing path 66:64. > [10421.169109] device-mapper: multipath: Failing path 68:16. > [10421.169450] device-mapper: multipath: Failing path 67:192. > [10421.178113] device-mapper: multipath: Failing path 68:0. > [10422.226466] device-mapper: multipath: Failing path 8:112. > [10422.233709] device-mapper: multipath: Failing path 8:96. > [10422.247912] device-mapper: multipath: Failing path 8:240. > [10422.248844] device-mapper: multipath: Failing path 66:80. > [10422.255501] device-mapper: multipath: Failing path 65:192. > [10422.261653] device-mapper: multipath: Failing path 66:160. > [10422.324643] device-mapper: multipath: Failing path 67:144. > [10423.297527] device-mapper: multipath: Failing path 66:48. > [10423.307128] device-mapper: multipath: Failing path 67:128. > [10423.314250] device-mapper: multipath: Failing path 67:64. > [10424.336170] device-mapper: multipath: Failing path 8:128. > [10424.336732] device-mapper: multipath: Failing path 65:80. > [10424.345244] device-mapper: multipath: Failing path 65:128. > [10424.345717] device-mapper: multipath: Failing path 65:64. > [10424.346791] device-mapper: multipath: Failing path 8:176. > [10424.354214] device-mapper: multipath: Failing path 67:96. > [10425.390756] scsi_io_completion: 22 callbacks suppressed > [10425.390886] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.390997] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10425.391099] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.391214] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10425.391321] blk_update_request: 22 callbacks suppressed > [10425.391415] blk_update_request: I/O error, dev sdj, sector 1952320 > [10425.391564] device-mapper: multipath: Failing path 8:144. > [10425.398476] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.398564] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10425.398632] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.398703] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00 > [10425.398767] blk_update_request: I/O error, dev sdz, sector 704 > [10425.398866] device-mapper: multipath: Failing path 65:144. > [10425.400705] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.400779] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10425.400845] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.400911] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10425.400969] blk_update_request: I/O error, dev sdk, sector 1952320 > [10425.401137] device-mapper: multipath: Failing path 8:160. > [10425.401616] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.401684] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] > [10425.401743] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.401807] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10425.401866] blk_update_request: I/O error, dev sdaa, sector 1952320 > [10425.401967] device-mapper: multipath: Failing path 65:160. > [10425.404794] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.404879] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10425.404952] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.405029] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 00 78 00 00 08 00 > [10425.405103] blk_update_request: I/O error, dev sdai, sector 120 > [10425.405273] device-mapper: multipath: Failing path 66:32. > [10425.409526] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.409609] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10425.409674] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.409741] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00 > [10425.409806] blk_update_request: I/O error, dev sdo, sector 704 > [10425.409971] device-mapper: multipath: Failing path 8:224. > [10425.411359] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.411471] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10425.411547] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.411614] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 e8 00 > [10425.411680] blk_update_request: I/O error, dev sdas, sector 768 > [10425.411770] device-mapper: multipath: Failing path 66:192. > [10425.417081] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.417156] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10425.417217] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.417283] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d cb b0 00 00 50 00 > [10425.417351] blk_update_request: I/O error, dev sdad, sector 1952688 > [10425.417512] device-mapper: multipath: Failing path 65:208. > [10425.483947] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.484013] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10425.484076] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.484127] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10425.484177] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10425.484245] device-mapper: multipath: Failing path 67:224. > [10425.497810] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.497866] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10425.497910] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.497957] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 18 00 00 08 00 > [10425.497999] blk_update_request: I/O error, dev sdar, sector 1952280 > [10425.498065] device-mapper: multipath: Failing path 66:176. > [10426.456675] device-mapper: multipath: Failing path 65:176. > [10426.457202] device-mapper: multipath: Failing path 66:208. > [10426.462722] device-mapper: multipath: Failing path 66:64. > [10426.472875] device-mapper: multipath: Failing path 68:16. > [10426.473257] device-mapper: multipath: Failing path 67:192. > [10426.477439] device-mapper: multipath: Failing path 68:0. > [10427.509583] device-mapper: multipath: Failing path 8:112. > [10427.516675] device-mapper: multipath: Failing path 8:96. > [10427.529858] device-mapper: multipath: Failing path 8:240. > [10427.530437] device-mapper: multipath: Failing path 66:80. > [10427.537627] device-mapper: multipath: Failing path 65:192. > [10427.544651] device-mapper: multipath: Failing path 66:160. > [10427.616573] device-mapper: multipath: Failing path 67:144. > [10428.580910] device-mapper: multipath: Failing path 66:48. > [10428.589846] device-mapper: multipath: Failing path 67:128. > [10428.601369] device-mapper: multipath: Failing path 67:64. > [10429.602027] device-mapper: multipath: Failing path 8:128. > [10429.607145] device-mapper: multipath: Failing path 65:80. > [10429.609131] device-mapper: multipath: Failing path 65:128. > [10429.619473] device-mapper: multipath: Failing path 8:176. > [10429.712106] device-mapper: multipath: Failing path 67:96. > [10430.675345] scsi_io_completion: 21 callbacks suppressed > [10430.675424] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.675493] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10430.675563] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.675645] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10430.675711] blk_update_request: 21 callbacks suppressed > [10430.675777] blk_update_request: I/O error, dev sdj, sector 1952320 > [10430.675880] device-mapper: multipath: Failing path 8:144. > [10430.677763] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.677884] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10430.677980] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.678092] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 08 00 00 00 08 00 > [10430.678189] blk_update_request: I/O error, dev sdz, sector 2048 > [10430.678408] device-mapper: multipath: Failing path 65:144. > [10430.680621] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.680774] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10430.680903] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.681010] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10430.681132] blk_update_request: I/O error, dev sdk, sector 1952320 > [10430.681357] device-mapper: multipath: Failing path 8:160. > [10430.686148] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.686267] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] > [10430.686364] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.686468] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 00 00 20 00 00 18 00 > [10430.686557] blk_update_request: I/O error, dev sdaa, sector 32 > [10430.687034] device-mapper: multipath: Failing path 65:160. > [10430.687938] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.688015] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10430.688077] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.688137] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10430.688193] blk_update_request: I/O error, dev sdu, sector 608 > [10430.688319] device-mapper: multipath: Failing path 65:64. > [10430.690959] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.691031] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10430.691091] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.691151] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 b8 00 > [10430.691211] blk_update_request: I/O error, dev sdo, sector 768 > [10430.691397] device-mapper: multipath: Failing path 8:224. > [10430.694975] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.695052] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10430.695115] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.695182] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 00 00 00 18 00 > [10430.695240] blk_update_request: I/O error, dev sdad, sector 1952256 > [10430.695480] device-mapper: multipath: Failing path 65:208. > [10430.776287] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.776347] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10430.776388] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.776430] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d cd b8 00 00 08 00 > [10430.776471] blk_update_request: I/O error, dev sdai, sector 1953208 > [10430.776537] device-mapper: multipath: Failing path 66:32. > [10430.779410] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.779467] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10430.779511] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.779562] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 18 00 00 08 00 > [10430.779606] blk_update_request: I/O error, dev sdas, sector 1952280 > [10430.779712] device-mapper: multipath: Failing path 66:192. > [10430.782147] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.782219] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10430.782279] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.782342] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10430.782402] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10430.782491] device-mapper: multipath: Failing path 67:224. > [10430.784545] device-mapper: multipath: Failing path 66:176. > [10431.767557] device-mapper: multipath: Failing path 65:176. > [10431.767921] device-mapper: multipath: Failing path 66:208. > [10431.773168] device-mapper: multipath: Failing path 66:64. > [10431.793497] device-mapper: multipath: Failing path 68:16. > [10431.794007] device-mapper: multipath: Failing path 67:192. > [10431.798234] device-mapper: multipath: Failing path 68:0. > [10432.827559] device-mapper: multipath: Failing path 8:112. > [10432.834048] device-mapper: multipath: Failing path 8:96. > [10432.847571] device-mapper: multipath: Failing path 8:240. > [10432.848142] device-mapper: multipath: Failing path 66:80. > [10432.854439] device-mapper: multipath: Failing path 65:192. > [10432.859782] device-mapper: multipath: Failing path 66:160. > [10432.925961] device-mapper: multipath: Failing path 67:144. > [10433.916020] device-mapper: multipath: Failing path 66:48. > [10433.923574] device-mapper: multipath: Failing path 67:128. > [10433.931521] device-mapper: multipath: Failing path 67:64. > [10434.944355] device-mapper: multipath: Failing path 8:128. > [10434.944780] device-mapper: multipath: Failing path 65:80. > [10434.946445] device-mapper: multipath: Failing path 67:96. > [10434.953279] device-mapper: multipath: Failing path 65:128. > [10434.961368] device-mapper: multipath: Failing path 8:176. > [10435.997587] scsi_io_completion: 22 callbacks suppressed > [10435.997666] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10435.997735] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10435.997800] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10435.997868] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10435.997932] blk_update_request: 22 callbacks suppressed > [10435.997992] blk_update_request: I/O error, dev sdj, sector 1952320 > [10435.998093] device-mapper: multipath: Failing path 8:144. > [10435.998848] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10435.998940] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10435.999015] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10435.999136] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10435.999212] blk_update_request: I/O error, dev sdz, sector 1952320 > [10435.999358] device-mapper: multipath: Failing path 65:144. > [10436.003810] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.003924] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10436.004034] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.004182] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10436.004300] blk_update_request: I/O error, dev sdk, sector 1952320 > [10436.004486] device-mapper: multipath: Failing path 8:160. > [10436.006466] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.006572] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] > [10436.006677] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.006835] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10436.006943] blk_update_request: I/O error, dev sdaa, sector 1952320 > [10436.007132] device-mapper: multipath: Failing path 65:160. > [10436.009417] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.009500] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10436.009563] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.009626] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 b8 00 00 48 00 > [10436.009686] blk_update_request: I/O error, dev sdu, sector 696 > [10436.009832] device-mapper: multipath: Failing path 65:64. > [10436.013058] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.013131] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10436.013213] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.013311] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 00 20 00 00 18 00 > [10436.013405] blk_update_request: I/O error, dev sdad, sector 32 > [10436.013575] device-mapper: multipath: Failing path 65:208. > [10436.016633] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.016704] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10436.016760] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.016820] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00 > [10436.016876] blk_update_request: I/O error, dev sdo, sector 704 > [10436.017086] device-mapper: multipath: Failing path 8:224. > [10436.022985] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.023062] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10436.023132] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.023201] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 70 00 00 60 00 > [10436.023264] blk_update_request: I/O error, dev sdas, sector 1952368 > [10436.023405] device-mapper: multipath: Failing path 66:192. > [10436.024977] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.025082] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10436.025178] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.025271] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d cb 20 00 00 30 00 > [10436.025328] blk_update_request: I/O error, dev sdai, sector 1952544 > [10436.025472] device-mapper: multipath: Failing path 66:32. > [10436.119145] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.119237] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10436.119305] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.119380] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10436.119486] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10436.119638] device-mapper: multipath: Failing path 67:224. > [10436.133091] device-mapper: multipath: Failing path 66:176. > [10437.082788] device-mapper: multipath: Failing path 65:176. > [10437.083248] device-mapper: multipath: Failing path 66:208. > [10437.083653] device-mapper: multipath: Failing path 68:16. > [10437.083928] device-mapper: multipath: Failing path 67:192. > [10437.087703] device-mapper: multipath: Failing path 68:0. > [10437.088448] device-mapper: multipath: Failing path 66:64. > [10438.129087] device-mapper: multipath: Failing path 8:112. > [10438.138623] device-mapper: multipath: Failing path 8:96. > [10438.151534] device-mapper: multipath: Failing path 8:240. > [10438.152048] device-mapper: multipath: Failing path 66:80. > [10438.158935] device-mapper: multipath: Failing path 65:192. > [10438.164948] device-mapper: multipath: Failing path 66:160. > [10438.232625] device-mapper: multipath: Failing path 67:144. > [10439.204539] device-mapper: multipath: Failing path 66:48. > [10439.212592] device-mapper: multipath: Failing path 67:128. > [10439.220192] device-mapper: multipath: Failing path 67:64. > [10440.234002] device-mapper: multipath: Failing path 8:128. > [10440.234585] device-mapper: multipath: Failing path 65:80. > [10440.237500] device-mapper: multipath: Failing path 67:96. > [10440.241884] device-mapper: multipath: Failing path 8:176. > [10440.241930] device-mapper: multipath: Failing path 65:128. > [10441.293227] scsi_io_completion: 22 callbacks suppressed > [10441.293301] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.293360] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10441.293420] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.293673] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10441.293733] blk_update_request: 22 callbacks suppressed > [10441.293785] blk_update_request: I/O error, dev sdj, sector 1952320 > [10441.293916] device-mapper: multipath: Failing path 8:144. > [10441.294596] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.294703] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10441.294799] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.294939] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10441.295032] blk_update_request: I/O error, dev sdz, sector 1952320 > [10441.295203] device-mapper: multipath: Failing path 65:144. > [10441.297186] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.297273] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10441.297341] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.297412] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10441.297518] blk_update_request: I/O error, dev sdk, sector 1952320 > [10441.297744] device-mapper: multipath: Failing path 8:160. > [10441.300255] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.300340] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] > [10441.300419] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.300535] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 00 00 38 00 00 08 00 > [10441.300626] blk_update_request: I/O error, dev sdaa, sector 56 > [10441.300861] device-mapper: multipath: Failing path 65:160. > [10441.302907] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.302972] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10441.303045] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.303132] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 70 00 00 60 00 > [10441.303215] blk_update_request: I/O error, dev sdu, sector 624 > [10441.303399] device-mapper: multipath: Failing path 65:64. > [10441.306683] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.306792] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10441.306862] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.306937] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10441.307000] blk_update_request: I/O error, dev sdo, sector 608 > [10441.307201] device-mapper: multipath: Failing path 8:224. > [10441.307621] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.307685] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10441.307739] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.307799] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10441.307855] blk_update_request: I/O error, dev sdad, sector 608 > [10441.307939] device-mapper: multipath: Failing path 65:208. > [10441.317400] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.317503] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10441.317591] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.317649] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d cb 20 00 00 30 00 > [10441.317704] blk_update_request: I/O error, dev sdai, sector 1952544 > [10441.317802] device-mapper: multipath: Failing path 66:32. > [10441.387950] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.388012] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10441.388060] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.388108] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10441.388156] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10441.388227] device-mapper: multipath: Failing path 67:224. > [10441.396210] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.396260] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10441.396308] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.396377] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10441.396436] blk_update_request: I/O error, dev sdas, sector 1952320 > [10441.396525] device-mapper: multipath: Failing path 66:192. > [10441.404557] device-mapper: multipath: Failing path 66:176. > [10442.380325] device-mapper: multipath: Failing path 65:176. > [10442.380917] device-mapper: multipath: Failing path 66:208. > [10442.385656] device-mapper: multipath: Failing path 66:64. > [10442.398393] device-mapper: multipath: Failing path 68:16. > [10442.398714] device-mapper: multipath: Failing path 67:192. > [10442.403546] device-mapper: multipath: Failing path 68:0. > [10443.438787] device-mapper: multipath: Failing path 8:112. > [10443.445929] device-mapper: multipath: Failing path 8:96. > [10443.452757] device-mapper: multipath: Failing path 67:144. > [10443.456211] device-mapper: multipath: Failing path 8:240. > [10443.457007] device-mapper: multipath: Failing path 66:80. > [10443.465293] device-mapper: multipath: Failing path 65:192. > [10443.470696] device-mapper: multipath: Failing path 66:160. > [10444.503706] device-mapper: multipath: Failing path 66:48. > [10444.510208] device-mapper: multipath: Failing path 67:128. > [10444.517049] device-mapper: multipath: Failing path 67:64. > [10445.536571] device-mapper: multipath: Failing path 8:128. > [10445.537057] device-mapper: multipath: Failing path 65:80. > [10445.544179] device-mapper: multipath: Failing path 65:128. > [10445.546885] device-mapper: multipath: Failing path 8:176. > [10445.551266] device-mapper: multipath: Failing path 67:96. > [10446.585026] scsi_io_completion: 22 callbacks suppressed > [10446.585106] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.585175] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10446.585238] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.585305] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10446.585368] blk_update_request: 22 callbacks suppressed > [10446.585427] blk_update_request: I/O error, dev sdj, sector 1952320 > [10446.585532] device-mapper: multipath: Failing path 8:144. > [10446.587225] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.587328] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10446.587409] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.587544] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 00 18 00 00 08 00 > [10446.587626] blk_update_request: I/O error, dev sdz, sector 24 > [10446.588770] device-mapper: multipath: Failing path 65:144. > [10446.596518] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.596664] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10446.596783] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.596934] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10446.597056] blk_update_request: I/O error, dev sdu, sector 608 > [10446.597236] device-mapper: multipath: Failing path 65:64. > [10446.612060] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.612156] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10446.612221] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.612292] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10446.612357] blk_update_request: I/O error, dev sdk, sector 1952320 > [10446.612575] device-mapper: multipath: Failing path 8:160. > [10446.613185] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.613324] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10446.613442] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.613576] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10446.613694] blk_update_request: I/O error, dev sdad, sector 1952320 > [10446.613897] device-mapper: multipath: Failing path 65:208. > [10446.618945] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.619076] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10446.619186] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.619329] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 00 20 00 00 18 00 > [10446.619458] blk_update_request: I/O error, dev sdai, sector 32 > [10446.619717] device-mapper: multipath: Failing path 66:32. > [10446.623887] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.624020] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10446.624136] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.624269] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10446.624374] blk_update_request: I/O error, dev sdo, sector 608 > [10446.624660] device-mapper: multipath: Failing path 8:224. > [10446.630687] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.630831] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10446.630956] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.631081] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 70 00 00 90 00 > [10446.631193] blk_update_request: I/O error, dev sdas, sector 1952368 > [10446.631370] device-mapper: multipath: Failing path 66:192. > [10446.701475] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.701555] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10446.701616] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.701679] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10446.701738] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10446.701834] device-mapper: multipath: Failing path 67:224. > [10446.722225] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.722330] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10446.722422] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.722506] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10446.722604] blk_update_request: I/O error, dev sdar, sector 1952320 > [10446.722745] device-mapper: multipath: Failing path 66:176. > [10447.703907] device-mapper: multipath: Failing path 65:160. > [10447.704418] device-mapper: multipath: Failing path 66:64. > [10447.711405] device-mapper: multipath: Failing path 65:176. > [10447.717415] device-mapper: multipath: Failing path 66:208. > [10447.736685] device-mapper: multipath: Failing path 68:16. > [10447.737169] device-mapper: multipath: Failing path 67:192. > [10447.742386] device-mapper: multipath: Failing path 68:0. > [10448.746828] device-mapper: multipath: Failing path 8:112. > [10448.752272] device-mapper: multipath: Failing path 8:96. > [10448.777442] device-mapper: multipath: Failing path 8:240. > [10448.778049] device-mapper: multipath: Failing path 66:80. > [10448.783876] device-mapper: multipath: Failing path 65:192. > [10448.784283] device-mapper: multipath: Failing path 66:160. > [10448.846609] device-mapper: multipath: Failing path 67:144. > [10449.839649] device-mapper: multipath: Failing path 66:48. > [10449.846890] device-mapper: multipath: Failing path 67:128. > [10449.855407] device-mapper: multipath: Failing path 67:64. > [10450.875424] device-mapper: multipath: Failing path 8:128. > [10450.875881] device-mapper: multipath: Failing path 65:80. > [10450.876903] device-mapper: multipath: Failing path 67:96. > [10450.883995] device-mapper: multipath: Failing path 65:128. > [10450.886613] device-mapper: multipath: Failing path 8:176. > [10451.926827] scsi_io_completion: 22 callbacks suppressed > [10451.926909] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.926974] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10451.927034] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.927099] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10451.927160] blk_update_request: 22 callbacks suppressed > [10451.927219] blk_update_request: I/O error, dev sdj, sector 1952320 > [10451.927337] device-mapper: multipath: Failing path 8:144. > [10451.930152] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.930249] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10451.930332] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.930451] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 00 78 00 00 08 00 > [10451.930536] blk_update_request: I/O error, dev sdz, sector 120 > [10451.930684] device-mapper: multipath: Failing path 65:144. > [10451.935021] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.935127] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10451.935186] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.935253] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10451.935309] blk_update_request: I/O error, dev sdu, sector 608 > [10451.935450] device-mapper: multipath: Failing path 65:64. > [10451.937343] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.937431] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10451.937506] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.937625] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10451.937705] blk_update_request: I/O error, dev sdk, sector 1952320 > [10451.937857] device-mapper: multipath: Failing path 8:160. > [10451.944256] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.944355] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10451.944434] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.944502] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 00 10 00 00 08 00 > [10451.944560] blk_update_request: I/O error, dev sdad, sector 16 > [10451.944650] device-mapper: multipath: Failing path 65:208. > [10451.948096] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.948166] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10451.948221] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.948282] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 03 60 00 00 60 00 > [10451.948338] blk_update_request: I/O error, dev sdo, sector 864 > [10451.948431] device-mapper: multipath: Failing path 8:224. > [10451.950866] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.950938] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10451.950992] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.951048] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d c9 80 00 00 80 00 > [10451.951102] blk_update_request: I/O error, dev sdai, sector 1952128 > [10451.951194] device-mapper: multipath: Failing path 66:32. > [10452.032024] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10452.032084] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10452.032122] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10452.032163] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10452.032199] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10452.032299] device-mapper: multipath: Failing path 67:224. > [10452.038388] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10452.038449] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10452.038500] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10452.038552] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10452.038596] blk_update_request: I/O error, dev sdas, sector 1952320 > [10452.038721] device-mapper: multipath: Failing path 66:192. > [10452.045632] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10452.045688] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10452.045733] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10452.045779] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00 > [10452.045822] blk_update_request: I/O error, dev sdar, sector 704 > [10452.045927] device-mapper: multipath: Failing path 66:176. > [10453.023588] device-mapper: multipath: Failing path 65:160. > [10453.024299] device-mapper: multipath: Failing path 66:64. > [10453.030250] device-mapper: multipath: Failing path 65:176. > [10453.036030] device-mapper: multipath: Failing path 66:208. > [10453.082442] device-mapper: multipath: Failing path 68:16. > [10453.082943] device-mapper: multipath: Failing path 67:192. > [10453.088366] device-mapper: multipath: Failing path 68:0. > [10454.101354] device-mapper: multipath: Failing path 8:112. > [10454.109728] device-mapper: multipath: Failing path 8:96. > [10454.122180] device-mapper: multipath: Failing path 8:240. > [10454.123095] device-mapper: multipath: Failing path 66:80. > [10454.133511] device-mapper: multipath: Failing path 65:192. > [10454.218507] device-mapper: multipath: Failing path 67:144. > [10454.225052] device-mapper: multipath: Failing path 66:160. > [10455.184119] device-mapper: multipath: Failing path 66:48. > [10455.192576] device-mapper: multipath: Failing path 67:128. > [10455.199838] device-mapper: multipath: Failing path 67:64. > [10456.219579] device-mapper: multipath: Failing path 8:128. > [10456.220649] device-mapper: multipath: Failing path 65:80. > [10456.224363] device-mapper: multipath: Failing path 8:176. > [10456.228154] device-mapper: multipath: Failing path 65:128. > [10456.236087] device-mapper: multipath: Failing path 67:96. > [10457.265228] scsi_io_completion: 22 callbacks suppressed > [10457.265309] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.265376] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10457.265441] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.265510] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10457.265573] blk_update_request: 22 callbacks suppressed > [10457.265631] blk_update_request: I/O error, dev sdj, sector 1952320 > [10457.265727] device-mapper: multipath: Failing path 8:144. > [10457.266608] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.266722] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10457.266805] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.266929] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d c1 e8 00 00 08 00 > [10457.267015] blk_update_request: I/O error, dev sdz, sector 1950184 > [10457.267159] device-mapper: multipath: Failing path 65:144. > [10457.273904] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.274003] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10457.274078] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.274161] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 88 00 00 60 00 > [10457.274236] blk_update_request: I/O error, dev sdu, sector 648 > [10457.274357] device-mapper: multipath: Failing path 65:64. > [10457.275488] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.275583] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10457.275670] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.275796] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10457.275874] blk_update_request: I/O error, dev sdk, sector 1952320 > [10457.276184] device-mapper: multipath: Failing path 8:160. > [10457.276668] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.276766] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10457.276841] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.276925] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10457.277005] blk_update_request: I/O error, dev sdad, sector 1952320 > [10457.277106] device-mapper: multipath: Failing path 65:208. > [10457.285381] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.285455] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10457.285512] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.285573] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 f0 00 00 10 00 > [10457.285817] blk_update_request: I/O error, dev sdo, sector 752 > [10457.285944] device-mapper: multipath: Failing path 8:224. > [10457.286392] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.286477] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10457.286552] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.286632] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 02 f0 00 00 10 00 > [10457.286707] blk_update_request: I/O error, dev sdai, sector 752 > [10457.286828] device-mapper: multipath: Failing path 66:32. > [10457.294129] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.294220] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10457.294295] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.294364] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d cb b0 00 00 50 00 > [10457.294424] blk_update_request: I/O error, dev sdas, sector 1952688 > [10457.294523] device-mapper: multipath: Failing path 66:192. > [10457.372572] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.372641] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10457.372688] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.372737] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10457.372783] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10457.372852] device-mapper: multipath: Failing path 67:224. > [10457.379743] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.379833] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10457.379889] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.379955] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10457.380011] blk_update_request: I/O error, dev sdar, sector 1952320 > [10457.380163] device-mapper: multipath: Failing path 66:176. > [10458.355184] device-mapper: multipath: Failing path 65:160. > [10458.355600] device-mapper: multipath: Failing path 66:64. > [10458.362158] device-mapper: multipath: Failing path 65:176. > [10458.367380] device-mapper: multipath: Failing path 66:208. > [10458.388284] device-mapper: multipath: Failing path 68:16. > [10458.389179] device-mapper: multipath: Failing path 67:192. > [10458.395063] device-mapper: multipath: Failing path 68:0. > [10459.421645] device-mapper: multipath: Failing path 8:112. > [10459.427728] device-mapper: multipath: Failing path 8:96. > [10459.433866] device-mapper: multipath: Failing path 67:144. > [10459.444039] device-mapper: multipath: Failing path 8:240. > [10459.444657] device-mapper: multipath: Failing path 66:80. > [10459.453974] device-mapper: multipath: Failing path 65:192. > [10459.454422] device-mapper: multipath: Failing path 66:160. > [10460.482933] device-mapper: multipath: Failing path 66:48. > [10460.488513] device-mapper: multipath: Failing path 67:128. > [10460.495239] device-mapper: multipath: Failing path 67:64. > [10461.513588] device-mapper: multipath: Failing path 8:128. > [10461.514034] device-mapper: multipath: Failing path 65:80. > [10461.521182] device-mapper: multipath: Failing path 65:128. > [10461.522550] device-mapper: multipath: Failing path 8:176. > [10461.528881] device-mapper: multipath: Failing path 67:96. > [10462.564110] scsi_io_completion: 22 callbacks suppressed > [10462.564202] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.564288] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10462.564368] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.564466] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10462.564557] blk_update_request: 22 callbacks suppressed > [10462.564640] blk_update_request: I/O error, dev sdj, sector 1952320 > [10462.564763] device-mapper: multipath: Failing path 8:144. > [10462.566732] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.566834] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10462.566929] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.567055] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 00 18 00 00 08 00 > [10462.567137] blk_update_request: I/O error, dev sdz, sector 24 > [10462.567301] device-mapper: multipath: Failing path 65:144. > [10462.573104] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.573192] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10462.573263] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.573332] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10462.573390] blk_update_request: I/O error, dev sdu, sector 608 > [10462.573467] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.573472] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10462.573477] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.573481] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10462.573485] blk_update_request: I/O error, dev sdk, sector 1952320 > [10462.573547] device-mapper: multipath: Failing path 8:160. > [10462.574255] device-mapper: multipath: Failing path 65:64. > [10462.574825] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.574884] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10462.574938] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.574989] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 08 00 00 00 08 00 > [10462.575044] blk_update_request: I/O error, dev sdad, sector 2048 > [10462.575121] device-mapper: multipath: Failing path 65:208. > [10462.581259] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.581316] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10462.581365] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.581413] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 b0 00 > [10462.581469] blk_update_request: I/O error, dev sdo, sector 768 > [10462.581594] device-mapper: multipath: Failing path 8:224. > [10462.583891] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.583954] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10462.584001] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.584053] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d c9 60 00 00 60 00 > [10462.584103] blk_update_request: I/O error, dev sdai, sector 1952096 > [10462.584219] device-mapper: multipath: Failing path 66:32. > [10462.686906] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.686982] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10462.687040] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.687097] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10462.687151] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10462.687234] device-mapper: multipath: Failing path 67:224. > [10462.692048] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.692112] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10462.692159] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.692210] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10462.692259] blk_update_request: I/O error, dev sdas, sector 1952320 > [10462.692330] device-mapper: multipath: Failing path 66:192. > [10462.699023] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.699089] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10462.699139] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.699190] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 60 00 > [10462.699246] blk_update_request: I/O error, dev sdar, sector 768 > [10462.699413] device-mapper: multipath: Failing path 66:176. > [10463.689681] device-mapper: multipath: Failing path 65:160. > [10463.690254] device-mapper: multipath: Failing path 66:64. > [10463.697821] device-mapper: multipath: Failing path 65:176. > [10463.705424] device-mapper: multipath: Failing path 66:208. > [10463.726975] device-mapper: multipath: Failing path 68:16. > [10463.727373] device-mapper: multipath: Failing path 67:192. > [10463.736754] device-mapper: multipath: Failing path 68:0. > [10464.765376] device-mapper: multipath: Failing path 8:112. > [10464.772388] device-mapper: multipath: Failing path 8:96. > [10464.784249] device-mapper: multipath: Failing path 8:240. > [10464.784747] device-mapper: multipath: Failing path 66:80. > [10464.792111] device-mapper: multipath: Failing path 65:192. > [10464.798740] device-mapper: multipath: Failing path 66:160. > [10464.852449] device-mapper: multipath: Failing path 67:144. > [10465.837027] device-mapper: multipath: Failing path 66:48. > [10465.840983] device-mapper: multipath: Failing path 67:128. > [10465.847920] device-mapper: multipath: Failing path 67:64. > [10466.866019] device-mapper: multipath: Failing path 8:128. > [10466.866759] device-mapper: multipath: Failing path 65:80. > [10466.872622] device-mapper: multipath: Failing path 8:176. > [10466.874797] device-mapper: multipath: Failing path 65:128. > [10466.882759] device-mapper: multipath: Failing path 67:96. > [10467.910924] scsi_io_completion: 22 callbacks suppressed > [10467.910999] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.911058] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10467.911116] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.911178] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10467.911235] blk_update_request: 22 callbacks suppressed > [10467.911287] blk_update_request: I/O error, dev sdj, sector 1952320 > [10467.911374] device-mapper: multipath: Failing path 8:144. > [10467.912763] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.912851] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10467.912921] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.912996] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 08 00 00 00 08 00 > [10467.913102] blk_update_request: I/O error, dev sdz, sector 2048 > [10467.913339] device-mapper: multipath: Failing path 65:144. > [10467.919680] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.919777] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10467.919873] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.919942] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10467.920021] blk_update_request: I/O error, dev sdu, sector 608 > [10467.920234] device-mapper: multipath: Failing path 65:64. > [10467.920949] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.921039] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10467.921110] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.921225] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10467.921310] blk_update_request: I/O error, dev sdk, sector 1952320 > [10467.921462] device-mapper: multipath: Failing path 8:160. > [10467.925848] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.925922] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10467.925987] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.926058] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 00 20 00 00 18 00 > [10467.926266] blk_update_request: I/O error, dev sdad, sector 32 > [10467.926765] device-mapper: multipath: Failing path 65:208. > [10467.931080] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.931152] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10467.931207] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.931262] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 88 00 > [10467.931316] blk_update_request: I/O error, dev sdo, sector 768 > [10467.931409] device-mapper: multipath: Failing path 8:224. > [10467.935526] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.935597] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10467.935657] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.935717] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d ca 00 00 00 18 00 > [10467.935772] blk_update_request: I/O error, dev sdai, sector 1952256 > [10467.935916] device-mapper: multipath: Failing path 66:32. > [10468.010601] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10468.010674] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10468.010723] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10468.010775] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10468.010823] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10468.010900] device-mapper: multipath: Failing path 67:224. > [10468.017011] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10468.017091] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10468.017156] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10468.017224] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 68 00 00 08 00 > [10468.017293] blk_update_request: I/O error, dev sdas, sector 1952360 > [10468.017447] device-mapper: multipath: Failing path 66:192. > [10468.018534] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10468.018613] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10468.018679] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10468.018750] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 18 00 00 08 00 > [10468.018815] blk_update_request: I/O error, dev sdar, sector 1952280 > [10468.018916] device-mapper: multipath: Failing path 66:176. > [10469.006483] device-mapper: multipath: Failing path 65:160. > [10469.006965] device-mapper: multipath: Failing path 66:64. > [10469.012283] device-mapper: multipath: Failing path 65:176. > [10469.016967] device-mapper: multipath: Failing path 66:208. > [10469.047129] device-mapper: multipath: Failing path 68:16. > [10469.047451] device-mapper: multipath: Failing path 67:192. > [10469.052379] device-mapper: multipath: Failing path 68:0. > [10470.072818] device-mapper: multipath: Failing path 8:112. > [10470.079865] device-mapper: multipath: Failing path 8:96. > [10470.101784] device-mapper: multipath: Failing path 8:240. > [10470.102397] device-mapper: multipath: Failing path 66:80. > [10470.111733] device-mapper: multipath: Failing path 65:192. > [10470.112277] device-mapper: multipath: Failing path 66:160. > [10470.169999] device-mapper: multipath: Failing path 67:144. > [10471.157267] device-mapper: multipath: Failing path 66:48. > [10471.166331] device-mapper: multipath: Failing path 67:128. > [10471.172806] device-mapper: multipath: Failing path 67:64. > [10472.191121] device-mapper: multipath: Failing path 8:128. > [10472.191627] device-mapper: multipath: Failing path 65:80. > [10472.192259] device-mapper: multipath: Failing path 8:176. > [10472.198851] device-mapper: multipath: Failing path 65:128. > [10472.204424] device-mapper: multipath: Failing path 67:96. > [10473.252723] scsi_io_completion: 22 callbacks suppressed > [10473.252798] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.252860] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10473.252918] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.252981] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10473.253045] blk_update_request: 22 callbacks suppressed > [10473.253099] blk_update_request: I/O error, dev sdj, sector 1952320 > [10473.253236] device-mapper: multipath: Failing path 8:144. > [10473.254825] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.254913] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10473.254982] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.255051] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d c1 e8 00 00 08 00 > [10473.255118] blk_update_request: I/O error, dev sdz, sector 1950184 > [10473.255219] device-mapper: multipath: Failing path 65:144. > [10473.262826] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.262921] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10473.262984] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.263052] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10473.263123] blk_update_request: I/O error, dev sdu, sector 608 > [10473.263268] device-mapper: multipath: Failing path 65:64. > [10473.264610] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.264682] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10473.264744] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.264807] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10473.264861] blk_update_request: I/O error, dev sdk, sector 1952320 > [10473.265012] device-mapper: multipath: Failing path 8:160. > [10473.265457] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.265532] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10473.265589] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.265648] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10473.265704] blk_update_request: I/O error, dev sdad, sector 1952320 > [10473.265790] device-mapper: multipath: Failing path 65:208. > [10473.272881] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.272984] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10473.273069] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.273166] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 02 00 00 00 60 00 > [10473.273255] blk_update_request: I/O error, dev sdai, sector 512 > [10473.273479] device-mapper: multipath: Failing path 66:32. > [10473.273980] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.274079] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10473.274168] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.274272] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 00 00 00 60 00 > [10473.274362] blk_update_request: I/O error, dev sdo, sector 512 > [10473.274532] device-mapper: multipath: Failing path 8:224. > [10473.286394] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.286470] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10473.286533] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.286597] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d cb b0 00 00 50 00 > [10473.286658] blk_update_request: I/O error, dev sdas, sector 1952688 > [10473.287437] device-mapper: multipath: Failing path 66:192. > [10473.363750] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.363816] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10473.363861] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.363908] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10473.363957] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10473.364066] device-mapper: multipath: Failing path 67:224. > [10473.365091] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.365138] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10473.365175] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.365215] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 28 00 00 08 00 > [10473.365269] blk_update_request: I/O error, dev sdar, sector 1952296 > [10473.365347] device-mapper: multipath: Failing path 66:176. > [10474.346552] device-mapper: multipath: Failing path 65:160. > [10474.347094] device-mapper: multipath: Failing path 66:64. > [10474.354623] device-mapper: multipath: Failing path 65:176. > [10474.362549] device-mapper: multipath: Failing path 66:208. > [10474.363551] device-mapper: multipath: Failing path 68:16. > [10474.364351] device-mapper: multipath: Failing path 67:192. > [10474.369349] device-mapper: multipath: Failing path 68:0. > [10475.402882] device-mapper: multipath: Failing path 8:112. > [10475.409928] device-mapper: multipath: Failing path 8:96. > [10475.422390] device-mapper: multipath: Failing path 8:240. > [10475.423089] device-mapper: multipath: Failing path 66:80. > [10475.431476] device-mapper: multipath: Failing path 65:192. > [10475.437756] device-mapper: multipath: Failing path 66:160. > [10475.497531] device-mapper: multipath: Failing path 67:144. > [10476.475289] device-mapper: multipath: Failing path 66:48. > [10476.481673] device-mapper: multipath: Failing path 67:128. > [10476.487874] device-mapper: multipath: Failing path 67:64. > [10477.508134] device-mapper: multipath: Failing path 8:128. > [10477.508759] device-mapper: multipath: Failing path 65:80. > [10477.509813] device-mapper: multipath: Failing path 67:96. > [10477.514796] device-mapper: multipath: Failing path 8:176. > [10477.515948] device-mapper: multipath: Failing path 65:128. > [10478.553204] scsi_io_completion: 22 callbacks suppressed > [10478.553279] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.553345] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10478.553413] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.553482] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10478.553554] blk_update_request: 22 callbacks suppressed > [10478.553611] blk_update_request: I/O error, dev sdj, sector 1952320 > [10478.553731] device-mapper: multipath: Failing path 8:144. > [10478.554502] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.554579] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10478.554637] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.554700] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10478.554757] blk_update_request: I/O error, dev sdz, sector 1952320 > [10478.555100] device-mapper: multipath: Failing path 65:144. > [10478.560256] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.560329] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10478.560385] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.560449] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10478.560503] blk_update_request: I/O error, dev sdk, sector 1952320 > [10478.560729] device-mapper: multipath: Failing path 8:160. > [10478.561184] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.561303] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10478.561404] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.561504] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10478.561610] blk_update_request: I/O error, dev sdad, sector 1952320 > [10478.561831] device-mapper: multipath: Failing path 65:208. > [10478.562889] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.562993] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10478.563053] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.563116] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 68 00 00 60 00 > [10478.563174] blk_update_request: I/O error, dev sdu, sector 616 > [10478.563274] device-mapper: multipath: Failing path 65:64. > [10478.563686] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.563771] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10478.563848] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.563931] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 00 00 18 00 00 08 00 > [10478.564008] blk_update_request: I/O error, dev sdas, sector 24 > [10478.564132] device-mapper: multipath: Failing path 66:192. > [10478.570509] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.570583] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10478.570643] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.570705] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 01 00 00 > [10478.570762] blk_update_request: I/O error, dev sdar, sector 768 > [10478.570853] device-mapper: multipath: Failing path 66:176. > [10478.571267] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.571333] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10478.571400] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.571463] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 01 00 00 > [10478.571522] blk_update_request: I/O error, dev sdai, sector 768 > [10478.571624] device-mapper: multipath: Failing path 66:32. > [10478.577878] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.577956] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10478.578016] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.578078] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 10 00 00 00 08 00 > [10478.578136] blk_update_request: I/O error, dev sdo, sector 4096 > [10478.578222] device-mapper: multipath: Failing path 8:224. > [10478.646314] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.646387] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10478.646439] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.646491] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d cc f0 00 00 08 00 > [10478.646541] blk_update_request: I/O error, dev sdbk, sector 1953008 > [10478.646616] device-mapper: multipath: Failing path 67:224. > [10479.627635] device-mapper: multipath: Failing path 65:160. > [10479.628207] device-mapper: multipath: Failing path 66:64. > [10479.636021] device-mapper: multipath: Failing path 65:176. > [10479.643095] device-mapper: multipath: Failing path 66:208. > [10479.677320] device-mapper: multipath: Failing path 68:16. > [10479.677851] device-mapper: multipath: Failing path 67:192. > [10479.683638] device-mapper: multipath: Failing path 68:0. > [10480.677063] device-mapper: multipath: Failing path 8:112. > [10480.681958] device-mapper: multipath: Failing path 8:96. > [10480.683898] device-mapper: multipath: Failing path 8:240. > [10480.684200] device-mapper: multipath: Failing path 66:80. > [10480.685455] device-mapper: multipath: Failing path 67:144. > [10480.689543] device-mapper: multipath: Failing path 65:192. > [10480.694580] device-mapper: multipath: Failing path 66:160. > [10481.719022] device-mapper: multipath: Failing path 66:48. > [10481.729178] device-mapper: multipath: Failing path 67:128. > [10481.734260] device-mapper: multipath: Failing path 67:64. > [10482.756839] device-mapper: multipath: Failing path 8:128. > [10482.763827] device-mapper: multipath: Failing path 65:80. > [10482.773300] device-mapper: multipath: Failing path 65:128. > [10482.780948] device-mapper: multipath: Failing path 8:176. > [10483.833855] scsi_io_completion: 21 callbacks suppressed > [10483.833924] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10483.833996] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10483.834059] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10483.834123] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10483.834192] blk_update_request: 21 callbacks suppressed > [10483.834252] blk_update_request: I/O error, dev sdad, sector 1952320 > [10483.834482] device-mapper: multipath: Failing path 65:208. > [10483.835438] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10483.835506] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10483.835566] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10483.835631] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10483.835696] blk_update_request: I/O error, dev sdas, sector 1952320 > [10483.835964] device-mapper: multipath: Failing path 66:192. > [10483.842954] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10483.843039] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10483.843118] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10483.843199] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 60 00 > [10483.843271] blk_update_request: I/O error, dev sdai, sector 768 > [10483.843552] device-mapper: multipath: Failing path 66:32. > [10484.844848] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10484.844932] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10484.845002] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10484.845081] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10484.845152] blk_update_request: I/O error, dev sdk, sector 1952320 > [10484.845737] device-mapper: multipath: Failing path 8:160. > [10509.568010] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358] > [10509.568088] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10509.575101] irq event stamp: 15233868 > [10509.575158] hardirqs last enabled at (15233867): [<ffffffff81578167>] _raw_spin_unlock_irq+0x27/0x40 > [10509.575265] hardirqs last disabled at (15233868): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10509.575370] softirqs last enabled at (15233850): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10509.575836] softirqs last disabled at (15233743): [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10509.575960] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10509.576023] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10509.576086] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10509.576149] RIP: 0010:[<ffffffff8157816f>] [<ffffffff8157816f>] _raw_spin_unlock_irq+0x2f/0x40 > [10509.576263] RSP: 0018:ffff880400fcbcd8 EFLAGS: 00000246 > [10509.576321] RAX: ffff88040120af80 RBX: ffff8800694542c0 RCX: 0000000000000001 > [10509.576383] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88040120af80 > [10509.576440] RBP: ffff880400fcbce0 R08: ffff8800694543c0 R09: 0000000000000000 > [10509.576498] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88040caf2290 > [10509.576557] R13: ffff880069453a00 R14: ffff8803b724ab80 R15: 0000000000000000 > [10509.576611] FS: 0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10509.576667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10509.576720] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10509.576772] Stack: > [10509.576821] ffff880455f0dcd0 ffff880400fcbd48 ffffffff813dbe28 ffff88040120af80 > [10509.577057] ffff88040caf25d8 ffff8804203cefa8 ffff880455f0dd50 0000000000000046 > [10509.577310] ffff880455f0dd00 ffff880069453a00 ffff880069453a00 0000000000000282 > [10509.577552] Call Trace: > [10509.577606] [<ffffffff813dbe28>] scsi_request_fn+0x118/0x600 > [10509.577666] [<ffffffff8128190e>] __blk_run_queue+0x2e/0x40 > [10509.577723] [<ffffffff81281175>] __elv_add_request+0x75/0x1f0 > [10509.577781] [<ffffffff81285b11>] blk_insert_cloned_request+0x101/0x190 > [10509.577845] [<ffffffffa0170eee>] map_request+0x18e/0x210 [dm_mod] > [10509.577909] [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod] > [10509.577969] [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0 > [10509.578028] [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60 > [10509.578085] [<ffffffff8107591a>] kthread+0xea/0x100 > [10509.578142] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10509.578199] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10509.578255] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10509.578310] Code: 00 00 48 89 e5 53 48 89 fb 48 8b 55 08 48 8d 7f 18 e8 26 cf b2 ff 48 89 df e8 8e 00 b3 ff e8 59 a8 b2 ff fb 65 ff 0d a1 2b a9 7e <5b> 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 > [10537.567212] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358] > [10537.567287] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10537.573758] irq event stamp: 30645910 > [10537.573810] hardirqs last enabled at (30645909): [<ffffffff81578167>] _raw_spin_unlock_irq+0x27/0x40 > [10537.573918] hardirqs last disabled at (30645910): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10537.574020] softirqs last enabled at (30645860): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10537.574122] softirqs last disabled at (30645755): [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10537.574223] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10537.574279] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10537.574335] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10537.574390] RIP: 0010:[<ffffffff8157816f>] [<ffffffff8157816f>] _raw_spin_unlock_irq+0x2f/0x40 > [10537.574492] RSP: 0018:ffff880400fcbcd8 EFLAGS: 00000246 > [10537.574543] RAX: ffff88040120af80 RBX: ffff880444673440 RCX: 0000000000000001 > [10537.574594] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88040120af80 > [10537.574645] RBP: ffff880400fcbce0 R08: ffff880444673540 R09: 0000000000000000 > [10537.575221] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8804426a4520 > [10537.575274] R13: ffff880444672b80 R14: ffff88033fd9a358 R15: 0000000000000000 > [10537.575327] FS: 0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10537.575382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10537.575435] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10537.575487] Stack: > [10537.575536] ffff88040161d388 ffff880400fcbd48 ffffffff813dbe28 ffff88040120af80 > [10537.575783] ffff8804426a4868 ffff880415a49c20 ffff88040161d408 0000000000000046 > [10537.576029] ffff88040161d3b8 ffff880444672b80 ffff880444672b80 0000000000000282 > [10537.576270] Call Trace: > [10537.576324] [<ffffffff813dbe28>] scsi_request_fn+0x118/0x600 > [10537.576384] [<ffffffff8128190e>] __blk_run_queue+0x2e/0x40 > [10537.576441] [<ffffffff81281175>] __elv_add_request+0x75/0x1f0 > [10537.576498] [<ffffffff81285b11>] blk_insert_cloned_request+0x101/0x190 > [10537.576561] [<ffffffffa0170eee>] map_request+0x18e/0x210 [dm_mod] > [10537.576624] [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod] > [10537.576685] [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0 > [10537.576742] [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60 > [10537.576798] [<ffffffff8107591a>] kthread+0xea/0x100 > [10537.576855] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10537.576911] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10537.576967] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10537.577022] Code: 00 00 48 89 e5 53 48 89 fb 48 8b 55 08 48 8d 7f 18 e8 26 cf b2 ff 48 89 df e8 8e 00 b3 ff e8 59 a8 b2 ff fb 65 ff 0d a1 2b a9 7e <5b> 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 > [10546.926940] INFO: rcu_sched self-detected stall on CPU > [10546.927120] 3-...: (6497 ticks this GP) idle=fb9/140000000000002/0 softirq=2044956/2045037 fqs=5414 > [10546.927219] (t=6500 jiffies g=219289 c=219288 q=7233211) > [10546.927401] Task dump for CPU 3: > [10546.927486] kdmwork-254:2 R running task 0 358 2 0x00000008 > [10546.927736] 0000000000000002 ffff88046f263848 ffffffff8108195f 0000000000000003 > [10546.928161] ffffffff81a40a40 ffff88046f263860 ffffffff81084742 0000000000000004 > [10546.928578] ffff88046f263890 ffffffff810b5e79 ffff88046f275400 ffffffff81a40a40 > [10546.928989] Call Trace: > [10546.929076] <IRQ> [<ffffffff8108195f>] sched_show_task+0xbf/0x150 > [10546.929253] [<ffffffff81084742>] dump_cpu_task+0x32/0x40 > [10546.929346] [<ffffffff810b5e79>] rcu_dump_cpu_stacks+0x89/0xe0 > [10546.929438] [<ffffffff810b9999>] rcu_check_callbacks+0x439/0x730 > [10546.929529] [<ffffffff8109e769>] ? cpuacct_account_field+0xd9/0x1b0 > [10546.929621] [<ffffffff8109e690>] ? cpuacct_charge+0x1a0/0x1a0 > [10546.929715] [<ffffffff810bc1c4>] update_process_times+0x34/0x60 > [10546.929807] [<ffffffff810caaa0>] tick_sched_handle.isra.18+0x20/0x50 > [10546.929899] [<ffffffff810cb148>] tick_sched_timer+0x38/0x70 > [10546.929991] [<ffffffff810cb110>] ? tick_sched_do_timer+0x30/0x30 > [10546.930083] [<ffffffff810bc865>] __hrtimer_run_queues+0xa5/0x1c0 > [10546.930176] [<ffffffff810bcef6>] hrtimer_interrupt+0xa6/0x1b0 > [10546.930267] [<ffffffff8109fb3d>] ? trace_hardirqs_off+0xd/0x10 > [10546.930360] [<ffffffff81038ba3>] smp_trace_apic_timer_interrupt+0x63/0x90 > [10546.930451] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10546.930544] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10546.930641] [<ffffffff815781b3>] ? _raw_spin_unlock_irqrestore+0x33/0x50 > [10546.930732] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10546.930835] [<ffffffff810b71c2>] ? call_rcu_sched+0x12/0x20 > [10546.930929] [<ffffffff8117baf8>] ? put_object+0x28/0x40 > [10546.931023] [<ffffffff810a2bf1>] ? debug_check_no_locks_freed+0xb1/0x140 > [10546.931117] [<ffffffff811d0a12>] ? kiocb_free+0x32/0x40 > [10546.931209] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10546.931301] [<ffffffff811d0a12>] kiocb_free+0x32/0x40 > [10546.931395] [<ffffffff811d1b75>] aio_complete+0x1e5/0x3c0 > [10546.931488] [<ffffffff811bef45>] dio_complete+0x75/0x1d0 > [10546.931582] [<ffffffff811bf11a>] dio_bio_end_aio+0x7a/0x130 > [10546.931675] [<ffffffff812800da>] bio_endio+0x3a/0x60 > [10546.931769] [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0 > [10546.931867] [<ffffffffa016fcc1>] end_clone_bio+0x41/0x70 [dm_mod] > [10546.931961] [<ffffffff812800da>] bio_endio+0x3a/0x60 > [10546.932052] [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0 > [10546.932147] [<ffffffff813da01e>] scsi_end_request+0x2e/0x1d0 > [10546.932238] [<ffffffff813dc444>] scsi_io_completion+0xb4/0x610 > [10546.932333] [<ffffffff813d3e0a>] scsi_finish_command+0xca/0x120 > [10546.932423] [<ffffffff813dbcf0>] scsi_softirq_done+0x120/0x140 > [10546.932516] [<ffffffff8128c742>] blk_done_softirq+0x72/0x90 > [10546.932607] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10546.932700] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10546.932792] [<ffffffff810065a5>] do_IRQ+0x65/0x110 > [10546.932884] [<ffffffff815793c9>] common_interrupt+0x89/0x89 > [10546.932973] <EOI> [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10546.933164] [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath] > [10546.933265] [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath] > [10546.933365] [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod] > [10546.933464] [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod] > [10546.933561] [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0 > [10546.933653] [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60 > [10546.933747] [<ffffffff8107591a>] kthread+0xea/0x100 > [10546.933841] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10546.933934] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10546.934033] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10573.566190] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358] > [10573.566286] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10573.575752] irq event stamp: 50535115 > [10573.575821] hardirqs last enabled at (50535114): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10573.575959] hardirqs last disabled at (50535115): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10573.576115] softirqs last enabled at (50535044): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10573.576259] softirqs last disabled at (50535061): [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10573.576410] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10573.576491] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10573.576571] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10573.576650] RIP: 0010:[<ffffffff815781b3>] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10573.576797] RSP: 0018:ffff88046f263b88 EFLAGS: 00000296 > [10573.576868] RAX: ffff88040120af80 RBX: 0000000000000296 RCX: 0000000000000000 > [10573.576938] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88040120af80 > [10573.577007] RBP: ffff88046f263b98 R08: ffff88046f263b08 R09: 0000000000000000 > [10573.577078] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880461584e00 > [10573.577162] R13: ffffea0009ec0400 R14: ffff880461584e00 R15: ffff880461488d80 > [10573.577240] FS: 0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10573.577323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10573.577399] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10573.577475] Stack: > [10573.577546] ffff880461584e00 0000000000000001 ffff88046f263c48 ffffffff811640d6 > [10573.577881] ffff88046f263bd8 ffff88040120af80 ffff880266ffaa80 ffff88027b011540 > [10573.578207] ffff88027b011540 ffff000109ec0400 0000000000000296 ffff88046f263bf0 > [10573.578538] Call Trace: > [10573.578608] <IRQ> > [10573.578690] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10573.578860] [<ffffffff810b71c2>] ? call_rcu_sched+0x12/0x20 > [10573.578947] [<ffffffff8117baf8>] ? put_object+0x28/0x40 > [10573.579029] [<ffffffff810a2bf1>] ? debug_check_no_locks_freed+0xb1/0x140 > [10573.579114] [<ffffffff811bef54>] ? dio_complete+0x84/0x1d0 > [10573.579203] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10573.579289] [<ffffffff811bef54>] dio_complete+0x84/0x1d0 > [10573.579372] [<ffffffff811bf11a>] dio_bio_end_aio+0x7a/0x130 > [10573.579453] [<ffffffff812800da>] bio_endio+0x3a/0x60 > [10573.579534] [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0 > [10573.579622] [<ffffffffa016fcc1>] end_clone_bio+0x41/0x70 [dm_mod] > [10573.579704] [<ffffffff812800da>] bio_endio+0x3a/0x60 > [10573.579784] [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0 > [10573.579866] [<ffffffff813da01e>] scsi_end_request+0x2e/0x1d0 > [10573.579946] [<ffffffff813dc444>] scsi_io_completion+0xb4/0x610 > [10573.580030] [<ffffffff813d3e0a>] scsi_finish_command+0xca/0x120 > [10573.580118] [<ffffffff813dbcf0>] scsi_softirq_done+0x120/0x140 > [10573.580210] [<ffffffff8128c742>] blk_done_softirq+0x72/0x90 > [10573.580296] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10573.580374] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10573.580453] [<ffffffff810065a5>] do_IRQ+0x65/0x110 > [10573.580535] [<ffffffff815793c9>] common_interrupt+0x89/0x89 > [10573.580614] <EOI> > [10573.580695] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10573.580853] [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath] > [10573.580942] [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath] > [10573.581032] [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod] > [10573.581122] [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod] > [10573.581205] [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0 > [10573.581284] [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60 > [10573.581367] [<ffffffff8107591a>] kthread+0xea/0x100 > [10573.581448] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10573.581530] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10573.581609] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10573.581691] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10601.565455] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358] > [10601.565532] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10601.572008] irq event stamp: 65965438 > [10601.572063] hardirqs last enabled at (65965437): [<ffffffff8157940f>] restore_regs_and_iret+0x0/0x1d > [10601.572175] hardirqs last disabled at (65965438): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10601.572985] softirqs last enabled at (65965386): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10601.573093] softirqs last disabled at (65965185): [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10601.573195] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10601.573250] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10601.573304] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10601.573357] RIP: 0010:[<ffffffff8157816f>] [<ffffffff8157816f>] _raw_spin_unlock_irq+0x2f/0x40 > [10601.573457] RSP: 0018:ffff880400fcbcd8 EFLAGS: 00000246 > [10601.573507] RAX: ffff88040120af80 RBX: ffff8800694542c0 RCX: 0000000000000001 > [10601.573557] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88040120af80 > [10601.573609] RBP: ffff880400fcbce0 R08: ffff8800694543c0 R09: 0000000000000000 > [10601.573670] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88040caf2290 > [10601.573724] R13: ffff880069453a00 R14: ffff88020d55f230 R15: 0000000000000000 > [10601.573779] FS: 0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10601.573835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10601.573888] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10601.573941] Stack: > [10601.573990] ffff880455f0dcd0 ffff880400fcbd48 ffffffff813dbe28 ffff88040120af80 > [10601.574230] ffff88040caf25d8 ffff8804203cefa8 ffff880455f0dd50 0000000000000046 > [10601.574464] ffff880455f0dd00 ffff880069453a00 ffff880069453a00 0000000000000282 > [10601.574694] Call Trace: > [10601.574746] [<ffffffff813dbe28>] scsi_request_fn+0x118/0x600 > [10601.574804] [<ffffffff8128190e>] __blk_run_queue+0x2e/0x40 > [10601.575744] [<ffffffff81281175>] __elv_add_request+0x75/0x1f0 > [10601.575810] [<ffffffff81285b11>] blk_insert_cloned_request+0x101/0x190 > [10601.575876] [<ffffffffa0170eee>] map_request+0x18e/0x210 [dm_mod] > [10601.575941] [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod] > [10601.576004] [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0 > [10601.576063] [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60 > [10601.576122] [<ffffffff8107591a>] kthread+0xea/0x100 > [10601.576181] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10601.576240] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10601.576299] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10601.576356] Code: 00 00 48 89 e5 53 48 89 fb 48 8b 55 08 48 8d 7f 18 e8 26 cf b2 ff 48 89 df e8 8e 00 b3 ff e8 59 a8 b2 ff fb 65 ff 0d a1 2b a9 7e <5b> 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 > [10629.564604] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358] > [10629.564704] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10629.574344] irq event stamp: 81405685 > [10629.574414] hardirqs last enabled at (81405684): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10629.574557] hardirqs last disabled at (81405685): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10629.574713] softirqs last enabled at (81405466): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10629.574858] softirqs last disabled at (81405547): [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10629.574999] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10629.575074] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10629.575157] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10629.575235] RIP: 0010:[<ffffffff815781b3>] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10629.575379] RSP: 0018:ffff88046f263b08 EFLAGS: 00000292 > [10629.575455] RAX: ffff88040120af80 RBX: 0000000000000292 RCX: 0000000000000000 > [10629.575529] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88040120af80 > [10629.575603] RBP: ffff88046f263b18 R08: ffff88046f263a88 R09: 0000000000000000 > [10629.575675] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880461585340 > [10629.575748] R13: ffffea000827d500 R14: ffff880461585340 R15: ffff880461489440 > [10629.575820] FS: 0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10629.575909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10629.575984] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10629.576062] Stack: > [10629.576134] ffff880461585340 0000000000000001 ffff88046f263bc8 ffffffff811640d6 > [10629.576472] ffff88046f263b58 ffff88040120af80 ffff880168a1fcd8 ffff880209f548c0 > [10629.576808] ffff880209f548c0 ffff00010827d500 0000000000000292 ffff88046f263b70 > [10629.577140] Call Trace: > [10629.577210] <IRQ> > [10629.577282] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10629.577430] [<ffffffff810b71c2>] ? call_rcu_sched+0x12/0x20 > [10629.577512] [<ffffffff8117baf8>] ? put_object+0x28/0x40 > [10629.577610] [<ffffffff810a2bf1>] ? debug_check_no_locks_freed+0xb1/0x140 > [10629.577697] [<ffffffff811d0a12>] ? kiocb_free+0x32/0x40 > [10629.577781] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10629.577870] [<ffffffff811d0a12>] kiocb_free+0x32/0x40 > [10629.577953] [<ffffffff811d1b75>] aio_complete+0x1e5/0x3c0 > [10629.578036] [<ffffffff811bef45>] dio_complete+0x75/0x1d0 > [10629.578118] [<ffffffff811bf11a>] dio_bio_end_aio+0x7a/0x130 > [10629.578200] [<ffffffff812800da>] bio_endio+0x3a/0x60 > [10629.578283] [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0 > [10629.578371] [<ffffffffa016fcc1>] end_clone_bio+0x41/0x70 [dm_mod] > [10629.578465] [<ffffffff812800da>] bio_endio+0x3a/0x60 > [10629.578544] [<ffffffff81284b7c>] blk_update_request+0x7c/0x2a0 > [10629.578629] [<ffffffff813da01e>] scsi_end_request+0x2e/0x1d0 > [10629.578710] [<ffffffff813dc444>] scsi_io_completion+0xb4/0x610 > [10629.578791] [<ffffffff813d3e0a>] scsi_finish_command+0xca/0x120 > [10629.578872] [<ffffffff813dbcf0>] scsi_softirq_done+0x120/0x140 > [10629.578954] [<ffffffff8128c742>] blk_done_softirq+0x72/0x90 > [10629.579034] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10629.579117] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10629.579197] [<ffffffff810065a5>] do_IRQ+0x65/0x110 > [10629.579280] [<ffffffff815793c9>] common_interrupt+0x89/0x89 > [10629.579357] <EOI> > [10629.579428] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10629.579590] [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath] > [10629.579680] [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath] > [10629.579771] [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod] > [10629.579860] [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod] > [10629.579946] [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0 > [10629.580032] [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60 > [10629.580114] [<ffffffff8107591a>] kthread+0xea/0x100 > [10629.580195] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10629.580276] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10629.580358] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10629.580438] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10657.563840] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358] > [10657.563911] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10657.570523] irq event stamp: 96839370 > [10657.570577] hardirqs last enabled at (96839369): [<ffffffff8157940f>] restore_regs_and_iret+0x0/0x1d > [10657.570688] hardirqs last disabled at (96839370): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10657.570792] softirqs last enabled at (96839350): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10657.570900] softirqs last disabled at (96839343): [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10657.571007] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10657.571066] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10657.571124] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10657.571180] RIP: 0010:[<ffffffff8157816f>] [<ffffffff8157816f>] _raw_spin_unlock_irq+0x2f/0x40 > [10657.571277] RSP: 0018:ffff880400fcbd50 EFLAGS: 00000246 > [10657.571328] RAX: ffff88040120af80 RBX: ffff8803fa3a6928 RCX: 0000000000000001 > [10657.571378] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88040120af80 > [10657.571427] RBP: ffff880400fcbd58 R08: ffff880408393a80 R09: 0000000000000000 > [10657.571485] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000001000 > [10657.571536] R13: ffff8802b3be3c70 R14: ffff88042dbaa358 R15: ffff8803fa3a6900 > [10657.571587] FS: 0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10657.571641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10657.571691] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10657.571742] Stack: > [10657.571791] ffff8803fa3a6928 ffff880400fcbdb0 ffffffffa010f815 ffff880407cf26c8 > [10657.572021] ffff880408393a80 0000000000000000 0000000000000000 ffff8802b3be3c28 > [10657.572251] ffff880037cf2610 ffff88042d0fd668 ffffc900030d5040 ffff8802b3be3c48 > [10657.572485] Call Trace: > [10657.572540] [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath] > [10657.572613] [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath] > [10657.572685] [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod] > [10657.572753] [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod] > [10657.572818] [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0 > [10657.572883] [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60 > [10657.572943] [<ffffffff8107591a>] kthread+0xea/0x100 > [10657.573005] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10657.573072] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10657.573132] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10657.573191] Code: 00 00 48 89 e5 53 48 89 fb 48 8b 55 08 48 8d 7f 18 e8 26 cf b2 ff 48 89 df e8 8e 00 b3 ff e8 59 a8 b2 ff fb 65 ff 0d a1 2b a9 7e <5b> 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 > [10693.562792] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [fio:7931] > [10693.562873] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10693.569186] irq event stamp: 30263533 > [10693.569241] hardirqs last enabled at (30263532): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10693.569355] hardirqs last disabled at (30263533): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10693.570408] softirqs last enabled at (7684746): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10693.570522] softirqs last disabled at (7684923): [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10693.570633] CPU: 3 PID: 7931 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10693.570693] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10693.570755] task: ffff880064ae2f80 ti: ffff8803f7578000 task.ti: ffff8803f7578000 > [10693.570816] RIP: 0010:[<ffffffff815781b3>] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10693.570932] RSP: 0018:ffff88046f263d78 EFLAGS: 00000292 > [10693.570989] RAX: ffff880064ae2f80 RBX: 0000000000000292 RCX: 0000000000000000 > [10693.571047] RDX: 0000000000000003 RSI: ffff880064ae3718 RDI: ffff880064ae2f80 > [10693.571105] RBP: ffff88046f263d88 R08: ffff88046f263cf8 R09: 0000000000000000 > [10693.571165] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10693.571223] R13: ffffea00069c1e00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10693.571280] FS: 00007f8a98e2f700(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10693.571339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10693.571397] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10693.571456] Stack: > [10693.571512] ffff88017fc01c00 0000000000000001 ffff88046f263e38 ffffffff811640d6 > [10693.571777] ffff88046f263dc8 ffffffff81001017 0000000100300030 ffff8801a707b7c8 > [10693.572047] ffff8801a707b7c8 ffff000164ae3718 0000000000000292 ffff880064ae3718 > [10693.572314] Call Trace: > [10693.572372] <IRQ> > [10693.572434] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10693.572557] [<ffffffff81001017>] ? trace_hardirqs_on_thunk+0x17/0x19 > [10693.572622] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10693.572690] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10693.572754] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10693.572820] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10693.572884] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10693.572948] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10693.573013] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10693.574184] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10693.574252] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10693.574315] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10693.574379] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10693.574443] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10693.574509] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10693.574573] <EOI> > [10693.574635] [<ffffffff81286284>] ? blk_flush_plug_list+0x224/0x240 > [10693.574759] [<ffffffff812865c7>] blk_finish_plug+0x27/0x40 > [10693.574824] [<ffffffff811d3136>] do_io_submit+0x3c6/0x770 > [10693.574887] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10693.574951] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10693.575014] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10693.575077] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10700.274209] sd 37:0:0:2: [sdbc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10700.274285] sd 37:0:0:2: [sdbc] tag#0 Sense Key : Not Ready [current] > [10700.274341] sd 37:0:0:2: [sdbc] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10700.274400] sd 37:0:0:2: [sdbc] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00 > [10700.274455] blk_update_request: I/O error, dev sdbc, sector 0 > [10700.274586] device-mapper: multipath: Failing path 67:96. > [10725.561889] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:7937] > [10725.561895] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [fio:7930] > [10725.561907] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE > [10725.561907] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [disk11_0:32753] > [10725.561909] nf_nat_masquerade_ipv4 > [10725.561910] Modules linked in: > [10725.561910] iptable_nat > [10725.561911] ib_srpt(O-) > [10725.561912] nf_nat_ipv4 > [10725.561913] scst_vdisk(O) > [10725.561914] nf_nat > [10725.561914] libcrc32c > [10725.561915] nf_conntrack_ipv4 > [10725.561916] scst(O) > [10725.561917] nf_defrag_ipv4 > [10725.561917] dlm > [10725.561920] xt_conntrack > [10725.561920] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kdmwork-254:2:358] > [10725.561922] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [fio:7934] > [10725.561923] Modules linked in: > [10725.561924] Modules linked in: > [10725.561924] ib_srpt(O-) > [10725.561925] ib_srpt(O-) > [10725.561926] scst_vdisk(O) > [10725.561926] scst_vdisk(O) > [10725.561927] libcrc32c > [10725.561928] libcrc32c > [10725.561928] scst(O) > [10725.561931] scst(O) > [10725.561931] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [fio:7935] > [10725.561932] dlm > [10725.561933] dlm > [10725.561933] netconsole > [10725.561934] netconsole > [10725.561934] configfs > [10725.561935] configfs > [10725.561936] Modules linked in: > [10725.561937] dm_queue_length > [10725.561938] ib_srpt(O-) > [10725.561939] brd > [10725.561939] scst_vdisk(O) > [10725.561940] nf_conntrack > [10725.561943] netconsole > [10725.561943] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [fio:7940] > [10725.561944] ipt_REJECT > [10725.561945] libcrc32c > [10725.561946] configfs > [10725.561947] xt_CHECKSUM > [10725.561948] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [fio:7938] > [10725.561949] Modules linked in: > [10725.561950] Modules linked in: > [10725.561950] ib_srpt(O-) > [10725.561951] dm_queue_length > [10725.561952] scst_vdisk(O) > [10725.561953] dm_queue_length > [10725.561954] xt_tcpudp > [10725.561954] ib_srpt(O-) > [10725.561955] libcrc32c > [10725.561956] scst_vdisk(O) > [10725.561957] scst(O) > [10725.561957] libcrc32c > [10725.561958] scst(O) > [10725.561959] dlm > [10725.561960] dlm > [10725.561961] brd > [10725.561961] netconsole > [10725.561962] tun > [10725.561963] netconsole > [10725.561963] scst(O) > [10725.561964] ebtable_filter > [10725.561965] xt_CHECKSUM > [10725.561966] dlm > [10725.561966] configfs > [10725.561967] iptable_mangle > [10725.561968] netconsole > [10725.561969] configfs > [10725.561969] configfs > [10725.561970] dm_queue_length > [10725.561971] dm_queue_length > [10725.561971] ipt_MASQUERADE > [10725.561972] dm_queue_length > [10725.561973] brd > [10725.561974] brd > [10725.561974] brd > [10725.561975] brd > [10725.561976] iptable_mangle > [10725.561976] xt_CHECKSUM > [10725.561977] ipt_MASQUERADE > [10725.561978] xt_CHECKSUM > [10725.561979] xt_CHECKSUM > [10725.561980] nf_nat_masquerade_ipv4 > [10725.561980] nf_nat_masquerade_ipv4 > [10725.561981] iptable_mangle > [10725.561982] iptable_nat > [10725.561982] iptable_mangle > [10725.561983] ebtables > [10725.561984] nf_nat_ipv4 > [10725.561985] ipt_MASQUERADE > [10725.561986] iptable_nat > [10725.561986] ip6table_filter > [10725.561987] nf_nat_ipv4 > [10725.561988] iptable_mangle > [10725.561989] ip6_tables > [10725.561990] xt_CHECKSUM > [10725.561991] ipt_MASQUERADE > [10725.561991] iptable_filter > [10725.561992] nf_nat_masquerade_ipv4 > [10725.561993] nf_nat > [10725.561993] ipt_MASQUERADE > [10725.561994] iptable_mangle > [10725.561995] nf_nat_masquerade_ipv4 > [10725.561996] ip_tables > [10725.561996] nf_nat > [10725.561997] nf_nat_masquerade_ipv4 > [10725.561998] x_tables > [10725.561999] ipt_MASQUERADE > [10725.561999] iptable_nat > [10725.562000] iptable_nat > [10725.562001] nf_nat_ipv4 > [10725.562001] af_packet > [10725.562002] nf_conntrack_ipv4 > [10725.562003] nf_nat_ipv4 > [10725.562004] nf_defrag_ipv4 > [10725.562004] iptable_nat > [10725.562005] nf_conntrack_ipv4 > [10725.562007] nf_nat_ipv4 > [10725.562008] nf_defrag_ipv4 > [10725.562009] nf_nat_masquerade_ipv4 > [10725.562010] xt_conntrack > [10725.562011] xt_conntrack > [10725.562011] bridge > [10725.562012] nf_conntrack > [10725.562013] iptable_nat > [10725.562014] nf_nat > [10725.562014] stp > [10725.562015] nf_conntrack_ipv4 > [10725.562016] nf_nat > [10725.562016] nf_nat > [10725.562017] nf_conntrack > [10725.562018] nf_conntrack_ipv4 > [10725.562019] ipt_REJECT > [10725.562019] nf_defrag_ipv4 > [10725.562020] xt_tcpudp > [10725.562021] ipt_REJECT > [10725.562021] nf_conntrack_ipv4 > [10725.562022] llc > [10725.562023] xt_conntrack > [10725.562024] tun > [10725.562024] nf_conntrack > [10725.562025] ebtable_filter > [10725.562026] nf_nat_ipv4 > [10725.562026] iscsi_ibft > [10725.562027] ebtables > [10725.562028] nf_nat > [10725.562029] nf_defrag_ipv4 > [10725.562029] ip6table_filter > [10725.562030] ipt_REJECT > [10725.562031] ip6_tables > [10725.562031] xt_tcpudp > [10725.562032] iptable_filter > [10725.562033] tun > [10725.562033] iscsi_boot_sysfs > [10725.562034] ebtable_filter > [10725.562035] ip_tables > [10725.562036] xt_tcpudp > [10725.562036] nf_conntrack_ipv4 > [10725.562037] xt_conntrack > [10725.562038] ebtables > [10725.562039] x_tables > [10725.562039] nf_conntrack > [10725.562040] ib_srp > [10725.562041] ipt_REJECT > [10725.562042] tun > [10725.562042] af_packet > [10725.562043] ip6table_filter > [10725.562044] xt_tcpudp > [10725.562045] scsi_transport_srp > [10725.562045] nf_defrag_ipv4 > [10725.562046] ib_ipoib > [10725.562047] xt_conntrack > [10725.562048] ip6_tables > [10725.562048] bridge > [10725.562049] iptable_filter > [10725.562050] rdma_ucm > [10725.562051] tun > [10725.562051] ib_ucm > [10725.562052] stp > [10725.562053] ebtable_filter > [10725.562054] llc > [10725.562054] ip_tables > [10725.562055] iscsi_ibft > [10725.562056] x_tables > [10725.562056] iscsi_boot_sysfs > [10725.562057] ebtables > [10725.562058] ib_srp > [10725.562058] af_packet > [10725.562059] scsi_transport_srp > [10725.562060] bridge > [10725.562061] ib_ipoib > [10725.562061] ebtable_filter > [10725.562062] rdma_ucm > [10725.562063] stp > [10725.562063] ib_ucm > [10725.562064] llc > [10725.562064] nf_conntrack > [10725.562065] nf_defrag_ipv4 > [10725.562066] ip6table_filter > [10725.562067] iscsi_ibft > [10725.562067] ib_uverbs > [10725.562068] iscsi_boot_sysfs > [10725.562069] ib_uverbs > [10725.562069] xt_conntrack > [10725.562070] msr > [10725.562071] ip6_tables > [10725.562072] ebtables > [10725.562073] ipt_REJECT > [10725.562074] ip6table_filter > [10725.562074] ib_umad > [10725.562075] iptable_filter > [10725.562076] nf_conntrack > [10725.562076] xt_tcpudp > [10725.562077] ipt_REJECT > [10725.562078] tun > [10725.562079] rdma_cm > [10725.562079] ib_srp > [10725.562080] msr > [10725.562081] scsi_transport_srp > [10725.562081] ib_umad > [10725.562082] ib_ipoib > [10725.562083] rdma_cm > [10725.562083] ebtable_filter > [10725.562084] xt_tcpudp > [10725.562085] ib_cm > [10725.562087] ebtables > [10725.562087] ip_tables > [10725.562088] iw_cm > [10725.562089] x_tables > [10725.562089] ib_cm > [10725.562090] ip6_tables > [10725.562091] af_packet > [10725.562092] tun > [10725.562092] ip6table_filter > [10725.562093] mlx4_ib > [10725.562093] rdma_ucm > [10725.562094] ib_sa > [10725.562095] iw_cm > [10725.562096] ib_ucm > [10725.562096] ib_mad > [10725.562097] iptable_filter > [10725.562098] ebtable_filter > [10725.562099] bridge > [10725.562099] ip6_tables > [10725.562100] ebtables > [10725.562101] iptable_filter > [10725.562101] ip6table_filter > [10725.562102] ip_tables > [10725.562103] ip6_tables > [10725.562103] ib_uverbs > [10725.562104] mlx4_ib > [10725.562105] x_tables > [10725.562106] iptable_filter > [10725.562106] af_packet > [10725.562107] ip_tables > [10725.562107] bridge > [10725.562108] x_tables > [10725.562109] stp > [10725.562109] ib_sa > [10725.562110] msr > [10725.562111] stp > [10725.562112] af_packet > [10725.562112] llc > [10725.562113] bridge > [10725.562114] iscsi_ibft > [10725.562115] ip_tables > [10725.562115] ib_umad > [10725.562116] ib_core > [10725.562116] rdma_cm > [10725.562117] ib_addr > [10725.562118] ib_mad > [10725.562119] x86_pkg_temp_thermal > [10725.562120] x_tables > [10725.562120] intel_powerclamp > [10725.562121] ib_cm > [10725.562121] coretemp > [10725.562122] stp > [10725.562123] llc > [10725.562123] kvm_intel > [10725.562124] iw_cm > [10725.562125] kvm > [10725.562125] mlx4_ib > [10725.562126] irqbypass > [10725.562127] ib_core > [10725.562127] crct10dif_pclmul > [10725.562128] ib_sa > [10725.562129] crc32_pclmul > [10725.562129] ib_mad > [10725.562130] crc32c_intel > [10725.562131] ib_core > [10725.562132] af_packet > [10725.562132] iscsi_ibft > [10725.562133] ib_addr > [10725.562134] ib_addr > [10725.562134] aesni_intel > [10725.562135] x86_pkg_temp_thermal > [10725.562135] tg3 > [10725.562136] intel_powerclamp > [10725.562137] iTCO_wdt > [10725.562137] x86_pkg_temp_thermal > [10725.562138] coretemp > [10725.562139] aes_x86_64 > [10725.562140] iscsi_boot_sysfs > [10725.562140] bridge > [10725.562141] lrw > [10725.562141] kvm_intel > [10725.562142] ipmi_ssif > [10725.562143] kvm > [10725.562143] ipmi_devintf > [10725.562144] irqbypass > [10725.562145] iscsi_boot_sysfs > [10725.562145] llc > [10725.562146] ib_srp > [10725.562147] ib_srp > [10725.562147] ptp > [10725.562148] crct10dif_pclmul > [10725.562148] iTCO_vendor_support > [10725.562149] crc32_pclmul > [10725.562150] gf128mul > [10725.562150] crc32c_intel > [10725.562151] stp > [10725.562152] scsi_transport_srp > [10725.562153] scsi_transport_srp > [10725.562153] iscsi_ibft > [10725.562154] ib_ipoib > [10725.562155] intel_powerclamp > [10725.562156] ib_ipoib > [10725.562156] iscsi_boot_sysfs > [10725.562157] rdma_ucm > [10725.562158] ib_srp > [10725.562158] rdma_ucm > [10725.562159] scsi_transport_srp > [10725.562160] glue_helper > [10725.562161] ib_ucm > [10725.562161] ib_ipoib > [10725.562162] ib_ucm > [10725.562163] coretemp > [10725.562164] mlx4_core > [10725.562164] llc > [10725.562165] ib_uverbs > [10725.562166] kvm_intel > [10725.562167] aesni_intel > [10725.562167] dcdbas > [10725.562168] tg3 > [10725.562168] ablk_helper > [10725.562169] iTCO_wdt > [10725.562169] pps_core > [10725.562171] aes_x86_64 > [10725.562172] kvm > [10725.562173] lrw > [10725.562173] cryptd > [10725.562174] msr > [10725.562175] rdma_ucm > [10725.562176] libphy > [10725.562176] irqbypass > [10725.562177] ib_uverbs > [10725.562178] ib_ucm > [10725.562178] msr > [10725.562179] ib_uverbs > [10725.562180] ib_umad > [10725.562181] crct10dif_pclmul > [10725.562181] ib_umad > [10725.562182] crc32_pclmul > [10725.562183] ipmi_ssif > [10725.562183] fjes > [10725.562184] ipmi_devintf > [10725.562185] 8250_fintek > [10725.562185] ptp > [10725.562186] sb_edac > [10725.562186] iTCO_vendor_support > [10725.562187] pcspkr > [10725.562188] rdma_cm > [10725.562189] crc32c_intel > [10725.562189] rdma_cm > [10725.562190] iscsi_ibft > [10725.562191] aesni_intel > [10725.562192] ib_cm > [10725.562192] msr > [10725.562193] ib_cm > [10725.562194] ib_umad > [10725.562194] iw_cm > [10725.562195] rdma_cm > [10725.562196] mlx4_ib > [10725.562196] iscsi_boot_sysfs > [10725.562197] iw_cm > [10725.562198] tg3 > [10725.562199] gf128mul > [10725.562199] ipmi_si > [10725.562200] ib_sa > [10725.562201] mlx4_ib > [10725.562202] ib_srp > [10725.562202] edac_core > [10725.562203] glue_helper > [10725.562204] ipmi_msghandler > [10725.562204] mlx4_core > [10725.562205] acpi_power_meter > [10725.562206] ib_cm > [10725.562206] ib_mad > [10725.562207] iw_cm > [10725.562208] scsi_transport_srp > [10725.562208] tpm_tis > [10725.562209] dcdbas > [10725.562210] iTCO_wdt > [10725.562211] ib_sa > [10725.562211] aes_x86_64 > [10725.562212] ib_mad > [10725.562213] ib_ipoib > [10725.562214] lrw > [10725.562214] ib_core > [10725.562215] tpm > [10725.562216] ablk_helper > [10725.562216] mlx4_ib > [10725.562217] ib_core > [10725.562218] ib_sa > [10725.562219] ib_addr > [10725.562219] pps_core > [10725.562220] lpc_ich > [10725.562221] rdma_ucm > [10725.562221] ib_mad > [10725.562222] ib_addr > [10725.562223] ib_core > [10725.562223] x86_pkg_temp_thermal > [10725.562224] cryptd > [10725.562225] ipmi_ssif > [10725.562226] libphy > [10725.562226] button > [10725.562227] fjes > [10725.562228] intel_powerclamp > [10725.562228] ib_addr > [10725.562229] coretemp > [10725.562230] x86_pkg_temp_thermal > [10725.562231] shpchp > [10725.562231] x86_pkg_temp_thermal > [10725.562232] ipmi_devintf > [10725.562233] intel_powerclamp > [10725.562234] ib_ucm > [10725.562234] mei_me > [10725.562235] 8250_fintek > [10725.562236] ptp > [10725.562237] kvm_intel > [10725.562238] iTCO_vendor_support > [10725.562239] coretemp > [10725.562239] intel_powerclamp > [10725.562240] mei > [10725.562241] sb_edac > [10725.562241] mfd_core > [10725.562242] pcspkr > [10725.562243] kvm_intel > [10725.562244] ipmi_si > [10725.562244] wmi > [10725.562245] edac_core > [10725.562245] dm_multipath > [10725.562246] ipmi_msghandler > [10725.562247] dm_mod > [10725.562247] acpi_power_meter > [10725.562248] hid_generic > [10725.562248] tpm_tis > [10725.562249] usbhid > [10725.562249] tpm > [10725.562250] mgag200 > [10725.562252] lpc_ich > [10725.562253] ib_uverbs > [10725.562253] kvm > [10725.562254] coretemp > [10725.562255] kvm > [10725.562255] kvm_intel > [10725.562256] gf128mul > [10725.562257] irqbypass > [10725.562258] glue_helper > [10725.562258] crct10dif_pclmul > [10725.562259] button > [10725.562260] i2c_algo_bit > [10725.562260] shpchp > [10725.562261] crc32_pclmul > [10725.562262] kvm > [10725.562262] irqbypass > [10725.562263] irqbypass > [10725.562264] crct10dif_pclmul > [10725.562264] crc32c_intel > [10725.562265] msr > [10725.562266] aesni_intel > [10725.562267] ib_umad > [10725.562268] tg3 > [10725.562268] mei_me > [10725.562269] iTCO_wdt > [10725.562270] mei > [10725.562270] drm_kms_helper > [10725.562271] mfd_core > [10725.562271] syscopyarea > [10725.562272] wmi > [10725.562273] sysfillrect > [10725.562274] rdma_cm > [10725.562274] mlx4_core > [10725.562275] ib_cm > [10725.562276] aes_x86_64 > [10725.562276] iw_cm > [10725.562277] dm_multipath > [10725.562278] lrw > [10725.562279] dcdbas > [10725.562279] dm_mod > [10725.562280] sysimgblt > [10725.562281] crc32_pclmul > [10725.562282] crct10dif_pclmul > [10725.562282] ipmi_ssif > [10725.562283] crc32_pclmul > [10725.562284] crc32c_intel > [10725.562285] ablk_helper > [10725.562285] aesni_intel > [10725.562286] crc32c_intel > [10725.562287] tg3 > [10725.562287] ipmi_devintf > [10725.562288] hid_generic > [10725.562289] fb_sys_fops > [10725.562289] usbhid > [10725.562290] pps_core > [10725.562291] mgag200 > [10725.562291] ptp > [10725.562292] i2c_algo_bit > [10725.562293] ttm > [10725.562293] aesni_intel > [10725.562294] iTCO_wdt > [10725.562295] tg3 > [10725.562295] aes_x86_64 > [10725.562296] drm > [10725.562297] drm_kms_helper > [10725.562297] ehci_pci > [10725.562298] syscopyarea > [10725.562299] sr_mod > [10725.562299] iTCO_vendor_support > [10725.562300] cryptd > [10725.562301] cdrom > [10725.562301] gf128mul > [10725.562302] mlx4_ib > [10725.562303] ehci_hcd > [10725.562304] sysfillrect > [10725.562304] ib_sa > [10725.562305] iTCO_wdt > [10725.562306] lrw > [10725.562307] sysimgblt > [10725.562307] usbcore > [10725.562308] ipmi_ssif > [10725.562309] aes_x86_64 > [10725.562309] ipmi_devintf > [10725.562310] lrw > [10725.562311] ptp > [10725.562311] libphy > [10725.562312] ib_mad > [10725.562313] iTCO_vendor_support > [10725.562313] ipmi_ssif > [10725.562314] gf128mul > [10725.562315] ipmi_devintf > [10725.562315] glue_helper > [10725.562316] ptp > [10725.562317] glue_helper > [10725.562318] iTCO_vendor_support > [10725.562318] mlx4_core > [10725.562319] gf128mul > [10725.562319] dcdbas > [10725.562320] glue_helper > [10725.562321] ablk_helper > [10725.562321] mlx4_core > [10725.562322] pps_core > [10725.562322] dcdbas > [10725.562323] cryptd > [10725.562325] fb_sys_fops > [10725.562326] fjes > [10725.562326] mlx4_core > [10725.562327] ttm > [10725.562328] usb_common > [10725.562328] drm > [10725.562329] sg > [10725.562329] ehci_pci > [10725.562329] > [10725.562331] 8250_fintek > [10725.562332] dcdbas > [10725.562332] irq event stamp: 11245005 > [10725.562333] libphy > [10725.562334] sb_edac > [10725.562335] ib_core > [10725.562335] fjes > [10725.562336] ablk_helper > [10725.562337] 8250_fintek > [10725.562337] pps_core > [10725.562338] sr_mod > [10725.562339] ablk_helper > [10725.562340] cryptd > [10725.562340] pcspkr > [10725.562341] cdrom > [10725.562342] hardirqs last enabled at (11245004): > [10725.562343] ehci_hcd > [10725.562343] sb_edac > [10725.562344] libphy > [10725.562350] pcspkr > [10725.562350] [<ffffffff8157940f>] restore_regs_and_iret+0x0/0x1d > [10725.562351] usbcore > [10725.562352] hardirqs last disabled at (11245005): > [10725.562354] usb_common > [10725.562354] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10725.562355] sg > [10725.562356] ipmi_si > [10725.562357] softirqs last enabled at (8566716): > [10725.562357] > [10725.562358] fjes > [10725.562359] ipmi_si > [10725.562359] 8250_fintek > [10725.562360] edac_core > [10725.562361] sb_edac > [10725.562361] ipmi_msghandler > [10725.562362] ib_addr > [10725.562363] pps_core > [10725.562363] pcspkr > [10725.562364] acpi_power_meter > [10725.562367] ipmi_si > [10725.562367] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10725.562368] cryptd > [10725.562369] edac_core > [10725.562370] softirqs last disabled at (8566949): > [10725.562370] irq event stamp: 122720091 > [10725.562372] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.562373] hardirqs last enabled at (122720090): > [10725.562375] x86_pkg_temp_thermal > [10725.562376] CPU: 7 PID: 7934 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10725.562378] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10725.562379] ipmi_msghandler > [10725.562380] hardirqs last disabled at (122720091): > [10725.562381] libphy > [10725.562381] edac_core > [10725.562382] tpm_tis > [10725.562383] intel_powerclamp > [10725.562383] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.562385] tpm > [10725.562385] task: ffff8804426faf80 ti: ffff8803f7598000 task.ti: ffff8803f7598000 > [10725.562387] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10725.562389] RIP: 0010:[<ffffffff815781b3>] > [10725.562391] softirqs last enabled at (119833606): > [10725.562391] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.562392] lpc_ich > [10725.562393] ipmi_msghandler > [10725.562394] button > [10725.562394] fjes > [10725.562396] acpi_power_meter > [10725.562396] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10725.562398] RSP: 0018:ffff88046f2e3d78 EFLAGS: 00000292 > [10725.562399] softirqs last disabled at (119835011): > [10725.562402] tpm_tis tpm > [10725.562402] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.562404] 8250_fintek > [10725.562404] CPU: 1 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10725.562405] shpchp > [10725.562406] acpi_power_meter > [10725.562406] mei_me > [10725.562407] tpm_tis > [10725.562407] mei > [10725.562408] tpm > [10725.562409] sb_edac > [10725.562410] lpc_ich > [10725.562411] pcspkr > [10725.562412] button > [10725.562412] RAX: ffff8804426faf80 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.562414] ipmi_si > [10725.562414] RDX: 0000000000000003 RSI: ffff8804426fb718 RDI: ffff8804426faf80 > [10725.562415] edac_core > [10725.562417] shpchp > [10725.562417] RBP: ffff88046f2e3d88 R08: ffff88046f2e3cf8 R09: 0000000000000000 > [10725.562418] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.562419] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.562420] mfd_core > [10725.562421] lpc_ich > [10725.562421] wmi > [10725.562422] coretemp > [10725.562423] ipmi_msghandler > [10725.562424] R13: ffffea0009ac0400 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.562425] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10725.562426] FS: 00007f8a9be35700(0000) GS:ffff88046f2e0000(0000) knlGS:0000000000000000 > [10725.562428] RIP: 0010:[<ffffffff815781b3>] > [10725.562429] mei_me > [10725.562429] dm_multipath > [10725.562430] button > [10725.562431] dm_mod > [10725.562432] kvm_intel > [10725.562433] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.562434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.562435] RSP: 0018:ffff88046f223d78 EFLAGS: 00000292 > [10725.562436] kvm > [10725.562437] mei > [10725.562437] hid_generic > [10725.562438] shpchp > [10725.562439] usbhid > [10725.562439] mei_me > [10725.562440] mgag200 > [10725.562441] irqbypass > [10725.562441] RAX: ffff88040120af80 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.562442] CR2: 00007ffeaad51d1c CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10725.562443] mei > [10725.562444] i2c_algo_bit > [10725.562445] mfd_core > [10725.562446] mfd_core > [10725.562446] drm_kms_helper > [10725.562447] wmi > [10725.562448] syscopyarea > [10725.562449] RDX: 0000000000000003 RSI: ffff88040120b718 RDI: ffff88040120af80 > [10725.562449] Stack: > [10725.562450] RBP: ffff88046f223d88 R08: ffff88046f223cf8 R09: 0000000000000000 > [10725.562452] ffff88017fc01c00 > [10725.562452] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.562453] 0000000000000001 > [10725.562453] R13: ffffea000bac6e00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.562455] ffff88046f2e3e38 > [10725.562455] FS: 0000000000000000(0000) GS:ffff88046f220000(0000) knlGS:0000000000000000 > [10725.562457] ffffffff811640d6 > [10725.562457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.562458] > [10725.562459] CR2: 00007ffadf28aa08 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10725.562460] acpi_power_meter > [10725.562461] wmi > [10725.562462] crct10dif_pclmul > [10725.562463] tpm_tis > [10725.562463] ffff88046f2e3dc8 > [10725.562463] Stack: > [10725.562464] 0000000000000246 > [10725.562465] ffff88017fc01c00 > [10725.562466] ffffffff811859b3 > [10725.562466] 0000000000000001 > [10725.562467] ffff88026b010aa0 > [10725.562467] ffff88046f223e38 > [10725.562467] > [10725.562469] crc32_pclmul > [10725.562470] tpm > [10725.562470] sysfillrect > [10725.562471] dm_multipath > [10725.562472] lpc_ich > [10725.562472] crc32c_intel > [10725.562473] dm_mod > [10725.562474] button > [10725.562475] ffff88026b010aa0 > [10725.562476] shpchp > [10725.562478] dm_multipath dm_mod > [10725.562478] mei_me > [10725.562479] aesni_intel > [10725.562480] ffff0001426fb718 > [10725.562480] ffffffff811640d6 > [10725.562481] 0000000000000292 > [10725.562482] sysimgblt > [10725.562482] hid_generic > [10725.562483] fb_sys_fops > [10725.562483] usbhid > [10725.562484] ttm > [10725.562485] mgag200 > [10725.562485] drm > [10725.562486] i2c_algo_bit > [10725.562486] ehci_pci > [10725.562487] hid_generic > [10725.562488] mei > [10725.562489] sr_mod > [10725.562489] > [10725.562490] ffff8804426fb718 > [10725.562491] cdrom > [10725.562492] drm_kms_helper > [10725.562492] ehci_hcd > [10725.562493] syscopyarea > [10725.562493] usbcore > [10725.562494] sysfillrect > [10725.562495] usb_common > [10725.562495] sysimgblt > [10725.562496] sg > [10725.562497] mfd_core > [10725.562497] fb_sys_fops > [10725.562497] > [10725.562499] ttm > [10725.562499] irq event stamp: 11251127 > [10725.562500] drm > [10725.562501] hardirqs last enabled at (11251126): > [10725.562501] ehci_pci > [10725.562505] wmi > [10725.562505] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10725.562506] > [10725.562507] ffff88046f223dc8 > [10725.562507] Call Trace: > [10725.562508] usbhid > [10725.562509] sr_mod > [10725.562510] hardirqs last disabled at (11251127): > [10725.562511] tg3 > [10725.562512] 0000000000000246 > [10725.562512] <IRQ> > [10725.562513] 0000000100300030 > [10725.562514] dm_multipath > [10725.562516] cdrom > [10725.562516] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10725.562520] ehci_hcd > [10725.562520] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10725.562524] ffff8802eb1bf238 > [10725.562524] [<ffffffff811859b3>] ? file_free_rcu+0x23/0x30 > [10725.562525] > [10725.562526] usbcore > [10725.562527] softirqs last enabled at (8589138): > [10725.562529] ffff8802eb1bf238 > [10725.562530] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10725.562533] ffff00010120b718 > [10725.562533] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.562536] 0000000000000292 > [10725.562536] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10725.562537] usb_common > [10725.562540] softirqs last disabled at (8589495): > [10725.562540] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.562541] dm_mod > [10725.562544] mgag200 > [10725.562544] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10725.562545] ffff88040120b718 > [10725.562547] i2c_algo_bit drm_kms_helper > [10725.562547] > [10725.562549] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10725.562551] iTCO_wdt > [10725.562552] sg > [10725.562552] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.562553] > [10725.562556] aes_x86_64 > [10725.562556] CPU: 2 PID: 7938 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10725.562557] hid_generic > [10725.562558] syscopyarea > [10725.562559] lrw > [10725.562559] irq event stamp: 11005237 > [10725.562560] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.562561] sysfillrect > [10725.562562] ipmi_ssif > [10725.562563] sysimgblt > [10725.562564] usbhid > [10725.562564] fb_sys_fops > [10725.562565] Call Trace: > [10725.562567] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10725.562569] hardirqs last enabled at (11005236): > [10725.562569] task: ffff880400708000 ti: ffff8803f7af8000 task.ti: ffff8803f7af8000 > [10725.562571] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10725.562573] RIP: 0010:[<ffffffff815781b3>] > [10725.562574] hardirqs last disabled at (11005237): > [10725.562575] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.562576] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10725.562578] mgag200 > [10725.562581] ipmi_devintf > [10725.562581] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10725.562583] <IRQ> > [10725.562584] ttm > [10725.562584] RSP: 0018:ffff88046f243d78 EFLAGS: 00000292 > [10725.562586] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.562589] drm > [10725.562589] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10725.562591] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10725.562592] ptp > [10725.562593] i2c_algo_bit > [10725.562594] RAX: ffff880400708000 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.562595] softirqs last enabled at (8447496): > [10725.562596] ehci_pci > [10725.562598] drm_kms_helper > [10725.562598] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10725.562600] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.562602] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.562605] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10725.562607] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.562609] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10725.562611] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10725.562613] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10725.562614] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10725.562615] RDX: 0000000000000003 RSI: ffff880400708798 RDI: ffff880400708000 > [10725.562617] softirqs last disabled at (8447895): > [10725.562617] RBP: ffff88046f243d88 R08: ffff88046f243cf8 R09: 0000000000000000 > [10725.562619] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.562621] syscopyarea > [10725.562623] iTCO_vendor_support > [10725.562623] CPU: 8 PID: 7940 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10725.562624] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.562625] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.562626] R13: ffffea000beaca00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.562628] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10725.562631] <EOI> > [10725.562631] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10725.562633] FS: 00007f8a99e31700(0000) GS:ffff88046f240000(0000) knlGS:0000000000000000 > [10725.562634] task: ffff8803fabddf00 ti: ffff8803f753c000 task.ti: ffff8803f753c000 > [10725.562635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.562637] RIP: 0010:[<ffffffff815781b3>] > [10725.562637] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10725.562639] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.562640] gf128mul > [10725.562641] sr_mod > [10725.562642] RSP: 0018:ffff88046f303d78 EFLAGS: 00000292 > [10725.562642] Stack: > [10725.562644] sysfillrect > [10725.562645] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10725.562648] [<ffffffff81286284>] ? blk_flush_plug_list+0x224/0x240 > [10725.562649] ffff88017fc01c00 > [10725.562649] RAX: ffff8803fabddf00 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.562650] 0000000000000001 > [10725.562651] sysimgblt > [10725.562652] ffff88046f243e38 > [10725.562653] RDX: 0000000000000003 RSI: ffff8803fabde698 RDI: ffff8803fabddf00 > [10725.562654] ffffffff811640d6 > [10725.562654] RBP: ffff88046f303d88 R08: ffff88046f303cf8 R09: 0000000000000000 > [10725.562657] [<ffffffff81286491>] blk_queue_bio+0x1f1/0x300 > [10725.562658] cdrom > [10725.562658] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.562659] > [10725.562660] R13: ffffea0009b52400 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.562662] ffff88046f243dc8 > [10725.562662] FS: 00007f8a96e2b700(0000) GS:ffff88046f300000(0000) knlGS:0000000000000000 > [10725.562664] glue_helper > [10725.562664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.562665] ffff88017fc07600 > [10725.562666] CR2: 00007fe88affadd0 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10725.562669] fb_sys_fops > [10725.562669] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10725.562671] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.562673] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10725.562674] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10725.562676] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10725.562679] mlx4_core > [10725.562679] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.562682] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10725.562684] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10725.562685] ehci_hcd > [10725.562686] 0000000100300030 > [10725.562687] usbcore > [10725.562689] ffff8802fab2b520 > [10725.562689] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10725.562690] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10725.562692] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10725.562694] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10725.562694] > [10725.562695] Stack: > [10725.562696] dcdbas > [10725.562698] ttm > [10725.562699] usb_common > [10725.562699] drm > [10725.562700] ablk_helper > [10725.562701] ehci_pci > [10725.562702] sg > [10725.562702] sr_mod > [10725.562703] pps_core > [10725.562704] ffff88017fc01c00 > [10725.562704] > [10725.562707] <EOI> > [10725.562707] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10725.562708] ffff8802fab2b520 > [10725.562709] 0000000000000001 > [10725.562710] ffff000100708798 > [10725.562711] cryptd > [10725.562714] cdrom > [10725.562714] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10725.562715] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10725.562717] ffff88046f303e38 > [10725.562717] 0000000000000292 > [10725.562718] ffffffff811640d6 > [10725.562721] libphy > [10725.562721] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10725.562725] [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath] > [10725.562727] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10725.562729] [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath] > [10725.562731] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10725.562736] [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod] > [10725.562739] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10725.562739] > [10725.562741] ffff880400708798 > [10725.562741] ehci_hcd > [10725.562742] ffff88046f303dc8 > [10725.562742] > [10725.562744] usbcore > [10725.562745] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10725.562749] [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod] > [10725.562751] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10725.562754] [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0 > [10725.562755] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10725.562756] Call Trace: > [10725.562757] 0000000000000246 > [10725.562758] usb_common > [10725.562759] ffffffff811859b3 > [10725.562759] irq event stamp: 10703453 > [10725.562761] ffff88026d492a80 > [10725.562762] <IRQ> > [10725.562764] fjes > [10725.562764] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10725.562766] [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60 > [10725.562767] Code: > [10725.562769] hardirqs last enabled at (10703452): > [10725.562769] [<ffffffff8107591a>] kthread+0xea/0x100 > [10725.562773] sg > [10725.562773] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10725.562777] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10725.562777] > [10725.562779] 8250_fintek > [10725.562781] fc > [10725.562781] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10725.562783] hardirqs last disabled at (10703453): > [10725.562784] ffff88026d492a80 > [10725.562785] 48 > [10725.562786] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10725.562788] 83 > [10725.562788] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10725.562789] c7 > [10725.562792] sb_edac > [10725.562792] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10725.562792] > [10725.562794] Code: > [10725.562796] 18 > [10725.562796] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10725.562797] ffff0001fabde698 > [10725.562798] 48 > [10725.562801] 0000000000000292 > [10725.562801] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.562802] irq event stamp: 9765723 > [10725.562803] pcspkr > [10725.562806] ffff8803fabde698 > [10725.562806] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.562807] ipmi_si > [10725.562807] > [10725.562809] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10725.562810] Call Trace: > [10725.562812] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10725.562815] <IRQ> > [10725.562815] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10725.562817] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10725.562818] 8b > [10725.562819] hardirqs last enabled at (9765722): > [10725.562820] edac_core > [10725.562820] 55 > [10725.562821] fc > [10725.562823] ipmi_msghandler > [10725.562824] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10725.562827] acpi_power_meter > [10725.562827] [<ffffffff811859b3>] ? file_free_rcu+0x23/0x30 > [10725.562828] 08 > [10725.562829] 48 > [10725.562830] 53 > [10725.562830] 83 > [10725.562831] 48 > [10725.562834] c7 > [10725.562834] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10725.562835] 89 > [10725.562836] 18 > [10725.562837] tpm_tis > [10725.562838] hardirqs last disabled at (9765723): > [10725.562839] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.562841] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10725.562842] 48 > [10725.562843] f3 > [10725.562843] 8b > [10725.562846] be > [10725.562846] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.562848] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10725.562850] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.562852] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10725.562853] softirqs last enabled at (8080444): > [10725.562856] softirqs last enabled at (8596058): > [10725.562856] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10725.562858] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10725.562859] softirqs last disabled at (8080721): > [10725.562860] softirqs last disabled at (8596459): > [10725.562862] tpm > [10725.562862] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.562863] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.562866] CPU: 6 PID: 7930 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10725.562868] CPU: 4 PID: 7935 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10725.562869] 55 > [10725.562869] 01 > [10725.562870] 08 > [10725.562871] 00 > [10725.562871] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.562872] 53 > [10725.562873] lpc_ich > [10725.562874] 48 > [10725.562874] task: ffff8803fbdd8000 ti: ffff8803f7524000 task.ti: ffff8803f7524000 > [10725.562875] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.562878] RIP: 0010:[<ffffffff815781b3>] > [10725.562878] task: ffff880064ae4740 ti: ffff8803f031c000 task.ti: ffff8803f031c000 > [10725.562880] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.562882] 89 > [10725.562882] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10725.562884] 00 > [10725.562884] RSP: 0018:ffff88046f283d78 EFLAGS: 00000292 > [10725.562886] RIP: 0010:[<ffffffff815781b3>] > [10725.562887] 00 > [10725.562887] f3 > [10725.562888] e8 > [10725.562888] be > [10725.562889] e1 > [10725.562889] 01 > [10725.562890] ce > [10725.562891] RAX: ffff8803fbdd8000 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.562892] 00 > [10725.562892] b2 > [10725.562893] 00 > [10725.562894] button > [10725.562895] RDX: 0000000000000003 RSI: ffff8803fbdd8798 RDI: ffff8803fbdd8000 > [10725.562896] 00 > [10725.562896] RBP: ffff88046f283d88 R08: ffff88046f283cf8 R09: 0000000000000000 > [10725.562899] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.562900] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10725.562903] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10725.562905] shpchp > [10725.562905] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10725.562906] e8 > [10725.562908] ff > [10725.562908] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.562910] <EOI> > [10725.562911] RSP: 0018:ffff88046f2c3d78 EFLAGS: 00000292 > [10725.562912] e1 > [10725.562912] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.562914] RAX: ffff880064ae4740 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.562915] 4c > [10725.562915] ce > [10725.562916] 89 > [10725.562916] b2 > [10725.562917] e7 > [10725.562918] ff > [10725.562920] e8 > [10725.562920] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10725.562922] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10725.562923] R13: ffffea000e1e9e00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.562925] RDX: 0000000000000003 RSI: ffff880064ae4ed8 RDI: ffff880064ae4740 > [10725.562926] 4c > [10725.562927] 49 > [10725.562928] 89 > [10725.562928] FS: 00007f8a9b634700(0000) GS:ffff88046f280000(0000) knlGS:0000000000000000 > [10725.562932] mei_me > [10725.562932] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10725.562933] e7 > [10725.562934] 00 > [10725.562934] e8 > [10725.562936] b3 > [10725.562937] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10725.562939] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10725.562941] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10725.562943] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10725.562947] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10725.562948] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.562951] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10725.562952] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10725.562955] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10725.562956] RBP: ffff88046f2c3d88 R08: ffff88046f2c3cf8 R09: 0000000000000000 > [10725.562957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.562959] mei > [10725.562959] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10725.562960] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.562961] Stack: > [10725.562963] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10725.562965] mfd_core > [10725.562965] R13: ffffea0009dcea00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.562966] ff > [10725.562967] 49 > [10725.562967] f6 > [10725.562968] 00 > [10725.562970] wmi > [10725.562970] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.562972] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10725.562973] ffff88017fc01c00 > [10725.562974] b3 > [10725.562976] dm_multipath > [10725.562976] FS: 00007f8a9862e700(0000) GS:ffff88046f2c0000(0000) knlGS:0000000000000000 > [10725.562979] 0000000000000001 > [10725.562980] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10725.562981] c7 > [10725.562983] ff > [10725.562983] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10725.562984] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10725.562986] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10725.562988] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10725.562989] ffff88046f283e38 > [10725.562990] dm_mod > [10725.562990] f6 > [10725.562991] 02 > [10725.562991] c7 > [10725.562992] 74 > [10725.562993] 02 > [10725.562993] 13 > [10725.562994] ffffffff811640d6 > [10725.562995] e8 > [10725.562997] 74 > [10725.562997] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10725.562999] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10725.563001] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10725.563004] <EOI> > [10725.563004] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10725.563006] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10725.563007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.563007] > [10725.563009] CR2: 00005633132c3fe8 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10725.563012] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10725.563013] 13 > [10725.563013] 0f > [10725.563014] ffff88046f283dc8 > [10725.563015] a8 > [10725.563015] 0000000000000246 > [10725.563016] hid_generic > [10725.563016] Stack: > [10725.563018] ffffffff811859b3 > [10725.563018] e8 > [10725.563019] b2 > [10725.563019] 0f > [10725.563020] ff > [10725.563021] usbhid > [10725.563022] ffff880387a78d48 > [10725.563022] mgag200 > [10725.563023] 53 > [10725.563023] a8 > [10725.563024] 9d > [10725.563025] b2 > [10725.563026] <65> > [10725.563027] i2c_algo_bit > [10725.563029] ffff88017fc01c00 > [10725.563029] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10725.563030] ff > [10725.563031] ff > [10725.563031] 53 > [10725.563033] 0d > [10725.563033] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10725.563034] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10725.563036] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10725.563039] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10725.563041] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10725.563043] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10725.563045] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10725.563047] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10725.563049] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10725.563051] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10725.563054] 0000000000000001 > [10725.563054] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10725.563056] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10725.563056] > [10725.563057] 56 > [10725.563058] ffff880387a78d48 > [10725.563060] ffff88046f2c3e38 > [10725.563060] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10725.563062] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10725.563063] Code: > [10725.563064] ffffffff811640d6 > [10725.563065] drm_kms_helper > [10725.563067] fc > [10725.563067] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10725.563068] 48 > [10725.563069] syscopyarea > [10725.563069] > [10725.563070] 2b > [10725.563070] 9d > [10725.563071] a9 > [10725.563073] <65> > [10725.563073] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10725.563075] 83 > [10725.563075] ffff88046f2c3dc8 > [10725.563076] sysfillrect > [10725.563077] ff > [10725.563077] 7e > [10725.563078] 0d > [10725.563079] ffff0001fbdd8798 > [10725.563079] 56 > [10725.563081] 0000000000000292 > [10725.563081] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10725.563084] c7 > [10725.563084] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10725.563087] 18 > [10725.563087] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10725.563089] 48 > [10725.563089] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10725.563091] 8b > [10725.563091] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10725.563093] 5b > [10725.563094] sysimgblt > [10725.563096] 0000000000000246 0000000100300030 > [10725.563096] 41 > [10725.563097] 2b > [10725.563098] 5c > [10725.563098] a9 > [10725.563099] 5d > [10725.563101] ffff8802773ab7c8 > [10725.563101] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10725.563102] > [10725.563103] fb_sys_fops > [10725.563104] 55 > [10725.563105] ffff8802773ab7c8 > [10725.563105] ttm > [10725.563106] ffff000164ae4ed8 > [10725.563107] ffff8803fbdd8798 > [10725.563108] 08 > [10725.563109] 0000000000000292 > [10725.563109] drm > [10725.563109] > [10725.563111] ffff880064ae4ed8 > [10725.563111] Call Trace: > [10725.563112] ehci_pci > [10725.563113] <IRQ> > [10725.563115] 53 > [10725.563115] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10725.563117] sr_mod > [10725.563117] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10725.563118] > [10725.563119] c3 > [10725.563120] Call Trace: > [10725.563121] cdrom > [10725.563121] 53 > [10725.563122] 7e > [10725.563125] 9d > [10725.563125] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10725.563126] 5b > [10725.563126] e8 > [10725.563127] 41 > [10725.563127] 6a > [10725.563128] 5c > [10725.563128] 79 > [10725.563131] 5d > [10725.563131] [<ffffffff811859b3>] ? file_free_rcu+0x23/0x30 > [10725.563133] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10725.563135] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10725.563137] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10725.563138] 48 > [10725.563139] c3 > [10725.563140] b2 > [10725.563140] 53 > [10725.563141] ff > [10725.563141] 89 > [10725.563142] Code: > [10725.563143] f3 > [10725.563145] fc > [10725.563145] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.563147] ehci_hcd > [10725.563148] 48 > [10725.563148] <IRQ> > [10725.563149] 9d > [10725.563149] eb > [10725.563150] e8 > [10725.563151] eb > [10725.563151] 6a > [10725.563151] > [10725.563154] 79 > [10725.563154] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.563155] usbcore > [10725.563156] 83 > [10725.563156] be > [10725.563157] c7 > [10725.563158] b2 > [10725.563161] usb_common > [10725.563161] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10725.563162] 18 > [10725.563163] sg > [10725.563165] ff > [10725.563165] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10725.563166] 48 > [10725.563167] 01 > [10725.563167] 8b > [10725.563168] 00 > [10725.563168] 55 > [10725.563169] 00 > [10725.563172] 08 > [10725.563172] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10725.563175] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10725.563177] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.563178] > [10725.563179] 00 > [10725.563180] 53 > [10725.563181] e8 > [10725.563181] irq event stamp: 11920625 > [10725.563182] eb > [10725.563183] hardirqs last enabled at (11920624): > [10725.563184] 48 > [10725.563186] e1 > [10725.563186] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.563187] eb > [10725.563188] 89 > [10725.563189] ce > [10725.563189] > [10725.563190] b2 > [10725.563190] f3 > [10725.563191] ff > [10725.563192] be > [10725.563192] 4c > [10725.563193] 01 > [10725.563193] 89 > [10725.563194] 00 > [10725.563194] e7 > [10725.563195] 00 > [10725.563195] e8 > [10725.563196] 00 > [10725.563197] 49 > [10725.563197] e8 > [10725.563198] 00 > [10725.563198] e1 > [10725.563199] b3 > [10725.563201] ce > [10725.563201] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10725.563204] b2 > [10725.563204] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10725.563207] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10725.563210] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10725.563212] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10725.563214] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10725.563215] ff > [10725.563216] ff > [10725.563217] f6 > [10725.563219] hardirqs last disabled at (11920625): > [10725.563219] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10725.563220] c7 > [10725.563222] 4c > [10725.563222] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.563224] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.563226] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10725.563227] 02 > [10725.563227] 89 > [10725.563228] 74 > [10725.563230] e7 > [10725.563230] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10725.563232] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.563233] e8 > [10725.563235] 13 > [10725.563235] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10725.563237] softirqs last enabled at (10133236): > [10725.563237] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10725.563240] e8 > [10725.563240] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10725.563241] 49 > [10725.563243] 0f > [10725.563243] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10725.563244] 00 > [10725.563245] a8 > [10725.563245] b3 > [10725.563246] <EOI> > [10725.563247] ff > [10725.563247] b2 > [10725.563249] f6 > [10725.563249] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10725.563251] c7 > [10725.563251] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.563280] [<ffffffffa05c9a57>] scst_check_blocked_dev+0x77/0x1c0 [scst] > [10725.563281] 02 > [10725.563281] ff > [10725.563282] 74 > [10725.563283] 53 > [10725.563283] 13 > [10725.563284] 9d > [10725.563284] e8 > [10725.563287] <65> > [10725.563287] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10725.563289] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10725.563291] softirqs last disabled at (10133267): > [10725.563292] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10725.563293] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.563296] 0f > [10725.563296] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10725.563298] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10725.563299] ff > [10725.563300] a8 > [10725.563302] 0d > [10725.563302] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10725.563305] <EOI> > [10725.563305] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10725.563307] CPU: 11 PID: 32753 Comm: disk11_0 Tainted: G O L 4.4.6-debug+ #2 > [10725.563308] 56 > [10725.563309] b2 > [10725.563311] 2b > [10725.563311] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10725.563312] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10725.563315] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10725.563317] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10725.563318] ff > [10725.563319] a9 > [10725.563319] 53 > [10725.563320] 7e > [10725.563322] 9d > [10725.563322] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10725.563323] <65> > [10725.563323] 5b > [10725.563324] ff > [10725.563324] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.563326] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10725.563327] 41 > [10725.563328] 0d > [10725.563328] 5c > [10725.563329] 56 > [10725.563329] 5d > [10725.563330] 2b > [10725.563331] c3 > [10725.563331] task: ffff880439375f00 ti: ffff880460d30000 task.ti: ffff880460d30000 > [10725.563332] 53 > [10725.563333] a9 > [10725.563335] 9d > [10725.563335] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10725.563337] RIP: 0010:[<ffffffff815781b3>] > [10725.563339] 7e > [10725.563339] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.563341] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10725.563342] RSP: 0018:ffff88046f363d68 EFLAGS: 00000292 > [10725.563344] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10725.563345] 5b > [10725.563348] e8 > [10725.563348] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10725.563349] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10725.563350] 41 > [10725.563351] 6a > [10725.563353] 5c > [10725.563353] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10725.563355] 5d > [10725.563355] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10725.563357] c3 > [10725.563359] 79 > [10725.563359] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10725.563361] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10725.563364] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10725.563365] RAX: ffff880439375f00 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.563367] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10725.563368] 53 > [10725.563368] b2 > [10725.563369] 9d > [10725.563369] ff > [10725.563370] e8 > [10725.563371] eb > [10725.563373] 6a > [10725.563373] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10725.563374] eb > [10725.563374] 79 > [10725.563375] > [10725.563377] b2 > [10725.563377] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10725.563379] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10725.563381] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10725.563382] RDX: 0000000000000003 RSI: ffff880439376698 RDI: ffff880439375f00 > [10725.563384] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10725.563386] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10725.563388] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10725.563390] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10725.563391] RBP: ffff88046f363d78 R08: ffff88046f363ce8 R09: 0000000000000000 > [10725.563393] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10725.563394] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10725.563396] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10725.563399] ff > [10725.563399] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10725.563400] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.563401] eb > [10725.563403] Code: > [10725.563403] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10725.563405] fc > [10725.563405] R13: ffffea000f204e00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.563408] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10725.563409] FS: 0000000000000000(0000) GS:ffff88046f360000(0000) knlGS:0000000000000000 > [10725.563410] 48 > [10725.563411] eb > [10725.563413] 83 > [10725.563413] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10725.563415] c7 > [10725.563415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.563417] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10725.563419] 18 > [10725.563419] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10725.563420] 48 > [10725.563421] > [10725.563422] CR2: 00005633130dd1a8 CR3: 0000000401468000 CR4: 00000000001406e0 > [10725.563424] 8b > [10725.563424] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10725.563425] 55 > [10725.563425] Stack: > [10725.563427] 08 > [10725.563427] ffff88017fc01c00 > [10725.563429] 53 > [10725.563429] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10725.563430] 48 > [10725.563431] 0000000000000001 > [10725.563433] 89 > [10725.563433] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10725.563434] f3 > [10725.563435] ffff88046f363e30 > [10725.563437] be > [10725.563437] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10725.563438] 01 > [10725.563440] ffffffff811640d6 > [10725.563440] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10725.563441] 00 > [10725.563441] > [10725.563442] 00 > [10725.563443] Code: > [10725.563444] 00 > [10725.563445] fc > [10725.563445] ffff88046f363db8 > [10725.563446] e8 > [10725.563447] 48 > [10725.563447] e1 > [10725.563448] ffff88016d2e0000 > [10725.563449] 83 > [10725.563449] ce > [10725.563450] c7 > [10725.563451] b2 > [10725.563451] 0000000000000000 > [10725.563452] ff > [10725.563452] 18 > [10725.563453] 4c > [10725.563454] 48 > [10725.563454] 89 > [10725.563455] 8b > [10725.563456] ffff8803c813ef90 > [10725.563456] e7 > [10725.563457] 55 > [10725.563457] e8 > [10725.563458] 08 > [10725.563459] 49 > [10725.563459] 53 > [10725.563460] 00 > [10725.563460] 48 > [10725.563461] > [10725.563462] b3 > [10725.563462] ffff8803c813ef90 > [10725.563463] ff > [10725.563464] 89 > [10725.563464] f6 > [10725.563465] f3 > [10725.563466] c7 > [10725.563466] 000000018157940f > [10725.563467] be > [10725.563467] 0000000000000292 > [10725.563468] 02 > [10725.563469] 01 > [10725.563469] 74 > [10725.563470] ffff880439376698 > [10725.563470] 13 > [10725.563471] > [10725.563472] 00 > [10725.563472] e8 > [10725.563472] Call Trace: > [10725.563473] 0f > [10725.563474] 00 > [10725.563475] a8 > [10725.563475] <IRQ> > [10725.563478] b2 > [10725.563478] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10725.563479] ff > [10725.563480] 00 > [10725.563482] 53 > [10725.563482] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10725.563485] 9d > [10725.563485] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.563488] e8 > [10725.563488] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10725.563489] e1 > [10725.563490] <65> > [10725.563492] ce > [10725.563492] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10725.563494] b2 > [10725.563494] ff > [10725.563497] ff > [10725.563497] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10725.563498] 4c > [10725.563500] 0d > [10725.563500] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10725.563503] 56 > [10725.563503] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10725.563505] 89 > [10725.563505] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.563506] 2b > [10725.563508] e7 > [10725.563508] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10725.563510] a9 > [10725.563510] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10725.563511] 7e > [10725.563512] e8 > [10725.563512] 5b > [10725.563515] 49 > [10725.563515] [<ffffffff810065a5>] do_IRQ+0x65/0x110 > [10725.563518] 41 > [10725.563518] [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20 > [10725.563519] 5c > [10725.563521] 00 > [10725.563521] [<ffffffff815793c9>] common_interrupt+0x89/0x89 > [10725.563523] b3 > [10725.563523] <EOI> > [10725.563524] 5d > [10725.563526] ff > [10725.563526] [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20 > [10725.563527] f6 > [10725.563530] c3 > [10725.563530] [<ffffffff8117bad7>] ? put_object+0x7/0x40 > [10725.563531] c7 > [10725.563533] 53 > [10725.563533] [<ffffffff8117bb48>] ? __delete_object+0x38/0x70 > [10725.563536] 9d > [10725.563536] [<ffffffff8117bb98>] delete_object_full+0x18/0x20 > [10725.563537] e8 > [10725.563540] 02 > [10725.563540] [<ffffffff8156dab1>] kmemleak_free+0x31/0x80 > [10725.563541] 74 > [10725.563542] 6a > [10725.563544] 13 > [10725.563544] [<ffffffff811642fa>] kmem_cache_free+0x7a/0x160 > [10725.563545] 79 > [10725.563546] e8 > [10725.563548] b2 > [10725.563548] [<ffffffff81113282>] mempool_free_slab+0x12/0x20 > [10725.563550] 0f > [10725.563552] ff > [10725.563552] [<ffffffff81113406>] mempool_free+0x26/0x80 > [10725.563553] eb > [10725.563555] a8 > [10725.563555] [<ffffffff8127e8c1>] bvec_free+0x31/0x40 > [10725.563556] eb > [10725.563556] b2 > [10725.563557] > [10725.563559] ff > [10725.563559] [<ffffffff8127ee54>] bio_free+0x34/0x60 > [10725.563561] 53 > [10725.563561] [<ffffffff8127ee9e>] bio_put+0x1e/0x30 > [10725.563568] 9d > [10725.563568] [<ffffffffa06f042d>] blockio_endio+0x2d/0xe0 [scst_vdisk] > [10725.563570] <65> > [10725.563570] [<ffffffff812800da>] bio_endio+0x3a/0x60 > [10725.563573] ff > [10725.563573] [<ffffffffa0560b08>] brd_make_request+0xe8/0x1ae [brd] > [10725.563576] 0d > [10725.563576] [<ffffffff812826fc>] ? generic_make_request_checks+0x10c/0x630 > [10725.563579] 56 > [10725.563579] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10725.563581] 2b > [10725.563581] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10725.563587] a9 7e > [10725.563587] [<ffffffffa06f0a33>] blockio_exec_rw+0x553/0x670 [scst_vdisk] > [10725.563593] 5b 41 5c > [10725.563593] [<ffffffffa06f0cb0>] ? blockio_exec_write_verify+0x50/0x50 [scst_vdisk] > [10725.563598] 5d > [10725.563598] [<ffffffffa06f0cbd>] blockio_exec_read+0xd/0x20 [scst_vdisk] > [10725.563602] c3 > [10725.563602] [<ffffffffa06e7773>] vdev_do_job+0xb3/0x310 [scst_vdisk] > [10725.563607] 53 9d > [10725.563607] [<ffffffffa06ebcc4>] blockio_exec+0x74/0x180 [scst_vdisk] > [10725.563610] e8 > [10725.563610] [<ffffffff810a2931>] ? trace_hardirqs_on_caller+0x121/0x1b0 > [10725.563632] 6a > [10725.563632] [<ffffffffa0609b12>] ? sgv_get_obj+0x72/0x1b0 [scst] > [10725.563649] 79 b2 ff > [10725.563649] [<ffffffffa05c7154>] scst_do_real_exec+0x94/0x3d0 [scst] > [10725.563664] eb > [10725.563664] [<ffffffffa05c01d3>] ? scst_is_not_reservation_holder+0x13/0x60 [scst] > [10725.563679] eb > [10725.563679] [<ffffffffa05c13ff>] ? __scst_check_local_events+0x19f/0x370 [scst] > [10725.563679] > [10725.563694] [<ffffffffa05ca8dc>] scst_exec_check_blocking+0xcc/0x340 [scst] > [10725.563709] [<ffffffffa05cac7d>] scst_exec_check_sn+0x12d/0x2c0 [scst] > [10725.563711] [<ffffffff8107a896>] ? finish_task_switch+0xa6/0x220 > [10725.563713] [<ffffffff8107a857>] ? finish_task_switch+0x67/0x220 > [10725.563728] [<ffffffffa05cf892>] scst_process_active_cmd+0x3e2/0x760 [scst] > [10725.563730] [<ffffffff810a29cd>] ? trace_hardirqs_on+0xd/0x10 > [10725.563744] [<ffffffffa05d1001>] scst_do_job_active+0xe1/0x160 [scst] > [10725.563759] [<ffffffffa05d11b2>] scst_cmd_thread+0x132/0x290 [scst] > [10725.563762] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10725.563776] [<ffffffffa05d1080>] ? scst_do_job_active+0x160/0x160 [scst] > [10725.563779] [<ffffffff8107591a>] kthread+0xea/0x100 > [10725.563782] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10725.563784] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10725.563785] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10725.563814] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10726.319704] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10726.380711] irq event stamp: 16370765 > [10726.381076] hardirqs last enabled at (16370764): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10726.382350] hardirqs last disabled at (16370765): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10726.383582] softirqs last enabled at (13717672): [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10726.384329] softirqs last disabled at (13717773): [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10726.384996] CPU: 0 PID: 7937 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10726.385483] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10726.385685] task: ffff88040070df00 ti: ffff8803f1678000 task.ti: ffff8803f1678000 > [10726.386036] RIP: 0010:[<ffffffff815781b3>] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10726.386575] RSP: 0018:ffff88046f203d78 EFLAGS: 00000292 > [10726.387228] RAX: ffff88040070df00 RBX: 0000000000000292 RCX: 0000000000000000 > [10726.387689] RDX: 0000000000000003 RSI: ffff88040070e698 RDI: ffff88040070df00 > [10726.388315] RBP: ffff88046f203d88 R08: ffff88046f203cf8 R09: 0000000000000000 > [10726.388558] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10726.389279] R13: ffffea000bef5800 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10726.389728] FS: 00007f8a9a632700(0000) GS:ffff88046f200000(0000) knlGS:0000000000000000 > [10726.390012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10726.390373] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406f0 > [10726.390881] Stack: > [10726.391294] ffff88017fc01c00 0000000000000001 ffff88046f203e38 ffffffff811640d6 > [10726.393205] ffff88046f203dc8 0000000000000246 ffffffff811859b3 ffff8802fbd64d08 > [10726.394507] ffff8802fbd64d08 ffff00010070e698 0000000000000292 ffff88040070e698 > [10726.397125] Call Trace: > [10726.397494] <IRQ> > [10726.397750] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10726.398542] [<ffffffff811859b3>] ? file_free_rcu+0x23/0x30 > [10726.398908] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10726.399384] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10726.399674] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10726.399952] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10726.400402] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10726.401001] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10726.401417] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10726.402363] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10726.402673] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10726.403097] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10726.403658] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10726.403861] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10726.404301] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10726.404600] <EOI> > [10726.405112] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10726.405768] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10726.405980] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10726.407030] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10726.407348] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10726.407859] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10726.408537] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10726.409417] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10726.409897] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10726.410140] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10726.410347] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10726.410703] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10726.411311] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10726.412019] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10726.412290] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10726.412748] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10726.413086] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10726.413349] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10726.413782] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10726.414050] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10726.414463] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10726.414818] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10726.415555] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10753.561101] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [disk11_0:32753] > [10753.561110] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [fio:7935] > [10753.561118] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs > [10753.561119] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [fio:7938] > [10753.561120] dm_queue_length > [10753.561121] Modules linked in: > [10753.561121] brd > [10753.561122] ib_srpt(O-) > [10753.561123] xt_CHECKSUM > [10753.561125] scst_vdisk(O) > [10753.561126] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kdmwork-254:2:358] > [10753.561127] libcrc32c > [10753.561128] iptable_mangle > [10753.561128] scst(O) > [10753.561129] ipt_MASQUERADE > [10753.561130] dlm > [10753.561130] Modules linked in: > [10753.561131] netconsole > [10753.561132] ib_srpt(O-) > [10753.561133] nf_nat_masquerade_ipv4 > [10753.561133] configfs > [10753.561134] iptable_nat > [10753.561135] scst_vdisk(O) > [10753.561136] nf_nat_ipv4 > [10753.561136] libcrc32c > [10753.561137] nf_nat > [10753.561138] scst(O) > [10753.561138] nf_conntrack_ipv4 > [10753.561141] dlm > [10753.561141] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [fio:7937] > [10753.561142] nf_defrag_ipv4 > [10753.561143] netconsole > [10753.561143] dm_queue_length > [10753.561144] Modules linked in: > [10753.561145] xt_conntrack > [10753.561146] ib_srpt(O-) > [10753.561147] brd > [10753.561147] scst_vdisk(O) > [10753.561148] nf_conntrack > [10753.561149] libcrc32c > [10753.561150] xt_CHECKSUM > [10753.561150] configfs > [10753.561151] ipt_REJECT > [10753.561152] scst(O) > [10753.561152] xt_tcpudp > [10753.561153] dlm > [10753.561154] tun > [10753.561155] netconsole > [10753.561155] dm_queue_length > [10753.561156] ebtable_filter > [10753.561157] brd > [10753.561157] ebtables > [10753.561158] iptable_mangle > [10753.561159] ip6table_filter > [10753.561160] configfs > [10753.561160] ipt_MASQUERADE > [10753.561161] dm_queue_length > [10753.561162] ip6_tables > [10753.561163] nf_nat_masquerade_ipv4 > [10753.561163] iptable_filter > [10753.561164] xt_CHECKSUM > [10753.561165] ip_tables > [10753.561165] brd > [10753.561166] iptable_nat > [10753.561167] iptable_mangle > [10753.561168] nf_nat_ipv4 > [10753.561168] xt_CHECKSUM > [10753.561169] ipt_MASQUERADE > [10753.561170] iptable_mangle > [10753.561171] x_tables > [10753.561171] ipt_MASQUERADE > [10753.561172] af_packet > [10753.561173] nf_nat_masquerade_ipv4 > [10753.561174] nf_nat_masquerade_ipv4 > [10753.561174] nf_nat > [10753.561175] iptable_nat > [10753.561176] iptable_nat > [10753.561177] bridge > [10753.561177] nf_nat_ipv4 > [10753.561178] nf_nat_ipv4 > [10753.561179] nf_nat > [10753.561179] nf_nat > [10753.561180] stp > [10753.561181] nf_conntrack_ipv4 > [10753.561181] llc > [10753.561182] nf_defrag_ipv4 > [10753.561183] nf_conntrack_ipv4 > [10753.561184] iscsi_ibft > [10753.561184] nf_defrag_ipv4 > [10753.561185] iscsi_boot_sysfs > [10753.561186] nf_conntrack_ipv4 > [10753.561187] xt_conntrack > [10753.561187] xt_conntrack > [10753.561188] nf_defrag_ipv4 > [10753.561189] ib_srp > [10753.561190] nf_conntrack > [10753.561190] xt_conntrack > [10753.561191] scsi_transport_srp > [10753.561192] nf_conntrack > [10753.561192] ipt_REJECT > [10753.561193] ib_ipoib > [10753.561194] nf_conntrack > [10753.561195] rdma_ucm > [10753.561195] xt_tcpudp > [10753.561196] ipt_REJECT > [10753.561197] ib_ucm > [10753.561197] tun > [10753.561198] ipt_REJECT > [10753.561199] xt_tcpudp > [10753.561199] ebtable_filter > [10753.561200] xt_tcpudp > [10753.561201] ib_uverbs > [10753.561201] tun > [10753.561202] ebtables > [10753.561203] ebtable_filter > [10753.561204] ip6table_filter > [10753.561204] ebtables > [10753.561205] tun > [10753.561206] msr > [10753.561206] ip6_tables > [10753.561207] ebtable_filter > [10753.561208] iptable_filter > [10753.561209] ebtables > [10753.561209] ib_umad > [10753.561210] ip_tables > [10753.561211] rdma_cm > [10753.561211] ip6table_filter > [10753.561212] ib_cm > [10753.561213] ip6_tables > [10753.561213] x_tables > [10753.561214] iptable_filter > [10753.561215] iw_cm > [10753.561215] af_packet > [10753.561216] mlx4_ib > [10753.561217] bridge > [10753.561218] ip6table_filter > [10753.561218] stp > [10753.561220] ip_tables > [10753.561221] ip6_tables > [10753.561222] ib_sa > [10753.561222] iptable_filter > [10753.561223] llc > [10753.561224] ib_mad > [10753.561224] ip_tables > [10753.561225] x_tables > [10753.561226] iscsi_ibft > [10753.561226] x_tables > [10753.561227] iscsi_boot_sysfs > [10753.561228] af_packet > [10753.561229] af_packet > [10753.561229] ib_srp > [10753.561230] bridge > [10753.561231] ib_core > [10753.561231] scsi_transport_srp > [10753.561232] ib_addr > [10753.561233] stp > [10753.561234] x86_pkg_temp_thermal > [10753.561234] llc > [10753.561235] ib_ipoib > [10753.561236] iscsi_ibft > [10753.561237] bridge > [10753.561237] iscsi_boot_sysfs > [10753.561238] intel_powerclamp > [10753.561239] ib_srp > [10753.561239] stp > [10753.561240] rdma_ucm > [10753.561241] llc > [10753.561241] coretemp > [10753.561242] ib_ucm > [10753.561243] kvm_intel > [10753.561243] iscsi_ibft > [10753.561244] ib_uverbs > [10753.561245] scsi_transport_srp > [10753.561246] msr > [10753.561246] iscsi_boot_sysfs > [10753.561247] ib_umad > [10753.561248] ib_srp > [10753.561249] ib_ipoib > [10753.561249] kvm > [10753.561250] rdma_ucm > [10753.561251] rdma_cm > [10753.561251] irqbypass > [10753.561252] ib_cm > [10753.561253] crct10dif_pclmul > [10753.561253] scsi_transport_srp > [10753.561254] crc32_pclmul > [10753.561255] iw_cm > [10753.561255] ib_ipoib > [10753.561256] crc32c_intel > [10753.561257] rdma_ucm > [10753.561258] mlx4_ib > [10753.561258] aesni_intel > [10753.561259] ib_sa > [10753.561260] tg3 > [10753.561260] ib_mad > [10753.561261] iTCO_wdt > [10753.561262] ib_ucm > [10753.561263] ib_core > [10753.561263] aes_x86_64 > [10753.561264] ib_uverbs > [10753.561265] lrw > [10753.561265] ib_ucm > [10753.561266] ib_addr > [10753.561267] ipmi_ssif > [10753.561268] x86_pkg_temp_thermal > [10753.561268] ipmi_devintf > [10753.561269] ib_uverbs > [10753.561270] ptp > [10753.561270] msr > [10753.561271] iTCO_vendor_support > [10753.561272] msr > [10753.561272] intel_powerclamp > [10753.561273] gf128mul > [10753.561274] coretemp > [10753.561275] ib_umad > [10753.561275] kvm_intel > [10753.561276] glue_helper > [10753.561277] rdma_cm > [10753.561277] mlx4_core > [10753.561278] ib_umad > [10753.561279] kvm > [10753.561279] dcdbas > [10753.561280] ib_cm > [10753.561281] ablk_helper > [10753.561281] irqbypass > [10753.561282] iw_cm > [10753.561283] rdma_cm > [10753.561283] pps_core > [10753.561284] crct10dif_pclmul > [10753.561285] cryptd > [10753.561285] crc32_pclmul > [10753.561286] mlx4_ib > [10753.561287] libphy > [10753.561287] ib_sa > [10753.561288] crc32c_intel > [10753.561289] fjes > [10753.561290] ib_mad > [10753.561290] aesni_intel > [10753.561291] ib_core > [10753.561292] 8250_fintek > [10753.561292] tg3 > [10753.561293] sb_edac > [10753.561294] ib_addr > [10753.561294] pcspkr > [10753.561295] iTCO_wdt > [10753.561296] ib_cm > [10753.561296] aes_x86_64 > [10753.561297] iw_cm > [10753.561298] lrw > [10753.561298] ipmi_si > [10753.561299] ipmi_ssif > [10753.561300] x86_pkg_temp_thermal > [10753.561301] ipmi_devintf > [10753.561301] mlx4_ib > [10753.561302] edac_core > [10753.561303] ptp > [10753.561303] ipmi_msghandler > [10753.561304] intel_powerclamp > [10753.561305] acpi_power_meter > [10753.561306] coretemp > [10753.561306] tpm_tis > [10753.561307] kvm_intel > [10753.561308] tpm > [10753.561308] ib_sa > [10753.561309] kvm > [10753.561310] ib_mad > [10753.561310] lpc_ich > [10753.561312] ib_core > [10753.561313] button > [10753.561313] irqbypass > [10753.561314] ib_addr > [10753.561315] iTCO_vendor_support > [10753.561315] shpchp > [10753.561316] gf128mul > [10753.561317] mei_me > [10753.561318] x86_pkg_temp_thermal > [10753.561318] glue_helper > [10753.561319] intel_powerclamp > [10753.561320] mei > [10753.561320] mlx4_core > [10753.561321] mfd_core > [10753.561322] coretemp > [10753.561322] wmi > [10753.561323] dcdbas > [10753.561324] kvm_intel > [10753.561324] ablk_helper > [10753.561325] kvm > [10753.561326] dm_multipath > [10753.561327] crct10dif_pclmul > [10753.561327] dm_mod > [10753.561328] pps_core > [10753.561329] hid_generic > [10753.561329] cryptd > [10753.561330] usbhid > [10753.561331] libphy > [10753.561331] mgag200 > [10753.561332] crc32_pclmul > [10753.561333] i2c_algo_bit > [10753.561334] crc32c_intel > [10753.561334] fjes > [10753.561335] irqbypass > [10753.561336] aesni_intel > [10753.561336] drm_kms_helper > [10753.561337] 8250_fintek > [10753.561338] syscopyarea > [10753.561338] sb_edac > [10753.561339] sysfillrect > [10753.561340] pcspkr > [10753.561340] tg3 > [10753.561341] crct10dif_pclmul > [10753.561342] ipmi_si > [10753.561343] iTCO_wdt > [10753.561343] edac_core > [10753.561344] crc32_pclmul > [10753.561345] ipmi_msghandler > [10753.561346] sysimgblt > [10753.561346] crc32c_intel > [10753.561347] fb_sys_fops > [10753.561348] aes_x86_64 > [10753.561348] ttm > [10753.561349] acpi_power_meter > [10753.561350] drm > [10753.561350] tpm_tis > [10753.561351] aesni_intel > [10753.561352] ehci_pci > [10753.561352] tpm > [10753.561353] lrw > [10753.561354] tg3 > [10753.561354] sr_mod > [10753.561355] lpc_ich > [10753.561356] cdrom > [10753.561356] iTCO_wdt > [10753.561357] button > [10753.561358] ehci_hcd > [10753.561359] ipmi_ssif > [10753.561359] aes_x86_64 > [10753.561360] shpchp > [10753.561361] usbcore > [10753.561361] ipmi_devintf > [10753.561362] usb_common > [10753.561363] lrw > [10753.561364] sg > [10753.561364] mei_me > [10753.561365] ipmi_ssif > [10753.561365] > [10753.561366] ptp > [10753.561367] irq event stamp: 10969989 > [10753.561368] mei > [10753.561369] hardirqs last enabled at (10969988): > [10753.561374] mfd_core > [10753.561375] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10753.561376] ipmi_devintf > [10753.561377] wmi > [10753.561377] iTCO_vendor_support > [10753.561378] hardirqs last disabled at (10969989): > [10753.561380] gf128mul > [10753.561381] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10753.561382] dm_multipath > [10753.561383] softirqs last enabled at (8596058): > [10753.561385] dm_mod > [10753.561386] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10753.561387] ptp > [10753.561388] glue_helper > [10753.561388] softirqs last disabled at (8596459): > [10753.561390] mlx4_core > [10753.561390] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10753.561391] hid_generic > [10753.561392] dcdbas > [10753.561392] usbhid > [10753.561394] ablk_helper > [10753.561395] CPU: 4 PID: 7935 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10753.561396] mgag200 > [10753.561397] pps_core > [10753.561397] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10753.561398] iTCO_vendor_support > [10753.561399] i2c_algo_bit > [10753.561400] cryptd > [10753.561400] task: ffff8803fbdd8000 ti: ffff8803f7524000 task.ti: ffff8803f7524000 > [10753.561401] libphy > [10753.561402] gf128mul > [10753.561403] drm_kms_helper > [10753.561404] RIP: 0010:[<ffffffff815781b3>] > [10753.561405] fjes > [10753.561406] glue_helper > [10753.561406] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10753.561408] syscopyarea > [10753.561408] RSP: 0018:ffff88046f283d78 EFLAGS: 00000292 > [10753.561410] 8250_fintek > [10753.561410] RAX: ffff8803fbdd8000 RBX: 0000000000000292 RCX: 0000000000000000 > [10753.561412] sb_edac > [10753.561412] sysfillrect > [10753.561414] mlx4_core > [10753.561414] RDX: 0000000000000003 RSI: ffff8803fbdd8798 RDI: ffff8803fbdd8000 > [10753.561415] dcdbas > [10753.561416] RBP: ffff88046f283d88 R08: ffff88046f283cf8 R09: 0000000000000000 > [10753.561418] ablk_helper > [10753.561418] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10753.561419] pcspkr > [10753.561420] sysimgblt > [10753.561421] ipmi_si > [10753.561421] fb_sys_fops > [10753.561422] pps_core > [10753.561423] ttm > [10753.561424] edac_core > [10753.561424] R13: ffffea000c5d8000 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10753.561425] drm > [10753.561426] ipmi_msghandler > [10753.561426] ehci_pci > [10753.561428] acpi_power_meter > [10753.561428] FS: 00007f8a9b634700(0000) GS:ffff88046f280000(0000) knlGS:0000000000000000 > [10753.561430] sr_mod > [10753.561430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10753.561431] cdrom > [10753.561432] cryptd > [10753.561432] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10753.561434] tpm_tis > [10753.561434] ehci_hcd > [10753.561435] tpm > [10753.561436] usbcore > [10753.561436] Stack: > [10753.561437] libphy > [10753.561438] usb_common > [10753.561438] ffff88017fc01c00 > [10753.561439] sg > [10753.561440] lpc_ich > [10753.561440] 0000000000000001 > [10753.561441] > [10753.561442] ffff88046f283e38 > [10753.561442] irq event stamp: 19364635 > [10753.561443] fjes > [10753.561444] ffffffff811640d6 > [10753.561445] button > [10753.561445] > [10753.561446] hardirqs last enabled at (19364634): > [10753.561447] ffff88046f283dc8 > [10753.561448] 8250_fintek > [10753.561448] shpchp > [10753.561452] ffff880319820078 > [10753.561452] [<ffffffff8157940f>] restore_regs_and_iret+0x0/0x1d > [10753.561453] sb_edac > [10753.561454] hardirqs last disabled at (19364635): > [10753.561456] ffff88017fc1ba80 > [10753.561456] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10753.561457] ffff880317607cd8 > [10753.561458] softirqs last enabled at (13717672): > [10753.561458] > [10753.561459] pcspkr > [10753.561460] mei_me > [10753.561462] ffff880317607cd8 > [10753.561462] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10753.561464] ffff0001fbdd8798 > [10753.561464] ipmi_si > [10753.561465] 0000000000000292 > [10753.561466] softirqs last disabled at (13717773): > [10753.561467] ffff8803fbdd8798 > [10753.561468] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10753.561468] > [10753.561469] mei > [10753.561470] edac_core > [10753.561470] Call Trace: > [10753.561472] CPU: 0 PID: 7937 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10753.561473] <IRQ> > [10753.561474] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10753.561477] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10753.561480] mfd_core > [10753.561480] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10753.561482] task: ffff88040070df00 ti: ffff8803f1678000 task.ti: ffff8803f1678000 > [10753.561485] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10753.561487] RIP: 0010:[<ffffffff815781b3>] > [10753.561489] wmi > [10753.561489] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10753.561492] dm_multipath > [10753.561492] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10753.561494] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10753.561496] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10753.561499] dm_mod > [10753.561499] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10753.561500] RSP: 0018:ffff88046f203d78 EFLAGS: 00000292 > [10753.561503] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10753.561504] RAX: ffff88040070df00 RBX: 0000000000000292 RCX: 0000000000000000 > [10753.561506] ipmi_msghandler > [10753.561506] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10753.561507] RDX: 0000000000000003 RSI: ffff88040070e698 RDI: ffff88040070df00 > [10753.561509] acpi_power_meter > [10753.561510] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10753.561511] tpm_tis > [10753.561511] RBP: ffff88046f203d88 R08: ffff88046f203cf8 R09: 0000000000000000 > [10753.561514] hid_generic > [10753.561514] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10753.561516] usbhid > [10753.561516] R13: ffffea0007054c00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10753.561518] tpm > [10753.561518] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10753.561520] FS: 00007f8a9a632700(0000) GS:ffff88046f200000(0000) knlGS:0000000000000000 > [10753.561521] mgag200 > [10753.561522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10753.561524] i2c_algo_bit > [10753.561525] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10753.561526] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406f0 > [10753.561528] drm_kms_helper > [10753.561528] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10753.561530] syscopyarea > [10753.561531] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10753.561531] Stack: > [10753.561532] lpc_ich > [10753.561533] <EOI> > [10753.561535] sysfillrect > [10753.561535] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10753.561536] button > [10753.561539] ffff88017fc01c00 > [10753.561539] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10753.561542] sysimgblt > [10753.561542] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10753.561543] fb_sys_fops > [10753.561545] 0000000000000001 > [10753.561546] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10753.561547] ffff88046f203e38 > [10753.561551] ttm > [10753.561551] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10753.561552] ffffffff811640d6 > [10753.561554] drm > [10753.561554] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10753.561555] > [10753.561557] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10753.561558] shpchp > [10753.561560] ffff88046f203dc8 > [10753.561560] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10753.561562] ehci_pci > [10753.561562] mei_me > [10753.561565] 0000000000000001 > [10753.561565] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10753.561569] 0000000000000001 > [10753.561569] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10753.561570] sr_mod > [10753.561572] mei > [10753.561572] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10753.561573] cdrom > [10753.561574] ffff8801c1537a30 > [10753.561576] mfd_core > [10753.561576] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10753.561579] wmi > [10753.561579] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10753.561580] dm_multipath > [10753.561580] > [10753.561584] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10753.561585] ffff8801c1537a30 > [10753.561587] dm_mod > [10753.561588] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10753.561590] ffff00010070e698 > [10753.561590] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10753.561592] 0000000000000292 > [10753.561592] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10753.561595] ehci_hcd > [10753.561596] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10753.561597] ffff88040070e698 > [10753.561599] usbcore > [10753.561599] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10753.561599] > [10753.561601] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10753.561602] hid_generic > [10753.561603] usb_common > [10753.561605] usbhid > [10753.561606] sg > [10753.561606] Call Trace: > [10753.561606] > [10753.561608] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10753.561610] <IRQ> > [10753.561610] irq event stamp: 14391137 > [10753.561613] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10753.561615] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10753.561618] mgag200 > [10753.561618] [<ffffffff810b9331>] ? rcu_irq_exit+0x41/0x50 > [10753.561619] i2c_algo_bit > [10753.561621] Code: > [10753.561621] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10753.561623] hardirqs last enabled at (14391136): > [10753.561625] fc > [10753.561625] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10753.561628] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10753.561629] drm_kms_helper > [10753.561630] hardirqs last disabled at (14391137): > [10753.561632] 48 > [10753.561632] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10753.561635] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10753.561636] syscopyarea > [10753.561638] 83 > [10753.561638] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10753.561639] c7 > [10753.561640] sysfillrect > [10753.561643] 18 > [10753.561643] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10753.561645] sysimgblt > [10753.561646] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10753.561648] 48 > [10753.561648] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10753.561651] softirqs last enabled at (8589138): > [10753.561651] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10753.561653] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10753.561654] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10753.561656] softirqs last disabled at (8589495): > [10753.561657] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10753.561659] fb_sys_fops > [10753.561659] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10753.561661] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10753.561663] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10753.561665] 8b > [10753.561665] CPU: 2 PID: 7938 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10753.561666] 55 > [10753.561667] ttm > [10753.561667] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10753.561669] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10753.561670] 08 > [10753.561671] drm > [10753.561672] 53 > [10753.561672] ehci_pci > [10753.561673] 48 > [10753.561674] sr_mod > [10753.561675] task: ffff880400708000 ti: ffff8803f7af8000 task.ti: ffff8803f7af8000 > [10753.561676] 89 > [10753.561677] <EOI> > [10753.561677] f3 > [10753.561679] cdrom > [10753.561679] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10753.561680] be > [10753.561683] ehci_hcd > [10753.561683] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10753.561684] 01 > [10753.561685] usbcore > [10753.561686] RIP: 0010:[<ffffffff815781b3>] > [10753.561688] 00 > [10753.561688] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10753.561689] 00 > [10753.561690] RSP: 0018:ffff88046f243d78 EFLAGS: 00000292 > [10753.561691] 00 > [10753.561691] usb_common > [10753.561693] e8 > [10753.561694] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10753.561696] e1 > [10753.561696] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10753.561697] sg > [10753.561698] ce > [10753.561698] > [10753.561702] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10753.561703] b2 > [10753.561703] irq event stamp: 126134521 > [10753.561705] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10753.561706] ff > [10753.561707] hardirqs last enabled at (126134520): > [10753.561709] 4c > [10753.561709] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10753.561712] [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10753.561714] 89 > [10753.561714] RAX: ffff880400708000 RBX: 0000000000000292 RCX: 0000000000000000 > [10753.561716] e7 > [10753.561716] RDX: 0000000000000003 RSI: ffff880400708798 RDI: ffff880400708000 > [10753.561718] e8 > [10753.561718] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10753.561719] 49 > [10753.561721] hardirqs last disabled at (126134521): > [10753.561722] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10753.561723] RBP: ffff88046f243d88 R08: ffff88046f243cf8 R09: 0000000000000000 > [10753.561725] 00 > [10753.561726] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10753.561728] b3 > [10753.561728] [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10753.561729] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10753.561731] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10753.561732] ff > [10753.561732] R13: ffffea0008065000 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10753.561734] f6 > [10753.561734] FS: 00007f8a99e31700(0000) GS:ffff88046f240000(0000) knlGS:0000000000000000 > [10753.561736] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10753.561738] c7 > [10753.561739] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10753.561742] 02 > [10753.561742] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10753.561744] 74 > [10753.561744] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10753.561746] 13 > [10753.561747] softirqs last enabled at (119833606): > [10753.561748] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10753.561750] e8 > [10753.561750] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10753.561751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10753.561754] 0f > [10753.561754] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10753.561756] [<ffffffff8105a55b>] __do_softirq+0x1cb/0x230 > [10753.561758] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10753.561760] a8 > [10753.561760] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10753.561762] b2 > [10753.561762] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10753.561764] ff > [10753.561764] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10753.561766] 53 > [10753.561766] softirqs last disabled at (119835011): > [10753.561766] Stack: > [10753.561768] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10753.561770] ffff88017fc01c00 > [10753.561771] CPU: 1 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10753.561772] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10753.561774] 9d > [10753.561774] 0000000000000001 > [10753.561775] <65> > [10753.561776] Code: > [10753.561777] ff > [10753.561777] ffff88046f243e38 > [10753.561778] fc > [10753.561779] 0d > [10753.561780] ffffffff811640d6 > [10753.561780] 48 > [10753.561781] 56 > [10753.561782] 83 > [10753.561782] > [10753.561783] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10753.561784] 2b > [10753.561784] c7 > [10753.561786] a9 > [10753.561786] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10753.561787] 7e > [10753.561788] 18 > [10753.561788] 5b > [10753.561789] 48 > [10753.561790] 41 > [10753.561790] 8b > [10753.561791] 5c > [10753.561792] 55 > [10753.561793] RIP: 0010:[<ffffffff815781b3>] > [10753.561794] 08 > [10753.561794] 5d > [10753.561795] 53 > [10753.561795] c3 > [10753.561796] 48 > [10753.561797] 53 > [10753.561799] 89 f3 > [10753.561799] 9d > [10753.561801] be > [10753.561801] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10753.561803] e8 > [10753.561803] ffff88046f243dc8 > [10753.561804] 6a > [10753.561804] 0000000000000000 > [10753.561805] 79 > [10753.561806] 01 > [10753.561806] ffff880400708000 > [10753.561807] b2 > [10753.561808] ffff880201943278 > [10753.561809] 00 > [10753.561809] RSP: 0018:ffff88046f223d78 EFLAGS: 00000292 > [10753.561811] ff > [10753.561811] RAX: ffff88040120af80 RBX: 0000000000000292 RCX: 0000000000000000 > [10753.561812] eb > [10753.561813] 00 > [10753.561814] eb > [10753.561814] RDX: 0000000000000003 RSI: ffff88040120b718 RDI: ffff88040120af80 > [10753.561814] > [10753.561815] RBP: ffff88046f223d88 R08: ffff88046f223cf8 R09: 0000000000000000 > [10753.561816] > [10753.561817] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10753.561818] 00 > [10753.561819] ffff880201943278 > [10753.561820] e8 > [10753.561820] R13: ffffea00060e8600 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10753.561821] ffff000100708798 > [10753.561823] e1 > [10753.561823] FS: 0000000000000000(0000) GS:ffff88046f220000(0000) knlGS:0000000000000000 > [10753.561824] ce > [10753.561824] 0000000000000292 > [10753.561826] b2 > [10753.561826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10753.561827] ff > [10753.561828] ffff880400708798 > [10753.561829] 4c > [10753.561829] CR2: 00007ffadf28aa08 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10753.561830] 89 > [10753.561830] Stack: > [10753.561831] e7 > [10753.561832] ffff88017fc01c00 > [10753.561833] e8 > [10753.561833] 0000000000000001 > [10753.561834] 49 > [10753.561835] ffff88046f223e38 > [10753.561835] 00 > [10753.561836] ffffffff811640d6 > [10753.561836] > [10753.561837] b3 > [10753.561837] > [10753.561838] ff > [10753.561839] ffff88046f223dc8 > [10753.561840] f6 > [10753.561840] 0000000000000246 > [10753.561841] c7 > [10753.561842] ffffffff811859b3 > [10753.561842] 02 > [10753.561843] Call Trace: > [10753.561844] 74 > [10753.561844] <IRQ> > [10753.561845] 13 > [10753.561846] ffff880183a1d258 > [10753.561846] e8 > [10753.561846] > [10753.561848] 0f > [10753.561848] ffff880183a1d258 > [10753.561851] a8 > [10753.561851] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10753.561854] b2 > [10753.561854] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10753.561857] ff > [10753.561857] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10753.561860] 53 > [10753.561860] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10753.561861] ffff00010120b718 > [10753.561862] 9d > [10753.561862] 0000000000000292 > [10753.561863] <65> > [10753.561864] ffff88040120b718 > [10753.561864] ff > [10753.561864] > [10753.561866] 0d > [10753.561866] Call Trace: > [10753.561868] 56 > [10753.561868] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10753.561871] 2b > [10753.561871] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10753.561874] a9 > [10753.561874] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10753.561875] <IRQ> > [10753.561878] 7e > [10753.561878] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10753.561881] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10753.561883] 5b > [10753.561883] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10753.561886] [<ffffffff811859b3>] ? file_free_rcu+0x23/0x30 > [10753.561888] 41 > [10753.561888] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10753.561890] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10753.561892] 5c > [10753.561893] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10753.561894] 5d > [10753.561895] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10753.561897] c3 > [10753.561897] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10753.561900] 53 > [10753.561900] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10753.561902] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10753.561904] 9d > [10753.561904] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10753.561906] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10753.561909] e8 > [10753.561909] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10753.561910] <EOI> > [10753.561914] 6a > [10753.561914] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10753.561916] 79 > [10753.561916] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10753.561919] b2 > [10753.561919] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10753.561921] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10753.561923] ff > [10753.561923] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10753.561926] eb > [10753.561926] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10753.561928] eb > [10753.561928] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10753.561928] > [10753.561930] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10753.561932] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10753.561934] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10753.561935] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10753.561937] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10753.561940] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10753.561941] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10753.561943] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10753.561946] <EOI> > [10753.561946] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10753.561948] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10753.561950] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10753.561953] [<ffffffffa010f815>] __multipath_map.isra.16+0x145/0x260 [dm_multipath] > [10753.561955] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10753.561957] [<ffffffffa010f962>] multipath_map+0x12/0x20 [dm_multipath] > [10753.561960] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10753.561965] [<ffffffffa0170da3>] map_request+0x43/0x210 [dm_mod] > [10753.561966] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10753.561971] [<ffffffffa0170f8d>] map_tio_request+0x1d/0x40 [dm_mod] > [10753.561973] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10753.561975] [<ffffffff81075a0d>] kthread_worker_fn+0x7d/0x1a0 > [10753.561977] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10753.561978] [<ffffffff81075990>] ? __init_kthread_worker+0x60/0x60 > [10753.561981] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10753.561983] [<ffffffff8107591a>] kthread+0xea/0x100 > [10753.561985] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10753.561986] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10753.561988] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10753.561990] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10753.561991] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10753.561993] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10753.561995] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10753.561998] Code: > [10753.561998] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10753.562000] fc > [10753.562000] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10753.562003] 48 83 > [10753.562003] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10753.562006] c7 > [10753.562006] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10753.562007] 18 48 > [10753.562008] Code: > [10753.562009] 8b > [10753.562009] fc > [10753.562010] 55 > [10753.562011] 48 > [10753.562012] 08 > [10753.562013] 83 > [10753.562013] 53 > [10753.562014] c7 > [10753.562014] 48 > [10753.562015] 18 > [10753.562016] 89 > [10753.562016] 48 > [10753.562017] f3 > [10753.562017] 8b > [10753.562018] be > [10753.562019] 55 > [10753.562019] 01 > [10753.562020] 08 > [10753.562020] 00 > [10753.562021] 53 > [10753.562022] 00 > [10753.562022] 48 > [10753.562023] 00 > [10753.562023] 89 > [10753.562024] e8 > [10753.562025] f3 > [10753.562025] e1 > [10753.562026] be > [10753.562026] ce > [10753.562027] 01 > [10753.562028] b2 > [10753.562028] 00 > [10753.562029] ff > [10753.562029] 00 > [10753.562030] 4c > [10753.562030] 00 > [10753.562031] 89 > [10753.562032] e8 > [10753.562032] e7 > [10753.562033] e1 > [10753.562033] e8 > [10753.562034] ce > [10753.562034] 49 > [10753.562035] b2 > [10753.562036] 00 > [10753.562036] ff > [10753.562037] b3 > [10753.562037] 4c > [10753.562038] ff > [10753.562038] 89 > [10753.562039] f6 > [10753.562040] e7 > [10753.562040] c7 > [10753.562041] e8 > [10753.562041] 02 > [10753.562042] 49 > [10753.562042] 74 > [10753.562043] 00 > [10753.562044] 13 > [10753.562044] b3 > [10753.562045] e8 > [10753.562045] ff > [10753.562046] 0f > [10753.562047] f6 > [10753.562047] a8 > [10753.562048] c7 > [10753.562048] b2 > [10753.562049] 02 > [10753.562049] ff > [10753.562050] 74 > [10753.562051] 53 > [10753.562051] 13 > [10753.562052] 9d > [10753.562052] e8 > [10753.562053] <65> > [10753.562054] 0f > [10753.562054] ff > [10753.562055] a8 > [10753.562055] 0d > [10753.562056] b2 > [10753.562056] 56 > [10753.562057] ff > [10753.562058] 2b > [10753.562058] 53 > [10753.562059] a9 > [10753.562059] 9d > [10753.562060] 7e > [10753.562061] <65> > [10753.562061] 5b > [10753.562062] ff > [10753.562062] 41 > [10753.562063] 0d > [10753.562064] 5c > [10753.562064] 56 > [10753.562065] 5d > [10753.562065] 2b > [10753.562066] c3 > [10753.562066] a9 > [10753.562067] 53 > [10753.562068] 7e > [10753.562068] 9d > [10753.562069] 5b > [10753.562069] e8 > [10753.562070] 41 > [10753.562071] 6a > [10753.562071] 5c > [10753.562072] 79 > [10753.562072] 5d > [10753.562073] b2 > [10753.562073] c3 > [10753.562074] ff > [10753.562075] 53 > [10753.562075] eb > [10753.562076] 9d > [10753.562076] eb > [10753.562077] e8 > [10753.562077] > [10753.562080] 6a 79 b2 ff eb eb > [10753.718766] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10753.737483] irq event stamp: 13918891 > [10753.737667] hardirqs last enabled at (13918890): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10753.738020] hardirqs last disabled at (13918891): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10753.738364] softirqs last enabled at (10133236): [<ffffffffa05c9a57>] scst_check_blocked_dev+0x77/0x1c0 [scst] > [10753.738765] softirqs last disabled at (10133267): [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10753.739063] CPU: 11 PID: 32753 Comm: disk11_0 Tainted: G O L 4.4.6-debug+ #2 > [10753.739256] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10753.739473] task: ffff880439375f00 ti: ffff880460d30000 task.ti: ffff880460d30000 > [10753.739599] RIP: 0010:[<ffffffff815781b3>] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10753.739990] RSP: 0018:ffff88046f363d68 EFLAGS: 00000292 > [10753.740149] RAX: ffff880439375f00 RBX: 0000000000000292 RCX: 0000000000000000 > [10753.740321] RDX: 0000000000000003 RSI: ffff880439376698 RDI: ffff880439375f00 > [10753.740439] RBP: ffff88046f363d78 R08: ffff88046f363ce8 R09: 0000000000000000 > [10753.740664] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10753.740772] R13: ffffea000e725600 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10753.740991] FS: 0000000000000000(0000) GS:ffff88046f360000(0000) knlGS:0000000000000000 > [10753.741189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10753.741321] CR2: 00005633130dd1a8 CR3: 0000000401468000 CR4: 00000000001406e0 > [10753.741465] Stack: > [10753.741613] ffff88017fc01c00 0000000000000001 ffff88046f363e30 ffffffff811640d6 > [10753.742334] ffff88046f363db8 ffffffff8157940f 0000000100300001 ffff88039c95bfc0 > [10753.743130] ffff88039c95bfc0 0000000100000282 0000000000000292 ffff880439376698 > [10753.743870] Call Trace: > [10753.744031] <IRQ> > [10753.744151] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10753.744464] [<ffffffff8157940f>] ? retint_kernel+0x10/0x10 > [10753.744618] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10753.744803] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10753.744935] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10753.745075] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10753.745209] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10753.745387] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10753.745585] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10753.745769] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10753.745920] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10753.746063] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10753.746262] [<ffffffff810065a5>] do_IRQ+0x65/0x110 > [10753.746437] [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20 > [10753.746653] [<ffffffff815793c9>] common_interrupt+0x89/0x89 > [10753.746803] <EOI> > [10753.746973] [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20 > [10753.747278] [<ffffffff8117bad7>] ? put_object+0x7/0x40 > [10753.747453] [<ffffffff8117bb48>] ? __delete_object+0x38/0x70 > [10753.747650] [<ffffffff8117bb98>] delete_object_full+0x18/0x20 > [10753.747797] [<ffffffff8156dab1>] kmemleak_free+0x31/0x80 > [10753.747987] [<ffffffff811642fa>] kmem_cache_free+0x7a/0x160 > [10753.748137] [<ffffffff81113282>] mempool_free_slab+0x12/0x20 > [10753.748284] [<ffffffff81113406>] mempool_free+0x26/0x80 > [10753.748423] [<ffffffff8127e8c1>] bvec_free+0x31/0x40 > [10753.748551] [<ffffffff8127ee54>] bio_free+0x34/0x60 > [10753.748737] [<ffffffff8127ee9e>] bio_put+0x1e/0x30 > [10753.748865] [<ffffffffa06f042d>] blockio_endio+0x2d/0xe0 [scst_vdisk] > [10753.749111] [<ffffffff812800da>] bio_endio+0x3a/0x60 > [10753.749289] [<ffffffffa0560b08>] brd_make_request+0xe8/0x1ae [brd] > [10753.749431] [<ffffffff812826fc>] ? generic_make_request_checks+0x10c/0x630 > [10753.749628] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10753.749762] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10753.749942] [<ffffffffa06f0a33>] blockio_exec_rw+0x553/0x670 [scst_vdisk] > [10753.750123] [<ffffffffa06f0cb0>] ? blockio_exec_write_verify+0x50/0x50 [scst_vdisk] > [10753.750325] [<ffffffffa06f0cbd>] blockio_exec_read+0xd/0x20 [scst_vdisk] > [10753.750472] [<ffffffffa06e7773>] vdev_do_job+0xb3/0x310 [scst_vdisk] > [10753.750631] [<ffffffffa06ebcc4>] blockio_exec+0x74/0x180 [scst_vdisk] > [10753.750759] [<ffffffff810a2931>] ? trace_hardirqs_on_caller+0x121/0x1b0 > [10753.750930] [<ffffffffa0609b12>] ? sgv_get_obj+0x72/0x1b0 [scst] > [10753.751195] [<ffffffffa05c7154>] scst_do_real_exec+0x94/0x3d0 [scst] > [10753.751372] [<ffffffffa05c01d3>] ? scst_is_not_reservation_holder+0x13/0x60 [scst] > [10753.751602] [<ffffffffa05c13ff>] ? __scst_check_local_events+0x19f/0x370 [scst] > [10753.751802] [<ffffffffa05ca8dc>] scst_exec_check_blocking+0xcc/0x340 [scst] > [10753.751983] [<ffffffffa05cac7d>] scst_exec_check_sn+0x12d/0x2c0 [scst] > [10753.752136] [<ffffffff8107a896>] ? finish_task_switch+0xa6/0x220 > [10753.752307] [<ffffffff8107a857>] ? finish_task_switch+0x67/0x220 > [10753.752548] [<ffffffffa05cf892>] scst_process_active_cmd+0x3e2/0x760 [scst] > [10753.752696] [<ffffffff810a29cd>] ? trace_hardirqs_on+0xd/0x10 > [10753.752875] [<ffffffffa05d1001>] scst_do_job_active+0xe1/0x160 [scst] > [10753.753142] [<ffffffffa05d11b2>] scst_cmd_thread+0x132/0x290 [scst] > [10753.753311] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10753.753524] [<ffffffffa05d1080>] ? scst_do_job_active+0x160/0x160 [scst] > [10753.753681] [<ffffffff8107591a>] kthread+0xea/0x100 > [10753.753836] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10753.753992] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10753.754165] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10753.754315] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10764.112230] scsi host50: SRP abort called > [10764.112455] scsi host50: Sending SRP abort for tag 0x0 > [10764.116228] [32754]: scst: scst_rx_mgmt_fn:7350:TM fn ABORT_TASK/0 (mcmd ffff88006892e788, initiator fe80:0000:0000:0000:e41d:2d03:000a:85b1, target fe80:0000:0000:0000:e41d:2d03:000a:6d51) > [10764.116401] [32754]: scst_rx_mgmt_fn:7364:sess=ffff88041067f400, tag_set 1, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff880454310548 > [10764.116635] [32754]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892e788 to active mgmt cmd list > [10764.117014] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892e788 from active cmd list > [10764.117333] [31019]: scst_abort_task:6762:Aborting task (cmd ffff8804543140a8, sn -20, set 1, tag 0, queue_type 1) > [10764.117488] [31019]: scst: scst_abort_cmd:5849:Aborting cmd ffff8804543140a8 (tag 0, op READ(10)) > [10764.117908] [31019]: scst_abort_cmd:5934:cmd ffff8804543140a8 (tag 0) is being executed > [10764.118220] [31019]: scst: scst_abort_cmd:5970:cmd ffff8804543140a8 (tag 0, sn 4294967276) being executed/xmitted (state EXEC_WAIT, op READ(10), proc time 63 sec., timeout 60 sec.), deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1, internal 0, mcmd fn 0 (mcmd ffff88006892e788), initiator fe80:0000:0000:0000:e41d:2d03:000a:85b1, target fe80:0000:0000:0000:e41d:2d03:000a:6d51) > [10764.118408] [31019]: scst: scst_set_mcmd_next_state:6011:cmd_done_wait_count(1) not 0, preparing to wait > [10765.270812] INFO: rcu_sched self-detected stall on CPU > [10765.271293] 4-...: (6499 ticks this GP) idle=ca3/140000000000002/0 softirq=1007789/1007789 fqs=1001 > [10765.271442] (t=6500 jiffies g=219292 c=219291 q=13711541) > [10765.271809] Task dump for CPU 0: > [10765.272032] fio R running task 0 7937 7744 0x00000008 > [10765.272691] ffff8803f167b898 ffff8804536bd140 0000000000000000 0000000000000000 > [10765.273812] 0000000000000000 0000000000000000 ffff88040070df00 0000000000000001 > [10765.274809] 0000000000000000 0000000000000001 ffff88040070df00 ffffffffffffff10 > [10765.275910] Call Trace: > [10765.276149] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.276362] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10765.276514] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10765.276828] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10765.277080] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10765.277216] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10765.277479] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10765.277651] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10765.277899] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10765.278145] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.278401] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10765.278610] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10765.278777] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10765.278941] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10765.279138] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10765.279261] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10765.279486] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10765.279661] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10765.279829] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10765.280114] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10765.280307] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10765.280564] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.280766] INFO: rcu_sched self-detected stall on CPU > [10765.280770] 0-...: (6412 ticks this GP) idle=911/140000000000002/0 softirq=1235409/1235409 fqs=1001 > [10765.280773] (t=6500 jiffies g=219292 c=219291 q=13711939) > [10765.280775] INFO: rcu_sched self-detected stall on CPU > [10765.280779] 2-...: (6500 ticks this GP) idle=203/140000000000002/0 softirq=1498700/1498700 fqs=1002 > [10765.280781] (t=6500 jiffies g=219292 c=219291 q=13711939) > [10765.281492] Task dump for CPU 2: > [10765.281636] fio R running task 0 7938 7744 0x00000008 > [10765.281940] ffff8803f7afb898 ffff8804536bd140 0000000000000000 0000000000000000 > [10765.282551] 0000000000000000 0000000000000000 ffff880400708000 0000000000000001 > [10765.283093] 0000000000000000 0000000000000001 ffff880400708000 ffffffffffffff10 > [10765.283618] Call Trace: > [10765.283746] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.283849] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10765.283977] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10765.284102] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10765.284238] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10765.284368] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10765.284486] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10765.284616] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10765.284722] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10765.284842] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.284985] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10765.285085] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10765.285202] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10765.285313] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10765.285428] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10765.285558] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10765.285688] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10765.285812] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10765.285932] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10765.286048] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10765.286193] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10765.286317] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.286415] Task dump for CPU 4: > [10765.286525] fio R running task 0 7935 7744 0x00000008 > [10765.286964] 0000000000001e40 ffff88046f283ab8 ffffffff8108195f 0000000000000004 > [10765.287498] ffffffff81a40a40 ffff88046f283ad0 ffffffff81084742 0000000000000005 > [10765.288065] ffff88046f283b00 ffffffff810b5e79 ffff88046f295400 ffffffff81a40a40 > [10765.288674] Call Trace: > [10765.288821] <IRQ> [<ffffffff8108195f>] sched_show_task+0xbf/0x150 > [10765.289152] [<ffffffff81084742>] dump_cpu_task+0x32/0x40 > [10765.289292] [<ffffffff810b5e79>] rcu_dump_cpu_stacks+0x89/0xe0 > [10765.289432] [<ffffffff810b9999>] rcu_check_callbacks+0x439/0x730 > [10765.289588] [<ffffffff810bc1c4>] update_process_times+0x34/0x60 > [10765.289730] [<ffffffff810caaa0>] tick_sched_handle.isra.18+0x20/0x50 > [10765.289896] [<ffffffff810cb148>] tick_sched_timer+0x38/0x70 > [10765.290036] [<ffffffff810cb110>] ? tick_sched_do_timer+0x30/0x30 > [10765.290173] [<ffffffff810bc865>] __hrtimer_run_queues+0xa5/0x1c0 > [10765.290390] [<ffffffff810bcef6>] hrtimer_interrupt+0xa6/0x1b0 > [10765.290529] [<ffffffff8109fb3d>] ? trace_hardirqs_off+0xd/0x10 > [10765.290631] [<ffffffff81038ba3>] smp_trace_apic_timer_interrupt+0x63/0x90 > [10765.290744] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10765.290855] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10765.290982] [<ffffffff815781b3>] ? _raw_spin_unlock_irqrestore+0x33/0x50 > [10765.291096] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10765.291208] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10765.291322] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10765.291449] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10765.291561] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10765.291687] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10765.291802] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10765.291915] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10765.292040] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10765.292204] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10765.292330] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10765.292492] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10765.292615] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10765.292751] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10765.292871] <EOI> [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.293139] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10765.293259] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10765.293392] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10765.293538] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10765.293670] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10765.293790] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10765.293910] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10765.294041] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10765.294151] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.294295] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10765.294446] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10765.294613] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10765.294747] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10765.294914] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10765.295062] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10765.295183] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10765.295317] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10765.295451] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10765.295576] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10765.295698] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10765.295830] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.295963] Task dump for CPU 0: > [10765.296100] fio R running task 0 7937 7744 0x00000008 > [10765.296734] ffff8803f167b898 ffff8804536bd140 0000000000000000 0000000000000000 > [10765.297665] 0000000000000000 0000000000000000 ffff88040070df00 0000000000000001 > [10765.298695] 0000000000000000 0000000000000001 ffff88040070df00 ffffffffffffff10 > [10765.299591] Call Trace: > [10765.299782] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.299953] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10765.300118] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10765.300363] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10765.300506] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10765.300650] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10765.300887] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10765.301094] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10765.301393] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10765.301577] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.301817] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10765.301951] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10765.302143] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10765.302359] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10765.302512] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10765.302664] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10765.302848] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10765.302981] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10765.303128] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10765.303255] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10765.303427] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10765.303656] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.303787] Task dump for CPU 2: > [10765.303918] fio R running task 0 7938 7744 0x00000008 > [10765.304354] 0000000000001e40 ffff88046f243ab8 ffffffff8108195f 0000000000000002 > [10765.305171] ffffffff81a40a40 ffff88046f243ad0 ffffffff81084742 0000000000000003 > [10765.306025] ffff88046f243b00 ffffffff810b5e79 ffff88046f255400 ffffffff81a40a40 > [10765.306874] Call Trace: > [10765.307035] <IRQ> [<ffffffff8108195f>] sched_show_task+0xbf/0x150 > [10765.307436] [<ffffffff81084742>] dump_cpu_task+0x32/0x40 > [10765.307729] [<ffffffff810b5e79>] rcu_dump_cpu_stacks+0x89/0xe0 > [10765.308054] [<ffffffff810b9999>] rcu_check_callbacks+0x439/0x730 > [10765.308207] [<ffffffff810bc1c4>] update_process_times+0x34/0x60 > [10765.308381] [<ffffffff810caaa0>] tick_sched_handle.isra.18+0x20/0x50 > [10765.308539] [<ffffffff810cb148>] tick_sched_timer+0x38/0x70 > [10765.308816] [<ffffffff810cb110>] ? tick_sched_do_timer+0x30/0x30 > [10765.309055] [<ffffffff810bc865>] __hrtimer_run_queues+0xa5/0x1c0 > [10765.309246] [<ffffffff810bcef6>] hrtimer_interrupt+0xa6/0x1b0 > [10765.309401] [<ffffffff8109fb3d>] ? trace_hardirqs_off+0xd/0x10 > [10765.309643] [<ffffffff81038ba3>] smp_trace_apic_timer_interrupt+0x63/0x90 > [10765.309759] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10765.309909] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10765.310054] [<ffffffff815781b3>] ? _raw_spin_unlock_irqrestore+0x33/0x50 > [10765.310226] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10765.310381] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10765.310522] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10765.310667] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10765.310860] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10765.311075] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10765.311285] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10765.311467] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10765.311660] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10765.311842] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10765.311991] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10765.312170] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10765.312381] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10765.312502] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10765.312632] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10765.312826] <EOI> [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.313135] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10765.313266] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10765.313424] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10765.313607] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10765.313738] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10765.313954] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10765.314094] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10765.314275] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10765.314388] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.314516] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10765.314738] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10765.314917] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10765.315116] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10765.315283] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10765.315415] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10765.315566] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10765.315707] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10765.315868] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10765.316010] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10765.316198] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10765.316322] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.316530] Task dump for CPU 4: > [10765.316737] fio R running task 0 7935 7744 0x00000008 > [10765.317203] 0000000000000001 00007f8a58031000 0000000000001000 000000000000001e > [10765.317947] ffff8803f7527de8 ffff8803fbdd8000 0000000000000246 ffff8803f7437300 > [10765.318778] 000000000000001e 0000000000000246 ffff8803f7527e38 ffff8803fbdd8000 > [10765.319525] Call Trace: > [10765.319669] [<ffffffff811d3066>] ? do_io_submit+0x2f6/0x770 > [10765.319853] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10765.319993] [<ffffffff811d34eb>] ? SyS_io_submit+0xb/0x10 > [10765.320150] [<ffffffff815788db>] ? entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.320304] Task dump for CPU 0: > [10765.320555] fio R running task 0 7937 7744 0x00000008 > [10765.321211] 0000000000001e40 ffff88046f203ab8 ffffffff8108195f 0000000000000000 > [10765.322400] ffffffff81a40a40 ffff88046f203ad0 ffffffff81084742 0000000000000001 > [10765.323303] ffff88046f203b00 ffffffff810b5e79 ffff88046f215400 ffffffff81a40a40 > [10765.324130] Call Trace: > [10765.324330] <IRQ> [<ffffffff8108195f>] sched_show_task+0xbf/0x150 > [10765.324936] [<ffffffff81084742>] dump_cpu_task+0x32/0x40 > [10765.325164] [<ffffffff810b5e79>] rcu_dump_cpu_stacks+0x89/0xe0 > [10765.325309] [<ffffffff810b9999>] rcu_check_callbacks+0x439/0x730 > [10765.325476] [<ffffffff810bc1c4>] update_process_times+0x34/0x60 > [10765.325659] [<ffffffff810caaa0>] tick_sched_handle.isra.18+0x20/0x50 > [10765.325980] [<ffffffff810cb148>] tick_sched_timer+0x38/0x70 > [10765.326220] [<ffffffff810cb110>] ? tick_sched_do_timer+0x30/0x30 > [10765.326459] [<ffffffff810bc865>] __hrtimer_run_queues+0xa5/0x1c0 > [10765.326699] [<ffffffff810bcef6>] hrtimer_interrupt+0xa6/0x1b0 > [10765.326895] [<ffffffff8109fb3d>] ? trace_hardirqs_off+0xd/0x10 > [10765.327086] [<ffffffff81038ba3>] smp_trace_apic_timer_interrupt+0x63/0x90 > [10765.327224] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10765.327407] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10765.327622] [<ffffffff815781b3>] ? _raw_spin_unlock_irqrestore+0x33/0x50 > [10765.327760] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10765.327916] [<ffffffff811859b3>] ? file_free_rcu+0x23/0x30 > [10765.328133] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10765.328434] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10765.328835] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10765.329043] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10765.329243] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10765.329487] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10765.329649] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10765.329865] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10765.330097] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10765.330272] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10765.330456] [<ffffffff81038ba8>] smp_trace_apic_timer_interrupt+0x68/0x90 > [10765.330696] [<ffffffff81038bd9>] smp_apic_timer_interrupt+0x9/0x10 > [10765.330836] [<ffffffff81579669>] apic_timer_interrupt+0x89/0x90 > [10765.330993] <EOI> [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.331359] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10765.331554] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10765.331757] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10765.332129] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10765.332320] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10765.332594] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10765.332749] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10765.333071] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10765.333244] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.333426] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10765.333644] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10765.333810] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10765.334020] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10765.334203] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10765.334348] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10765.334474] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10765.334656] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10765.334972] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10765.335179] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10765.335430] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10765.335757] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.335928] Task dump for CPU 2: > [10765.336319] fio R running task 0 7938 7744 0x00000008 > [10765.337078] ffff8803f7afb898 ffff8804536bd140 0000000000000000 0000000000000000 > [10765.338168] 0000000000000000 0000000000000000 ffff880400708000 0000000000000001 > [10765.339181] 0000000000000000 0000000000000001 ffff880400708000 ffffffffffffff10 > [10765.340163] Call Trace: > [10765.340325] [<ffffffff8157816f>] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.340515] [<ffffffff81283717>] get_request+0x447/0x8e0 > [10765.340721] [<ffffffff812832d0>] ? blk_cleanup_queue+0x270/0x270 > [10765.340843] [<ffffffff812b2401>] ? __delay+0x1/0x10 > [10765.341044] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10765.341210] [<ffffffff812863b1>] blk_queue_bio+0x111/0x300 > [10765.341364] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10765.341552] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10765.341776] [<ffffffff8127ec39>] ? bio_set_pages_dirty+0x49/0x60 > [10765.341929] [<ffffffff811c2218>] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.342070] [<ffffffff811bc6f0>] ? I_BDEV+0x10/0x10 > [10765.342378] [<ffffffff811c2b8e>] __blockdev_direct_IO+0x3e/0x40 > [10765.342599] [<ffffffff811bd013>] blkdev_direct_IO+0x53/0x80 > [10765.342798] [<ffffffff81111f4c>] generic_file_read_iter+0x48c/0x5c0 > [10765.343017] [<ffffffff810a303e>] ? __lock_acquire+0x3be/0x1ab0 > [10765.343241] [<ffffffff811bd490>] ? blkdev_write_iter+0x100/0x100 > [10765.343401] [<ffffffff811bd4c0>] blkdev_read_iter+0x30/0x40 > [10765.343562] [<ffffffff811d1e72>] aio_run_iocb+0x122/0x2a0 > [10765.343696] [<ffffffff811d3066>] do_io_submit+0x2f6/0x770 > [10765.343873] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10765.344044] [<ffffffff811d34eb>] SyS_io_submit+0xb/0x10 > [10765.344231] [<ffffffff815788db>] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.344385] Task dump for CPU 4: > [10765.344560] fio R running task 0 7935 7744 0x00000008 > [10765.345208] 0000000000000001 00007f8a58031000 0000000000001000 000000000000001e > [10765.345981] ffff8803f7527de8 ffff8803fbdd8000 0000000000000246 ffff8803f7437300 > [10765.347051] 000000000000001e 0000000000000246 ffff8803f7527e38 ffff8803fbdd8000 > [10765.347947] Call Trace: > [10765.348145] [<ffffffff811d3066>] ? do_io_submit+0x2f6/0x770 > [10765.348313] [<ffffffff811d2f03>] ? do_io_submit+0x193/0x770 > [10765.348448] [<ffffffff811d34eb>] ? SyS_io_submit+0xb/0x10 > [10765.348880] [<ffffffff815788db>] ? entry_SYSCALL_64_fastpath+0x16/0x6f > [10777.682813] systemd-journald[8985]: File /var/log/journal/6d2d7eb462015b615a423a1056c6183d/system.journal corrupted or uncleanly shut down, renaming and replacing. > [10778.058082] systemd-coredump[8980]: Detected coredump of the journal daemon itself, diverted to /var/lib/systemd/coredump/core.systemd-journal.0.fd2308d1f77847cf9c32b20535a1d348.499.1458676877000000.xz. > [10781.560307] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [disk11_0:32753] > [10781.560440] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10781.573088] irq event stamp: 20161671 > [10781.573201] hardirqs last enabled at (20161670): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10781.573436] hardirqs last disabled at (20161671): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10781.573698] softirqs last enabled at (10133236): [<ffffffffa05c9a57>] scst_check_blocked_dev+0x77/0x1c0 [scst] > [10781.573951] softirqs last disabled at (10133267): [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10781.574196] CPU: 11 PID: 32753 Comm: disk11_0 Tainted: G O L 4.4.6-debug+ #2 > [10781.574316] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10781.574425] task: ffff880439375f00 ti: ffff880460d30000 task.ti: ffff880460d30000 > [10781.574531] RIP: 0010:[<ffffffff815781b3>] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10781.574742] RSP: 0018:ffff88046f363d68 EFLAGS: 00000292 > [10781.574854] RAX: ffff880439375f00 RBX: 0000000000000292 RCX: 0000000000000000 > [10781.574969] RDX: 0000000000000003 RSI: ffff880439376698 RDI: ffff880439375f00 > [10781.575105] RBP: ffff88046f363d78 R08: ffff88046f363ce8 R09: 0000000000000000 > [10781.575209] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10781.575310] R13: ffffea0001036200 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10781.575429] FS: 0000000000000000(0000) GS:ffff88046f360000(0000) knlGS:0000000000000000 > [10781.575542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10781.575676] CR2: 00005633130dd1a8 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10781.575786] Stack: > [10781.575896] ffff88017fc01c00 0000000000000001 ffff88046f363e30 ffffffff811640d6 > [10781.576422] ffff88046f363db8 ffffffff811640c0 0000000100300001 ffff880040d8b7c8 > [10781.576944] ffff880040d8b7c8 0000000113d14e00 0000000000000292 ffff880439376698 > [10781.577475] Call Trace: > [10781.577585] <IRQ> > [10781.577690] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10781.577918] [<ffffffff811640c0>] ? __slab_free+0xb0/0x270 > [10781.578050] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10781.578176] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10781.578323] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10781.578447] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10781.578584] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10781.578697] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10781.578805] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10781.578927] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10781.579038] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10781.579164] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10781.579276] [<ffffffff810065a5>] do_IRQ+0x65/0x110 > [10781.579407] [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20 > [10781.579518] [<ffffffff815793c9>] common_interrupt+0x89/0x89 > [10781.579629] <EOI> > [10781.579746] [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20 > [10781.579977] [<ffffffff8117bad7>] ? put_object+0x7/0x40 > [10781.580090] [<ffffffff8117bb48>] ? __delete_object+0x38/0x70 > [10781.580212] [<ffffffff8117bb98>] delete_object_full+0x18/0x20 > [10781.580328] [<ffffffff8156dab1>] kmemleak_free+0x31/0x80 > [10781.580430] [<ffffffff811642fa>] kmem_cache_free+0x7a/0x160 > [10781.580553] [<ffffffff81113282>] mempool_free_slab+0x12/0x20 > [10781.580672] [<ffffffff81113406>] mempool_free+0x26/0x80 > [10781.580798] [<ffffffff8127e8c1>] bvec_free+0x31/0x40 > [10781.580916] [<ffffffff8127ee54>] bio_free+0x34/0x60 > [10781.581037] [<ffffffff8127ee9e>] bio_put+0x1e/0x30 > [10781.581162] [<ffffffffa06f042d>] blockio_endio+0x2d/0xe0 [scst_vdisk] > [10781.581287] [<ffffffff812800da>] bio_endio+0x3a/0x60 > [10781.581404] [<ffffffffa0560b08>] brd_make_request+0xe8/0x1ae [brd] > [10781.581521] [<ffffffff812826fc>] ? generic_make_request_checks+0x10c/0x630 > [10781.581631] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10781.581746] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10781.581871] [<ffffffffa06f0a33>] blockio_exec_rw+0x553/0x670 [scst_vdisk] > [10781.581992] [<ffffffffa06f0cb0>] ? blockio_exec_write_verify+0x50/0x50 [scst_vdisk] > [10781.582106] [<ffffffffa06f0cbd>] blockio_exec_read+0xd/0x20 [scst_vdisk] > [10781.582260] [<ffffffffa06e7773>] vdev_do_job+0xb3/0x310 [scst_vdisk] > [10781.582398] [<ffffffffa06ebcc4>] blockio_exec+0x74/0x180 [scst_vdisk] > [10781.582514] [<ffffffff810a2931>] ? trace_hardirqs_on_caller+0x121/0x1b0 > [10781.582652] [<ffffffffa0609b12>] ? sgv_get_obj+0x72/0x1b0 [scst] > [10781.582804] [<ffffffffa05c7154>] scst_do_real_exec+0x94/0x3d0 [scst] > [10781.582952] [<ffffffffa05c01d3>] ? scst_is_not_reservation_holder+0x13/0x60 [scst] > [10781.583109] [<ffffffffa05c13ff>] ? __scst_check_local_events+0x19f/0x370 [scst] > [10781.583245] [<ffffffffa05ca8dc>] scst_exec_check_blocking+0xcc/0x340 [scst] > [10781.583387] [<ffffffffa05cac7d>] scst_exec_check_sn+0x12d/0x2c0 [scst] > [10781.583520] [<ffffffff8107a896>] ? finish_task_switch+0xa6/0x220 > [10781.583646] [<ffffffff8107a857>] ? finish_task_switch+0x67/0x220 > [10781.583777] [<ffffffffa05cf892>] scst_process_active_cmd+0x3e2/0x760 [scst] > [10781.583920] [<ffffffff810a29cd>] ? trace_hardirqs_on+0xd/0x10 > [10781.584054] [<ffffffffa05d1001>] scst_do_job_active+0xe1/0x160 [scst] > [10781.584193] [<ffffffffa05d11b2>] scst_cmd_thread+0x132/0x290 [scst] > [10781.584316] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10781.584458] [<ffffffffa05d1080>] ? scst_do_job_active+0x160/0x160 [scst] > [10781.584590] [<ffffffff8107591a>] kthread+0xea/0x100 > [10781.584706] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10781.584827] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10781.584948] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10781.585086] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10788.304454] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b2-611 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.304852] scsi host60: ib_srp: DREQ received - connection closed > [10788.305727] ib_srpt: RDMA t 5 for idx 5 failed with status 5. If this has not been triggered by a cable pull, please check the involved IB HCA's and cables. > [10788.305926] ib_srpt: sending response for idx 5 failed with status 5 > [10788.306061] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b2-611. > [10788.328967] [560]: scst_unregister_session:7901:Unregistering session ffff880416bf0000 (wait 0) > [10788.329005] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b2-617 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.329770] [560]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892d810) > [10788.329839] scsi host60: ib_srp: DREQ received - connection closed > [10788.330132] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b2-623 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.330698] [574]: scst_unregister_session:7901:Unregistering session ffff880416bf0e80 (wait 0) > [10788.330921] [560]: scst_rx_mgmt_fn:7364:sess=ffff880416bf0000, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.331187] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b2-617. > [10788.331547] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b2-629 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.332133] [574]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892e5d0) > [10788.332434] [560]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892d810 to active mgmt cmd list > [10788.332490] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d52-610 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.332907] [574]: scst_rx_mgmt_fn:7364:sess=ffff880416bf0e80, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.333332] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892d810 from active cmd list > [10788.333516] [586]: scst_unregister_session:7901:Unregistering session ffff880416bf1d00 (wait 0) > [10788.333707] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf0000 (mcmd ffff88006892d810) > [10788.333977] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d52-616 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.334379] [574]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892e5d0 to active mgmt cmd list > [10788.334553] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.334780] [586]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892d9c8) > [10788.335366] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.335483] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d52-621 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.335955] [586]: scst_rx_mgmt_fn:7364:sess=ffff880416bf1d00, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.336287] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892d810) finished, status 0 > [10788.336554] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d52-627 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.337339] [586]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892d9c8 to active mgmt cmd list > [10788.337480] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892e5d0 from active cmd list > [10788.337707] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d51-613 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.337926] [603]: scst_unregister_session:7901:Unregistering session ffff880416bf4880 (wait 0) > [10788.338031] [554]: scst_unregister_session:7901:Unregistering session ffff880400faba00 (wait 0) > [10788.338437] [603]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892d810) > [10788.339050] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d51-619 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.339367] [570]: scst_unregister_session:7901:Unregistering session ffff880416bf6cc0 (wait 0) > [10788.339774] [555]: scst: scst_cmd_thread:5499:Processing thread disk134_0 finished > [10788.339973] [570]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892e418) > [10788.340435] [603]: scst_rx_mgmt_fn:7364:sess=ffff880416bf4880, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.340624] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf0e80 (mcmd ffff88006892e5d0) > [10788.340759] [554]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.340994] [603]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892d810 to active mgmt cmd list > [10788.341354] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.341536] [554]: scst_rx_mgmt_fn:7364:sess=ffff880400faba00, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.342115] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.342324] [593]: scst_unregister_session:7901:Unregistering session ffff880416bf4fc0 (wait 0) > [10788.342613] [554]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.342920] [593]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892dd38) > [10788.343926] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892e5d0) finished, status 0 > [10788.344411] [593]: scst_rx_mgmt_fn:7364:sess=ffff880416bf4fc0, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.344785] [570]: scst_rx_mgmt_fn:7364:sess=ffff880416bf6cc0, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.345000] [593]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892dd38 to active mgmt cmd list > [10788.345266] [570]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892e418 to active mgmt cmd list > [10788.345403] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892d9c8 from active cmd list > [10788.345667] [566]: scst_unregister_session:7901:Unregistering session ffff880416bf0740 (wait 0) > [10788.345840] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf1d00 (mcmd ffff88006892d9c8) > [10788.346315] [566]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892e5d0) > [10788.346665] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.346879] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.347005] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892d9c8) finished, status 0 > [10788.347253] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892d810 from active cmd list > [10788.347462] [566]: scst_rx_mgmt_fn:7364:sess=ffff880416bf0740, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.347810] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf4880 (mcmd ffff88006892d810) > [10788.347858] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d51-624 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.348180] [566]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892e5d0 to active mgmt cmd list > [10788.348448] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.348697] [579]: scst_unregister_session:7901:Unregistering session ffff880416bf15c0 (wait 0) > [10788.349339] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.349487] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892d810) finished, status 0 > [10788.349699] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d51-630 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.349946] [579]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892d9c8) > [10788.350246] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.350560] [579]: scst_rx_mgmt_fn:7364:sess=ffff880416bf15c0, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.350823] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880400faba00 (mcmd ffff88006892db80) > [10788.350953] scsi host58: ib_srp: DREQ received - connection closed > [10788.351102] [579]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892d9c8 to active mgmt cmd list > [10788.351139] scsi host58: ib_srp: DREQ received - connection closed > [10788.351454] scsi host58: ib_srp: DREQ received - connection closed > [10788.351938] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.352233] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.352645] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.352944] [588]: scst_unregister_session:7901:Unregistering session ffff880416bf5700 (wait 0) > [10788.353527] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892e418 from active cmd list > [10788.353540] scsi host58: ib_srp: DREQ received - connection closed > [10788.353933] scsi host61: ib_srp: DREQ received - connection closed > [10788.354074] scsi host61: ib_srp: DREQ received - connection closed > [10788.354416] scsi host61: ib_srp: DREQ received - connection closed > [10788.354762] [588]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.354868] scsi host61: ib_srp: DREQ received - connection closed > [10788.355077] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-610. > [10788.355101] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-616. > [10788.355141] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-621. > [10788.357117] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-627. > [10788.357120] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf6cc0 (mcmd ffff88006892e418) > [10788.357183] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.357185] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.357190] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892e418) finished, status 0 > [10788.357202] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892dd38 from active cmd list > [10788.357276] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf4fc0 (mcmd ffff88006892dd38) > [10788.357337] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.357339] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.357343] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892dd38) finished, status 0 > [10788.357353] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892e5d0 from active cmd list > [10788.357390] [588]: scst_rx_mgmt_fn:7364:sess=ffff880416bf5700, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.357393] [588]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.357445] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf0740 (mcmd ffff88006892e5d0) > [10788.357502] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.357504] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.357508] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892e5d0) finished, status 0 > [10788.357517] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892d9c8 from active cmd list > [10788.357748] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf15c0 (mcmd ffff88006892d9c8) > [10788.357806] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.357808] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.357811] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892d9c8) finished, status 0 > [10788.357821] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.357901] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf5700 (mcmd ffff88006892db80) > [10788.357958] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.357960] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.357964] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.357980] scsi host60: ib_srp: DREQ received - connection closed > [10788.358287] scsi host60: ib_srp: DREQ received - connection closed > [10788.359468] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b2-623. > [10788.359491] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b2-629. > [10788.360123] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b1-612 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.360815] scsi host59: ib_srp: DREQ received - connection closed > [10788.361642] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b1-618 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.361991] [564]: scst_unregister_session:7901:Unregistering session ffff880416bf7400 (wait 0) > [10788.362038] [564]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.362042] [564]: scst_rx_mgmt_fn:7364:sess=ffff880416bf7400, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.362045] [564]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.362094] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-612. > [10788.362118] scsi host59: ib_srp: DREQ received - connection closed > [10788.362616] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.362838] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf7400 (mcmd ffff88006892db80) > [10788.362899] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.362901] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.362906] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.364202] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-618. > [10788.370985] [576]: scst_unregister_session:7901:Unregistering session ffff880416bf6580 (wait 0) > [10788.371047] [576]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.371050] [576]: scst_rx_mgmt_fn:7364:sess=ffff880416bf6580, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.371053] [576]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.371169] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b1-625 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.371592] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.371644] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf6580 (mcmd ffff88006892db80) > [10788.371705] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.371707] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.371711] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.386343] [601]: scst_unregister_session:7901:Unregistering session ffff880416bf2b80 (wait 0) > [10788.386502] [601]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.386505] [601]: scst_rx_mgmt_fn:7364:sess=ffff880416bf2b80, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.386508] [601]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.386522] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.386590] scsi host59: ib_srp: DREQ received - connection closed > [10788.386724] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf2b80 (mcmd ffff88006892db80) > [10788.386785] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.386787] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.386791] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.387386] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-625. > [10788.396186] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-613. > [10788.396511] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-619. > [10788.396758] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-624. > [10788.397245] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-630. > [10788.397967] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b1-631 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.399003] scsi host59: ib_srp: DREQ received - connection closed > [10788.399155] [590]: scst_unregister_session:7901:Unregistering session ffff880416bf2440 (wait 0) > [10788.399349] [590]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.399353] [590]: scst_rx_mgmt_fn:7364:sess=ffff880416bf2440, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.399356] [590]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.399369] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.399437] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf2440 (mcmd ffff88006892db80) > [10788.399501] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.399504] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.399508] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.409181] ib_srpt: Closing channel fe80:0000:0000:0000:0002:c903:00a0:2dc2-657 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.409978] [606]: scst_unregister_session:7901:Unregistering session ffff880416bf32c0 (wait 0) > [10788.410171] [606]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.410174] [606]: scst_rx_mgmt_fn:7364:sess=ffff880416bf32c0, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.410177] [606]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.410190] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.410296] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf32c0 (mcmd ffff88006892db80) > [10788.410359] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.410361] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.410365] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.418821] ib_srpt: Closing channel fe80:0000:0000:0000:0002:c903:00a0:2dc1-658 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.419326] [2408]: scst_unregister_session:7901:Unregistering session ffff8803f8f17400 (wait 0) > [10788.419798] [2408]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.419801] [2408]: scst_rx_mgmt_fn:7364:sess=ffff8803f8f17400, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.419804] [2408]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.419815] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.419981] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff8803f8f17400 (mcmd ffff88006892db80) > [10788.420042] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.420045] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.420049] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.427954] [2410]: scst_unregister_session:7901:Unregistering session ffff8803f8f10740 (wait 0) > [10788.428188] [2410]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.428192] [2410]: scst_rx_mgmt_fn:7364:sess=ffff8803f8f10740, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.428195] [2410]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.428208] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.428513] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff8803f8f10740 (mcmd ffff88006892db80) > [10788.428576] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.428579] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.428583] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.479546] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.479914] [573]: scst: scst_cmd_thread:5499:Processing thread disk138_0 finished > [10788.484823] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.486170] [585]: scst: scst_cmd_thread:5499:Processing thread disk142_0 finished > [10788.490385] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-618. > [10788.490785] scsi host59: ib_srp: connection closed > [10788.490788] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:0002:c903:00a0:2dc2-657. > [10788.490850] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:0002:c903:00a0:2dc1-658. > [10788.490894] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-631. > [10788.493602] scsi host59: ib_srp: connection closed > [10788.493990] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-612. > [10788.520887] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.521524] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-625. > [10788.522148] scsi host59: ib_srp: connection closed > [10788.526290] [602]: scst: scst_cmd_thread:5499:Processing thread disk147_0 finished > [10788.532024] [553]: scst: scst_cmd_thread:5499:Processing thread disk133_0 finished > [10788.547718] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.548481] [569]: scst: scst_cmd_thread:5499:Processing thread disk137_0 finished > [10788.551854] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.600121] scsi host59: ib_srp: connection closed > [10788.620114] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-631. > [10788.670189] [581]: scst_unregister_session:7901:Unregistering session ffff880416bf5e40 (wait 0) > [10788.671892] [581]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.673417] [581]: scst_rx_mgmt_fn:7364:sess=ffff880416bf5e40, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.673860] [581]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.674072] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.674441] [592]: scst: scst_cmd_thread:5499:Processing thread disk145_0 finished > [10788.675123] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf5e40 (mcmd ffff88006892db80) > [10788.676436] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.676676] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.676914] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.698994] [565]: scst: scst_cmd_thread:5499:Processing thread disk136_0 finished > [10788.703046] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.703357] [578]: scst: scst_cmd_thread:5499:Processing thread disk140_0 finished > [10788.708079] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.708352] [587]: scst: scst_cmd_thread:5499:Processing thread disk143_0 finished > [10788.712911] [563]: scst: scst_cmd_thread:5499:Processing thread disk135_0 finished > [10788.716807] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10790.350077] scsi host60: ib_srp: connection closed > [10790.370069] scsi host60: ib_srp: connection closed > [10790.400086] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-621. > [10790.400090] scsi host60: ib_srp: connection closed > [10790.400103] scsi host60: ib_srp: connection closed > [10790.400685] scsi host61: ib_srp: connection closed > [10790.400701] scsi host61: ib_srp: connection closed > [10790.400715] scsi host61: ib_srp: connection closed > [10790.400729] scsi host61: ib_srp: connection closed > [10790.402597] scsi host58: ib_srp: connection closed > [10790.403206] scsi host58: ib_srp: connection closed > [10790.403613] scsi host58: ib_srp: connection closed > [10790.404086] scsi host58: ib_srp: connection closed > [10790.440067] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-630. > [10790.540711] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:0002:c903:00a0:2dc1-658. > [10790.541056] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:0002:c903:00a0:2dc2-657. > [10793.419992] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1: waiting for session unregistration ... > [10793.421559] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d52-621: state disconnected; 0 commands in progress > [10793.422461] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d51-630: state disconnected; 0 commands in progress > [10793.422928] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-618: state disconnected; 0 commands in progress > [10793.424120] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-625: state disconnected; 0 commands in progress > [10793.424912] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-631: state disconnected; 0 commands in progress > [10793.425992] ib_srpt: fe80:0000:0000:0000:0002:c903:00a0:2dc2-657: state disconnected; 0 commands in progress > [10793.427069] ib_srpt: fe80:0000:0000:0000:0002:c903:00a0:2dc1-658: state disconnected; 0 commands in progress > [10798.310258] scsi host60: ib_srp: Sending CM DREQ failed > [10798.310968] scsi host60: ib_srp: Sending CM DREQ failed > [10798.311772] scsi host60: ib_srp: Sending CM DREQ failed > [10798.312243] scsi host60: ib_srp: Sending CM DREQ failed > [10798.369850] scsi host58: ib_srp: Sending CM DREQ failed > [10798.369851] scsi host61: ib_srp: Sending CM DREQ failed > [10798.369854] scsi host61: ib_srp: Sending CM DREQ failed > [10798.369856] scsi host61: ib_srp: Sending CM DREQ failed > [10798.369858] scsi host61: ib_srp: Sending CM DREQ failed > [10798.371422] scsi host58: ib_srp: Sending CM DREQ failed > [10798.372037] scsi host58: ib_srp: Sending CM DREQ failed > [10798.372303] scsi host58: ib_srp: Sending CM DREQ failed > [10798.389849] scsi host59: ib_srp: Sending CM DREQ failed > [10798.390128] scsi host59: ib_srp: Sending CM DREQ failed > [10798.390336] scsi host59: ib_srp: Sending CM DREQ failed > [10798.390551] scsi host59: ib_srp: Sending CM DREQ failed > [10798.429858] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1: waiting for session unregistration ... > [10798.430095] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d52-621: state disconnected; 0 commands in progress > [10798.430326] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d51-630: state disconnected; 0 commands in progress > [10798.430632] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-618: state disconnected; 0 commands in progress > [10798.431060] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-625: state disconnected; 0 commands in progress > [10798.431522] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-631: state disconnected; 0 commands in progress > [10798.432378] ib_srpt: fe80:0000:0000:0000:0002:c903:00a0:2dc2-657: state disconnected; 0 commands in progress > [10798.432607] ib_srpt: fe80:0000:0000:0000:0002:c903:00a0:2dc1-658: state disconnected; 0 commands in progress > [10798.593380] scsi host60: ib_srp: REJ received > [10798.594137] scsi host60: REJ reason 0x8 > [10798.595188] scsi host60: reconnect attempt 1 failed (-104) > [10798.609908] scsi host61: ib_srp: REJ received > [10798.610291] scsi host61: REJ reason 0x8 > [10798.610828] scsi host61: reconnect attempt 1 failed (-104) > [10798.612989] scsi host58: ib_srp: REJ received > [10798.613366] scsi host58: REJ reason 0x8 > [10798.613752] scsi host58: reconnect attempt 1 failed (-104) > [10798.629912] [575]: scst: scst_cmd_thread:5499:Processing thread disk139_0 finished > [10798.633770] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10798.634435] [600]: scst: scst_cmd_thread:5499:Processing thread disk146_0 finished > [10798.661741] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10798.661942] [589]: scst: scst_cmd_thread:5499:Processing thread disk144_0 finished > [10798.665387] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10798.665829] [604]: scst: scst_cmd_thread:5499:Processing thread disk148_0 finished > [10798.677166] [2407]: scst: scst_cmd_thread:5499:Processing thread disk169_0 finished > [10798.680754] [2409]: scst: scst_cmd_thread:5499:Processing thread disk170_0 finished > [10798.683539] scsi host59: ib_srp: REJ received > [10798.683665] scsi host59: REJ reason 0x8 > [10798.683743] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10798.684732] scsi host59: reconnect attempt 1 failed (-104) > [10803.350085] fast_io_fail_tmo expired for SRP port-60:1 / host60. > [10803.354685] sd 60:0:0:0: rejecting I/O to offline device > [10803.358943] sd 60:0:0:0: [sdbx] killing request > [10803.359803] sd 60:0:0:0: rejecting I/O to offline device > [10803.360324] sd 60:0:0:0: [sdbx] killing request > [10803.361332] sd 60:0:0:0: rejecting I/O to offline device > [10803.362291] sd 60:0:0:0: rejecting I/O to offline device > [10803.362295] device-mapper: multipath: Failing path 68:176. > [10803.363197] sd 60:0:0:0: rejecting I/O to offline device > [10803.363526] sd 60:0:0:0: rejecting I/O to offline device > [10803.363812] sd 60:0:0:0: rejecting I/O to offline device > [10803.363983] sd 60:0:0:0: rejecting I/O to offline device > [10803.364277] sd 60:0:0:0: rejecting I/O to offline device > [10803.364472] sd 60:0:0:0: [sdbx] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK > [10803.364762] sd 60:0:0:0: [sdbx] CDB: Read(10) 28 00 00 00 1a 58 00 00 08 00 > [10803.365071] blk_update_request: I/O error, dev sdbx, sector 6744 > [10803.365345] sd 60:0:0:0: [sdbx] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK > [10803.365650] sd 60:0:0:0: [sdbx] CDB: Read(10) 28 00 00 00 5c 38 00 00 08 00 > [10803.365915] blk_update_request: I/O error, dev sdbx, sector 23608 > [10803.399704] fast_io_fail_tmo expired for SRP port-59:1 / host59. > [10803.399708] fast_io_fail_tmo expired for SRP port-58:1 / host58. > [10803.400381] sd 59:0:0:0: rejecting I/O to offline device > [10803.400514] fast_io_fail_tmo expired for SRP port-61:1 / host61. > [10803.401006] sd 59:0:0:0: rejecting I/O to offline device > [10803.401008] device-mapper: multipath: Failing path 68:192. > [10803.401448] sd 59:0:0:0: rejecting I/O to offline device > [10803.401660] sd 59:0:0:0: rejecting I/O to offline device > [10803.401808] sd 59:0:0:0: rejecting I/O to offline device > [10803.401999] sd 59:0:0:0: rejecting I/O to offline device > [10803.402200] sd 59:0:0:0: rejecting I/O to offline device > [10803.402387] sd 59:0:0:0: rejecting I/O to offline device > [10803.402586] sd 59:0:0:0: rejecting I/O to offline device > [10803.429706] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1: waiting for session unregistration ... > [10803.430138] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d52-621: state disconnected; 0 commands in progress > [10803.904798] device-mapper: multipath: Failing path 68:160. > [10803.909315] device-mapper: multipath: Failing path 68:208. > [10808.429596] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1: waiting for session unregistration ... > [10808.430002] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d52-621: state disconnected; 0 commands in progress > [10808.609582] scsi host60: ib_srp: Sending CM DREQ failed > [10808.609770] scsi host60: ib_srp: Sending CM DREQ failed > [10808.609938] scsi host60: ib_srp: Sending CM DREQ failed > [10808.610378] scsi host60: ib_srp: Sending CM DREQ failed > [10808.639569] scsi host58: ib_srp: Sending CM DREQ failed > [10808.639765] scsi host58: ib_srp: Sending CM DREQ failed > [10808.640049] scsi host58: ib_srp: Sending CM DREQ failed > [10808.640217] scsi host58: ib_srp: Sending CM DREQ failed > [10808.660821] scsi host61: ib_srp: Sending CM DREQ failed > [10808.661050] scsi host61: ib_srp: Sending CM DREQ failed > [10808.661794] scsi host61: ib_srp: Sending CM DREQ failed > [10808.661961] scsi host61: ib_srp: Sending CM DREQ failed > [10808.690042] [580]: scst: scst_cmd_thread:5499:Processing thread disk141_0 finished > [10808.693028] [8149]: scst_suspend_activity:965:suspend_count 0 > [10808.693136] [8149]: scst: scst_suspend_activity:990:Waiting for 10 active commands to complete... > [10808.709633] scsi host59: ib_srp: Sending CM DREQ failed > [10808.709729] scsi host59: ib_srp: Sending CM DREQ failed > [10808.709807] scsi host59: ib_srp: Sending CM DREQ failed > [10808.709892] scsi host59: ib_srp: Sending CM DREQ failed > [10808.730162] scsi host60: ib_srp: REJ received > [10808.730248] scsi host60: REJ reason 0x8 > [10808.730374] scsi host60: reconnect attempt 2 failed (-104) > [10808.755642] scsi host61: ib_srp: REJ received > [10808.755727] scsi host61: REJ reason 0x8 > [10808.755874] scsi host61: reconnect attempt 2 failed (-104) > [10808.757781] scsi host58: ib_srp: REJ received > [10808.757861] scsi host58: REJ reason 0x8 > [10808.757991] scsi host58: reconnect attempt 2 failed (-104) > [10808.789571] scsi host59: ib_srp: REJ received > [10808.789652] scsi host59: REJ reason 0x8 > [10808.789734] scsi host59: reconnect attempt 2 failed (-104) > [10809.402402] [550]: scst_translate_lun:4957:FLAG SUSPENDED set, skipping > [10809.559516] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [disk11_0:32753] > [10809.559635] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10809.570166] irq event stamp: 29993531 > [10809.570263] hardirqs last enabled at (29993530): [<ffffffff815781b1>] _raw_spin_unlock_irqrestore+0x31/0x50 > [10809.570452] hardirqs last disabled at (29993531): [<ffffffff81579664>] apic_timer_interrupt+0x84/0x90 > [10809.570640] softirqs last enabled at (10133236): [<ffffffffa05c9a57>] scst_check_blocked_dev+0x77/0x1c0 [scst] > [10809.570851] softirqs last disabled at (10133267): [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10809.571039] CPU: 11 PID: 32753 Comm: disk11_0 Tainted: G O L 4.4.6-debug+ #2 > [10809.571140] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10809.571239] task: ffff880439375f00 ti: ffff880460d30000 task.ti: ffff880460d30000 > [10809.571338] RIP: 0010:[<ffffffff815781b3>] [<ffffffff815781b3>] _raw_spin_unlock_irqrestore+0x33/0x50 > [10809.571530] RSP: 0018:ffff88046f363d68 EFLAGS: 00000292 > [10809.571626] RAX: ffff880439375f00 RBX: 0000000000000292 RCX: 0000000000000000 > [10809.571722] RDX: 0000000000000003 RSI: ffff880439376698 RDI: ffff880439375f00 > [10809.571818] RBP: ffff88046f363d78 R08: ffff88046f363ce8 R09: 0000000000000000 > [10809.571914] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10809.572010] R13: ffffea0008392800 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10809.572106] FS: 0000000000000000(0000) GS:ffff88046f360000(0000) knlGS:0000000000000000 > [10809.572204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10809.572298] CR2: 00005633130dd1a8 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10809.572392] Stack: > [10809.572488] ffff88017fc01c00 0000000000000001 ffff88046f363e30 ffffffff811640d6 > [10809.572943] ffff88046f363db8 ffffffff8157940f 0000000100300001 ffff88020e4a3fc0 > [10809.573396] ffff88020e4a3fc0 0000000100000282 0000000000000292 ffff880439376698 > [10809.573845] Call Trace: > [10809.573938] <IRQ> > [10809.574034] [<ffffffff811640d6>] __slab_free+0xc6/0x270 > [10809.574236] [<ffffffff8157940f>] ? retint_kernel+0x10/0x10 > [10809.574338] [<ffffffff8116431d>] ? kmem_cache_free+0x9d/0x160 > [10809.574441] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10809.574543] [<ffffffff8117b8d9>] ? free_object_rcu+0x89/0x90 > [10809.574644] [<ffffffff811643d9>] kmem_cache_free+0x159/0x160 > [10809.574746] [<ffffffff8117b850>] ? memory_failure_work_func+0xb0/0xb0 > [10809.574847] [<ffffffff8117b8d9>] free_object_rcu+0x89/0x90 > [10809.574948] [<ffffffff810b7480>] rcu_process_callbacks+0x2b0/0x5f0 > [10809.575049] [<ffffffff810b743f>] ? rcu_process_callbacks+0x26f/0x5f0 > [10809.575149] [<ffffffff8105a49f>] __do_softirq+0x10f/0x230 > [10809.575248] [<ffffffff8105a748>] irq_exit+0xa8/0xb0 > [10809.575349] [<ffffffff810065a5>] do_IRQ+0x65/0x110 > [10809.575450] [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20 > [10809.575549] [<ffffffff815793c9>] common_interrupt+0x89/0x89 > [10809.575650] <EOI> > [10809.575748] [<ffffffff81113282>] ? mempool_free_slab+0x12/0x20 > [10809.575949] [<ffffffff8117bad7>] ? put_object+0x7/0x40 > [10809.576051] [<ffffffff8117bb48>] ? __delete_object+0x38/0x70 > [10809.576153] [<ffffffff8117bb98>] delete_object_full+0x18/0x20 > [10809.576254] [<ffffffff8156dab1>] kmemleak_free+0x31/0x80 > [10809.576355] [<ffffffff811642fa>] kmem_cache_free+0x7a/0x160 > [10809.576456] [<ffffffff81113282>] mempool_free_slab+0x12/0x20 > [10809.576558] [<ffffffff81113406>] mempool_free+0x26/0x80 > [10809.576659] [<ffffffff8127e8c1>] bvec_free+0x31/0x40 > [10809.576758] [<ffffffff8127ee54>] bio_free+0x34/0x60 > [10809.576858] [<ffffffff8127ee9e>] bio_put+0x1e/0x30 > [10809.576962] [<ffffffffa06f042d>] blockio_endio+0x2d/0xe0 [scst_vdisk] > [10809.577067] [<ffffffff812800da>] bio_endio+0x3a/0x60 > [10809.577168] [<ffffffffa0560b08>] brd_make_request+0xe8/0x1ae [brd] > [10809.577273] [<ffffffff812826fc>] ? generic_make_request_checks+0x10c/0x630 > [10809.577378] [<ffffffff81284342>] generic_make_request+0xd2/0x290 > [10809.577482] [<ffffffff81284562>] submit_bio+0x62/0x140 > [10809.577587] [<ffffffffa06f0a33>] blockio_exec_rw+0x553/0x670 [scst_vdisk] > [10809.577697] [<ffffffffa06f0cb0>] ? blockio_exec_write_verify+0x50/0x50 [scst_vdisk] > [10809.577807] [<ffffffffa06f0cbd>] blockio_exec_read+0xd/0x20 [scst_vdisk] > [10809.577915] [<ffffffffa06e7773>] vdev_do_job+0xb3/0x310 [scst_vdisk] > [10809.578024] [<ffffffffa06ebcc4>] blockio_exec+0x74/0x180 [scst_vdisk] > [10809.578130] [<ffffffff810a2931>] ? trace_hardirqs_on_caller+0x121/0x1b0 > [10809.578253] [<ffffffffa0609b12>] ? sgv_get_obj+0x72/0x1b0 [scst] > [10809.578374] [<ffffffffa05c7154>] scst_do_real_exec+0x94/0x3d0 [scst] > [10809.578500] [<ffffffffa05c01d3>] ? scst_is_not_reservation_holder+0x13/0x60 [scst] > [10809.578629] [<ffffffffa05c13ff>] ? __scst_check_local_events+0x19f/0x370 [scst] > [10809.578750] [<ffffffffa05ca8dc>] scst_exec_check_blocking+0xcc/0x340 [scst] > [10809.578876] [<ffffffffa05cac7d>] scst_exec_check_sn+0x12d/0x2c0 [scst] > [10809.578982] [<ffffffff8107a896>] ? finish_task_switch+0xa6/0x220 > [10809.579084] [<ffffffff8107a857>] ? finish_task_switch+0x67/0x220 > [10809.579202] [<ffffffffa05cf892>] scst_process_active_cmd+0x3e2/0x760 [scst] > [10809.579319] [<ffffffff810a29cd>] ? trace_hardirqs_on+0xd/0x10 > [10809.579435] [<ffffffffa05d1001>] scst_do_job_active+0xe1/0x160 [scst] > [10809.579560] [<ffffffffa05d11b2>] scst_cmd_thread+0x132/0x290 [scst] > [10809.579680] [<ffffffff81098660>] ? wake_atomic_t_function+0x60/0x60 > [10809.579797] [<ffffffffa05d1080>] ? scst_do_job_active+0x160/0x160 [scst] > [10809.579906] [<ffffffff8107591a>] kthread+0xea/0x100 > [10809.580009] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10809.580114] [<ffffffff81578c7f>] ret_from_fork+0x3f/0x70 > [10809.580219] [<ffffffff81075830>] ? kthread_create_on_node+0x1f0/0x1f0 > [10809.580321] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb
Powered by blists - more mailing lists