linux-kernel - Re: LTP test related to virtio releasing and reassigning resource leads to guest hung

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <CAJSP0QXjzxjXcsfLMsKQmOvuzRbqxpT70iGsSdDu+qANCd729w@mail.gmail.com>
Date:   Tue, 15 Aug 2023 11:18:42 -0400
From:   Stefan Hajnoczi <stefanha@...il.com>
To:     "longguang.yue" <kvmluck@....com>
Cc:     Michael Tokarev <mjt@....msk.ru>,
        "mst@...hat.com" <mst@...hat.com>,
        qemu-devel <qemu-devel@...gnu.org>,
        linux-kernel <linux-kernel@...r.kernel.org>
Subject: Re: LTP test related to virtio releasing and reassigning resource
 leads to guest hung

On Thu, 10 Aug 2023 at 22:27, longguang.yue <kvmluck@....com> wrote:
>
>
> 一）
> Can you post the guest kernel messages (dmesg)? If the guest is hanging
> then it may be easiest to configure a serial console so the kernel
> messages are sent to the host where you can see them.
>
> Does the hang occur during the LTP code you linked or afterwards when
> the PCI device is bound to a virtio driver?
>
>
> >   I used conosle, the hang occurred afterwards.   dmesg shows that tpci test is finished without error.
> LTP test case: https://github.com/linux-test-project/ltp/blob/522d7fba4afc84e07b252aa4cd91b241e81d6613/testcases/kernel/device-drivers/pci/tpci_kernel/ltp_tpci.c#L428
> kernel 5.10, qemu 6.2
>
> different guest-configuration tests show different results.  guest did not crash if hung-task-panic=0, in my case  i enable hung-task-panic in order to trace.
>
> test case 1:
> xml machine pc,virtio disk, virtio net ——  guest's io hung, network broke down, though console is avilable but io operation hung.
>
> #ps -aux| grep D
> root           7  0.0  0.0      0     0 ?        D    14:37   0:00 [kworker/u16:0+flush-253:0]
> root         483  0.0  0.0      0     0 ?      D    14:37   0:00 [jbd2/vda3-8]
>
> test case 2:
> xml machine q35,virtio/q35,scsi ——disk did not hung but network broke down. ping errors though everything looks ok and no crash and no kernel error
>
>
>
> 二）
> I didn't see your original email so I missed the panic. I'd still like
> to see the earlier kernel messages before the panic in order to
> understand how the PCI device is bound.
>
> Is the vda device with hung I/O the same device that was accessed by
> the LTP test earlier? I guess the LTP test runs against the device and
> then the virtio driver binds to the device again afterwards?
>
> > the test is
> ```
> // iterate all devices
> ……
> for (i = 0; i < 7; ++i) {  // iterate current device's resources
>   if (r->flags & IORESOURCE_MEM &&
>   r->flags & IORESOURCE_PREFETCH) {
>   pci_release_resource(dev, i);
>   ret = pci_assign_resource(dev, i);
>   prk_info("assign resource to '%d', ret '%d'", i, ret);
>   rc |= (ret < 0 && ret != -EBUSY) ? TFAIL : TPASS;
>   }
> }
> ```
> test does not do virtio device unbind and  bind.
> I only notice mem resource changed. see 'test-case 12'
>
> ———————————
> [   88.905705] ltp_tpci: test-case 12
> [   88.905706] ltp_tpci: assign resources
> [   88.905706] ltp_tpci: assign resource #0
> [   88.905707] ltp_tpci: name = 0000:00:07.0, flags = 262401, start 0xc080, end 0xc0ff
> [   88.905707] ltp_tpci: assign resource #1
> [   88.905708] ltp_tpci: name = 0000:00:07.0, flags = 262656, start 0xfebd4000, end 0xfebd4fff
> [   88.905709] ltp_tpci: assign resource #2
> [   88.905709] ltp_tpci: name = 0000:00:07.0, flags = 0, start 0x0, end 0x0
> [   88.905710] ltp_tpci: assign resource #3
> [   88.905710] ltp_tpci: name = 0000:00:07.0, flags = 0, start 0x0, end 0x0
> [   88.905711] ltp_tpci: assign resource #4
> [   88.905711] ltp_tpci: name = 0000:00:07.0, flags = 1319436, start 0xfe00c000, end 0xfe00ffff
> [   88.905713] virtio-pci 0000:00:07.0: BAR 4: releasing [mem 0xfe00c000-0xfe00ffff 64bit pref]
> [   88.905715] virtio-pci 0000:00:07.0: BAR 4: assigned [mem 0x24000c000-0x24000ffff 64bit pref]
> [   88.906693] ltp_tpci: assign resource to '4', ret '0'
> [   88.906694] ltp_tpci: assign resource #5
> [   88.906694] ltp_tpci: name = (null), flags = 0, start 0x0, end 0x0
> [   88.906695] ltp_tpci: assign resource #6
> [   88.906695] ltp_tpci: name = 0000:00:07.0, flags = 0, start 0x0, end 0x0
>
> [   88.906800] ltp_tpci: test-case 13

I don't know. Maybe the test case is leaving the device is a state
that conflicts with the virtio drivers that are bound after testing
finishes.

One approach is to trace the PCI BAR accesses after the test runs and
compare against a trace when the tpci driver hasn't been loaded. That
way you might be able to find out what is different.

Stefan