Message-ID: <cda84c4d-0f4c-1dd8-44f6-68f211e5de6d@redhat.com>
Date:   Thu, 14 May 2020 14:19:51 +0200
From:   David Hildenbrand <david@...hat.com>
To:     teawater <teawaterz@...ux.alibaba.com>
Cc:     LKML <linux-kernel@...r.kernel.org>, Linux MM <linux-mm@...ck.org>,
        virtio-dev@...ts.oasis-open.org,
        Linux Virtualization <virtualization@...ts.linux-foundation.org>,
        kvm@...r.kernel.org, Michal Hocko <mhocko@...nel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        "Michael S . Tsirkin" <mst@...hat.com>,
        Sebastien Boeuf <sebastien.boeuf@...el.com>,
        Samuel Ortiz <samuel.ortiz@...el.com>,
        Robert Bradford <robert.bradford@...el.com>,
        Luiz Capitulino <lcapitulino@...hat.com>,
        Pankaj Gupta <pankaj.gupta.linux@...il.com>,
        Igor Mammedov <imammedo@...hat.com>,
        "Dr . David Alan Gilbert" <dgilbert@...hat.com>,
        Alexander Duyck <alexander.h.duyck@...ux.intel.com>,
        Alexander Potapenko <glider@...gle.com>,
        Anshuman Khandual <anshuman.khandual@....com>,
        Anthony Yznaga <anthony.yznaga@...cle.com>,
        Dan Williams <dan.j.williams@...el.com>,
        Dave Young <dyoung@...hat.com>,
        Jason Wang <jasowang@...hat.com>,
        Johannes Weiner <hannes@...xchg.org>,
        Juergen Gross <jgross@...e.com>,
        Konrad Rzeszutek Wilk <konrad.wilk@...cle.com>,
        Len Brown <lenb@...nel.org>,
        Mel Gorman <mgorman@...hsingularity.net>,
        Michal Hocko <mhocko@...e.com>,
        Mike Rapoport <rppt@...ux.ibm.com>,
        Oscar Salvador <osalvador@...e.com>,
        Oscar Salvador <osalvador@...e.de>,
        Pavel Tatashin <pasha.tatashin@...een.com>,
        Pavel Tatashin <pavel.tatashin@...rosoft.com>,
        Pingfan Liu <kernelfans@...il.com>, Qian Cai <cai@....pw>,
        "Rafael J. Wysocki" <rafael@...nel.org>,
        "Rafael J. Wysocki" <rjw@...ysocki.net>,
        Stefan Hajnoczi <stefanha@...hat.com>,
        Vlastimil Babka <vbabka@...e.cz>,
        Wei Yang <richard.weiyang@...il.com>
Subject: Re: [virtio-dev] [PATCH v3 00/15] virtio-mem: paravirtualized memory

On 14.05.20 13:47, David Hildenbrand wrote:
> On 14.05.20 13:10, David Hildenbrand wrote:
>> On 14.05.20 12:12, David Hildenbrand wrote:
>>> On 14.05.20 12:02, teawater wrote:
>>>>
>>>>
>>>>> 2020年5月14日 16:48，David Hildenbrand <david@...hat.com> 写道：
>>>>>
>>>>> On 14.05.20 08:44, teawater wrote:
>>>>>> Hi David,
>>>>>>
>>>>>> I got a kernel warning with v2 and v3.
>>>>>
>>>>> Hi Hui,
>>>>>
>>>>> thanks for playing with the latest versions. Surprisingly, I can
>>>>> reproduce even by hotplugging a DIMM instead as well - that's good, so
>>>>> it's not related to virtio-mem, lol. Seems to be some QEMU setup issue
>>>>> with older machine types.
>>>>>
>>>>> Can you switch to a newer qemu machine version, especially
>>>>> pc-i440fx-5.0? Both, hotplugging DIMMs and virtio-mem works for me with
>>>>> that QEMU machine just fine.
>>>>
>>>> I still could reproduce this issue with pc-i440fx-5.0 or pc.  Did I miss anything?
>>>>
>>>
>>> Below I don't even see virtio_mem. I had to repair the image (filesystem
>>> fsck) because it was broken, can you try that as well?
>>>
>>> Also, it would be great if you could test with v4.
>>>
>>
>> Correction, something seems to be broken either in QEMU or the kernel. Once I
>> define a DIMM so it's added and online during boot, I get these issues:
>>
>> (I have virtio-mem v4 installed in the guest)
>>
>> #! /bin/bash
>> sudo x86_64-softmmu/qemu-system-x86_64 \
>>     -machine pc-i440fx-5.0,accel=kvm,usb=off \
>>     -cpu host \
>>     -no-reboot \
>>     -nographic \
>>     -device ide-hd,drive=hd \
>>     -drive if=none,id=hd,file=/home/dhildenb/git/Fedora-Cloud-Base-31-1.9.x86_64.qcow2,format=qcow2 \
>>     -m 1g,slots=10,maxmem=2G \
>>     -smp 1 \
>>     -object memory-backend-ram,id=mem0,size=256m \
>>     -device pc-dimm,id=dimm0,memdev=mem0 \
>>     -s \
>>     -monitor unix:/var/tmp/monitor,server,nowait
>>
>>
>> Without the DIMM it seems to work just fine.
>>
> 
> And another correction. 
> 
> Using QEMU v5.0.0, Linux 5.7-rc5, untouched
> Fedora-Cloud-Base-32-1.6.x86_64.qcow2, I get even without any memory hotplug:
> 
> #! /bin/bash
> sudo x86_64-softmmu/qemu-system-x86_64 \
>     -machine pc-i440fx-5.0,accel=kvm,usb=off \
>     -cpu host \
>     -no-reboot \
>     -nographic \
>     -device ide-hd,drive=hd \
>     -drive if=none,id=hd,file=/home/dhildenb/git/Fedora-Cloud-Base-32-1.6.x86_64.qcow2,format=qcow2 \
>     -m 5g,slots=10,maxmem=6G \
>     -smp 1 \
>     -s \
>     -kernel /home/dhildenb/git/linux/arch/x86/boot/bzImage \
>     -append "console=ttyS0 rd.shell nokaslr swiotlb=noforce" \
>     -monitor unix:/var/tmp/monitor,server,nowait
> 
> 
> Observe how big the initial RAM even is!
> 
> 
> So this is no DIMM/hotplug/virtio_mem issue. With memory hotplug, it seems to get
> more likely to trigger if "swiotlb=noforce" is not specified.
> 
> "swiotlb=noforce" seems to trigger some pre-existing issue here. Without
> "swiotlb=noforce", I was only able to observe this via pc-i440fx-2.1,
> 

(talking to myself :) )

I think I finally understood why using "swiotlb=noforce" with hotplugged
memory is wrong - or with memory > 3GB. Via "swiotlb=noforce" you tell
the system to "Never use bounce buffers (for debugging)". This works as
long as all memory is DMA memory (e.g., < 3GB) AFAIK.

"If specified, trying to map memory that cannot be used with DMA will
fail, and a rate-limited warning will be printed."

Hotplugged memory (under QEMU) is never added below 4GB, because of the
PCI hole. So both, memory from DIMMs and from virtio-mem will end up at
or above 4GB. To make a device use that memory, you need bounce buffers.

Hotplugged memory is never DMA memory.

-- 
Thanks,

David / dhildenb

Powered by blists - more mailing lists