| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Tue, 4 Jun 2013 23:54:45 +0200
From: Frank Mehnert <frank.mehnert@...cle.com>
To: Michal Hocko <mhocko@...e.cz>
Cc: Robin Holt <holt@....com>, linux-mm@...ck.org,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
Hugh Dickins <hughd@...gle.com>
Subject: Re: Handling NUMA page migration
On Tuesday 04 June 2013 20:17:02 Frank Mehnert wrote:
> On Tuesday 04 June 2013 16:02:30 Michal Hocko wrote:
> > On Tue 04-06-13 14:14:45, Frank Mehnert wrote:
> > > On Tuesday 04 June 2013 13:58:07 Robin Holt wrote:
> > > > This is probably more appropriate to be directed at the linux-mm
> > > > mailing list.
> > > >
> > > > On Tue, Jun 04, 2013 at 09:22:10AM +0200, Frank Mehnert wrote:
> > > > > Hi,
> > > > >
> > > > > our memory management on Linux hosts conflicts with NUMA page
> > > > > migration. I assume this problem existed for a longer time but
> > > > > Linux 3.8 introduced automatic NUMA page balancing which makes the
> > > > > problem visible on multi-node hosts leading to kernel oopses.
> > > > >
> > > > > NUMA page migration means that the physical address of a page
> > > > > changes. This is fatal if the application assumes that this never
> > > > > happens for that page as it was supposed to be pinned.
> > > > >
> > > > > We have two kind of pinned memory:
> > > > >
> > > > > A) 1. allocate memory in userland with mmap()
> > > > >
> > > > > 2. madvise(MADV_DONTFORK)
> > > > > 3. pin with get_user_pages().
> > > > > 4. flush dcache_page()
> > > > > 5. vm_flags |= (VM_DONTCOPY | VM_LOCKED)
> > > > >
> > > > > (resulting flags are VM_MIXEDMAP | VM_DONTDUMP |
> > > > > VM_DONTEXPAND
> > > > >
> > > > > VM_DONTCOPY | VM_LOCKED | 0xff)
> > > >
> > > > I don't think this type of allocation should be affected. The
> > > > get_user_pages() call should elevate the pages reference count which
> > > > should prevent migration from completing. I would, however, wait for
> > > > a more definitive answer.
> > >
> > > Thanks Robin! Actually case B) is more important for us so I'm waiting
> > > for more feedback :)
> >
> > The manual node migration code seems to be OK in case B as well because
> > Reserved are skipped (check check_pte_range called from down the
> > do_migrate_pages path).
> >
> > Maybe auto-numa code is missing this check assuming that it cannot
> > encounter reserved pages.
> >
> > migrate_misplaced_page relies on numamigrate_isolate_page which relies
> > on isolate_lru_page and that one expects a LRU page. Is your Reserved
> > page on the LRU list? That would be a bit unexpected.
>
> I will check this.
I tested this now. When the Oops happens, PageLRU() of the corresponding
page struct is NOT set! I've patched the kernel to find that out. This is
case B from my original mail (alloc_pages(), SetPageReserved(), vm_mmap(),
vm_insert_page(), vm_flags |= (VM_DONTEXPAND | VM_DONTDUMP)) and PageLRU()
was clear after vm_insert_page().
Example of such an oops (the present bits of PMD and PTE are clear):
BUG: unable to handle kernel paging request at 00007ff493c7eff8
IP: [<ffffffffa039e17f>] 0xffffffffa039e17e
PGD 201b068067 PUD 381c082067 PMD 20063d2166 PTE 8000002005da9166
Oops: 0000 [#1] SMP
Modules linked in: pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF)
vboxdrv(OF) md4 nls_utf8 cifs fscache vesafb kvm_amd kvm psmouse serio_raw
microcode ib_mthca ib_mad ib_core amd64_edac_mod edac_core k10temp
edac_mce_amd joydev shpchp mac_hid lp parport i2c_nforce2 hid_generic usbhid
hid mptsas mptscsih mptbase scsi_transport_sas e1000 pata_acpi pata_amd
CPU 24
Pid: 2058, comm: EMT Tainted: GF O 3.8.0-23-generic #34 Sun
Microsystems Sun Fire X4600 M2/Sun Fire X4600 M2
RIP: 0010:[<ffffffffa039e17f>] [<ffffffffa039e17f>] 0xffffffffa039e17e
RSP: 0018:ffff88381bac1968 EFLAGS: 00010202
RAX: 00007ff493c7eff8 RBX: ffff88381bac1998 RCX: 0000000000000000
RDX: 0000000000000ff8 RSI: 0000000000000000 RDI: ffff88381bac1a18
RBP: ffff88381bac1988 R08: ffffc90029981000 R09: ffffc9002999c000
R10: ffff88381bac1998 R11: ffffffffa037aee0 R12: ffffc9002999c000
R13: ffffffffa002f98d R14: ffffffffa002f98d R15: ffffc9002999c000
FS: 00007ff4f59b7700(0000) GS:ffff883827c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007ff493c7eff8 CR3: 000000201b06f000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process EMT (pid: 2058, threadinfo ffff88381bac0000, task ffff88381b840000)
Stack:
0000000000000000 ffff88381bac1a60 ffff88381bac1ab8 ffffffffa002f98d
ffff88381bac1a28 ffffffffa039e5bd ffffffffa002f98d 0000000000000000
0000000000000000 0000000000000000 00007ff493c7e000 00007ff493c7eff8
Any more ideas? I'm happy to perform more tests.
Thanks,
Frank
> In the meantime I verified that my testcase does not fail if I pass
> 'numa_balancing=false' to the kernel, so it's definitely a NUMA balancing
> problem.
>
> I also did 'get_page()' on all pages of method B but the testcase so this
> didn't help.
>
> Frank
--
Dr.-Ing. Frank Mehnert | Software Development Director, VirtualBox
ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt, Germany
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603
Geschäftsführer: Jürgen Kunz
Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
Download attachment "signature.asc " of type "application/pgp-signature" (199 bytes)
Powered by blists - more mailing lists