linux-kernel - Re: Allocated large blocks of memory on 64 bit linux.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Wed, 20 Sep 2006 15:20:16 +0300
From:	Avi Kivity <avi@...o.co.il>
To:	Chris Jefferson <chris@...blescope.net>
CC:	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: Allocated large blocks of memory on 64 bit linux.

Chris Jefferson wrote:
>
> I apologise for this slightly off-topic message, but I believe it can
> best be answered here, and hope the question may be interesting.
>
> Many libraries have some kind of dynamically sized container (for
> example C++'s std::vector). When the container is full a new block of
> memory, typically double the original size, is allocated and the old
> data copied across.
>
> On a 64 bit architecture, where the memory space is massive, it seems
> at first glance a sensible thing to do might be to first make a buffer
> of size 4k, and then when this fills up, just straight to something
> huge, like 1MB or even 1GB, as the memory space is effectively
> infinate compared to the physical memory. Obvious most of this buffer
> may never be written to, as the object never grows large enough to
> fill it.
>
> What is the overhead of allocating memory which is never used? Is this
>

A 1MB virtual area which has just one page instantiated has (amortized) 
2KB cost in page tables, while a similar 1GB mapping has 8KB cost. 
That's a 50%-200% overhead which is quite bad.  Also cache line usage is 
worse since each pte needs a full cache line (two for the 1GB version) now.

In addition, the virtual address space is not infinite. On x86-64, 
userspace has 47 bits = 128 TB, enough for 128K of these 1G mappings, so 
your program would exhaust it after allocating 128,000 buffers, which is 
less than a gigabyte of physical RAM.

-- 
error compiling committee.c: too many arguments to function

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/