linux-kernel - Continuous mmap & unmap of kernel buffers leading to memory leak on linux kernel 2.6.38

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <11309A76DCEC0041BEB858FDC83272E6015B1A73E8@inbmail01.lsi.com>
Date:	Mon, 4 Jul 2011 15:37:58 +0530
From:	"Jaju, Ganesh" <Ganesh.Jaju@....com>
To:	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Continuous mmap & unmap of kernel buffers leading to memory leak on
 linux kernel 2.6.38

Hi,
I have a driver that supports mmap of kernel buffers to the user space.
The user space application continuously mmaps and unmaps lot of memory.
This piece of code works fine on linux kernel 2.6.9 while same code gives huge memory leaks on linux kernel 2.6.38 finally leading to system memory allocation failure.
I am suspecting that the mmap’ed pages are not getting unmapped at all.

The nopage() routine (fault() routine for linux kernel version >= 2.6.26) of the vma operations increments the page count of the page that has been requested/accessed before returning the required page.
As I am seeing it, on linux kernel 2.6.9, when the vmaclose() routine gets called, the page count in 2.6.9 kernel is ‘0’ while in 2.6.38 it is ‘1’.
To my knowledge, the unmap routine itself decrements the page count of each of the pages. But, somehow for 2.6.38, the page count is ‘1’ even when the vmaclose() routine gets called.
I even tried decrementing the page count in vmaclose() routine for 2.6.38 kernel. But, the memory leak continues.

Am I missing something here or has something changed in the linux kernel in this path?

Thanks & Regards
Ganesh Jaju.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/