netdev - Re: possible memory leak in ipc

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Fri, 26 Aug 2016 11:25:56 -0400 (EDT)
From:   CAI Qian <caiqian@...hat.com>
To:     Linus Torvalds <torvalds@...ux-foundation.org>
Cc:     Thomas Graf <tgraf@...g.ch>,
        Herbert Xu <herbert@...dor.apana.org.au>,
        Eric Dumazet <edumazet@...gle.com>,
        Network Development <netdev@...r.kernel.org>
Subject: Re: possible memory leak in ipc



----- Original Message -----
> From: "Linus Torvalds" <torvalds@...ux-foundation.org>
> To: "CAI Qian" <caiqian@...hat.com>, "Thomas Graf" <tgraf@...g.ch>, "Herbert Xu" <herbert@...dor.apana.org.au>
> Cc: "Eric Dumazet" <edumazet@...gle.com>, "Network Development" <netdev@...r.kernel.org>
> Sent: Thursday, August 25, 2016 6:20:03 PM
> Subject: Re: possible memory leak in ipc
> 
> On Thu, Aug 25, 2016 at 1:17 PM, CAI Qian <caiqian@...hat.com> wrote:
> > I am unsure if it is really a memleak (could be a security issue due to
> > eventually OOM and DoS) or just a soft lockup with in kmemlock code with
> > false alarm.
> 
> Hmm. The reported leaks look like
> 
> unreferenced object 0xffffc90004857000 (size 4608):
>   comm "kworker/16:0", pid 110, jiffies 4294705908 (age 883.925s)
>   hex dump (first 32 bytes):
>     c0 05 3d 5e 08 88 ff ff ff ff ff ff 00 00 dc 6e  ..=^...........n
>     ff ff ff ff ff ff ff ff 28 c7 46 83 ff ff ff ff  ........(.F.....
>   backtrace:
>     [<ffffffff817d95ba>] kmemleak_alloc+0x4a/0xa0
>     [<ffffffff8123df4e>] __vmalloc_node_range+0x1de/0x2f0
>     [<ffffffff8123e324>] vmalloc+0x54/0x60
>     [<ffffffff81404934>] alloc_bucket_locks.isra.7+0xd4/0xf0
>     [<ffffffff814049a8>] bucket_table_alloc+0x58/0x100
>     [<ffffffff8140538e>] rht_deferred_worker+0x10e/0x890
>     [<ffffffff810c30a8>] process_one_work+0x218/0x750
>     [<ffffffff810c3705>] worker_thread+0x125/0x4a0
>     [<ffffffff810ca8b1>] kthread+0x101/0x120
>     [<ffffffff817e70af>] ret_from_fork+0x1f/0x40
>     [<ffffffffffffffff>] 0xffffffffffffffff
> 
> which would indicate that it's a rhashtable resize event where we
> perhaps haven't free'd the old hash table when we create a new one.
> 
> The actually freeing of the old one is done RCU-deferred from
> rhashtable_rehash_table(), but that itself is also deferred by a
> worker thread (rht_deferred_worker).
> 
> I'm not seeing anything wrong in the logic, but let's bring in Thomas
> Graf and Herbert Xu.
> 
> Hmm. The size (4608) is always the same and doesn't change, so maybe
> it's not actually a rehash events per se - it's somebody creating a
> rhashtable, but perhaps not freeing it?
> 
> Sadly, all but one of the traces are that kthread one, and the one
> that isn't that might give an idea about what code triggers this is:
> 
> unreferenced object 0xffffc900048b6000 (size 4608):
>   comm "modprobe", pid 2485, jiffies 4294727633 (age 862.590s)
>   hex dump (first 32 bytes):
>     00 9c 49 21 00 ea ff ff 00 d5 59 21 00 ea ff ff  ..I!......Y!....
>     00 a5 7d 21 00 ea ff ff c0 da 74 21 00 ea ff ff  ..}!......t!....
>   backtrace:
>     [<ffffffff817d95ba>] kmemleak_alloc+0x4a/0xa0
>     [<ffffffff8123df4e>] __vmalloc_node_range+0x1de/0x2f0
>     [<ffffffff8123e324>] vmalloc+0x54/0x60
>     [<ffffffff81404934>] alloc_bucket_locks.isra.7+0xd4/0xf0
>     [<ffffffff814049a8>] bucket_table_alloc+0x58/0x100
>     [<ffffffff81404d8d>] rhashtable_init+0x1ed/0x390
>     [<ffffffffa05b201b>] 0xffffffffa05b201b
>     [<ffffffff81002190>] do_one_initcall+0x50/0x190
>     [<ffffffff811e6eed>] do_init_module+0x60/0x1f3
>     [<ffffffff81155107>] load_module+0x1487/0x1ca0
>     [<ffffffff81155b56>] SYSC_finit_module+0xa6/0xf0
>     [<ffffffff81155bbe>] SyS_finit_module+0xe/0x10
>     [<ffffffff81003c4c>] do_syscall_64+0x6c/0x1e0
>     [<ffffffff817e6f3f>] return_from_SYSCALL_64+0x0/0x7a
>     [<ffffffffffffffff>] 0xffffffffffffffff
> 
> so it comes from some module init code, but since the module hasn't
> fully initialized, the kallsym code doesn't find the symbol name
> either. Annoying.
> 
> Maybe the above just makes one of the rhashtable people go "Oh, that's
> obvious".
> 
>                  Linus
> 
FYI, this also happened while compiling gcc.
$ make -j 56

$ cat /sys/kernel/debug/kmemleak
unreferenced object 0xffffc9000485a000 (size 4608):
  comm "kworker/7:1", pid 368, jiffies 4294835499 (age 1033.075s)
  hex dump (first 32 bytes):
    5f 65 76 65 6e 74 5f 69 74 65 6d 2e 68 00 05 00  _event_item.h...
    00 76 6d 73 74 61 74 2e 68 00 05 00 00 70 6c 69  .vmstat.h....pli
  backtrace:
    [<ffffffff8268aa6a>] kmemleak_alloc+0x4a/0xa0
    [<ffffffff815e4d78>] __vmalloc_node_range+0x378/0x700
    [<ffffffff815e51a4>] vmalloc+0x54/0x60
    [<ffffffff81ab48c8>] alloc_bucket_locks.isra.7+0x188/0x220
    [<ffffffff81ab4a0c>] bucket_table_alloc+0xac/0x290
    [<ffffffff81ab6955>] rht_deferred_worker+0x8c5/0x1890
    [<ffffffff811ee711>] process_one_work+0x731/0x16c0
    [<ffffffff811ef77c>] worker_thread+0xdc/0xf10
    [<ffffffff81201be3>] kthread+0x223/0x2f0
    [<ffffffff8269e82f>] ret_from_fork+0x1f/0x40
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc90004861000 (size 4608):
  comm "kworker/10:1", pid 380, jiffies 4294843418 (age 1025.169s)
  hex dump (first 32 bytes):
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  backtrace:
    [<ffffffff8268aa6a>] kmemleak_alloc+0x4a/0xa0
    [<ffffffff815e4d78>] __vmalloc_node_range+0x378/0x700
    [<ffffffff815e51a4>] vmalloc+0x54/0x60
    [<ffffffff81ab48c8>] alloc_bucket_locks.isra.7+0x188/0x220
    [<ffffffff81ab4a0c>] bucket_table_alloc+0xac/0x290
    [<ffffffff81ab7664>] rht_deferred_worker+0x15d4/0x1890
    [<ffffffff811ee711>] process_one_work+0x731/0x16c0
    [<ffffffff811ef77c>] worker_thread+0xdc/0xf10
    [<ffffffff81201be3>] kthread+0x223/0x2f0
    [<ffffffff8269e82f>] ret_from_fork+0x1f/0x40
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc90004864000 (size 4608):
  comm "kworker/27:0", pid 176, jiffies 4294866756 (age 1002.065s)
  hex dump (first 32 bytes):
    83 e0 07 83 c0 03 38 d0 7c 08 84 d2 0f 85 00 15  ......8.|.......
    00 00 48 8b 85 f0 fe ff ff 45 85 e4 8b 40 4c 74  ..H......E...@Lt
  backtrace:
    [<ffffffff8268aa6a>] kmemleak_alloc+0x4a/0xa0
    [<ffffffff815e4d78>] __vmalloc_node_range+0x378/0x700
    [<ffffffff815e51a4>] vmalloc+0x54/0x60
    [<ffffffff81ab48c8>] alloc_bucket_locks.isra.7+0x188/0x220
    [<ffffffff81ab4a0c>] bucket_table_alloc+0xac/0x290
    [<ffffffff81ab6955>] rht_deferred_worker+0x8c5/0x1890
    [<ffffffff811ee711>] process_one_work+0x731/0x16c0
    [<ffffffff811ef77c>] worker_thread+0xdc/0xf10
    [<ffffffff81201be3>] kthread+0x223/0x2f0
    [<ffffffff8269e82f>] ret_from_fork+0x1f/0x40
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc9000486a000 (size 4608):
  comm "kworker/3:1", pid 365, jiffies 4294867142 (age 1001.782s)
  hex dump (first 32 bytes):
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
    10 10 d2 2a 04 88 ff ff 10 10 d2 2a 04 88 ff ff  ...*.......*....
  backtrace:
    [<ffffffff8268aa6a>] kmemleak_alloc+0x4a/0xa0
    [<ffffffff815e4d78>] __vmalloc_node_range+0x378/0x700
    [<ffffffff815e51a4>] vmalloc+0x54/0x60
    [<ffffffff81ab48c8>] alloc_bucket_locks.isra.7+0x188/0x220
    [<ffffffff81ab4a0c>] bucket_table_alloc+0xac/0x290
    [<ffffffff81ab6955>] rht_deferred_worker+0x8c5/0x1890
    [<ffffffff811ee711>] process_one_work+0x731/0x16c0
    [<ffffffff811ef77c>] worker_thread+0xdc/0xf10
    [<ffffffff81201be3>] kthread+0x223/0x2f0
    [<ffffffff8269e82f>] ret_from_fork+0x1f/0x40
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc90004a71000 (size 4608):
  comm "kworker/16:1", pid 376, jiffies 4294891121 (age 977.869s)
  hex dump (first 32 bytes):
    00 00 00 00 00 00 00 00 00 00 00 00 fe 01 00 00  ................
    00 98 c3 1c 00 ea ff ff 40 9a c3 1c 00 ea ff ff  ........@.......
  backtrace:
    [<ffffffff8268aa6a>] kmemleak_alloc+0x4a/0xa0
    [<ffffffff815e4d78>] __vmalloc_node_range+0x378/0x700
    [<ffffffff815e51a4>] vmalloc+0x54/0x60
    [<ffffffff81ab48c8>] alloc_bucket_locks.isra.7+0x188/0x220
    [<ffffffff81ab4a0c>] bucket_table_alloc+0xac/0x290
    [<ffffffff81ab7664>] rht_deferred_worker+0x15d4/0x1890
    [<ffffffff811ee711>] process_one_work+0x731/0x16c0
    [<ffffffff811ef77c>] worker_thread+0xdc/0xf10
    [<ffffffff81201be3>] kthread+0x223/0x2f0
    [<ffffffff8269e82f>] ret_from_fork+0x1f/0x40
    [<ffffffffffffffff>] 0xffffffffffffffff