bugtraq - gcc 4.1 bug miscompiles pointer range checks, may place you at risk

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date: Mon, 17 Apr 2006 22:03:54 +0200
From: Felix von Leitner <felix-bugtraq@...e.de>
To: bugtraq@...urityfocus.com
Subject: gcc 4.1 bug miscompiles pointer range checks, may place you at risk

I wrote a small library of functions to do typical range checks as they
are needed in code that handles incoming packets or messages from
untrusted sources.  My impetus was SMB code, in case you want to know.

Here is one of my functions:

static inline int range_ptrinbuf(const void* buf,unsigned long len,const void* ptr) {
  register const char* c=(const char*)buf;      /* no pointer arithmetic on void* */
  return (c && c+len>c && (const char*)ptr-c<len);
}

Of course, when developing security critical code like this, you also
write a good test suite for it, that exercises all the cases.  Here is
part of my test suite:

  assert(range_ptrinbuf(buf,(unsigned long)-1,buf+1)==0);

Imagine my surprise when this assertion failed.  I had compiled the
code with gcc 4.1 and compiled it without optimizing (I mention this
because for most gcc bugs, a workaround is disabling the optimizer).

gcc 3 compiles this code correctly.  I tested this on x86 and amd64.
I mention this here because "c+len>c" is the code with which you would
typically check for integer overflows, which is a check that for example
an IP stack would do, or Samba.  So, if you compiled your kernel with
gcc 4.1, or your Samba, or some other packet handling code in a security
relevant context, you might want to recompile with gcc 3.

Felix