linux-kernel - Linux doesn't follow x86/x86-64 ABI wrt direction flag

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20080305153020.GA24631@volta.aurel32.net>
Date:	Wed, 5 Mar 2008 16:30:20 +0100
From:	Aurelien Jarno <aurelien@...el32.net>
To:	linux-kernel@...r.kernel.org
Cc:	gcc@....gnu.org
Subject: Linux doesn't follow x86/x86-64 ABI wrt direction flag

Hi all,

Since version 4.3, gcc changed its behaviour concerning the x86/x86-64 
ABI and the direction flag, that is it now assumes that the direction 
flag is cleared at the entry of a function and it doesn't clear once 
more if needed.

This causes some problems with the Linux kernel which does not clear
the direction flag when entering a signal handler. The small code below
(for x86-64) demonstrates that. 

If the signal handler is using code that need the direction flag cleared
(for example bzero() or memset()), the code is incorrectly executed.

I guess this has to be fixed on the kernel side, but also gcc-4.3 could
revert back to the old behaviour, that is clearing the direction flag
when entering a routine that touches it until most people are running a
fixed kernel.

Kind regards,
Aurelien

[1] http://gcc.gnu.org/gcc-4.3/changes.html

#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#include <signal.h>

void handler(int signal) {
	uint64_t rflags;

	asm volatile("pushfq ; popq %0" : "=g" (rflags));

	if (rflags & (1 << 10))
		printf("DF = 1\n");
	else
		printf("DF = 0\n");
}

int main() {
	signal(SIGUSR1, handler);

	while(1)
	{
		asm volatile("std\r\n");
	}

	return 0;
}

-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32@...ian.org         | aurelien@...el32.net
   `-    people.debian.org/~aurel32 | www.aurel32.net
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/