linux-kernel - 4.4-rc5 Setting trap flag inside nmi handler results in HARD LOCKUP

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAO6TR8UPzM9J+JDhpZpBseNLX7mV35d-tAEk6dj4AM=O=+qJdA@mail.gmail.com>
Date:	Wed, 16 Dec 2015 11:19:21 -0700
From:	Jeff Merkey <linux.mdb@...il.com>
To:	LKML <linux-kernel@...r.kernel.org>
Cc:	tglx@...utronix.de, mingo@...hat.com, hpa@...or.com,
	x86@...nel.org, peterz@...radead.org, luto@...nel.org
Subject: 4.4-rc5 Setting trap flag inside nmi handler results in HARD LOCKUP

Setting the (trap flag | resume flag) inside of an nmi handler results
in a hard lockup while setting the resume flag works fine.

The watchdog detector fails to detect the lockup.  I am currently
examining the trap gate and interrupt gate setup on Linux and if
anyone has any ideas it would be nice to be able to debug and step
through the nmi handlers.  I got breakpoints to work.  I noticed
kgdb/kdb just punts here and refuses to allow someone to step inside
an nmi handler.

There is no reason Linux should not allow this to work since windows
does and every other OS out there.  I have seen this across some rex64
sysret calls as well this lockup behavior.

Anyone who is an intel expert with any clues would love some input if
you know about this problem.

Jeff
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/