linux-kernel - Re: Emulating unsupported x86 opcodes in software

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 25 Jan 2012 13:17:06 +0100
From:	Mikael Pettersson <mikpe@...uu.se>
To:	Stefan Seidel <stefans@...roid-x86.org>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: Emulating unsupported x86 opcodes in software

Stefan Seidel writes:
 > Hello,
 > 
 > I'm looking for advice regarding the emulation of CPU instructions
 > which the current processor doesn't support. Specifically, I want to
 > emulate SSE3 instructions on non-SSE3-capable CPUs. The OSx86
 > project[1] has developed an SSE3 emulation based on SSE2 instructions.
 > 
 > The general idea is:
 >  - install a "stage 1" trap handler for INT 6h (invalid opcode)
 >  - "stage 1" handler returns the address of the "stage 2" handler, the
 > actual emulation routine
 >  - "stage 2" handler runs in user-mode, reads the invalid (SSE3)
 > instruction and operands and emulates it
 >  - "stage 2" handler stores the result in the correct destination
 > (register or memory) and restores all dirty registers to their
 > original state
 > More in-depth description is available as PDF[2].
 > 
 > Now it is not a problem to replace the INT 6h handler, but these are
 > my open questions:
 > 1. How can I have the stage 2 handler run in user-mode (ring 3)
 > 2. How do I allocate/obtain the address of the code segment containing
 > the stage 2 handler OR
 > 3. Is it even possible to for code which is implemented in the kernel
 > to be called and executed in user-mode, and if not, what is the
 > alternative.
 > 
 > Both stage 1 and 2 handlers are available as binary (currently, I'm
 > trying to contact the original author for the source code).
 > 
 > Thank you very much for any help.
 > 
 > Stefan Seidel
 > P.S.: CC'ing me in your answer would be appreciated.
 > 
 > [1] http://code.google.com/p/xnu-dev/
 > [2] http://code.google.com/p/xnu-dev/downloads/detail?name=SSE3%20Emulator.pdf

Just two general comments, not specifically related to emulating SSE3:

1. Deploying the emulator from userspace would be preferable.  LD_PRELOAD a
   library containing the SSE3 emulation and init code to set up a SIGILL
   handler.  This solves all the ring 0 vs 3 problems and avoids needing
   any kernel modifications.

   For static binaries you can use a debugger-like launcher to insert
   and activate the emulation code.

2. It's not always the case that new instructions in later generation x86
   processors trap in earlier generation processors.  E.g., several years
   ago I discovered by accident that a PIII would run SSE (or was it SSE2?)
   instructions w/o trapping, it just ignored the additional prefix and
   executed the corresponding MMX instruction instead.  Maybe that was an
   isolated case, I don't know.

   In the worst case you'll need binary rewriting (static or dynamic), which
   also is implementable entirely in userspace without kernel modifications.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/