Message-ID: <ffr673gcremzfvcmjnt5qigfjfkrgchipgungjgnzqnf6kc7y6@n4kdu7nxoaw4>
Date: Wed, 26 Feb 2025 12:52:38 +0100
From: Maciej Wieczor-Retman <maciej.wieczor-retman@...el.com>
To: Andrey Konovalov <andreyknvl@...il.com>
CC: <kees@...nel.org>, <julian.stecklina@...erus-technology.de>,
	<kevinloughlin@...gle.com>, <peterz@...radead.org>, <tglx@...utronix.de>,
	<justinstitt@...gle.com>, <catalin.marinas@....com>,
	<wangkefeng.wang@...wei.com>, <bhe@...hat.com>, <ryabinin.a.a@...il.com>,
	<kirill.shutemov@...ux.intel.com>, <will@...nel.org>, <ardb@...nel.org>,
	<jason.andryuk@....com>, <dave.hansen@...ux.intel.com>,
	<pasha.tatashin@...een.com>, <ndesaulniers@...gle.com>,
	<guoweikang.kernel@...il.com>, <dwmw@...zon.co.uk>, <mark.rutland@....com>,
	<broonie@...nel.org>, <apopple@...dia.com>, <bp@...en8.de>,
	<rppt@...nel.org>, <kaleshsingh@...gle.com>, <richard.weiyang@...il.com>,
	<luto@...nel.org>, <glider@...gle.com>, <pankaj.gupta@....com>,
	<pawan.kumar.gupta@...ux.intel.com>, <kuan-ying.lee@...onical.com>,
	<tony.luck@...el.com>, <tj@...nel.org>, <jgross@...e.com>,
	<dvyukov@...gle.com>, <baohua@...nel.org>, <samuel.holland@...ive.com>,
	<dennis@...nel.org>, <akpm@...ux-foundation.org>,
	<thomas.weissschuh@...utronix.de>, <surenb@...gle.com>,
	<kbingham@...nel.org>, <ankita@...dia.com>, <nathan@...nel.org>,
	<ziy@...dia.com>, <xin@...or.com>, <rafael.j.wysocki@...el.com>,
	<andriy.shevchenko@...ux.intel.com>, <cl@...ux.com>, <jhubbard@...dia.com>,
	<hpa@...or.com>, <scott@...amperecomputing.com>, <david@...hat.com>,
	<jan.kiszka@...mens.com>, <vincenzo.frascino@....com>, <corbet@....net>,
	<maz@...nel.org>, <mingo@...hat.com>, <arnd@...db.de>, <ytcoode@...il.com>,
	<xur@...gle.com>, <morbo@...gle.com>, <thiago.bauermann@...aro.org>,
	<linux-doc@...r.kernel.org>, <kasan-dev@...glegroups.com>,
	<linux-kernel@...r.kernel.org>, <llvm@...ts.linux.dev>, <linux-mm@...ck.org>,
	<linux-arm-kernel@...ts.infradead.org>, <x86@...nel.org>
Subject: Re: [PATCH v2 13/14] x86: runtime_const used for KASAN_SHADOW_END

On 2025-02-25 at 22:37:37 +0100, Andrey Konovalov wrote:
>On Tue, Feb 25, 2025 at 6:16 PM Maciej Wieczor-Retman
><maciej.wieczor-retman@...el.com> wrote:
>>
>> I mean in my tests, with setting offset in runtime, everything works correctly
>> in inline mode. Even though hwasan-mapping-offset ends up empty and doesn't end
>> up in CFLAGS_KASAN. I assume this means that the inline mode is pretty much the
>> same as outline mode with the runtime offset setting?
>>
>> I also tested if hwasan-mapping-offset does anything if I passed random values
>> to it by hardcoding them in the makefile and still everything seemed to work
>> just fine. Therefore I assumed that this option doesn't have any effect on x86.
>
>Hm that's weird. I wonder if inline instrumentation somehow gets auto-disabled.
>
>> Hmm indeed it does. Then I'm not sure why I didn't crash when I started putting
>> in random variables. I'll dive into assembly and see what's up in there.
>
>Please do, I'm curious what's going on there.

I think I figured it out.

After adding
	kasan_params += hwasan-instrument-with-calls=0
to Makefile.kasan just under
	kasan_params += hwasan-mapping-offset=$(KASAN_SHADOW_OFFSET)
inline works properly in x86. I looked into assembly and before there were just
calls to __hwasan_load/store. After adding the the
hwasan-instrument-with-calls=0 I can see no calls and the KASAN offset is now
inlined, plus all functions that were previously instrumented now have the
kasan_check_range inlined in them.

My LLVM investigation lead me to
	bool shouldInstrumentWithCalls(const Triple &TargetTriple) {
	  return optOr(ClInstrumentWithCalls, TargetTriple.getArch() == Triple::x86_64);
	}
which I assume defaults to "1" on x86? So even with inline mode it doesn't care
and still does an outline version.

I checked how arm64 reacts to adding the hwasan-instrument-with-calls=0 by cross
compiling and I don't see any differences in output assembly.

>
>> But anyway I have an idea how to setup the x86 offset for tag-based mode so it
>> works for both paging modes. I did some testing and value
>>         0xffeffc0000000000
>> seems to work fine and has at least some of the benefits I was hoping for when
>> doing the runtime_const thing. It works in both paging modes because in 5 levels
>> it's just a little bit below the 0xffe0000000000000 that I was thinking about
>> first and in 4 levels, because of LAM, it becomes 0xfffffc0000000000 (because in
>> 4 level paging bits 62:48 are masked from address translation. So it's the same
>> as the end of generic mode shadow memory space.
>>
>> The alignment doesn't fit the shadow memory size so it's not optimal but I'm not
>> sure it can be if we want to have the inline mode and python scripts working at
>> the same time. At the very least I think the KASAN_SHADOW_END won't collide with
>> other things in the tab-based mode in 5 level paging mode, so no extra steps are
>> needed (arch/x86/mm/kasan_init_64.c in kasan_init()).
>
>What do you mean by "The alignment doesn't fit the shadow memory size"?

Maybe that's the wrong way to put it. I meant that KASAN_SHADOW_END and
KASAN_SHADOW_END aren't aligned to the size of shadow memory.

>
>> Do you see any problems with this offset for x86 tag-based mode?
>
>I don't, but I think someone who understands the x86 memory layout
>better needs to look at this.
>
>> Btw I think kasan_check_range() can be optimized on x86 if we use
>> addr_has_metadata() that doesn't use KASAN_SHADOW_START. Getting rid of it from
>> the implementation will remove pgtable_l5_enabled() which is pretty slow so
>> kasan_check_range() which is called a lot would probably work much faster.
>> Do you see any way in which addr_has_metadata() will make sense but won't use
>> KASAN_SHADOW_START? Every one of my ideas ends up using pgtable_l5_enabled()
>> because the metadata can have 6 or 15 bits depending on paging level.
>
>What if we turn pgtable_l5_enabled() into using a read-only static key
>(DEFINE_STATIC_KEY_FALSE_RO) instead of a bool variable? Or if that is
>not acceptable, we could cache its value in a KASAN-specific static
>key.

I think this was a false alarm, sorry. I asked Kirill about turning
pgtable_l5_enabled() into a runtime_const value but it turns out it's already
patched by alternative code during boot. I just saw a bunch more stuff there
because I was looking at the assembly output and the code isn't patched there
yet.

-- 
Kind regards
Maciej Wieczór-Retman

Powered by blists - more mailing lists