Date:	Wed, 9 Sep 2015 08:56:21 +0200
From:	Sedat Dilek <sedat.dilek@...il.com>
To:	Baoquan He <bhe@...hat.com>
Cc:	Denys Vlasenko <dvlasenk@...hat.com>, Tejun Heo <tj@...nel.org>,
	Christoph Lameter <cl@...ux.com>,
	LKML <linux-kernel@...r.kernel.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	David Rientjes <rientjes@...gle.com>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Peter Zijlstra <peterz@...radead.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Thomas Graf <tgraf@...g.ch>, Ingo Molnar <mingo@...nel.org>,
	"the arch/x86 maintainers" <x86@...nel.org>
Subject: Re: [llvmlinux] percpu | bitmap issue? (Cannot boot on bare metal due
 to a kernel NULL pointer dereference)

On Wed, Sep 9, 2015 at 5:46 AM, Sedat Dilek <sedat.dilek@...il.com> wrote:
> On Wed, Sep 9, 2015 at 5:25 AM, Baoquan He <bhe@...hat.com> wrote:
>> Hi Sedat,
>>
>> On 09/09/15 at 04:51am, Sedat Dilek wrote:
>>> On Wed, Sep 9, 2015 at 4:29 AM, Baoquan He <bhe@...hat.com> wrote:
>>> commit 1a1d48a4a8fde49aedc045d894efe67173d59fe0
>>> "linux/bitmap: Force inlining of bitmap weight functions"
>>>
>>> ...on top of Linux v4.2.
>>>
>>> This resulted in the same call-trace in QEMU.
>>>
>>> I hacked around to only re-build mm/percpu.c with GCC (rest with
>>> CLANG) with some guidance from Linus (compiler warapper-script) etc.
>>
>> Sorry, from below log message and code flow I didn't get what's wrong
>> with it. I am working on another issue which gives me much headache,
>> don't hvae time to look into the the disassembling code now. But if GCC
>> built code works, it should be related to compiler issues. You can try
>> more tests, e.g build percpu.c bitmap.c with GCC. Maybe other people
>> can give suggestions.
>>
>> Sorry again, Sedat.
>>
>
> Hi Baoquan,
>
> OK, I understand your situation and kernel folks are within
> merge-window or you have to concentrate on your daily work etc.
>
> I have found some more playground
>
> The article "Force a function to be inline in Clang/LLVM" from
> <http://stackoverflow.com> in [1] says...
>
> "There is a good solution if compiling with C99 which is Clang's
> default. Its simply using inline attribute.
>
> inline void foo() {}
>
> [...]
>
> So in order to guarantee that the function is inlined:
>
> 1. Don’t use static inline.
> 2. Don’t add another implementation for the function that doesn't have
> inline attribute.
> 3. You must use optimization. But even if there isn't optimization the
> compilation will fail which is good.
> 4. Make sure not to compile with GNU89."
>
> So I added to Linux main Makefile...
>
> $ git diff
> diff --git a/Makefile b/Makefile
> index 0e333fd142a1..e1723367ece4 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -702,6 +702,14 @@ KBUILD_CFLAGS += $(call cc-disable-warning,
> tautological-compare)
>  # See modpost pattern 2
>  KBUILD_CFLAGS += $(call cc-option, -mno-global-merge,)
>
> +# Force a function to be inline in Clang/LLVM
> +# http://stackoverflow.com/questions/25602813/force-a-function-to-be-inline-in-clang-llvm
> +# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66122
> +# http://marc.info/?t=144156156700001&r=1&w=2
> +ifdef CONFIG_OPTIMIZE_INLINING
> +KBUILD_CFLAGS += $(call cc-option,-always-inline)
> +endif
> +
>  else
>
>  KBUILD_CFLAGS += $(call cc-option,-fno-delete-null-pointer-checks,)
>
> Some comments below...
>
> Thanks in advance!
>
>> setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:1 nr_node_ids:1
>>
>> arch/x86/kernel/setup_percpu.c:setup_per_cpu_areas() ->
>> mm/percpu.c : pcpu_embed_first_chunk() ->
>> mm/percpu.c:pcpu_build_alloc_info() ->
>> include/linux/cpumask.h:
>> #define num_possible_cpus()     cpumask_weight(cpu_possible_mask)
>> static inline unsigned int cpumask_weight(const struct cpumask *srcp)
>> {
>>         return bitmap_weight(cpumask_bits(srcp), nr_cpumask_bits);
>> }
>>
>
> s/inline/__always_inline/ ???
>
>> include/linux/bitmap.h:
>> static inline int bitmap_weight(const unsigned long *src, unsigned int nbits)
>> {
>>         if (small_const_nbits(nbits))
>>                 return hweight_long(*src & BITMAP_LAST_WORD_MASK(nbits));
>>         return __bitmap_weight(src, nbits);
>> }
>>
>
> s/inline/__always_inline/ ???
>
> Already realized in "linux/bitmap: Force inlining of bitmap weight
> functions" upstream (see [3], but not available for Linux v4.2).
>
>> lib/bitmap.c:
>> int __bitmap_weight(const unsigned long *bitmap, unsigned int bits)
>> {
>>         unsigned int k, lim = bits/BITS_PER_LONG;
>>         int w = 0;
>>
>>         for (k = 0; k < lim; k++)
>>                 w += hweight_long(bitmap[k]);
>>
>>         if (bits % BITS_PER_LONG)
>>                 w += hweight_long(bitmap[k] & BITMAP_LAST_WORD_MASK(bits));
>>
>>         return w;
>> }
>> EXPORT_SYMBOL(__bitmap_weight);
>>
>
> Thanks again.
>
> - Sedat -
>
> [1] http://stackoverflow.com/questions/25602813/force-a-function-to-be-inline-in-clang-llvm
> [2] http://git.linuxfoundation.org/?p=llvmlinux.git;a=blob;f=arch/all/patches/compiler-gcc.patch
> [3] http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=1a1d48a4a8fde49aedc045d894efe67173d59fe0

None of the force-inlining patches helped.

      cpumask: Force inlining of cpumask_weight function
      nodemask: Force inlining of __nodes_weight function

If I recompile lib/bitmap.o with GCC v4.9 (see attached file) and
re-invoke make I can boot into this mixture of clang-gcc-compiled
Linux-kernel.

I have here CONFIG_OPTIMIZE_INLINING=n, but I doubt this had an effect.

CONCLUSION:

The problem exists in lib/bitmap.c - cannot say what is wrong with it
when compiling with CLANG v3.7.

Regards,
- Sedat -

View attachment "dmesg_4.2.0-4-llvmlinux-small.txt" of type "text/plain" (60529 bytes)

Download attachment "bitmap_o-cmdline.sh" of type "application/x-sh" (4096 bytes)

Download attachment "config-4.2.0-4-llvmlinux-small" of type "application/octet-stream" (128624 bytes)

View attachment "4.2.0-4-llvmlinux-small.patch" of type "text/x-patch" (38028 bytes)

Powered by blists - more mailing lists