Message-ID: <0bf90fc0-2287-4ce0-b810-6e383e695981@linux.dev>
Date: Wed, 29 Jan 2025 23:10:14 -0800
From: Yonghong Song <yonghong.song@...ux.dev>
To: Josh Poimboeuf <jpoimboe@...nel.org>,
 Peter Zijlstra <peterz@...radead.org>
Cc: Song Liu <song@...nel.org>, LKML <linux-kernel@...r.kernel.org>,
 yonghong.song@...ux.dev, Kernel Team <kernel-team@...com>
Subject: objtool failure caused some kernel functionality not working

In Meta, when I tried to use llvm19 compiler to build a kernel with PGO 
([1]) support for 6.13 kernel, Ihit the following objtool warning:   
warning: objtool: __htab_map_lookup_elem+0x3fb: can't find switch jump 
table

The kernel is built successfully, but after booting the kernel, 
/sys/kernel/debug/tracing/available_filter_functionsand 
/sys/kernel/debug/tracing/available_filter_functions_addrs are empty. I 
did some investigation and found that the warning is at func 
add_jump_table(): static int add_jump_table(struct objtool_file *file, 
struct instruction *insn, struct reloc *next_table) { struct symbol 
*pfunc = insn_func(insn)->pfunc; struct reloc *table = 
insn_jump_table(insn); struct instruction *dest_insn; unsigned int 
prev_offset = 0; struct reloc *reloc = table; struct alternative *alt; 
/* * Each @reloc is a switch table relocation which points to the target 
* instruction. */ for_each_reloc_from(table->sec, reloc) { /* Check for 
the end of the table: */ if (reloc != table && reloc == next_table) 
break; /* Make sure the table entries are consecutive: */ if 
(prev_offset && reloc_offset(reloc) != prev_offset + 8) break; /* Detect 
function pointers from contiguous objects: */ if (reloc->sym->sec == 
pfunc->sec && reloc_addend(reloc) == pfunc->offset) break; dest_insn = 
find_insn(file, reloc->sym->sec, reloc_addend(reloc)); <===== find_insn 
return a NULL dest_insn with prev_offset == 0 if (!dest_insn) break; /* 
Make sure the destination is in the same function: */ if 
(!insn_func(dest_insn) || insn_func(dest_insn)->pfunc != pfunc) break; 
alt = malloc(sizeof(*alt)); if (!alt) { WARN("malloc failed"); return 
-1; } alt->insn = dest_insn; alt->next = insn->alts; insn->alts = alt; 
prev_offset = reloc_offset(reloc); } if (!prev_offset) { WARN_INSN(insn, 
"can't find switch jump table"); <===== error message here. return -1; } 
return 0; } The find_insn failed for 'prev_offset == 0' at the second 
iteration and finally the function return -1, which propagated to 
decode_sections() in func check(). Since decode_sections() got an error 
ret the later create_mcount_loc_sections() will not be called. This 
caused the above available_filter_functions issue.

The following are some codes related to func__htab_map_lookup_elem():

Disassembly of section .text.hot.__htab_map_lookup_elem:
        
0000000000000000 <__htab_map_lookup_elem>:
        0: e8 00 00 00 00                callq   0x5 <__htab_map_lookup_elem+0x5>
                 0000000000000001:  R_X86_64_PLT32       __fentry__-0x4
        5: 55                            pushq   %rbp
...
      3f3: e9 03 fe ff ff                jmp     0x1fb <__htab_map_lookup_elem+0x1fb>
      3f8: 41 ff c8                      decl    %r8d
      3fb: 42 ff 24 c5 00 00 00 00       jmpq    *(,%r8,8)
                 00000000000003ff:  R_X86_64_32S .rodata.hot.__htab_map_lookup_elem
      403: 44 0f b6 42 0a                movzbl  0xa(%rdx), %r8d
      408: 41 c1 e0 10                   shll    $0x10, %r8d
...
      45b: 44 0f b6 42 02                movzbl  0x2(%rdx), %r8d
      460: 41 c1 e0 10                   shll    $0x10, %r8d
      464: 44 01 c0                      addl    %r8d, %eax
      467: e9 26 fe ff ff                jmp     0x292 <__htab_map_lookup_elem+0x292>
      46c: 41 89 d8                      movl    %ebx, %r8d
      46f: 89 c1                         movl    %eax, %ecx
      471: 89 c5                         movl    %eax, %ebp
      473: 48 89 f2                      movq    %rsi, %rdx
      476: e9 0d fe ff ff                jmp     0x288 <__htab_map_lookup_elem+0x288>
      47b: 44 0f b6 42 0b                movzbl  0xb(%rdx), %r8d
      480: 41 c1 e0 18                   shll    $0x18, %r8d
      484: 44 01 c5                      addl    %r8d, %ebp
      487: e9 77 ff ff ff                jmp     0x403 <__htab_map_lookup_elem+0x403>

Disassembly of section .text.unlikely.bpf_percpu_hash_update:

So the func __htab_map_lookup_elem() size is 0x48c.

Here, insn 0x3fb is a jump table insn, the following is the actual jump table:
RELOCATION RECORDS FOR [.rodata.hot.__htab_map_lookup_elem]:
OFFSET           TYPE                     VALUE
0000000000000000 R_X86_64_64              .text.hot.__htab_map_lookup_elem+0x29e
0000000000000008 R_X86_64_64              .text.hot.__htab_map_lookup_elem+0x48c
                                           <==== problematic one
0000000000000010 R_X86_64_64              .text.hot.__htab_map_lookup_elem+0x45b
0000000000000018 R_X86_64_64              .text.hot.__htab_map_lookup_elem+0x44f
0000000000000020 R_X86_64_64              .text.hot.__htab_map_lookup_elem+0x447
0000000000000028 R_X86_64_64              .text.hot.__htab_map_lookup_elem+0x43b
0000000000000030 R_X86_64_64              .text.hot.__htab_map_lookup_elem+0x42f
0000000000000038 R_X86_64_64              .text.hot.__htab_map_lookup_elem+0x423
0000000000000040 R_X86_64_64              .text.hot.__htab_map_lookup_elem+0x41b
0000000000000048 R_X86_64_64              .text.hot.__htab_map_lookup_elem+0x40f
0000000000000050 R_X86_64_64              .text.hot.__htab_map_lookup_elem+0x403
0000000000000058 R_X86_64_64              .text.hot.__htab_map_lookup_elem+0x47b

Note that for the second entry which has .text.hot.__htab_map_lookup_elem+0x48c
which is right after the last insn. This caused the problem for objtool since
jump target is not within the function, rather immediately after the function.

There are quite some similar cases in my build.

The llvm18 does not have issues. I tried to bisect what changed in llvm19 and
found the following llvm patch is responsible:

https://github.com/llvm/llvm-project/pull/96089

Basically, the compiler might be able to create a jump target which actually
not possible at runtime. For example, in one of examples in the above llvm patch, something
like
     
    if i >= 3 goto out;  /* i unsigned */
    switch i, label default_unreachable:
      case 0:  goto label1;
      case 1:  goto label2;
      case 2:  goto label3;
    label1: ...; return;
    label2: ...; return;
    label3: ...; return;
    default_unreachable:

I think that this should be a valid code from compiler perspective.

Can we fix objtool to handle jump target which is immediately after the func body?Thanks, Yonghong


   [1] 
https://patchwork.kernel.org/project/linux-kbuild/patch/20210407211704.367039-1-morbo@google.com/#24246189

Powered by blists - more mailing lists