Message-ID: <7d306fb3-43ba-4691-9302-47d69f8ab183@redhat.com>
Date: Thu, 17 Oct 2024 15:55:33 +0200
From: Veronika Molnarova <vmolnaro@...hat.com>
To: Arnaldo Carvalho de Melo <acme@...nel.org>
Cc: linux-perf-users@...r.kernel.org, acme@...hat.com, mpetlan@...hat.com,
 peterz@...radead.org, mingo@...hat.com, namhyung@...nel.org,
 mark.rutland@....com, alexander.shishkin@...ux.intel.com, jolsa@...nel.org,
 irogers@...gle.com, adrian.hunter@...el.com, kan.liang@...ux.intel.com,
 linux-kernel@...r.kernel.org
Subject: Re: [PATCH] perf test: Handle perftool-testsuite_probe failure due to
 broken DWARF



On 10/16/24 18:15, Arnaldo Carvalho de Melo wrote:
> On Wed, Oct 16, 2024 at 11:19:30AM +0200, vmolnaro@...hat.com wrote:
>> From: Veronika Molnarova <vmolnaro@...hat.com>
>>
>> Test case test_adding_blacklisted ends in failure if the blacklisted
>> probe is of an assembler function with no DWARF available. At the same
>> time, probing the blacklisted function with ASM DWARF doesn't test the
>> blacklist itself as the failure is a result of the broken DWARF.
>>
>> When the broken DWARF output is encountered, check if the probed
>> function was compiled by the assembler. If so, the broken DWARF message
>> is expected and does not report a perf issue, else report a failure.
>> If the ASM DWARF affected the probe, try the next probe on the blacklist.
>> If the first 5 probes are defective due to broken DWARF, skip the test
>> case.
>>
>> Fixes: def5480d63c1e847 ("perf testsuite probe: Add test for blacklisted kprobes handling")
>> Signed-off-by: Veronika Molnarova <vmolnaro@...hat.com>
>> ---
>>  .../base_probe/test_adding_blacklisted.sh     | 74 +++++++++++++++----
>>  1 file changed, 59 insertions(+), 15 deletions(-)
>>
>> diff --git a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
>> index b5dc10b2a73810b3..f67b3b267ac55269 100755
>> --- a/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
>> +++ b/tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
>> @@ -19,35 +19,79 @@
>>  TEST_RESULT=0
>>  
>>  # skip if not supported
>> -BLACKFUNC=`head -n 1 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2`
>> -if [ -z "$BLACKFUNC" ]; then
>> +BLACKFUNC_LIST=`head -n 5 /sys/kernel/debug/kprobes/blacklist 2> /dev/null | cut -f2`
>> +if [ -z "$BLACKFUNC_LIST" ]; then
>>  	print_overall_skipped
>>  	exit 0
>>  fi
>>  
>> +# chceck if locate is present to find vmlinux with DWARF debug info
> 
> check
> 
>> +locate --help 2&> /dev/null
>> +if [ $? -eq 0 ]; then
>> +	VMLINUX_FILE=$(locate -r '/vmlinux$' | xargs -I{} sh -c 'test -f "{}" && echo "{}"' | grep "$(uname -r)")
>> +fi
> 
> And that matches the running kernel? Maybe its better to use 'perf
> probe' itself to find it:
> 
>   root@...ber:~# perf probe -V icmp_rcv
>   Available variables at icmp_rcv
>           @<icmp_rcv+0>
>                   struct sk_buff* skb
>   root@...ber:~# perf probe -v -V icmp_rcv |& grep "Using.*for symbols"
>   Using /usr/lib/debug/lib/modules/6.11.3-200.fc40.x86_64/vmlinux for symbols
>   root@...ber:~# perf probe -v -V icmp_rcv |& grep "Using.*for symbols" | sed -r 's/^Using (.*) for symbols$/\1/'
>   /usr/lib/debug/lib/modules/6.11.3-200.fc40.x86_64/vmlinux
>   root@...ber:~#
> 
Well, it was the best solution we could find, but using 'perf probe' might be
a better solution to it. It seems that 'perf probe' prints the using log line
even if the probe doesn't exist:

    [root@...el-sharkbay-mb-01 ~]# perf probe -v random_probe
    probe-definition(0): random_probe 
    symbol:random_probe file:(null) line:0 offset:0 return:0 lazy:(null)
    0 arguments
    Looking at the vmlinux_path (8 entries long)
    Using /usr/lib/debug/lib/modules/5.14.0-364.3026_999908015.el9.x86_64/vmlinux for symbols
    Open Debuginfo file: /usr/lib/debug/.build-id/97/52ebee3cc94ca35fbbaafc9abf3cbacb34d4d1.debug
    Try to find probe point from debuginfo.
    Probe point 'random_probe' not found.
      Error: Failed to add events. Reason: No such device (Code: -19)

Thanks, will rework that to grep the vmlinux from the 'perf probe -v' output instead.
> I'll probably add something like this to 'perf buildid-list':
> 
> root@...ber:~# pahole --running_kernel_vmlinux
> /usr/lib/debug/lib/modules/6.11.3-200.fc40.x86_64/vmlinux
> root@...ber:~# readelf -wi `pahole --running_kernel_vmlinux` | head -12
> Contents of the .debug_info section:
> 
>   Compilation Unit @ offset 0:
>    Length:        0x2b082 (32-bit)
>    Version:       5
>    Unit Type:     DW_UT_compile (1)
>    Abbrev Offset: 0
>    Pointer Size:  8
>  <0><c>: Abbrev Number: 246 (DW_TAG_compile_unit)
>     <e>   DW_AT_producer    : (indirect string, offset: 0x4eda9f): GNU C11 14.2.1 20240912 (Red Hat 14.2.1-3) -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -mindirect-branch=thunk-extern -mindirect-branch-register -mindirect-branch-cs-prefix -mfunction-return=thunk-extern -mharden-sls=all -mrecord-mcount -mfentry -march=x86-64 -g -O2 -std=gnu11 -p -fshort-wchar -funsigned-char -fno-common -fno-PIE -fno-strict-aliasing -fcf-protection=branch -falign-jumps=1 -falign-loops=1 -fno-asynchronous-unwind-tables -fno-jump-tables -fpatchable-function-entry=16,16 -fno-delete-null-pointer-checks -fno-allow-store-data-races -fstack-protector-strong -ftrivial-auto-var-init=zero -fno-stack-clash-protection -fmin-function-alignment=16 -fstrict-flex-arrays=3 -fno-strict-overflow -fstack-check=no -fconserve-stack -fno-function-sections -fno-data-sections -fsanitize=bounds-strict -fsanitize=shift
>     <12>   DW_AT_language    : 29	(C11)
>     <13>   DW_AT_name        : (indirect line string, offset: 0xa546): init/main.c
> root@...ber:~#
> 
>> +
>>  # remove all previously added probes
>>  clear_all_probes
>>  
>>  
>>  ### adding blacklisted function
>> -
>> -# functions from blacklist should be skipped by perf probe
>> -! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err
>> -PERF_EXIT_CODE=$?
>> -
>>  REGEX_SCOPE_FAIL="Failed to find scope of probe point"
>>  REGEX_SKIP_MESSAGE=" is blacklisted function, skip it\."
>> -REGEX_NOT_FOUND_MESSAGE="Probe point \'$BLACKFUNC\' not found."
>> +REGEX_NOT_FOUND_MESSAGE="Probe point \'$RE_EVENT\' not found."
>>  REGEX_ERROR_MESSAGE="Error: Failed to add events."
>>  REGEX_INVALID_ARGUMENT="Failed to write event: Invalid argument"
>>  REGEX_SYMBOL_FAIL="Failed to find symbol at $RE_ADDRESS"
>> -REGEX_OUT_SECTION="$BLACKFUNC is out of \.\w+, skip it"
>> -../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err
>> -CHECK_EXIT_CODE=$?
>> -
>> -print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC"
>> -(( TEST_RESULT += $? ))
>> -
>> +REGEX_OUT_SECTION="$RE_EVENT is out of \.\w+, skip it"
>> +REGEX_MISSING_DECL_LINE="A function DIE doesn't have decl_line. Maybe broken DWARF?"
>> +
>> +BLACKFUNC=""
>> +SKIP_DWARF=0
>> +
>> +for BLACKFUNC in $BLACKFUNC_LIST; do
>> +	echo "Probing $BLACKFUNC"
>> +
>> +	# functions from blacklist should be skipped by perf probe
>> +	! $CMD_PERF probe $BLACKFUNC > $LOGS_DIR/adding_blacklisted.log 2> $LOGS_DIR/adding_blacklisted.err
>> +	PERF_EXIT_CODE=$?
>> +
>> +	# check for bad DWARF polluting the result
>> +	../common/check_all_patterns_found.pl "$REGEX_MISSING_DECL_LINE" >/dev/null < $LOGS_DIR/adding_blacklisted.err
>> +
>> +	if [ $? -eq 0 ]; then
>> +		SKIP_DWARF=1
>> +
>> +		# confirm that the broken DWARF comes from assembler
>> +		if [ -n "$VMLINUX_FILE" ]; then
>> +			readelf -wi "$VMLINUX_FILE" |
> 
> Can you cache the output of 'readelf -wi' for the whole vmlinux file so
> that we don't do it more than once?
> 
> Doing it for each assembly function in the blacklist probably will add
> up on this test run time.
> 
I am not sure if it is possible to cache the whole output somehow
as my tested system takes a minute if I try to print the output
of the 'readlef -wi'. To add to that, I do not need the whole
vmlinux file as I am exiting as soon I as find the blacklisted_probe in
the output.

About the test run time, I checked how long it takes to find the probe in
the 'readelf -wi' output using awk and it is about 2 seconds per probe:

    [root@...el-sharkbay-mb-01 ~]# time readelf -wi "$VMLINUX_FILE" | awk -v probe="asm_exc_device_not_available" '/DW_AT_language/ { comp_lang = $0 } $0 ~ probe { if (comp_lang) { print comp_lang }; exit }'
        <41fc04>   DW_AT_language    : 29	(C11)

    real	0m1.509s
    user	0m1.979s
    sys		0m0.173s
 
This depends on the probe so some might take longer than that. Doing it
repeatedly for multiple probes is not the default behavior anyway, as
this only happens if there are multiple assembly functions at the
start of the blacklist in a row. Currently, we know only about one
of such functions, which is warn_thunk_thunk. If the run time will be
the issue, I think we would have to address the issue differently.

>> +			awk -v probe="$BLACKFUNC" '/DW_AT_language/ { comp_lang = $0 }
>> +										$0 ~ probe { if (comp_lang) { print comp_lang }; exit }' |
>> +			grep -q "MIPS assembler"
>> +
>> +			CHECK_EXIT_CODE=$?
>> +			if [ $CHECK_EXIT_CODE -ne 0 ]; then
>> +				SKIP_DWARF=0 # broken DWARF while available
>> +				break
>> +			fi
>> +		else
>> +			echo "Result polluted by broken DWARF, trying another probe"
>> +		fi
>> +
>> +	else
>> +		../common/check_all_lines_matched.pl "$REGEX_SKIP_MESSAGE" "$REGEX_NOT_FOUND_MESSAGE" "$REGEX_ERROR_MESSAGE" "$REGEX_SCOPE_FAIL" "$REGEX_INVALID_ARGUMENT" "$REGEX_SYMBOL_FAIL" "$REGEX_OUT_SECTION" < $LOGS_DIR/adding_blacklisted.err
>> +		CHECK_EXIT_CODE=$?
>> +
>> +		SKIP_DWARF=0
>> +		break
>> +	fi
>> +done
>> +
>> +if [ $SKIP_DWARF -eq 1 ]; then
>> +	print_testcase_skipped "adding blacklisted function $BLACKFUNC"
>> +else
>> +	print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "adding blacklisted function $BLACKFUNC"
>> +	(( TEST_RESULT += $? ))
>> +fi
>>  
>>  ### listing not-added probe
>>  
>> -- 
>> 2.43.0
>>
> 

Powered by blists - more mailing lists