| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20241115060542.lqpu3sdqnsxass6q@basti-XPS-13-9310>
Date: Fri, 15 Nov 2024 07:05:42 +0100
From: Sebastian Fricke <sebastian.fricke@...labora.com>
To: Randy Dunlap <rdunlap@...radead.org>
Cc: Jonathan Corbet <corbet@....net>, bagasdotme@...il.com,
linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org,
linux-media@...r.kernel.org, laurent.pinchart@...asonboard.com,
hverkuil-cisco@...all.nl, mchehab@...nel.org, kernel@...labora.com,
bob.beckett@...labora.com, nicolas.dufresne@...labora.com
Subject: Re: [PATCH v2 1/2] docs: Add debugging section to process
Hey Randy,
Thank you for the review, I'll apply these changes, I just found one
advice below not terribly helpful maybe you can clarify ...
On 14.11.2024 21:06, Randy Dunlap wrote:
>
>
>On 11/13/24 3:17 AM, Sebastian Fricke wrote:
>> This idea was formed after noticing that new developers experience
>> certain difficulty to navigate within the multitude of different
>> debugging options in the Kernel and while there often is good
>> documentation for the tools, the developer has to know first that they
>> exist and where to find them.
>> Add a general debugging section to the Kernel documentation, as an
>> easily locatable entry point to other documentation and as a general
>> guideline for the topic.
>>
>> Signed-off-by: Sebastian Fricke <sebastian.fricke@...labora.com>
>> ---
>> .../driver_development_debugging_guide.rst | 214 ++++++++++++++++
>> Documentation/process/debugging/index.rst | 65 +++++
>> .../debugging/userspace_debugging_guide.rst | 278 +++++++++++++++++++++
>> Documentation/process/index.rst | 8 +-
>> 4 files changed, 562 insertions(+), 3 deletions(-)
>>
>
>
>> diff --git a/Documentation/process/debugging/userspace_debugging_guide.rst b/Documentation/process/debugging/userspace_debugging_guide.rst
>> new file mode 100644
>> index 000000000000..a7c94407bcae
>> --- /dev/null
>> +++ b/Documentation/process/debugging/userspace_debugging_guide.rst
>> @@ -0,0 +1,278 @@
>> +.. SPDX-License-Identifier: GPL-2.0
>> +
>> +==========================
>> +Userspace debugging advice
>> +==========================
>> +
>> +A brief overview of common tools to debug the Linux Kernel from userspace.
>
>Make that a sentence?
Can you clarify this?
This could either mean:
- What you even bother to make a sentence out of that?
- Please make a proper sentence out of this because it is hard to
understand
- Please go into more detail because this is too brief
Or maybe something completely different :)
Regards,
Sebastian
>
>> +For debugging advice aimed at driver developer go :doc:`here
>> +</process/debugging/driver_development_debugging_guide>`.
>> +For general debugging advice, see :doc:`general advice document
>> +</process/debugging/index>`.
>> +
>> +.. contents::
>> + :depth: 3
>> +
>> +The following sections show you the available tools.
>> +
>> +Dynamic debug
>> +-------------
>> +
>> +Mechanism to filter what ends up in the kernel log by dis-/en-abling log
>> +messages.
>> +
>> +Prerequisite: ``CONFIG_DYNAMIC_DEBUG``
>> +
>> +Dynamic debug is only able to target:
>> +
>> +- pr_debug()
>> +- dev_dbg()
>> +- print_hex_dump_debug()
>> +- print_hex_dump_bytes()
>> +
>> +Therefore the usability of this tool is, as of now, quite limited as there is
>> +no uniform rule for adding debug prints to the codebase, resulting in a variety
>> +of ways these prints are implemented.
>> +
>> +Also, note that most debug statements are implemented as a variation of
>> +dprintk(), which have to be activated via a parameter in respective module,
>
> in the respective module;
>
>> +dynamic debug is unable to do that step for you.
>> +
>> +Here is one example, that enables all available pr_debug() 's within the file::
>
> no space ^
>
>> +
>> + $ alias ddcmd='echo $* > /proc/dynamic_debug/control'
>> + $ ddcmd '-p; file v4l2-h264.c +p'
>> + $ grep =p /proc/dynamic_debug/control
>> + drivers/media/v4l2-core/v4l2-h264.c:372 [v4l2_h264]print_ref_list_b =p
>> + "ref_pic_list_b%u (cur_poc %u%c) %s"
>> + drivers/media/v4l2-core/v4l2-h264.c:333 [v4l2_h264]print_ref_list_p =p
>> + "ref_pic_list_p (cur_poc %u%c) %s\n"
>> +
>> +**When should you use this over Ftrace ?**
>> +
>> +- When the code contains one of the valid print statements (see above) or when
>> + you have added multiple pr_debug() statements during development
>> +- When timing is not an issue, meaning if multiple pr_debug() statements in
>> + the code won't cause delays
>> +- When you care more about receiving specific log messages than tracing the
>> + pattern of how a function is called
>> +
>> +For the full documentation see :doc:`/admin-guide/dynamic-debug-howto`
>> +
>> +Ftrace
>> +------
>> +
>> +Prerequisite: ``CONFIG_DYNAMIC_FTRACE``
>> +
>> +This tool uses the tracefs file system for the control files and output files,
>
> files.
>
>> +that file system will be mounted as a ``tracing`` folder, which can be found in
>
> That
>
>> +either ``/sys/kernel/`` or ``/sys/debug/kernel/``.
>> +
>> +Some of the most important operations for debugging are:
>> +
>> +- You can perform a function trace by adding a function name to the
>> + ``set_ftrace_filter`` file (which accepts any function name found within the
>> + ``available_filter_functions`` file) or you can specifically disable certain
>> + functions by adding their names to the ``set_ftrace_notrace`` file (More info
>
> more
>
>> + at: :ref:`trace/ftrace:dynamic ftrace`).
>> +- In order to find out where the calls originates from you can activate the
>
> where calls originate from
>
>> + ``func_stack_trace`` option under ``options/func_stack_trace``.
>> +- Tracing the children of a function call and showing the return values is
>
> are
>
>> + possible by adding the desired function in the ``set_graph_function`` file
>> + (requires config ``FUNCTION_GRAPH_RETVAL``) more info at
>
> );
>
>> + :ref:`trace/ftrace:dynamic ftrace with the function graph tracer`.
>> +
>> +For the full Ftrace documentation see :doc:`/trace/ftrace`
>> +
>> +Or you could also trace for specific events by :ref:`using event tracing
>> +<trace/events:2. using event tracing>`, which can be defined as described here:
>> +:ref:`Creating a custom Ftrace tracepoint
>> +<process/debugging/driver_development_debugging_guide:ftrace>`.
>> +
>> +For the full Ftrace event tracing documentation see :doc:`/trace/events`
>> +
>> +.. _read_ftrace_log:
>> +
>> +Reading the ftrace log
>> +~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +The ``trace`` file can be read just like any other file (``cat``, ``tail``, ``head``,
>> +``vim``, etc.), the size of the file is limited by the ``buffer_size_kb`` (``echo
>> +1000 > buffer_size_kb``). The :ref:`trace/ftrace:trace_pipe` will behave
>> +similar to the ``trace`` file, but whenever you read from the file the content is
>
> similarly
>IMO but not a big deal.
>
>> +consumed.
>> +
>> +Kernelshark
>> +~~~~~~~~~~~
>> +
>> +A GUI interface to visualize the traces as a graph and list view from the
>> +output of the `trace-cmd
>> +<https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/>`__ application.
>> +
>> +For the full documentation see `<https://kernelshark.org/Documentation.html>`__
>> +
>> +Perf & alternatives
>> +-------------------
>> +
>> +The tools mentioned above provide ways to inspect kernel code, results, variable values, etc.
>> +Sometimes you have to find out first where to look and for those cases, a box of
>> +performance tracking tools can help you to frame the issue.
>> +
>> +Why should you do a performance analysis?
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +A performance analysis is a good first step when among other reasons:
>> +
>> +- you cannot define the issue
>> +- you do not know where it occurs
>> +- the running system should not be interrupted or it is a remote system, where
>> + you cannot install a new module/kernel
>> +
>> +How to do a simple analysis with linux tools?
>
> Linux
>
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +For the start of a performance analysis, you can start with the usual tools
>> +like:
>> +
>> +- ``top`` / ``htop`` / ``atop`` (*get an overview of the system load, see spikes on
>> + specific processes*)
>> +- ``mpstat -P ALL`` (*look at the load distribution among CPUs*)
>> +- ``iostat -x`` (*observe input and output devices utilization and performance*)
>> +- ``vmstat`` (*overview of memory usage on the system*)
>> +- ``pidstat`` (*similar to* ``vmstat`` *but per process, to dial it down to the
>> + target*)
>> +- ``strace -tp $PID`` (*once you know the process, you can figure out how it
>> + communicates with the Kernel*)
>> +
>> +These should help to narrow down the areas to look at sufficiently.
>> +
>> +Diving deeper with perf
>> +~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +The **perf** tool provides a series of metrics and events to further dial down
>> +on issues.
>> +
>> +Prerequisite: build or install perf on your system
>> +
>> +Gather statistics data for finding all files starting with ``gcc`` in ``/usr``::
>> +
>> + # perf stat -d find /usr -name 'gcc*' | wc -l
>> +
>> + Performance counter stats for 'find /usr -name gcc*':
>> +
>> + 1277.81 msec task-clock # 0.997 CPUs utilized
>> + 9 context-switches # 7.043 /sec
>> + 1 cpu-migrations # 0.783 /sec
>> + 704 page-faults # 550.943 /sec
>> + 766548897 cycles # 0.600 GHz (97.15%)
>> + 798285467 instructions # 1.04 insn per cycle (97.15%)
>> + 57582731 branches # 45.064 M/sec (2.85%)
>> + 3842573 branch-misses # 6.67% of all branches (97.15%)
>> + 281616097 L1-dcache-loads # 220.390 M/sec (97.15%)
>> + 4220975 L1-dcache-load-misses # 1.50% of all L1-dcache accesses (97.15%)
>> + <not supported> LLC-loads
>> + <not supported> LLC-load-misses
>> +
>> + 1.281746009 seconds time elapsed
>> +
>> + 0.508796000 seconds user
>> + 0.773209000 seconds sys
>> +
>> +
>> + 52
>> +
>> +The availability of events and metrics depends on the system you are running.
>> +
>> +For the full documentation see
>> +`<https://perf.wiki.kernel.org/index.php/Main_Page>`__
>> +
>> +Perfetto
>> +~~~~~~~~
>> +
>> +A set of tools to measure and analyze how well applications and systems perform.
>> +You can use it to:
>> +
>> +* identify bottlenecks
>> +* optimize code
>> +* make software run faster and more efficiently.
>> +
>> +**What is the difference between perfetto and perf?**
>> +
>> +* perf is tool as part of and specialized for the Linux Kernel and has CLI user
>> + interface.
>> +* perfetto cross-platform performance analysis stack, has extended
>> + functionality into userspace and provides a WEB user interface.
>> +
>> +For the full documentation see `<https://perfetto.dev/docs/>`__
>
>config PSI
> bool "Pressure stall information tracking"
>might also be useful here.
>
>> +
>> +Kernel panic analysis tools
>> +---------------------------
>> +
>> + To capture the crash dump please use ``Kdump`` & ``Kexec``. Below you can find
>> + some advice for analysing the data.
>> +
>> + For the full documentation see the :doc:`/admin-guide/kdump/kdump`
>> +
>> + In order to find the corresponding line in the code you can use `faddr2line
>> + <https://elixir.bootlin.com/linux/v6.11.6/source/scripts/faddr2line>`__, note
>
> ; note
>
>> + that you need to enable ``CONFIG_DEBUG_INFO`` for that to work.
>> +
>> + An alternative to using ``faddr2line`` is the use of ``objdump`` (and it's
>
> its
>
>> + derivatives for the different platforms like ``aarch64-linux-gnu-objdump``),
>
> ).
>
>> + take this line as an example:
>
> Take
>
>> +
>> + ``[ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec]``.
>> +
>> + We can find the corresponding line of code by executing::
>> +
>> + aarch64-linux-gnu-objdump -dS drivers/staging/media/rkvdec/rockchip-vdec.ko | grep rkvdec_device_run\>: -A 40
>> + 0000000000000ac8 <rkvdec_device_run>:
>> + ac8: d503201f nop
>> + acc: d503201f nop
>> + {
>> + ad0: d503233f paciasp
>> + ad4: a9bd7bfd stp x29, x30, [sp, #-48]!
>> + ad8: 910003fd mov x29, sp
>> + adc: a90153f3 stp x19, x20, [sp, #16]
>> + ae0: a9025bf5 stp x21, x22, [sp, #32]
>> + const struct rkvdec_coded_fmt_desc *desc = ctx->coded_fmt_desc;
>> + ae4: f9411814 ldr x20, [x0, #560]
>> + struct rkvdec_dev *rkvdec = ctx->dev;
>> + ae8: f9418015 ldr x21, [x0, #768]
>> + if (WARN_ON(!desc))
>> + aec: b4000654 cbz x20, bb4 <rkvdec_device_run+0xec>
>> + ret = pm_runtime_resume_and_get(rkvdec->dev);
>> + af0: f943d2b6 ldr x22, [x21, #1952]
>> + ret = __pm_runtime_resume(dev, RPM_GET_PUT);
>> + af4: aa0003f3 mov x19, x0
>> + af8: 52800081 mov w1, #0x4 // #4
>> + afc: aa1603e0 mov x0, x22
>> + b00: 94000000 bl 0 <__pm_runtime_resume>
>> + if (ret < 0) {
>> + b04: 37f80340 tbnz w0, #31, b6c <rkvdec_device_run+0xa4>
>> + dev_warn(rkvdec->dev, "Not good\n");
>> + b08: f943d2a0 ldr x0, [x21, #1952]
>> + b0c: 90000001 adrp x1, 0 <rkvdec_try_ctrl-0x8>
>> + b10: 91000021 add x1, x1, #0x0
>> + b14: 94000000 bl 0 <_dev_warn>
>> + *bad = 1;
>> + b18: d2800001 mov x1, #0x0 // #0
>> + ...
>> +
>> + Meaning, in this line from the crash dump::
>> +
>> + [ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec]
>> +
>> + I can take the ``0x50`` as offset, which I have to add to the base address
>> + of the corresponding function, which I find in this line::
>> +
>> + 0000000000000ac8 <rkvdec_device_run>:
>> +
>> + The result of ``0xac8 + 0x50 = 0xb18``
>> + And when I search for that address within the function I get the
>> + following line::
>> +
>> + *bad = 1;
>> + b18: d2800001 mov x1, #0x0
>> +
>> +**Copyright** ©2024 : Collabora
>> diff --git a/Documentation/process/index.rst b/Documentation/process/index.rst
>> index 6455eba3ef0c..aa12f2660194 100644
>> --- a/Documentation/process/index.rst
>> +++ b/Documentation/process/index.rst
>> @@ -72,13 +72,15 @@ beyond).
>> Dealing with bugs
>> -----------------
>>
>> -Bugs are a fact of life; it is important that we handle them properly.
>> -The documents below describe our policies around the handling of a couple
>> -of special classes of bugs: regressions and security problems.
>> +Bugs are a fact of life; it is important that we handle them properly. The
>> +documents below provide general advice about debugging and describe our
>> +policies around the handling of a couple of special classes of bugs:
>> +regressions and security problems.
>>
>> .. toctree::
>> :maxdepth: 1
>>
>> + debugging/index
>> handling-regressions
>> security-bugs
>> cve
>>
>
>Thanks.
>
>--
>~Randy
>
>
Powered by blists - more mailing lists