netdev - Re: [PATCH][v3] rtnetlink: add rtnl

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20210512083913.733f9803@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
Date:   Wed, 12 May 2021 08:39:13 -0700
From:   Jakub Kicinski <kuba@...nel.org>
To:     Rocco yue <rocco.yue@...iatek.com>
Cc:     "David S . Miller" <davem@...emloft.net>,
        Matthias Brugger <matthias.bgg@...il.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Masahiro Yamada <masahiroy@...nel.org>,
        Nick Desaulniers <ndesaulniers@...gle.com>,
        Andy Shevchenko <andriy.shevchenko@...ux.intel.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Tetsuo Handa <penguin-kernel@...ove.SAKURA.ne.jp>,
        <linux-kernel@...r.kernel.org>, <netdev@...r.kernel.org>,
        <linux-arm-kernel@...ts.infradead.org>,
        <linux-mediatek@...ts.infradead.org>, <wsd_upstream@...iatek.com>
Subject: Re: [PATCH][v3] rtnetlink: add rtnl_lock debug log

On Tue, 11 May 2021 19:32:57 +0800 Rocco yue wrote:
> We often encounter system hangs caused by certain process
> holding rtnl_lock for a long time. Even if there is a lock
> detection mechanism in Linux, it is a bit troublesome and
> affects the system performance. We hope to add a lightweight
> debugging mechanism for detecting rtnl_lock.
> 
> Up to now, we have discovered and solved some potential bugs
> through this lightweight rtnl_lock debugging mechanism, which
> is helpful for us.
> 
> When you say Y for RTNL_LOCK_DEBUG, then the kernel will
> detect if any function hold rtnl_lock too long and some key
> information will be printed out to help locate the problem.
> 
> i.e: from the following logs, we can clearly know that the
> pid=2206 RfxSender_4 process holds rtnl_lock for a long time,
> causing the system to hang. And we can also speculate that the
> delay operation may be performed in devinet_ioctl(), resulting
> in rtnl_lock was not released in time.

You can achieve that with a pair of fexit/fentry hooks or kprobes,
and maybe a bit of BPF. No need for config options, and hardcoded
parameters..