linux-kernel - Re: BUG: ib_mad ftrace event unsupported migration

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 2 Nov 2022 07:44:57 -0400
From:   Steven Rostedt <rostedt@...dmis.org>
To:     Leonid Ravich <leonid.ravich@...anetworks.com>
Cc:     "linux-trace-kernel@...r.kernel.org" 
        <IMCEAMAILTO-linux-trace-kernel+40vger+2Ekernel+2Eorg@...prd02.prod.outlook.com>,
        "mingo@...hat.com" <mingo@...hat.com>,
        "linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        Yigal Korman <yigal.korman@...anetworks.com>
Subject: Re: BUG:  ib_mad ftrace event unsupported migration

On Wed, 2 Nov 2022 10:49:26 +0000
Leonid Ravich <leonid.ravich@...anetworks.com> wrote:

> after investigation  looks like there is a broken assumption of rb_end_commit and rb_start_commit are on same CPU (no migration) during trace 
> looking in ib_mad trace  in include/trace/events/ib_mad.h there is a call to create_mad_addr_info during  TP assign which sleeps (at least under mlx5 ib driver)

Wait, WHAT?

> 
> so the scenario looks :
> 
> rb_start_commit - buffer 0xffffa0984777e400  CPU 1 pid 1368
> switch from 1368 to 1605
> thread 1368 () is migrating from 1 to 0
> rb_end_commit - buffer 0xffffa09847c14a00  CPU 0 pid 1368

That's a MAJOR bug with the use case of tracepoints!

> 
> before starting throwing some patch into the the air  I would like to align with you the approach we should take here. 
> 
> my suggestion here : 
> - ftrace infra should verify no migration happen  (end and start happens on same CPU)  in case not we will  throw warning for the issue  .

The scheduler should have. On entering the ring buffer code
ring_buffer_lock_reserver() it disables preemption and does not
re-enable it until ring_buffer_unlock_commit().

The only way to migrate is if you re-enable preemption. WHICH IS A BUG!

> - ftrace users will be responsible to avoid migration during TP assign (RDMA in my case)

Yes, it is responsible to do that. Just like it's up to users of
spin_locks to not enable preemption and migrate while holding one.
Entering the ring buffer code is the same as holding a spin_lock except
that the ring buffer code allows for reentrancy.

> 
> please let me know what do you think 

I think that whatever enabled preemption in the ib_mad trace event
needs to be fixed.

-- Steve