linux-kernel - Re: [PATCHv2] perf/x86/intel: Account interrupts for PEBS errors

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <alpine.DEB.2.11.1612152033420.15312@pianoman.cluster.toy>
Date:   Thu, 15 Dec 2016 20:37:46 -0500 (EST)
From:   Vince Weaver <vince@...ter.net>
To:     Jiri Olsa <jolsa@...hat.com>
cc:     Peter Zijlstra <peterz@...radead.org>,
        Andi Kleen <andi@...stfloor.org>,
        lkml <linux-kernel@...r.kernel.org>,
        Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
        Ingo Molnar <mingo@...nel.org>
Subject: Re: [PATCHv2] perf/x86/intel: Account interrupts for PEBS errors

On Thu, 15 Dec 2016, Jiri Olsa wrote:

> It's possible to setup PEBS events and get only errors and not
> a single data, like on SNB-X (model 45) and IVB-EP (model 62)
> via 2 perf commands running simultaneously:
> 
>     taskset -c 1 ./perf record -c 4 -e branches:pp -j any -C 10
> 
> This leads to soft lock up, because the error path of the
> intel_pmu_drain_pebs_nhm does not account event->hw.interrupt
> for error PEBS interrupts so the event is not eventually
> stopped when it gets over the max_samples_per_tick limit.
> 
>   NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [perf_fuzzer:5816]
>   ...
>   task: ffff880273148000 task.stack: ffffc90002d58000
>   RIP: 0010:[<ffffffff81159232>]  [<ffffffff81159232>] smp_call_function_single+0xe2/0x140
>   RSP: 0018:ffffc90002d5bd60  EFLAGS: 00000202
>   ...
>   Call Trace:
>    ? trace_hardirqs_on_caller+0xf5/0x1b0
>    ? perf_cgroup_attach+0x70/0x70
>    perf_install_in_context+0x199/0x1b0
>    ? ctx_resched+0x90/0x90
>    SYSC_perf_event_open+0x641/0xf90
>    SyS_perf_event_open+0x9/0x10
>    do_syscall_64+0x6c/0x1f0
>    entry_SYSCALL64_slow_path+0x25/0x25

I'll have to try this, with all the recent fixes I am down to NMI lockups 
like this being the major cause of fuzzer issues on my intel machines.

My AMD and ARM machines are now fuzzing for weeks w/o problems.

I also finally got a power8 machine and it crashes really quickly when 
fuzzing, but I haven't had a chance to track dthings own yet because it 
sounds like a jet plane taking off and I can't really leave it fuzzing 
like that when students are sitting nearby.  Maybe over break.

Vince