linux-kernel - Re: [PATCH 0/2] genirq/affinity: try to make sure online CPU is assgined to irq vector

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20180201103651.GA29263@ming.t460p>
Date:   Thu, 1 Feb 2018 18:36:57 +0800
From:   Ming Lei <ming.lei@...hat.com>
To:     Don Brace <don.brace@...rosemi.com>
Cc:     Laurence Oberman <loberman@...hat.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        Christoph Hellwig <hch@...radead.org>,
        Jens Axboe <axboe@...com>,
        "linux-block@...r.kernel.org" <linux-block@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        Mike Snitzer <snitzer@...hat.com>
Subject: Re: [PATCH 0/2] genirq/affinity: try to make sure online CPU is
 assgined to irq vector

On Tue, Jan 16, 2018 at 03:22:18PM +0000, Don Brace wrote:
> > -----Original Message-----
> > From: Laurence Oberman [mailto:loberman@...hat.com]
> > Sent: Tuesday, January 16, 2018 7:29 AM
> > To: Thomas Gleixner <tglx@...utronix.de>; Ming Lei <ming.lei@...hat.com>
> > Cc: Christoph Hellwig <hch@...radead.org>; Jens Axboe <axboe@...com>;
> > linux-block@...r.kernel.org; linux-kernel@...r.kernel.org; Mike Snitzer
> > <snitzer@...hat.com>; Don Brace <don.brace@...rosemi.com>
> > Subject: Re: [PATCH 0/2] genirq/affinity: try to make sure online CPU is assgined
> > to irq vector
> > 
> > > > It is because of irq_create_affinity_masks().
> > >
> > > That still does not answer the question. If the interrupt for a queue
> > > is
> > > assigned to an offline CPU, then the queue should not be used and
> > > never
> > > raise an interrupt. That's how managed interrupts have been designed.
> > >
> > > Thanks,
> > >
> > >       tglx
> > >
> > >
> > >
> > >
> > 
> > I captured a full boot log for this issue for Microsemi, I will send it
> > to Don Brace.
> > I enabled all the HPSA debug and here is snippet
> > 
> > 
> > ..
> > ..
> > ..
> >   246.751135] INFO: task systemd-udevd:413 blocked for more than 120
> > seconds.
> > [  246.788008]       Tainted: G          I      4.15.0-rc4.noming+ #1
> > [  246.822380] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > disables this message.
> > [  246.865594] systemd-udevd   D    0   413    411 0x80000004
> > [  246.895519] Call Trace:
> > [  246.909713]  ? __schedule+0x340/0xc20
> > [  246.930236]  schedule+0x32/0x80
> > [  246.947905]  schedule_timeout+0x23d/0x450
> > [  246.970047]  ? find_held_lock+0x2d/0x90
> > [  246.991774]  ? wait_for_completion_io+0x108/0x170
> > [  247.018172]  io_schedule_timeout+0x19/0x40
> > [  247.041208]  wait_for_completion_io+0x110/0x170
> > [  247.067326]  ? wake_up_q+0x70/0x70
> > [  247.086801]  hpsa_scsi_do_simple_cmd+0xc6/0x100 [hpsa]
> > [  247.114315]  hpsa_scsi_do_simple_cmd_with_retry+0xb7/0x1c0 [hpsa]
> > [  247.146629]  hpsa_scsi_do_inquiry+0x73/0xd0 [hpsa]
> > [  247.174118]  hpsa_init_one+0x12cb/0x1a59 [hpsa]
> 
> This trace comes from internally generated discovery commands. No SCSI devices have
> been presented to the SML yet.
> 
> At this point we should be running on only one CPU. These commands are meant to use
> reply queue 0 which are tied to CPU 0. It's interesting that the patch helps.
> 
> However, I was wondering if you could inspect the iLo IML logs and send the
> AHS logs for inspection.

Hello Don,

Now the patch has been merged to linus tree as:

84676c1f21e8ff54b ("genirq/affinity: assign vectors to all possible CPUs")

and it breaks Laurence's machine completely, :-(

I just take a look at HPSA's code, and found that reply queue is chosen
in the following way in most of code path:

        if (likely(reply_queue == DEFAULT_REPLY_QUEUE))
                cp->ReplyQueue = smp_processor_id() % h->nreply_queues;

h->nreply_queues is the msix vector number which is returned from
pci_alloc_irq_vectors(), and now some of vectors may be mapped to all
offline CPUs, for example, one processor isn't plugged to socket.

If I understand correctly, 'cp->ReplyQueue' is aligned to one irq
vector, and the command is expected by handled via that irq vector,
is it right?

If yes, now I guess this way can't work any more if number of online
CPUs is >= h->nreply_queues, and you may need to check the cpu affinity
of one vector before choosing the reply queue, and block/blk-mq-pci.c
may be helpful for you.

Thanks,
Ming