Message-ID: <55B88A1D.2000601@cp.jp.nec.com>
Date:	Wed, 29 Jul 2015 08:09:01 +0000
From:	Taichi Kageyama <t-kageyama@...jp.nec.com>
To:	Peter Hurley <peter@...leysoftware.com>
CC:	Prarit Bhargava <prarit@...hat.com>,
	"gregkh@...uxfoundation.org" <gregkh@...uxfoundation.org>,
	"linux-serial@...r.kernel.org" <linux-serial@...r.kernel.org>,
	"jslaby@...e.cz" <jslaby@...e.cz>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"Naoya Horiguchi" <n-horiguchi@...jp.nec.com>
Subject: Re: [PATCH 2/2] serial: 8250: Allow to skip autoconfig_irq() for a
 console

On 2015/07/28 4:44, Peter Hurley wrote:
> On 07/21/2015 05:44 AM, Taichi Kageyama wrote:
>> On 2015/07/21 1:36, Peter Hurley wrote:
>>> On 07/16/2015 05:58 AM, Taichi Kageyama wrote:
>>>> On 2015/07/15 4:29, Peter Hurley wrote:
>>>>> On 07/13/2015 09:16 PM, Taichi Kageyama wrote:
>>>>>> On 2015/07/11 9:12, Peter Hurley wrote:
>>>>>>> On 07/09/2015 01:32 AM, Taichi Kageyama wrote:
>>>>>>>> On 2015/07/08 23:00, Prarit Bhargava wrote:
>>>>>>>>> On 07/08/2015 09:51 AM, Peter Hurley wrote:
>>>>>>>>>> On 07/08/2015 08:53 AM, Prarit Bhargava wrote:
>>>>>>>>>>> On 07/08/2015 07:55 AM, Peter Hurley wrote:
>>>>>>>>>>>> On 06/05/2015 06:03 AM, Taichi Kageyama wrote:
>>>>>>>>>>>>> This patch provides a new parameter as a workaround of the following
>>>>>>>>>>>>> problem. It allows us to skip autoconfig_irq() and to use a well-known irq
>>>>>>>>>>>>> number for a console even if CONFIG_SERIAL_8250_DETECT_IRQ is defined.
>>>>>>>>>>>>>
>>>>>>>>>>>>> There're cases where autoconfig_irq() fails during boot.
>>>>>>>>>>>>> In these cases, the console doesn't work in interrupt mode,
>>>>>>>>>>>>> the mode cannot be changed anymore, and "input overrun"
>>>>>>>>>>>>> (which can make operation mistakes) happens easily.
>>>>>>>>>>>>> This problem happens with high rate every boot once it occurs
>>>>>>>>>>>>> because the boot sequence is always almost same.
>>>>>>>>>>>>>
>>>>>>>>>>>>> autoconfig_irq() assumes that a CPU can handle an interrupt from a serial
>>>>>>>>>>>>> during the waiting time, but there're some cases where the CPU cannot
>>>>>>>>>>>>> handle the interrupt for longer than the time. It completely depends on
>>>>>>>>>>>>> how other functions work on the CPU. Ideally, autoconfig_irq() should be
>>>>>>>>>>>>> fixed
>>>>>>>>>>>>
>>>>>>>>
>>>>>>>> Thank you for your comments.
>>>>>>>>
>>>>>>>>>>>> It completely depends on how long some other driver has interrupts disabled,
>>>>>>>>
>>>>>>>> Agree.
>>>>>>>>
>>>>>>>>>>>> which is a problem that needs fixed _in that driver_. autoconfig_irq() does not
>>>>>>>>>>>> need fixing.
>>>>>>>>
>>>>>>>> Peter, ideally, you're right.
>>>>>>>> However, we cannot assume how long other drivers disable interrupts.
>>>>>>>> That's why I introduced this workaround.
>>>>>>>> In my opinion, a console is important and always should be available
>>>>>>>> even if other drivers have a bad behavior.
>>>>>>>
>>>>>>> I have no problem with wanting to make the console more robust, but
>>>>>>> rather with the hacky way this is being done.
>>>>>>
>>>>>> Hi Peter,
>>>>>>
>>>>>> Thank you for your advice.
>>>>>> If there is other way to fix this problem simply,
>>>>>> I also think it's better than the dirty hack.
>>>>>
>>>>> While module parameters seem like "simple" solutions at the time,
>>>>> they add real maintenance burden, because they establish userspace
>>>>> requirements that must be preserved forever to avoid breakage.
>>>>
>>>> Yeah, I agree with you.
>>>>
>>>>>>> Better solutions:
>>>>>>> 1. Fix autoprobing to force irq affinity to autoprobing cpu
>>>>>>
>>>>>> I couldn't make sure which CPU handled serial interrupt
>>>>>> on all platforms before irq# was not known.
>>>>>> Do you know the way to detect which CPU is used for console serial?
>>>>>
>>>>>
>>>>> The basic idea would be:
>>>>> 1. disable preemption
>>>>> 2. for each irq descriptor selected for autoprobing, set the irq
>>>>>      affinity to the current processor.
>>>>> 3. probe the i/o port as is done now
>>>>> 4. stop probing
>>>>> 5. re-enable preemption.
>>>>
>>>> Thanks, I think it works.
>>>>
>>>>> With this solution, your patch 1/2 wouldn't be required either
>>>>> because the worker thread that disabled interrupts wouldn't be
>>>>> running on the cpu detecting the triggered irq(s).
>>>>
>>>> I still need my patch 1/2 which fixes also other cases (see case2 & 3).
>>>> I think both port->lock and console_lock are required in your solution.
>>>> to avoid deadlock because printk() can be called on every context.
>>>>
>>>>> I would imagine most or all of this would be done in
>>>>> probe_irq_on(), possibly refactored to perform the preemption
>>>>> disable and irq affinity.
>>>>
>>>> I think introducing new function like "probe_irq_set_affinity()" is better
>>>> than modifying probe_irq_on(). I cannot test all legacy devices and
>>>> I don't have any reason to break the code which works for other devices.
>>>
>>> That's fine, although most of the arguments for fixing this in the serial
>>> driver apply equally to other users of probe_irq_on().
>>>
>>>
>>>>>> The way is safe for all platforms?
>>>>>
>>>>> Please understand though, autoprobing is not safe, period.
>>>>> Even says so in Kconfig.
>>>>
>>>> OK, I'll try to create new patch which makes autoprobing safer as possible.
>>>> New patch is going to be like below.
>>>>    1. console and port lock
>>>>    2. probe_irq_on()
>>>>    3. probe_irq_set_affinity(&cpumask)
>>>>    4. probe_irq_off()
>>>>    5. port and console unlock
>>>
>>> The port->lock can't be taken in this context because hard irq
>>> has to be disabled with port->lock which defeats the purpose of
>>> pinning the irq affinity to the current cpu.
>>
>> My test code uses spin_lock() instead of spin_lock_irqsave().
>>
>>> What are you concerned about being concurrent with autoconfig_irq()?
>>> Many operations are excluded by the port->mutex.
>>
>> Actually I don't have any concerns as long as console_lock() is used,
>> but I thought protecting port was better during auto_irq
>> or register operations as same as autoconfig().
>>
>> I was thinking they are used as the following purposes;
>>    console_lock()
>>     + Make sure serial8250_console_write() doesn't disable interrupt,
>>       try to get port->lock or touch the ctrl register of the port.
>>       # serial8250_console_write() can be called in any context.
>>    spin_lock()
>>     + Make sure the probing runs on the current CPU only
>>       to handle a serial irq by itself after setting irq affinity.
>>     + Make sure any other CPUs don't touch the ctrl register of the port.
>>
>> It seems my test code has been working fine so far,
>> but let me know if you have any concerns about using spin_lock()
>> instead of preempt_disable().
> 
> If you turn on lockdep, taking the port->lock without disabling irq will
> assert.

You're right. 
I've changed my code, thank you!

I actually have tested preempt_disable version too.
It has been working on my machine, but I've found it doesn't work on KVM with v4.1.
Now I'm lost about how irq affinity should be set during interrupt probing phase...
I guess more fixes are required to achieve this. 
I'll send v2 as RFC.

> A quick static analysis shows autoconfig_irq() reachable via 2 different
> call trees:
> 
> [1]  uart_add_one_port()
>         lock global port_mutex (to prevent concurrent port add/remove)
>         lock port->mutex
>         uart_configure_port()
>            ops->config_port => serial8250_config_port()
>               autoconfig_irq()
> 
> [2]  ioctl(TIOCSERCONFIG)
>         uart_do_autoconfig()
>            lock port->mutex
>            uart_shutdown()
>            ops->config_port => serial8250_config_port()
>               autoconfig_irq()
> 
> Call tree #1 cannot execute concurrently with any other driver function
> because the tty device doesn't even exist at that time.
> 
> ioctl(TIOCSERCONFIG) -- call tree #2 -- is pretty much a hack and tries
> to do its best to prevent concurrent driver function/hardware access.
> So it takes the port->mutex which prevents many concurrent operations,
> and shuts down the port hardware. In other words, the autoconfig operation
> is intended to be exclusive of any other concurrent hardware access (except
> console).
> 
> I say 'intended' because this is broken if the line discipline is echoing;
> I just fixed this in uart_close() and now realize it's possible wherever
> uart_shutdown() is called -- so I need to fix that harder. But my point is
> that no other lock should not be necessary.
> 
> Please feel free to double-check my work.

Thank you for the details. I understand. 
I've updated commit log.


Thanks,
Taichi



> 
> Regards,
> Peter Hurley
> 
> PS -I attached a catalog of concurrent operations excluded by port->mutex.
> 

Powered by blists - more mailing lists