Date:	Tue, 05 Oct 2010 09:08:47 -0700
From:	John Fastabend <john.r.fastabend@...el.com>
To:	Eric Dumazet <eric.dumazet@...il.com>
CC:	"bhutchings@...arflare.com" <bhutchings@...arflare.com>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	"therbert@...gle.com" <therbert@...gle.com>
Subject: Re: [net-next-2.6 PATCH] net: netif_set_real_num_rx_queues may cap
 num_rx_queues at init time

On 10/4/2010 10:35 PM, Eric Dumazet wrote:
> Le lundi 04 octobre 2010 à 15:00 -0700, John Fastabend a écrit :
>> The logic for netif_set_real_num_rx_queues is the following,
>>
>> netif_set_real_num_rx_queues(dev, rxq)
>> {
>> 	...
>> 	if (dev->reg_state == NETREG_REGISTERED) {
>> 		...
>> 	} else {
>> 		dev->num_rx_queues = rxq;
>> 	}
>>
>> 	dev->real_num_rx_queues = rxq;
>> 	return 0;
>> }
>>
>> Some drivers init path looks like the following,
>>
>> alloc_etherdev_mq(priv_sz, max_num_queues_ever);
>> ...
>> netif_set_real_num_rx_queues(dev, queues_to_use_now);
>> ...
>> register_netdev(dev);
>> ...
>>
>> Because netif_set_real_num_rx_queues sets num_rx_queues if the
>> reg state is not NETREG_REGISTERED we end up with the incorrect
>> max number of rx queues. This patch proposes to remove the else
>> clause above so this does not occur.  Also just reading the
>> function set_real_num it seems a bit unexpected that num_rx_queues
>> gets set.
>>
> 
> You dont tell why its "incorrect".
> 

OK that is a poor description.

> Why should we keep num_rx_queues > real_num_rx_queues ?
> 

If we do not ever need them then we should not keep them I agree.
But having netif_set_real_num_rx_queues set something other then
'real_num_rx_queues' does not seem right to me at least. Also
netif_set_real_num_tx_queues and netif_set_real_num_rx_queues have
different behavior. It would be nice if this weren't the case but
they allocate queues in two places.

The drivers Ben modified seem to be split between these two flows

alloc_etherdev_mq(priv_sz, max_num_queues_ever);
register_netdev(dev);
netif_set_real_num_rx_queues(dev, queues_to_use_now);

and

alloc_etherdev_mq(priv_sz, max_num_queues_ever);
netif_set_real_num_rx_queues(dev, queues_to_use_now);
register_netdev(dev);

In the first case we never set 'num_rx_queues' because its after
the register so that leaves the second case. All the remaining
drivers except ixgbe, igb, and myri10ge know or could easily find
out the number of rx queues at alloc_etherdev_mq and are really trying
to explicitly set the num_rx_queues. Adding a num_rx_queues param to
alloc_etherdev_mq might make more sense in this case.

The myri10ge is querying the firmware which seems to be tangled up with
the netdev priv space, but otherwise isn't using any new knowledge. And
ixgbe/igb may end up capping the max number of queues because it is
trying to set the number of queues it intends to use now not the max it
will ever consume.

My point with this patch is I do not see any cases where we really do
not know the max number rx queues until after alloc_etherdev_mq() but before
register_netdev(). The piece I missed is if a driver wants to set rx queues
and tx queues separately they either need to explicitly set rx_num_queues or
use netif_set_real_num_rx_queues before registering the netdev. This syntax
seems error prone to me, and it would be better to set this in alloc_etherdev_mq().
But, maybe you disagree?

So I can either go rework the ixgbe/igb init flow so it does what I want.
Or add a parameter to alloc_etherdev_mq for rx queues, remove the
netif_set_real_num_rx_queues where it is not needed and modify the drivers
to use this interface. I like the second case because it makes the
netif_set_real_num_[rx|tx}_queues() behave the same but could do the first
just as easily.

Thanks,
John 

> It creates /sys files, and Qdisc stuff for nothing...
> 
> 
> 
>> CC: Ben Hutchings <bhutchings@...arflare.com>
>>
>> Signed-off-by: John Fastabend <john.r.fastabend@...el.com>
>> ---
>>
>>  net/core/dev.c |    2 --
>>  1 files changed, 0 insertions(+), 2 deletions(-)
>>
>> diff --git a/net/core/dev.c b/net/core/dev.c
>> index a313bab..f78d996 100644
>> --- a/net/core/dev.c
>> +++ b/net/core/dev.c
>> @@ -1592,8 +1592,6 @@ int netif_set_real_num_rx_queues(struct net_device *dev, unsigned int rxq)
>>  						  rxq);
>>  		if (rc)
>>  			return rc;
>> -	} else {
>> -		dev->num_rx_queues = rxq;
>>  	}
>>  
>>  	dev->real_num_rx_queues = rxq;
>>
>> --
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists