Date:   Mon, 28 Feb 2022 09:46:26 -0800
From:   Jakub Kicinski <kuba@...nel.org>
To:     Thorsten Leemhuis <regressions@...mhuis.info>
Cc:     David Miller <davem@...emloft.net>,
        netdev <netdev@...r.kernel.org>,
        "regressions@...ts.linux.dev" <regressions@...ts.linux.dev>,
        Luiz Augusto von Dentz <luiz.dentz@...il.com>,
        Steffen Klassert <steffen.klassert@...unet.com>
Subject: Re: Regression are sometimes merged slowly, maybe optimize the
 downstream interaction?

On Mon, 28 Feb 2022 14:45:47 +0100 Thorsten Leemhuis wrote:
> Hi Jakub, Hi Dave!

Let's CC Luiz and Steffen.

> I was wondering if you and your downstream maintainers could consider
> slightly optimizing your working habits to get regression fixes from
> downstream git repos a bit quicker into mainline. A slightly different
> timing afaics might already help a lot; or some timing optimizations in
> the interaction with downstream maintainers.
> 
> I ask, because in my regression tracking work I noticed that quite a few
> regression fixes take a long time towards mainline when they need to go
> through net.git; that imho is especially bad for regressions caused by
> commits in earlier development cycles, as they can only be fixed in new
> stable releases after the fix was mainlined.
> 
> Often the fixes progress slowly due to the habits of the downstream
> maintainers -- some for example are imho not asking you often enough to
> pull fixes. I guess that might need to be discussed down the road as
> well, but there is something else that imho needs to be addressed first.
> 
> At least from the outside it often looks like bad timing is the reason
> why some fixes take quite long journey to mainline. Take for example the
> latest pull requests for bluetooth and ipsec:
> 
> https://lore.kernel.org/netdev/20220224210838.197787-1-luiz.dentz@gmail.com/
> https://lore.kernel.org/netdev/20220225074733.118664-1-steffen.klassert@secunet.com/

Yeah, we also narrowly missed the BPF pr a week back :/
Or should I say BPF pr missed the net pr..

> One is from Thursday, the other from early Friday; both contain fixes
> for regressions in earlier mainline releases that afaics need to get
> backported to stable and longterm releases to finally get the regression
> erased from this world. The ipsec fix has been in -next already for a
> while, the bluetooth fix afaics wasn't.
> 
> Sadly, both patch sets missed rc6 as Jakub already had sent his pull
> request to Linus on Thursday:
> https://lore.kernel.org/all/20220224195305.1584666-1-kuba@kernel.org/
> 
> This is not the first time I noticed such bad timing. That made me
> wonder: would it be possible for you to optimize the workflow here?
> Maybe a simple advice to downstream maintainers could do the trick, e.g.
> "ideally sent pull request by Friday morning[some timezone] at the
> latest, as then the net maintainers can review, merge, and sent them
> onwards to Linus in a pull request later that day".

These are fair complaints. We've been sending PRs with fixes every
Thursday for, I'd say, a year or so now. If the sub-tree PR is posted 
by Wednesday it will definitely make the cut. Either folks don't know 
this or they want changes to sit in the networking tree for a couple
of days? Hm.

> FWIW, I don't know anything about the inner working of your subsystem,
> if you need more time to review or process merge requests from
> downstream maintainers the "Friday morning" obviously needs to be adjusted.
> 
> Or is there something like that already and the timing just has been bad
> a few times when I looked closer?

I think it's a particularly unfortunate time with a few "missed prs"
in a short span of time. When Dave was handling all the prs he used
to decide the timing based on contents of the tree, maybe that's 
a better model for prioritizing fixes getting to Linus, but I lack 
the skills necessary to make such calls.

I'll try to advertise the Wednesday rule more, although creating
deadlines has proven to lead to rushed work. Which IMHO is much 
worse :(

BTW there's also something weird with the flow of fixes lately.
Maybe it's just my perception but I feel like it has been disrupted
in Dec and hasn't fully recovered. Normally we get an influx of "new
code / regression fixes" after rc2, and they peter out around rc5.
This release it feels like the fixes _started_ flowing at rc5 :S

Anyway, thanks for raising the issue, and please keep us posted on how
things look from your perspective. It's a balancing act, it'd be great
if we can improve things over time without sudden changes.

Thanks again!

Powered by blists - more mailing lists