Hi stable team - please don't take patches for fs/bcachefs/ except from myself; I'll be doing backports and sending pull requests after stuff has been tested by my CI.
Thanks, and let me know if there's any other workflow things I should know about -Kent
On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
Hi stable team - please don't take patches for fs/bcachefs/ except from myself; I'll be doing backports and sending pull requests after stuff has been tested by my CI.
Thanks, and let me know if there's any other workflow things I should know about
Sure, we can ignore fs/bcachefs/ patches.
Note that the proposed workflow would only work through patches coming through your tree, but patches in other subsystems that could affect bcachefs might break it in stable trees without being caught by your CI.
I'd recommend integrating your pre-release tests with something like kernelci, which would let us catch bcachefs-affecting issues coming from other sources.
What more, if we do the above, we could in the future avoid special casing bcachefs :)
On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
Hi stable team - please don't take patches for fs/bcachefs/ except from myself; I'll be doing backports and sending pull requests after stuff has been tested by my CI.
Thanks, and let me know if there's any other workflow things I should know about
Sure, we can ignore fs/bcachefs/ patches.
I see that you even acked this.
What the fuck?
Note that the proposed workflow would only work through patches coming through your tree, but patches in other subsystems that could affect bcachefs might break it in stable trees without being caught by your CI.
I'd recommend integrating your pre-release tests with something like kernelci, which would let us catch bcachefs-affecting issues coming from other sources.
What more, if we do the above, we could in the future avoid special casing bcachefs :)
-- Thanks, Sasha
On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
Hi stable team - please don't take patches for fs/bcachefs/ except from myself; I'll be doing backports and sending pull requests after stuff has been tested by my CI.
Thanks, and let me know if there's any other workflow things I should know about
Sure, we can ignore fs/bcachefs/ patches.
I see that you even acked this.
What the fuck?
Accidents happen, you were copied on those patches. I'll go drop them now, not a big deal.
thanks,
greg k-h
On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
Hi stable team - please don't take patches for fs/bcachefs/ except from myself; I'll be doing backports and sending pull requests after stuff has been tested by my CI.
Thanks, and let me know if there's any other workflow things I should know about
Sure, we can ignore fs/bcachefs/ patches.
I see that you even acked this.
What the fuck?
Accidents happen, you were copied on those patches. I'll go drop them now, not a big deal.
Wait, why are you doing "Fixes:" with an empty tag in your commits like 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
That's messing with scripts and doesn't make much sense. Please put a real git id in there as the documentation suggests to.
thanks,
greg k-h
On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
Hi stable team - please don't take patches for fs/bcachefs/ except from myself; I'll be doing backports and sending pull requests after stuff has been tested by my CI.
Thanks, and let me know if there's any other workflow things I should know about
Sure, we can ignore fs/bcachefs/ patches.
I see that you even acked this.
What the fuck?
Accidents happen, you were copied on those patches. I'll go drop them now, not a big deal.
Wait, why are you doing "Fixes:" with an empty tag in your commits like 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
That's messing with scripts and doesn't make much sense. Please put a real git id in there as the documentation suggests to.
There isn't always a clear-cut commit when a regression was introduced (it might not have been a regresison at all). I could dig and make something up, but that's slowing down your workflow, and I thought I was going to be handling all the stable backports for fs/bcachefs/, so - ?
On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
Hi stable team - please don't take patches for fs/bcachefs/ except from myself; I'll be doing backports and sending pull requests after stuff has been tested by my CI.
Thanks, and let me know if there's any other workflow things I should know about
Sure, we can ignore fs/bcachefs/ patches.
I see that you even acked this.
What the fuck?
Accidents happen, you were copied on those patches. I'll go drop them now, not a big deal.
Wait, why are you doing "Fixes:" with an empty tag in your commits like 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
That's messing with scripts and doesn't make much sense. Please put a real git id in there as the documentation suggests to.
There isn't always a clear-cut commit when a regression was introduced (it might not have been a regresison at all). I could dig and make something up, but that's slowing down your workflow, and I thought I was going to be handling all the stable backports for fs/bcachefs/, so - ?
Doesn't matter, please do not put "fake" tags in commit messages like this. It hurts all of the people that parse commit logs. Just don't put a fixes tag at all as the documentation states that after "Fixes:" a commit id belongs.
thanks,
greg k-h
On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote: > Hi stable team - please don't take patches for fs/bcachefs/ except from > myself; I'll be doing backports and sending pull requests after stuff > has been tested by my CI. > > Thanks, and let me know if there's any other workflow things I should > know about
Sure, we can ignore fs/bcachefs/ patches.
I see that you even acked this.
What the fuck?
Accidents happen, you were copied on those patches. I'll go drop them now, not a big deal.
Wait, why are you doing "Fixes:" with an empty tag in your commits like 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
That's messing with scripts and doesn't make much sense. Please put a real git id in there as the documentation suggests to.
There isn't always a clear-cut commit when a regression was introduced (it might not have been a regresison at all). I could dig and make something up, but that's slowing down your workflow, and I thought I was going to be handling all the stable backports for fs/bcachefs/, so - ?
Doesn't matter, please do not put "fake" tags in commit messages like this. It hurts all of the people that parse commit logs. Just don't put a fixes tag at all as the documentation states that after "Fixes:" a commit id belongs.
Then there's a gap, because I need a tag that I can stick in a commit message that says "this is a bugfix I need to consider backporting later", and the way you want the Fixes: tag used doesn't meet my needs.
On Tue, Feb 20, 2024 at 03:22:59PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote: > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote: > > Hi stable team - please don't take patches for fs/bcachefs/ except from > > myself; I'll be doing backports and sending pull requests after stuff > > has been tested by my CI. > > > > Thanks, and let me know if there's any other workflow things I should > > know about > > Sure, we can ignore fs/bcachefs/ patches.
I see that you even acked this.
What the fuck?
Accidents happen, you were copied on those patches. I'll go drop them now, not a big deal.
Wait, why are you doing "Fixes:" with an empty tag in your commits like 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
That's messing with scripts and doesn't make much sense. Please put a real git id in there as the documentation suggests to.
There isn't always a clear-cut commit when a regression was introduced (it might not have been a regresison at all). I could dig and make something up, but that's slowing down your workflow, and I thought I was going to be handling all the stable backports for fs/bcachefs/, so - ?
Doesn't matter, please do not put "fake" tags in commit messages like this. It hurts all of the people that parse commit logs. Just don't put a fixes tag at all as the documentation states that after "Fixes:" a commit id belongs.
Then there's a gap, because I need a tag that I can stick in a commit message that says "this is a bugfix I need to consider backporting later", and the way you want the Fixes: tag used doesn't meet my needs.
Then use patchwork or something else, but please do not override a 15+ year old tag for just one small portion of the kernel.
Or better yet, use the fixes: tag with the commit id you are fixing, that way all other kernel workflows work properly with this filesystem. No need to be unique here :)
thanks,
greg k-h
On Tue, Feb 20, 2024 at 09:39:03PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 03:22:59PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote: > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote: > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote: > > > Hi stable team - please don't take patches for fs/bcachefs/ except from > > > myself; I'll be doing backports and sending pull requests after stuff > > > has been tested by my CI. > > > > > > Thanks, and let me know if there's any other workflow things I should > > > know about > > > > Sure, we can ignore fs/bcachefs/ patches. > > I see that you even acked this. > > What the fuck?
Accidents happen, you were copied on those patches. I'll go drop them now, not a big deal.
Wait, why are you doing "Fixes:" with an empty tag in your commits like 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
That's messing with scripts and doesn't make much sense. Please put a real git id in there as the documentation suggests to.
There isn't always a clear-cut commit when a regression was introduced (it might not have been a regresison at all). I could dig and make something up, but that's slowing down your workflow, and I thought I was going to be handling all the stable backports for fs/bcachefs/, so - ?
Doesn't matter, please do not put "fake" tags in commit messages like this. It hurts all of the people that parse commit logs. Just don't put a fixes tag at all as the documentation states that after "Fixes:" a commit id belongs.
Then there's a gap, because I need a tag that I can stick in a commit message that says "this is a bugfix I need to consider backporting later", and the way you want the Fixes: tag used doesn't meet my needs.
Then use patchwork or something else, but please do not override a 15+ year old tag for just one small portion of the kernel.
Or better yet, use the fixes: tag with the commit id you are fixing, that way all other kernel workflows work properly with this filesystem. No need to be unique here :)
And stop trying to blame this on your scripts or the fixes tag, there's a real lack of communication there; you can't say you're going to do one thing and then do the complete opposite.
On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote: > Hi stable team - please don't take patches for fs/bcachefs/ except from > myself; I'll be doing backports and sending pull requests after stuff > has been tested by my CI. > > Thanks, and let me know if there's any other workflow things I should > know about
Sure, we can ignore fs/bcachefs/ patches.
I see that you even acked this.
What the fuck?
Accidents happen, you were copied on those patches. I'll go drop them now, not a big deal.
Wait, why are you doing "Fixes:" with an empty tag in your commits like 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
That's messing with scripts and doesn't make much sense. Please put a real git id in there as the documentation suggests to.
There isn't always a clear-cut commit when a regression was introduced (it might not have been a regresison at all). I could dig and make something up, but that's slowing down your workflow, and I thought I was going to be handling all the stable backports for fs/bcachefs/, so - ?
Doesn't matter, please do not put "fake" tags in commit messages like this. It hurts all of the people that parse commit logs. Just don't put a fixes tag at all as the documentation states that after "Fixes:" a commit id belongs.
So you manually repicked a subset of my pull request, and of the two patches you silently dropped, one was a security fix - and you _never communicated_ what you were doing.
Greg, this isn't working. How are we going to fix this?
On Tue, Feb 20, 2024 at 03:39:16PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote: > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote: > > Hi stable team - please don't take patches for fs/bcachefs/ except from > > myself; I'll be doing backports and sending pull requests after stuff > > has been tested by my CI. > > > > Thanks, and let me know if there's any other workflow things I should > > know about > > Sure, we can ignore fs/bcachefs/ patches.
I see that you even acked this.
What the fuck?
Accidents happen, you were copied on those patches. I'll go drop them now, not a big deal.
Wait, why are you doing "Fixes:" with an empty tag in your commits like 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
That's messing with scripts and doesn't make much sense. Please put a real git id in there as the documentation suggests to.
There isn't always a clear-cut commit when a regression was introduced (it might not have been a regresison at all). I could dig and make something up, but that's slowing down your workflow, and I thought I was going to be handling all the stable backports for fs/bcachefs/, so - ?
Doesn't matter, please do not put "fake" tags in commit messages like this. It hurts all of the people that parse commit logs. Just don't put a fixes tag at all as the documentation states that after "Fixes:" a commit id belongs.
So you manually repicked a subset of my pull request, and of the two patches you silently dropped, one was a security fix - and you _never communicated_ what you were doing.
I explicitly said "Not all of these applied properly, please send me the remaining ones". I can go back and get the message-id if you want reciepts :)
Greg, this isn't working. How are we going to fix this?
Please send a set of backported commits that you wish to have applied to the stable trees. All other subsystems do this fairly easily, it's no different from sending a patch series out for anything else.
Worst case, I can take a git tree, BUT I will then turn that git tree into individual commits as that is what we MUST deal with for the stable trees, we can not work with direct pull requests for obvious reasons of how the tree needs to be managed (i.e. rebasing all the time would never work.)
thanks,
greg k-h
On Tue, Feb 20, 2024 at 09:51:20PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 03:39:16PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote: > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote: > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote: > > > Hi stable team - please don't take patches for fs/bcachefs/ except from > > > myself; I'll be doing backports and sending pull requests after stuff > > > has been tested by my CI. > > > > > > Thanks, and let me know if there's any other workflow things I should > > > know about > > > > Sure, we can ignore fs/bcachefs/ patches. > > I see that you even acked this. > > What the fuck?
Accidents happen, you were copied on those patches. I'll go drop them now, not a big deal.
Wait, why are you doing "Fixes:" with an empty tag in your commits like 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
That's messing with scripts and doesn't make much sense. Please put a real git id in there as the documentation suggests to.
There isn't always a clear-cut commit when a regression was introduced (it might not have been a regresison at all). I could dig and make something up, but that's slowing down your workflow, and I thought I was going to be handling all the stable backports for fs/bcachefs/, so - ?
Doesn't matter, please do not put "fake" tags in commit messages like this. It hurts all of the people that parse commit logs. Just don't put a fixes tag at all as the documentation states that after "Fixes:" a commit id belongs.
So you manually repicked a subset of my pull request, and of the two patches you silently dropped, one was a security fix - and you _never communicated_ what you were doing.
I explicitly said "Not all of these applied properly, please send me the remaining ones". I can go back and get the message-id if you want reciepts :)
I gave you a _signed pull request_, and there were no merge conflicts.
Greg, this isn't working. How are we going to fix this?
Please send a set of backported commits that you wish to have applied to the stable trees. All other subsystems do this fairly easily, it's no different from sending a patch series out for anything else.
Worst case, I can take a git tree, BUT I will then turn that git tree into individual commits as that is what we MUST deal with for the stable trees, we can not work with direct pull requests for obvious reasons of how the tree needs to be managed (i.e. rebasing all the time would never work.)
You rebase these trees? Why? Are they not public?
Look, I need to know that the code I send you is the same as the code that gets published in stable releases. If you're going to be rebasing the trees I send you, _all_ the mechanisms we have for doing that validation break and I'm back to manual verification.
And given that we've got mechanisms for avoiding that - not rebasing so that we can verify by the sha1, gpg signing - why is stable special here?
On Tue, Feb 20, 2024 at 04:00:18PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 09:51:20PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 03:39:16PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote: > On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote: > > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote: > > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote: > > > > Hi stable team - please don't take patches for fs/bcachefs/ except from > > > > myself; I'll be doing backports and sending pull requests after stuff > > > > has been tested by my CI. > > > > > > > > Thanks, and let me know if there's any other workflow things I should > > > > know about > > > > > > Sure, we can ignore fs/bcachefs/ patches. > > > > I see that you even acked this. > > > > What the fuck? > > Accidents happen, you were copied on those patches. I'll go drop them > now, not a big deal.
Wait, why are you doing "Fixes:" with an empty tag in your commits like 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
That's messing with scripts and doesn't make much sense. Please put a real git id in there as the documentation suggests to.
There isn't always a clear-cut commit when a regression was introduced (it might not have been a regresison at all). I could dig and make something up, but that's slowing down your workflow, and I thought I was going to be handling all the stable backports for fs/bcachefs/, so - ?
Doesn't matter, please do not put "fake" tags in commit messages like this. It hurts all of the people that parse commit logs. Just don't put a fixes tag at all as the documentation states that after "Fixes:" a commit id belongs.
So you manually repicked a subset of my pull request, and of the two patches you silently dropped, one was a security fix - and you _never communicated_ what you were doing.
I explicitly said "Not all of these applied properly, please send me the remaining ones". I can go back and get the message-id if you want reciepts :)
I gave you a _signed pull request_, and there were no merge conflicts.
Greg, this isn't working. How are we going to fix this?
Please send a set of backported commits that you wish to have applied to the stable trees. All other subsystems do this fairly easily, it's no different from sending a patch series out for anything else.
Worst case, I can take a git tree, BUT I will then turn that git tree into individual commits as that is what we MUST deal with for the stable trees, we can not work with direct pull requests for obvious reasons of how the tree needs to be managed (i.e. rebasing all the time would never work.)
You rebase these trees? Why? Are they not public?
Look, I need to know that the code I send you is the same as the code that gets published in stable releases. If you're going to be rebasing the trees I send you, _all_ the mechanisms we have for doing that validation break and I'm back to manual verification.
And given that we've got mechanisms for avoiding that - not rebasing so that we can verify by the sha1, gpg signing - why is stable special here?
Hi,
This is the friendly email-bot of the stable kernel team. I've detected that you are asking a question that has been discussed many times in the past. If you wish to contact the stable developers directly, please use their phone hotline, 1-800-382-5633.
thanks,
stable email bot
----------------
*beep beep beep beep beep beep beep beep beep beep beep*
*ring* *ring*
This is the Linux stable kernel team's phone line.
Please listen to the following menu fully as the items have changed recently.
To contact the Linux stable developer's directly, please press 1.
To contact the Linux stable develop— *1*
All of the current stable developers are busy handling other kernel developer issues or reviewing properly tagged kernel patches. Please wait for the next available developer, or press 0 to return to the main menu.
You are caller number SEVENTY FIVE in the queue with an expected wait time of ELEVEN HUNDRED AND TWENTY FIVE MINUTES.
Please enjoy the smooth sounds of Enya while you wait.
"Who can say where the road goes? Where the day flows? Only time. And who can say— *0*
This is the Linux stable kernel team's phone line.
Please listen to the following menu fully as the items have changed recently.
To contact the Linux stable developer's directly, please press 1.
To contact the Linux stable developer's legal department to lodge a complaint, please press 2.
To listen to the— *2*
You are now being transferred to the legal department, to return to the main menu, please press 0
....
Welcome to the law firm of Dewey Cheatem & Howe. Unfortunately, due to a large outstanding bill, we no longer represent the Linux stable kernel team as it turned out they were not being paid for their services, and so didn't feel like paying for ours either. If you wish to hire us to represent you in an accident claim, stay on the line and someone will be with you— *0*
This is the Linux stable kernel team's phone line.
Please listen to the following menu fully as the items have changed recently.
To contact the Linux stable developer's directly, please press 1.
To contact the Linux stable developer's legal department to lodge a complaint, please press 2.
To listen to the often asked question of "why don't you accept pull requests directly", please press 3.
For all other questions, please write an email and it will be handled in the order in which it is received.
This menu will now repeat for your enjoyment.
This is the Linux stable—
*3*
The reason the stable kernel team can not accept pull requests directly is that they accumulate many patches from many different developers and trees and combine them all into a set of patches to be applied on top of the last release. If a git tree were used, the ordering, reshuffling, removing, and adding of patches in the list would not work properly at all as it would require constant rebasing. This can be seen directly by following the often-rebased linux-stable-rc git tree, which is only present for CI systems to consume if they can not handle a set of quilt patches directly.
The stable developer workflow has been working well for over 15 years as a patch queue, no convincing of "but you must accept my pull request" is going to work well, given the current requirements of how the stable trees are generated.
A pull request can be handled, by the stable developer taking it, exporting the changes as individual patches (using 'git format-patch') and then importing them into the stable patch queue directly. If a subsystem maintainer feels that this is the best way to coordinate with the stable team, that is fine, but note that individual patches will be the end result, so perhaps just using 'git send-email' in the first place would be simpler for everyone involved?
Having to process a git pull takes additional time and energy and slows down the patch acceptance rate. It is generally postponed as there are lots of properly tagged commits that are easier and simpler for them to handle.
Given the huge patch volume that the stable tree manages (30-40 changes accepted a day, 7 days a week), any one kernel subsystem that wishes to do something different only slows down everyone else. Yes, your subsystem is special and unique, just like everyone else's, we know, remember, we maintain kernel subsystems too. Just use patches and email, it's the lowest common denominator here that works well given the often-disconnected nature of the stable developers as they travel the globe racking up frequent flyer points and talking in generic conference center meeting rooms about hardware roadmaps and their applicability to the kernel.
Thank you for listening to the reasoning here, and as we generally assume this will not have calmed you down much at all, please feel free to stay on the line in the phone queue which currently has EIGHTY FIVE callers ahead of you, with a expected wait time of TWELVE HUNDRED AND SEVENTY MINUTES
Please continue to enjoy the smooth sounds of Enya while you wait.
"Let me sail, let me sail Let the orinoco flow" Let me reach, let me beach—
*CLICK*
On středa 21. února 2024 15:53:11 CET Greg KH wrote:
Given the huge patch volume that the stable tree manages (30-40 changes accepted a day, 7 days a week), any one kernel subsystem that wishes to do something different only slows down everyone else.
Lower down the volume then? Raise the bar for what gets backported? Stable kernel releases got unnecessarily big [1] (Jiří is in Cc). Those 40 changes a day cannot get a proper review. Each stable release tries to mimic -rc except -rc is in consistent state while "stable" is just a bunch of changes picked here and there.
[1] https://lwn.net/Articles/962131/
On Wed, Feb 21, 2024 at 05:00:05PM +0100, Oleksandr Natalenko wrote:
On středa 21. února 2024 15:53:11 CET Greg KH wrote:
Given the huge patch volume that the stable tree manages (30-40 changes accepted a day, 7 days a week), any one kernel subsystem that wishes to do something different only slows down everyone else.
Lower down the volume then? Raise the bar for what gets backported? Stable kernel releases got unnecessarily big [1] (Jiří is in Cc). Those 40 changes a day cannot get a proper review. Each stable release tries to mimic -rc except -rc is in consistent state while "stable" is just a bunch of changes picked here and there.
If you can point out any specific commits that we should not be taking, please let us know.
Personally I think we are not taking enough, and are still missing real fixes. Overall, this is only a very small % of what goes into Linus's tree every day, so by that measure alone, we know we are missing things.
thanks,
greg k-h
On 2/21/24 18:57, Greg KH wrote:
On Wed, Feb 21, 2024 at 05:00:05PM +0100, Oleksandr Natalenko wrote:
On středa 21. února 2024 15:53:11 CET Greg KH wrote:
Given the huge patch volume that the stable tree manages (30-40 changes accepted a day, 7 days a week), any one kernel subsystem that wishes to do something different only slows down everyone else.
Lower down the volume then? Raise the bar for what gets backported? Stable kernel releases got unnecessarily big [1] (Jiří is in Cc). Those 40 changes a day cannot get a proper review. Each stable release tries to mimic -rc except -rc is in consistent state while "stable" is just a bunch of changes picked here and there.
If you can point out any specific commits that we should not be taking, please let us know.
Personally I think we are not taking enough, and are still missing real fixes. Overall, this is only a very small % of what goes into Linus's tree every day, so by that measure alone, we know we are missing things.
What % of what goes into Linus's tree do you think fits within the rules stated in Documentation/process/stable-kernel-rules.rst ? I don't know but "very small" would be my guess, so we should be fine as it is?
Or are the rules actually still being observed? I doubt e.g. many of the AUTOSEL backports fit them? Should we rename the file to stable-rules-nonsense.rst?
thanks,
greg k-h
On Wed, Feb 21, 2024 at 07:10:02PM +0100, Vlastimil Babka wrote:
On 2/21/24 18:57, Greg KH wrote:
On Wed, Feb 21, 2024 at 05:00:05PM +0100, Oleksandr Natalenko wrote:
On středa 21. února 2024 15:53:11 CET Greg KH wrote:
Given the huge patch volume that the stable tree manages (30-40 changes accepted a day, 7 days a week), any one kernel subsystem that wishes to do something different only slows down everyone else.
Lower down the volume then? Raise the bar for what gets backported? Stable kernel releases got unnecessarily big [1] (Jiří is in Cc). Those 40 changes a day cannot get a proper review. Each stable release tries to mimic -rc except -rc is in consistent state while "stable" is just a bunch of changes picked here and there.
If you can point out any specific commits that we should not be taking, please let us know.
Personally I think we are not taking enough, and are still missing real fixes. Overall, this is only a very small % of what goes into Linus's tree every day, so by that measure alone, we know we are missing things.
What % of what goes into Linus's tree do you think fits within the rules stated in Documentation/process/stable-kernel-rules.rst ? I don't know but "very small" would be my guess, so we should be fine as it is?
Or are the rules actually still being observed? I doubt e.g. many of the AUTOSEL backports fit them? Should we rename the file to stable-rules-nonsense.rst?
Yeah, I'd say around half of the backports I see being done really had no justification at all - i.e. were for fixes for bugs that weren't present on the kernel being backported to, including most of the autosel patches.
There's clearly a balance to be struck with what we backport - but it doesn't seem like Greg and Sasha are trying to find that balance, it seems to be all pedal-to-the-metal backport-everything.
And the "process" seems to be whatever makes things most convenient for Greg and Sasha, and they _really_ want to be doing everything themselves. That form letter response quite illustrates that.
This doesn't scale.
And Greg not taking signed pull requests is a _real_ what the fuck. If we care about supply chain attacks at all, surely we care about them for stable, because those are the kernels most people actually run!
Greg, you're doing an end run around a lot of the process the _community_ has built up.
On Wed, Feb 21, 2024 at 07:10:02PM +0100, Vlastimil Babka wrote:
On 2/21/24 18:57, Greg KH wrote:
On Wed, Feb 21, 2024 at 05:00:05PM +0100, Oleksandr Natalenko wrote:
On středa 21. února 2024 15:53:11 CET Greg KH wrote:
Given the huge patch volume that the stable tree manages (30-40 changes accepted a day, 7 days a week), any one kernel subsystem that wishes to do something different only slows down everyone else.
Lower down the volume then? Raise the bar for what gets backported? Stable kernel releases got unnecessarily big [1] (Jiří is in Cc). Those 40 changes a day cannot get a proper review. Each stable release tries to mimic -rc except -rc is in consistent state while "stable" is just a bunch of changes picked here and there.
If you can point out any specific commits that we should not be taking, please let us know.
Personally I think we are not taking enough, and are still missing real fixes. Overall, this is only a very small % of what goes into Linus's tree every day, so by that measure alone, we know we are missing things.
What % of what goes into Linus's tree do you think fits within the rules stated in Documentation/process/stable-kernel-rules.rst ? I don't know but "very small" would be my guess, so we should be fine as it is?
Or are the rules actually still being observed? I doubt e.g. many of the AUTOSEL backports fit them? Should we rename the file to stable-rules-nonsense.rst?
Hey, I have an exercise for you which came up last week during the whole CVE thing!
Take a look at a random LTS kernel (I picked 5.10), in particular at the CVEs assigned to the kernel (in my case I relied on https://github.com/nluedtke/linux_kernel_cves/blob/master/data/5.10/5.10_sec...).
See how many of those actually have a stable@ tag to let us know that we need to pull that commit. (spoiler alert: in the 5.10 case it was ~33%)
Do you have a better way for us to fish for the remaining 67%?
Yeah, some have a Fixes tag, (it's not in stable-kernel-rules.rst!), and in the 5.10 case it would have helped with about half of the commits, but even then - what do we do with the remaining half?
The argument you're making is in favor of just ignoring it until they get a CVE assigned (and even then, would we take them if it goes against stable-kernel-rules.rst?), but then we end up leaving users exposed for *years* as evidenced by some CVEs.
So if we go with the current workflow, folks complain that we take too many patches. If we were to lean strictly to what stable-kernel-rules.rst says, we'd apparently miss most of the (security) issues affecting users.
On Wed, Feb 21, 2024 at 05:58:30PM -0500, Sasha Levin wrote:
On Wed, Feb 21, 2024 at 07:10:02PM +0100, Vlastimil Babka wrote:
On 2/21/24 18:57, Greg KH wrote:
On Wed, Feb 21, 2024 at 05:00:05PM +0100, Oleksandr Natalenko wrote:
On středa 21. února 2024 15:53:11 CET Greg KH wrote:
Given the huge patch volume that the stable tree manages (30-40 changes accepted a day, 7 days a week), any one kernel subsystem that wishes to do something different only slows down everyone else.
Lower down the volume then? Raise the bar for what gets backported? Stable kernel releases got unnecessarily big [1] (Jiří is in Cc). Those 40 changes a day cannot get a proper review. Each stable release tries to mimic -rc except -rc is in consistent state while "stable" is just a bunch of changes picked here and there.
If you can point out any specific commits that we should not be taking, please let us know.
Personally I think we are not taking enough, and are still missing real fixes. Overall, this is only a very small % of what goes into Linus's tree every day, so by that measure alone, we know we are missing things.
What % of what goes into Linus's tree do you think fits within the rules stated in Documentation/process/stable-kernel-rules.rst ? I don't know but "very small" would be my guess, so we should be fine as it is?
Or are the rules actually still being observed? I doubt e.g. many of the AUTOSEL backports fit them? Should we rename the file to stable-rules-nonsense.rst?
Hey, I have an exercise for you which came up last week during the whole CVE thing!
Take a look at a random LTS kernel (I picked 5.10), in particular at the CVEs assigned to the kernel (in my case I relied on https://github.com/nluedtke/linux_kernel_cves/blob/master/data/5.10/5.10_sec...).
See how many of those actually have a stable@ tag to let us know that we need to pull that commit. (spoiler alert: in the 5.10 case it was ~33%)
Do you have a better way for us to fish for the remaining 67%?
Yeah, some have a Fixes tag, (it's not in stable-kernel-rules.rst!), and in the 5.10 case it would have helped with about half of the commits, but even then - what do we do with the remaining half?
The argument you're making is in favor of just ignoring it until they get a CVE assigned (and even then, would we take them if it goes against stable-kernel-rules.rst?), but then we end up leaving users exposed for *years* as evidenced by some CVEs.
No, I'm not in favor of ignoring things either. What I want is better collaboration between subsystem maintainers and you guys. Awhile back I was proposing some sort of shared dashboard where we could all track and annotate which patches should be considered for backporting when, because the current email based workflow sucks.
We need to be working together better, which is why I'm so _damn annoyed_ over the crap with the Fixes: tag, and Greg not taking my signed pull request as an atomic unit.
You guys shouldn't be doing all this work yourselves; the subsystem maintainers, the people who know the code best, should be involved. But your attitude 100% pushes people away.
I need a workflow that works for me, if I'm going to do the backporting for bcachefs patches, as I intend to: fuckups in filesystem land can have _far_ too high a cost for me to leave this work to anyone else.
What I need is:
- A way to unambigiously tag a patch as something that I need to look at later when I'm doing backports; and I do _not_ want to have to go git blame spelunking when I'm doing that, distracting me from whatever else I'm working on; if you insist on the Fixes: tag conforming to your tools then this won't get done and bugfixes will get lost.
- Signed pull requests to not get tweaked and rebased.
Tweaking and hand editing patches is one thing when it's being done by the subsystem maintainer, the person who at least nominally knows the code best and is best qualified to review those changes.
You guys should not be doing that, because then I have to go back and check your work (and even if you swear you aren't changing the code; how do I know unless I look at the diffs? Remember all the talk about supply chain attacks?) and I'm just not going to do that. That adds too much to my workload, and there's no reason for it.
And please, you guys, make a bit more of an effort to work with people, and listen to and address their concerns. I hear _way_ too much grousing in private about -stable bullshit, and despite that I went in to this optimistic that we'd be able to cut past the bullshit and establish a good working relationship. Let's see if we still can...
On Wed, Feb 21, 2024 at 06:12:42PM -0500, Kent Overstreet wrote:
The argument you're making is in favor of just ignoring it until they get a CVE assigned (and even then, would we take them if it goes against stable-kernel-rules.rst?), but then we end up leaving users exposed for *years* as evidenced by some CVEs.
No, I'm not in favor of ignoring things either. What I want is better collaboration between subsystem maintainers and you guys. Awhile back I was proposing some sort of shared dashboard where we could all track and annotate which patches should be considered for backporting when, because the current email based workflow sucks.
I too would like a pony, but for now, we work with what we have, right?
We need to be working together better, which is why I'm so _damn annoyed_ over the crap with the Fixes: tag, and Greg not taking my signed pull request as an atomic unit.
Fixes: has a long long long history and to think that your subsystem can ignore it and do something else with it feels very odd to me. Your subsystem lives in the same ecosystem as all other 4000+ developers. There's a reason we have standards/rules, you can't just ignore them.
You guys shouldn't be doing all this work yourselves; the subsystem maintainers, the people who know the code best, should be involved. But your attitude 100% pushes people away.
The goal of the stable/lts kernels is that no developer or maintainer that does NOT want to help out, has to do anything more than just a simple "cc: stable@vger.kernel.org" addition to a commit when they apply it to their tree.
Anything above that is wonderful, but again, not something I can ever tell someone to do.
I need a workflow that works for me, if I'm going to do the backporting for bcachefs patches, as I intend to: fuckups in filesystem land can have _far_ too high a cost for me to leave this work to anyone else.
What I need is:
- A way to unambigiously tag a patch as something that I need to look at later when I'm doing backports; and I do _not_ want to have to go git blame spelunking when I'm doing that, distracting me from whatever else I'm working on; if you insist on the Fixes: tag conforming to your tools then this won't get done and bugfixes will get lost.
Why can't you use Fixes like all of the 10's of thousands of other developers have over the past 15+ years?
And if you don't like it, that's fine, but again, you can not redefine it to mean something else for your tiny subsystem, sorry, that's not how projects of any size work well and survive.
- Signed pull requests to not get tweaked and rebased.
As-is, I can NOT take your signed pull request as it did not include the needed information in it, as I said at the time (i.e. no reference to the commits that you were backporting.)
What I DID do is dig through your pull request and take the individual commits that DID apply after looking up, by hand, the proper upstream git commit id. I then did NOT take the 2 commits that you had modified from their upstream version, as there was no indication as to why the changes were made, or even that any change was made at all, from what is in Linus's tree. And at the time I told you all of this, so there was no question of what happened, and what was expected.
And remember, Linus's tree is the "canonical" version here. We rely on signed tags for that, and then we can safely cherry-pick and compare to those commits to verify that "yes, this is the exact same change that we already approved and accepted".
Tweaking and hand editing patches is one thing when it's being done by the subsystem maintainer, the person who at least nominally knows the code best and is best qualified to review those changes.
Wonderful, then say you tweaked and hand-edited the patches when you submit them to us and we are fine with it. But you didn't do that, so in the interest of saftey and stability, I did NOT take the changes that were modified. And here I would think you would be happy we do validation and verification...
You guys should not be doing that, because then I have to go back and check your work (and even if you swear you aren't changing the code; how do I know unless I look at the diffs? Remember all the talk about supply chain attacks?) and I'm just not going to do that. That adds too much to my workload, and there's no reason for it.
If I was a more paranoid person, I would have thought that the modified changes you sent us with no indication that the changes were modified, was a "supply chain attack" that you were attempting to do on us.
And please, you guys, make a bit more of an effort to work with people, and listen to and address their concerns. I hear _way_ too much grousing in private about -stable bullshit, and despite that I went in to this optimistic that we'd be able to cut past the bullshit and establish a good working relationship. Let's see if we still can...
This goes both ways please.
Again, I can not take pull requests, it does not work at all with our workflow as we NEED and REQUIRE actual individual commits, both for verification and validation, as well as to actually be able to apply to our trees.
We NEED and REQUIRE the git commit ids of the changes you are asking for including to be in the commit message itself (or somewhere that I can then add it), as that is how we, and everyone else, tracks what gets applied to where, and to be able to validate and ensure that the commit really is what you say it is.
We NEED and REQUIRE you, if you do modify a commit from what is in Linus's tree, to say "hey, this is modified because of X and Y", not to just not say anything and assume that we should blindly take a modified change. You don't want us to do that, right?
So, to work with the stable trees, you have a number of simple options, here they are going from least-amount-of-work to most-amount-of-work: - ignore the stable trees and insist that everyone only use Linus's releases for your subsystem (some subsystems do want this, for whatever reason, that's on them.) - add only "Fixes:" tags to commits, with the correct git id. These will eventually get picked up by a stable developer and a semi-best effort done to apply them were relevant, but we can not guarantee it. - add a "Cc: stable@vger.kernel.org" tag to a commit when you apply it to your tree. This is the simplest, and recommended way. Bonus if you also include a "Fixes:" tag so we know how far back to apply it, AND you will get an automated message for when we can NOT apply it that far back successfully. - Send us a list of git ids you want to have us backport. - Send us a patch series that you want to have applied, with the git id of the original commit in the commit body, and if the change has been modified from the original, a small note somewhere (usually in the signed-off-by area) saying it has been modified and why. - Send us a pull request of patches built up the same way as the above option. This puts more work on us as we then need to turn the pull request into a set of individual patches and review them "by hand" in a tool other than our mail readers, but eventually the patches will get applied.
Note that no where is a "send a pull request that consists of patches with no references to upstream git ids or notification that anything has changed" option. That just will not work for all of the reasons speficied above, sorry.
Hope this helps explain all of this better.
greg k-h
On Thu, Feb 22, 2024 at 06:48:58AM +0100, Greg KH wrote:
- A way to unambigiously tag a patch as something that I need to look at later when I'm doing backports; and I do _not_ want to have to go git blame spelunking when I'm doing that, distracting me from whatever else I'm working on; if you insist on the Fixes: tag conforming to your tools then this won't get done and bugfixes will get lost.
Why can't you use Fixes like all of the 10's of thousands of other developers have over the past 15+ years?
I've explained that multiple times and you're not listening.
That's not how you work with people, Greg; cut the form letter response bullshit and try and have a real conversation.
You've been repeatedly stonewalling and shutting down my every attempt to find a solution here.
But honestly, this isn't even the main issue.
And if you don't like it, that's fine, but again, you can not redefine it to mean something else for your tiny subsystem, sorry, that's not how projects of any size work well and survive.
- Signed pull requests to not get tweaked and rebased.
As-is, I can NOT take your signed pull request as it did not include the needed information in it, as I said at the time (i.e. no reference to the commits that you were backporting.)
You communicated that one, and I redid the cherry picks with -x...
...And they still showed up in your tree as completely different commits.
What I DID do is dig through your pull request and take the individual commits that DID apply after looking up, by hand, the proper upstream git commit id. I then did NOT take the 2 commits that you had modified from their upstream version, as there was no indication as to why the changes were made, or even that any change was made at all, from what is in Linus's tree. And at the time I told you all of this, so there was no question of what happened, and what was expected.
Why would you silently redo work that I sent you?
All you're doing is making more work for yourself. If I send you a pull request that you can't use, kick it back to me and tell me why!
But silently redo it and drop a security fix? Can we please not?
If I was a more paranoid person, I would have thought that the modified changes you sent us with no indication that the changes were modified, was a "supply chain attack" that you were attempting to do on us.
Of my own code?
Look, this isn't about not trusting _you_, it's that the further up the food chain commits go the less it is possible and reasonable to expect anything to be caught by review.
And please, you guys, make a bit more of an effort to work with people, and listen to and address their concerns. I hear _way_ too much grousing in private about -stable bullshit, and despite that I went in to this optimistic that we'd be able to cut past the bullshit and establish a good working relationship. Let's see if we still can...
This goes both ways please.
Again, I can not take pull requests, it does not work at all with our workflow as we NEED and REQUIRE actual individual commits, both for verification and validation, as well as to actually be able to apply to our trees.
We NEED and REQUIRE the git commit ids of the changes you are asking for including to be in the commit message itself (or somewhere that I can then add it), as that is how we, and everyone else, tracks what gets applied to where, and to be able to validate and ensure that the commit really is what you say it is.
We NEED and REQUIRE you, if you do modify a commit from what is in Linus's tree, to say "hey, this is modified because of X and Y", not to just not say anything and assume that we should blindly take a modified change. You don't want us to do that, right?
I can provide commit messages in the format you need - but also: _none_ of this is documented in stable-kernel-rules.rst, so I'm going to want something clear and specific I can go off of.
(And why are you not specifying the original sha1 in the format cherry-pick -x produces? Why is that not documented?)
But not taking signed pull requests is going to be a sticking point, as well as the fact that you only took part of the pull request.
We've had multiple bugs lately stemming from related patches not being carried together - _nasty_ bugs, as in "I can't access my filesystem" or "My data is being corrupted".
That was the case in this pull request too; one of the patches you dropped, IIRC, was a locking fix for an earlier fix by Al; those patches should have gone together.
This sort of thing is a good part of why I'm insisting on doing things myself.
So: in the interest of avoiding issues like this, can I at least ask you to start taking signed pull requests if the patch metadata is all in the format you need?
On Thu, Feb 22, 2024 at 01:30:01AM -0500, Kent Overstreet wrote:
On Thu, Feb 22, 2024 at 06:48:58AM +0100, Greg KH wrote:
We NEED and REQUIRE you, if you do modify a commit from what is in Linus's tree, to say "hey, this is modified because of X and Y", not to just not say anything and assume that we should blindly take a modified change. You don't want us to do that, right?
I can provide commit messages in the format you need - but also: _none_ of this is documented in stable-kernel-rules.rst, so I'm going to want something clear and specific I can go off of.
This part is explicitly documented in stable-kernel-rules.rst. In particular, it lists the 3 ways one could send us commits for inclusion (https://docs.kernel.org/process/stable-kernel-rules.html#procedure-for-submi...). To summarize, it's your choice of:
1. Tag it with a stable@ tag. OR: 2. Send us a mail to the stable mailing list with commit IDs for us to cherry pick. OR: 3. Send the patches themselves (which is the option you've picked), and then the doc proceeds to call out how to include the upstream commit ID in the patch (https://docs.kernel.org/process/stable-kernel-rules.html#option-3).
On středa 21. února 2024 23:58:30 CET Sasha Levin wrote:
On Wed, Feb 21, 2024 at 07:10:02PM +0100, Vlastimil Babka wrote:
On 2/21/24 18:57, Greg KH wrote:
On Wed, Feb 21, 2024 at 05:00:05PM +0100, Oleksandr Natalenko wrote:
On středa 21. února 2024 15:53:11 CET Greg KH wrote:
Given the huge patch volume that the stable tree manages (30-40 changes accepted a day, 7 days a week), any one kernel subsystem that wishes to do something different only slows down everyone else.
Lower down the volume then? Raise the bar for what gets backported? Stable kernel releases got unnecessarily big [1] (Jiří is in Cc). Those 40 changes a day cannot get a proper review. Each stable release tries to mimic -rc except -rc is in consistent state while "stable" is just a bunch of changes picked here and there.
If you can point out any specific commits that we should not be taking, please let us know.
Personally I think we are not taking enough, and are still missing real fixes. Overall, this is only a very small % of what goes into Linus's tree every day, so by that measure alone, we know we are missing things.
What % of what goes into Linus's tree do you think fits within the rules stated in Documentation/process/stable-kernel-rules.rst ? I don't know but "very small" would be my guess, so we should be fine as it is?
Or are the rules actually still being observed? I doubt e.g. many of the AUTOSEL backports fit them? Should we rename the file to stable-rules-nonsense.rst?
Hey, I have an exercise for you which came up last week during the whole CVE thing!
Take a look at a random LTS kernel (I picked 5.10), in particular at the CVEs assigned to the kernel (in my case I relied on https://github.com/nluedtke/linux_kernel_cves/blob/master/data/5.10/5.10_sec...).
See how many of those actually have a stable@ tag to let us know that we need to pull that commit. (spoiler alert: in the 5.10 case it was ~33%)
Do you have a better way for us to fish for the remaining 67%?
With all due respect, once a measure becomes a target, it ceases to be a good measure.
Yeah, some have a Fixes tag, (it's not in stable-kernel-rules.rst!), and in the 5.10 case it would have helped with about half of the commits, but even then - what do we do with the remaining half?
The argument you're making is in favor of just ignoring it until they get a CVE assigned (and even then, would we take them if it goes against stable-kernel-rules.rst?), but then we end up leaving users exposed for *years* as evidenced by some CVEs.
So if we go with the current workflow, folks complain that we take too many patches. If we were to lean strictly to what stable-kernel-rules.rst says, we'd apparently miss most of the (security) issues affecting users.
It's not a catastrophic problem to miss fixes, you will never be able to reach 100% anyway, guaranteed. Introducing a new, untested and not reviewed code on scale is a bigger problem. Yes, backports should be considered as a new code that needs appropriate review. Regardless of how skilled you two are, it's not a task for you. There must be another way of doing this.
Hi!
Personally I think we are not taking enough, and are still missing real fixes. Overall, this is only a very small % of what goes into Linus's tree every day, so by that measure alone, we know we are missing things.
What % of what goes into Linus's tree do you think fits within the rules stated in Documentation/process/stable-kernel-rules.rst ? I don't know but "very small" would be my guess, so we should be fine as it is?
Or are the rules actually still being observed? I doubt e.g. many of the AUTOSEL backports fit them? Should we rename the file to stable-rules-nonsense.rst?
There seems to be just one rule being observed: "It or an equivalent fix must already exist in Linus' tree (upstream).". Every other rule is broken pretty much all the time.
AUTOSEL is a problem.
Plus there's problem with dependencies -- if a patch A is need for fix B, the rules pretty much go out of the window, huge patches are applied, whitespace fixes are applied, etc.
There are even known-bad patches being applied, and then reverted. Greg explained that it heps his process somehow.
For example in 6.1.53 review, my notes say 30% of the patches did not match the documented rules. 42% for v6.1.76.
OTOH ammount of patches that cause "real" problems is not that great, and we seem to have enough testing. Still, updating the documentation to match the reality would be good (perhaps explaining that stable does not have manpower to re-do the dependencies, and how "apply bad and revert" works).
Best regards, Pavel
On Thu, Feb 22, 2024 at 08:19:06PM +0100, Pavel Machek wrote:
Hi!
Personally I think we are not taking enough, and are still missing real fixes. Overall, this is only a very small % of what goes into Linus's tree every day, so by that measure alone, we know we are missing things.
What % of what goes into Linus's tree do you think fits within the rules stated in Documentation/process/stable-kernel-rules.rst ? I don't know but "very small" would be my guess, so we should be fine as it is?
Or are the rules actually still being observed? I doubt e.g. many of the AUTOSEL backports fit them? Should we rename the file to stable-rules-nonsense.rst?
There seems to be just one rule being observed: "It or an equivalent fix must already exist in Linus' tree (upstream).". Every other rule is broken pretty much all the time.
AUTOSEL is a problem.
Plus there's problem with dependencies -- if a patch A is need for fix B, the rules pretty much go out of the window, huge patches are applied, whitespace fixes are applied, etc.
There are even known-bad patches being applied, and then reverted. Greg explained that it heps his process somehow.
This seems to be a pretty consistent theme theme - thins are done baesd on whatever makes Greg's process easier, not input from the people stable ought to be working with. Pretty questionable set of priorities if you ask me.
Hi!
There seems to be just one rule being observed: "It or an equivalent fix must already exist in Linus' tree (upstream).". Every other rule is broken pretty much all the time.
AUTOSEL is a problem.
Plus there's problem with dependencies -- if a patch A is need for fix B, the rules pretty much go out of the window, huge patches are applied, whitespace fixes are applied, etc.
There are even known-bad patches being applied, and then reverted. Greg explained that it heps his process somehow.
This seems to be a pretty consistent theme theme - thins are done baesd on whatever makes Greg's process easier, not input from the people stable ought to be working with. Pretty questionable set of priorities if you ask me.
Well, I'd not mind stable process following the documented rules.
But fixing the documentation to match the reality would also be an improvement, because some people actually read it and expect it to be followed.
Best regards, Pavel
On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
Hi stable team - please don't take patches for fs/bcachefs/ except from myself; I'll be doing backports and sending pull requests after stuff has been tested by my CI.
Thanks, and let me know if there's any other workflow things I should know about
Sure, we can ignore fs/bcachefs/ patches.
I see that you even acked this.
What the fuck?
Is this really how you communicate with other humans?
So what happened here is that my script apparently crashes on empty "Fixes:" annotations (it can deal with invalid commits, various combinations of sha1 and description, but apparently not with just nothing at all).
And so I've fixed my scripts, so hopefully this shouldn't happen again, but I'd ask for 2 things:
1. Next time you should mail me, could you try and be more civil?
2. Consider fixing your scripts .https://docs.kernel.org/process/submitting-patches.html#using-reported-by-te... is pretty explicit as to how "Fixes:" tags are supposed to look like.
On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
Hi stable team - please don't take patches for fs/bcachefs/ except from myself; I'll be doing backports and sending pull requests after stuff has been tested by my CI.
Now done: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/comm...
We will ignore it for any "Fixes:" tags, or AUTOSEL, but if you explicitly add a "cc: stable@" in the signed-off-by area, we will pick that up.
Thanks, and let me know if there's any other workflow things I should know about
This is going to cause you more work, but less for us, so thanks!
greg k-h
On Tue, Jan 16, 2024 at 03:13:08PM +0100, Greg KH wrote:
On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
Hi stable team - please don't take patches for fs/bcachefs/ except from myself; I'll be doing backports and sending pull requests after stuff has been tested by my CI.
Now done: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/comm...
We will ignore it for any "Fixes:" tags, or AUTOSEL, but if you explicitly add a "cc: stable@" in the signed-off-by area, we will pick that up.
Would it work for your process to ignore cc: stable@ as well?
I want a tag that I can have tooling grep for later that means "this patch should be backported later, after seeing sufficient testing", and cc: stable@ has that meaning in current usage.
Or if you'd like that to be reserved for yourself we could think of a new one.
Thanks, and let me know if there's any other workflow things I should know about
This is going to cause you more work, but less for us, so thanks!
per our conversation on IRC I'm going to write some simple tooling for grepping through the git log for patches that may want to be backported and haven't yet made it to a stable tree.
Also, Sasha, is there any reason your autosel tool couldn't be made available for others to use that way?
Yes, it's more work for myself, but I have strong opinions that I as the person who knows the code ought to be picking the patches to backport, and doing the backports and resolving merge conflicts when necessary; but collaborating on tooling would be _fantastic_.
On Tue, Jan 16, 2024 at 12:26:38PM -0500, Kent Overstreet wrote:
On Tue, Jan 16, 2024 at 03:13:08PM +0100, Greg KH wrote:
On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
Hi stable team - please don't take patches for fs/bcachefs/ except from myself; I'll be doing backports and sending pull requests after stuff has been tested by my CI.
Now done: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/comm...
We will ignore it for any "Fixes:" tags, or AUTOSEL, but if you explicitly add a "cc: stable@" in the signed-off-by area, we will pick that up.
Would it work for your process to ignore cc: stable@ as well?
Nope! That's explicit, you add it for when you want us to pick up a change when it hits Linus's tree. If you don't want that to happen, then don't add it.
I want a tag that I can have tooling grep for later that means "this patch should be backported later, after seeing sufficient testing", and cc: stable@ has that meaning in current usage.
Just use a "Fixes:" tag then, that's what networking did for years before they gave up and now use cc: stable.
thanks,
greg k-h
linux-stable-mirror@lists.linaro.org
-
Greg KH -
Kent Overstreet -
Oleksandr Natalenko -
Pavel Machek -
Sasha Levin -
Vlastimil Babka