This is a first draft at a process for getting things integrated into the Linaro Stable Kernel releases. Any feedback is appreciated, hopefully this is all fairly boring and uncontroversial so there shouldn't be too many surprises.
The first thing to say here is that all LSK releases will be based off the latest generic Linux stable kernel so the best way to get a change into a Linaro release is to get it into the generic Linux stable kernel using the standard processes. This will maximise the number of people who can use the change and is just generally good practice.
New features ------------
Features for the stable kernel are agreed by the TSC. Once a feature has been agreed by the TSC there should be an owner assigned to deliver a feature branch into the stable kernel and work with the stable kernel team to resolve any integration issues at least up until the feature has been included in a release. This will be done per kernel version.
These feature branches should be based on the relevant upstream kernel as far as possible (any dependencies on other branches should be discussed with the stable kernel team). Some information about where the code came fromm should be included along with the code, in order of preference:
1. Commit IDs from the standard kernel in the changelogs of the individual patches. 2. A description of how the equivalent change was made upstream or why it isn't required in LSK (eg, explaining that this is taken care of by features not present in the stable kernel). 3. References to where out of tree development is happening including contact information for followup.
The code should be sent as a pull request or patches, with review by the stable team following normal kernel process and focusing on backporting and integration issues. Relevant testing infrastructure should also be available in LAVA for QA and the review will also include ensuring that the testsuite passes with the changes integrated.
Once the code has been accepted it will be stored as a branch in the LSK tree and the submission branch can be deleted.
Updating code in the LSK ------------------------
The LSK can be updated either by replacing an existing topic branch or by submitting incremental patches. Replacement would be most useful in cases where a feature has not yet been merged into the standard kernel and is still being redeveloped there but otherwise incremental updates are preferred. The process for submitting changes is the same as for new features with the exception that incremental updates should be based on the topic branch in the LSK git rather than the standard kernel.
Hi Mark -
It sounds good to me.
There's some extra things to consider in terms of internal consumers of this tree though. They're probably Andrey's problems not the LTS kernel directly but we should discuss.
1) At the moment the Landing Team kernels all (?) use llct as a basis. From there on top of mainline we get some upstream-headed things, and some less upstream-headed things like Androidization.
So in addition to having this backport-focused "clean" mainline tree, someone, probably Andrey, will need to look after a "flavoured" llc-LTS version which follows your mainline -focused tree but has the best updated versions of the original llct content as well. Depending on what the original llct merges were, LTS maintenance of those may look like swapping out the original series for backporting the actual series of the same patches that went upstream, because those are the ones that will get fixes going on.
One way to do that is retain the original merge sources and Andrey recomposes on top of the LTS kernel each update... that makes it easy to swap out the content of the series he is merging, and it should be fairly turnkey most of the time because almost everything is normally locked to the same version that it all applied to originally.
2) I'm not sure there's anything to do about it but having an LTS kernel at all also generates a new tension between tracking at the member, and the desire to forget about that irritating work of keeping their patch load and kernel current and "getting lost" in the LTS version until the next LTS version. We've experienced being asked to do that and no matter the reasoning, you end up with a rotting pile of unmaintained patches with nowhere to go, no matter how good back in the day (cf, tilt-3.4). So it's a worry people look at the LTS kernel and don't understand why they should bother keeping their kernel and OOT patches up to date and at least with the possibility to be polished and offered to mainline.
3) Once people who don't fully understand the issues get a taste of backport candy, it becomes the answer to everything, for example, "we have so much investment in 3.0 let's backport big.LITTLE support to that". That kind of thing just digs the grave deeper and wastes engineering resources, so I hope (given who's doing it, actually, I assume ^^) we have a calibrated approach in mind for NAKing things that someone may have gotten themselves in a position they really want, but which definitely go in a bad direction for everyone, even the people asking.
-Andy
On 20 June 2013 00:32, Mark Brown broonie@linaro.org wrote:
This is a first draft at a process for getting things integrated into the Linaro Stable Kernel releases. Any feedback is appreciated, hopefully this is all fairly boring and uncontroversial so there shouldn't be too many surprises.
The first thing to say here is that all LSK releases will be based off the latest generic Linux stable kernel so the best way to get a change into a Linaro release is to get it into the generic Linux stable kernel using the standard processes. This will maximise the number of people who can use the change and is just generally good practice.
New features
Features for the stable kernel are agreed by the TSC. Once a feature has been agreed by the TSC there should be an owner assigned to deliver a feature branch into the stable kernel and work with the stable kernel team to resolve any integration issues at least up until the feature has been included in a release. This will be done per kernel version.
These feature branches should be based on the relevant upstream kernel as far as possible (any dependencies on other branches should be discussed with the stable kernel team). Some information about where the code came fromm should be included along with the code, in order of preference:
1. Commit IDs from the standard kernel in the changelogs of the individual patches. 2. A description of how the equivalent change was made upstream or why it isn't required in LSK (eg, explaining that this is taken care of by features not present in the stable kernel). 3. References to where out of tree development is happening including contact information for followup.The code should be sent as a pull request or patches, with review by the stable team following normal kernel process and focusing on backporting and integration issues. Relevant testing infrastructure should also be available in LAVA for QA and the review will also include ensuring that the testsuite passes with the changes integrated.
Once the code has been accepted it will be stored as a branch in the LSK tree and the submission branch can be deleted.
Updating code in the LSK
The LSK can be updated either by replacing an existing topic branch or by submitting incremental patches. Replacement would be most useful in cases where a feature has not yet been merged into the standard kernel and is still being redeveloped there but otherwise incremental updates are preferred. The process for submitting changes is the same as for new features with the exception that incremental updates should be based on the topic branch in the LSK git rather than the standard kernel.
linaro-kernel mailing list linaro-kernel@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-kernel
On Thu, Jun 20, 2013 at 10:29:43AM +0800, Andy Green wrote:
Hi Mark -
Please don't top post on upstream style lists such as this one.
- At the moment the Landing Team kernels all (?) use llct as a basis.
From there on top of mainline we get some upstream-headed things, and some less upstream-headed things like Androidization.
So in addition to having this backport-focused "clean" mainline tree, someone, probably Andrey, will need to look after a "flavoured" llc-LTS version which follows your mainline -focused tree but has the best updated versions of the original llct content as well. Depending
Is there some use case of the LTS kernels where someone would want to do this? I don't think I'm aware of it if there is.
- I'm not sure there's anything to do about it but having an LTS
kernel at all also generates a new tension between tracking at the member, and the desire to forget about that irritating work of keeping their patch load and kernel current and "getting lost" in the LTS version until the next LTS version. We've experienced being asked to do that and no matter the reasoning, you end up with a rotting pile of unmaintained patches with nowhere to go, no matter how good back in the day (cf, tilt-3.4). So it's a worry people look at the LTS kernel and don't understand why they should bother keeping their kernel and OOT patches up to date and at least with the possibility to be polished and offered to mainline.
This is happening anyway and will continue to happen for the forseeable future in the consumer electronics space, it's just a function of shipping product on something other than -next really. I'd actually hope that our strong policy on using upstream code and backporting it might help a little here, keeping things more in sync. It's not a particular problem to work on a backport for a product so long as the backport and upstream are reasonably in sync.
- Once people who don't fully understand the issues get a taste of
backport candy, it becomes the answer to everything, for example, "we have so much investment in 3.0 let's backport big.LITTLE support to that". That kind of thing just digs the grave deeper and wastes engineering resources, so I hope (given who's doing it, actually, I assume ^^) we have a calibrated approach in mind for NAKing things that someone may have gotten themselves in a position they really want, but which definitely go in a bad direction for everyone, even the people asking.
I don't think the people you're worrying about here have any intention of running a LTS kernel directly, the broad plan is that they will take the LTS and use it as a basis in the same way they currently do with upstream.
The investment in old kernels thing that you mention is not really an issue these days thanks to Android, it's got people used to an upgrade cycle. The reason people standardise on v3.4 right now is more things like being more comfortable using whatever Google were using for their lead device and it giving all the component makers a target to aim at so the product people don't have integration hassle.
On 20 June 2013 19:16, Mark Brown broonie@sirena.org.uk wrote:
On Thu, Jun 20, 2013 at 10:29:43AM +0800, Andy Green wrote:
Hi Mark -
Please don't top post on upstream style lists such as this one.
shrug
- At the moment the Landing Team kernels all (?) use llct as a basis.
From there on top of mainline we get some upstream-headed things, and some less upstream-headed things like Androidization.
So in addition to having this backport-focused "clean" mainline tree, someone, probably Andrey, will need to look after a "flavoured" llc-LTS version which follows your mainline -focused tree but has the best updated versions of the original llct content as well. Depending
Is there some use case of the LTS kernels where someone would want to do this? I don't think I'm aware of it if there is.
- I'm not sure there's anything to do about it but having an LTS
kernel at all also generates a new tension between tracking at the member, and the desire to forget about that irritating work of keeping their patch load and kernel current and "getting lost" in the LTS version until the next LTS version. We've experienced being asked to do that and no matter the reasoning, you end up with a rotting pile of unmaintained patches with nowhere to go, no matter how good back in the day (cf, tilt-3.4). So it's a worry people look at the LTS kernel and don't understand why they should bother keeping their kernel and OOT patches up to date and at least with the possibility to be polished and offered to mainline.
This is happening anyway and will continue to happen for the forseeable future in the consumer electronics space, it's just a function of shipping product on something other than -next really. I'd actually hope that our strong policy on using upstream code and backporting it might help a little here, keeping things more in sync. It's not a particular problem to work on a backport for a product so long as the backport and upstream are reasonably in sync.
Linaro works with its members to maintain kernels for them via Landing Teams. Linaro has always had a "strong policy on upstream code" but that is the art of the possible when you work directly with member companies who variously may not want things upstreamed because somebody might learn something about their IP, have a large number of engineers already supposedly upstreaming things or not even understand why upstreaming things is worth the time.
I'm sure you have a good handle on nice clean, sane upstream world but I'm telling you something specific about Linaro and LT experience with members.
- Once people who don't fully understand the issues get a taste of
backport candy, it becomes the answer to everything, for example, "we have so much investment in 3.0 let's backport big.LITTLE support to that". That kind of thing just digs the grave deeper and wastes engineering resources, so I hope (given who's doing it, actually, I assume ^^) we have a calibrated approach in mind for NAKing things that someone may have gotten themselves in a position they really want, but which definitely go in a bad direction for everyone, even the people asking.
I don't think the people you're worrying about here have any intention of running a LTS kernel directly, the broad plan is that they will take the LTS and use it as a basis in the same way they currently do with upstream.
You clearly do not know the people I am worrying about.
The investment in old kernels thing that you mention is not really an issue these days thanks to Android, it's got people used to an upgrade cycle. The reason people standardise on v3.4 right now is more things like being more comfortable using whatever Google were using for their lead device and it giving all the component makers a target to aim at so the product people don't have integration hassle.
What that causes is a large number of engineers spending time on "disposable kernels" that are not aligned with, and don't feed upstream. We saw it at TI before many of the "large number of engineers" wasting time on that got canned (it didn't stop them just slowed them down ^^). People are still working on 3.0 even and 3.4 now, sometimes without DT, when 3.10 will soon be here, it's actually not a good situation and not in itself what Linaro is meant to encourage. To the extent an LTS kernel will encourage / enable / legitimize that when people could be on tracking kernels, it's a problem. In other cases LTS usage is legitimate especially if the platform is mature.
In the Landing Teams, it's Linaro's policy to encourage the members to make sure their stuff is tracking Linus' HEAD so they always have a contemporary kernel with all the pieces, reducing the barrier to upstreaming the content.
-Andy
On Thu, Jun 20, 2013 at 08:02:54PM +0800, Andy Green wrote:
On 20 June 2013 19:16, Mark Brown broonie@sirena.org.uk wrote:
Please don't top post on upstream style lists such as this one.
shrug
It's important to follow good practice here; it helps set a good example for people of upstream processes.
This is happening anyway and will continue to happen for the forseeable future in the consumer electronics space, it's just a function of shipping product on something other than -next really. I'd actually hope that our strong policy on using upstream code and backporting it might help a little here, keeping things more in sync. It's not a particular problem to work on a backport for a product so long as the backport and upstream are reasonably in sync.
Linaro works with its members to maintain kernels for them via Landing Teams. Linaro has always had a "strong policy on upstream code" but that is the art of the possible when you work directly with member companies who variously may not want things upstreamed because somebody might learn something about their IP, have a large number of engineers already supposedly upstreaming things or not even understand why upstreaming things is worth the time.
I'm sure you have a good handle on nice clean, sane upstream world but I'm telling you something specific about Linaro and LT experience with members.
So, my previous job was all about shipping into smartphones so I'm more than familiar with the situation here - this is in no way specific to Linaro, it's a pressure felt by everyone shipping into the consumer electronics space. To me part of teaching people about the value of upstreaming is teaching them how that process interacts with and helps the process of stabalisation for release.
I don't think the people you're worrying about here have any intention of running a LTS kernel directly, the broad plan is that they will take the LTS and use it as a basis in the same way they currently do with upstream.
You clearly do not know the people I am worrying about.
I'm pretty sure I've actually been one of them, at least from the point of view of a commercial pressure/delivery point of view.
The investment in old kernels thing that you mention is not really an issue these days thanks to Android, it's got people used to an upgrade cycle. The reason people standardise on v3.4 right now is more things like being more comfortable using whatever Google were using for their lead device and it giving all the component makers a target to aim at so the product people don't have integration hassle.
What that causes is a large number of engineers spending time on "disposable kernels" that are not aligned with, and don't feed upstream. We saw it at TI before many of the "large number of
Sure, it's not ideal and one of the things you guys on the landing teams need to be teaching people is how to do this effectively. The issue here isn't that people are picking a kernel and stabalising for release on that, it's that they're frequently doing so in an ineffective fashion. What you're talking about here isn't sticking to a single kernel and staying with it for ever more, it's something different to that. More on this below...
engineers" wasting time on that got canned (it didn't stop them just slowed them down ^^). People are still working on 3.0 even and 3.4 now, sometimes without DT, when 3.10 will soon be here, it's actually
Essentially everyone in the Android space will be working on v3.4 up until the release of Key Lime Pie. That's the stable kernel for Jelly Bean, that's what everyone works with. For ICS it was v3.0, though that's pretty much dead at this point for anything except support.
As far as device tree goes... it's not really solving a problem that people doing phones have. People will use it (and IME do use it) when it's a standard feature in the baseline kernel but there's no real pull to backport it, if anything it's likely to cause problems with third party code as nobody else is using it on that kernel version so there's unlikely to be DT support in their drivers.
not a good situation and not in itself what Linaro is meant to encourage. To the extent an LTS kernel will encourage / enable / legitimize that when people could be on tracking kernels, it's a problem. In other cases LTS usage is legitimate especially if the platform is mature.
In the Landing Teams, it's Linaro's policy to encourage the members to make sure their stuff is tracking Linus' HEAD so they always have a contemporary kernel with all the pieces, reducing the barrier to upstreaming the content.
So what I think the landing teams need to be doing here is unchanged - you need to be showing people how to develop in such a way that the backport/release code and upstream code don't shift too far apart so work done in one can be readily applied to the other, and helping them make sure that things are kept in sync so this is tractable.
I would actually hope that in producing a stable kernel Linaro will provide a positive example of how this can be done effectively that people can look to when trying to understand what's going on. When we keep our stable trees trackable back to current upstream work we will be showing this process in action which the landing teams will be able to point to when making the case for this approach to their member companies.
The other thing that is often a good example here is going through the process of moving to a new kernel version when you've been tracking upstream - this often brings home the advantages to people. I've seen some companies in this space go from just not caring about upstream to strongly advocating for upstream after going through this process.
On 20 June 2013 17:22, Mark Brown broonie@linaro.org wrote:
On Thu, Jun 20, 2013 at 08:02:54PM +0800, Andy Green wrote:
On 20 June 2013 19:16, Mark Brown broonie@sirena.org.uk wrote:
Please don't top post on upstream style lists such as this one.
shrug
It's important to follow good practice here; it helps set a good example for people of upstream processes.
This is happening anyway and will continue to happen for the forseeable future in the consumer electronics space, it's just a function of shipping product on something other than -next really. I'd actually hope that our strong policy on using upstream code and backporting it might help a little here, keeping things more in sync. It's not a particular problem to work on a backport for a product so long as the backport and upstream are reasonably in sync.
Linaro works with its members to maintain kernels for them via Landing Teams. Linaro has always had a "strong policy on upstream code" but that is the art of the possible when you work directly with member companies who variously may not want things upstreamed because somebody might learn something about their IP, have a large number of engineers already supposedly upstreaming things or not even understand why upstreaming things is worth the time.
I'm sure you have a good handle on nice clean, sane upstream world but I'm telling you something specific about Linaro and LT experience with members.
So, my previous job was all about shipping into smartphones so I'm more than familiar with the situation here - this is in no way specific to Linaro, it's a pressure felt by everyone shipping into the consumer electronics space. To me part of teaching people about the value of upstreaming is teaching them how that process interacts with and helps the process of stabalisation for release.
I don't think the people you're worrying about here have any intention of running a LTS kernel directly, the broad plan is that they will take the LTS and use it as a basis in the same way they currently do with upstream.
You clearly do not know the people I am worrying about.
I'm pretty sure I've actually been one of them, at least from the point of view of a commercial pressure/delivery point of view.
The investment in old kernels thing that you mention is not really an issue these days thanks to Android, it's got people used to an upgrade cycle. The reason people standardise on v3.4 right now is more things like being more comfortable using whatever Google were using for their lead device and it giving all the component makers a target to aim at so the product people don't have integration hassle.
What that causes is a large number of engineers spending time on "disposable kernels" that are not aligned with, and don't feed upstream. We saw it at TI before many of the "large number of
Sure, it's not ideal and one of the things you guys on the landing teams need to be teaching people is how to do this effectively. The issue here isn't that people are picking a kernel and stabalising for release on that, it's that they're frequently doing so in an ineffective fashion. What you're talking about here isn't sticking to a single kernel and staying with it for ever more, it's something different to that. More on this below...
engineers" wasting time on that got canned (it didn't stop them just slowed them down ^^). People are still working on 3.0 even and 3.4 now, sometimes without DT, when 3.10 will soon be here, it's actually
Essentially everyone in the Android space will be working on v3.4 up until the release of Key Lime Pie. That's the stable kernel for Jelly Bean, that's what everyone works with. For ICS it was v3.0, though that's pretty much dead at this point for anything except support.
As far as device tree goes... it's not really solving a problem that people doing phones have. People will use it (and IME do use it) when it's a standard feature in the baseline kernel but there's no real pull to backport it, if anything it's likely to cause problems with third party code as nobody else is using it on that kernel version so there's unlikely to be DT support in their drivers.
not a good situation and not in itself what Linaro is meant to encourage. To the extent an LTS kernel will encourage / enable / legitimize that when people could be on tracking kernels, it's a problem. In other cases LTS usage is legitimate especially if the platform is mature.
In the Landing Teams, it's Linaro's policy to encourage the members to make sure their stuff is tracking Linus' HEAD so they always have a contemporary kernel with all the pieces, reducing the barrier to upstreaming the content.
So what I think the landing teams need to be doing here is unchanged - you need to be showing people how to develop in such a way that the backport/release code and upstream code don't shift too far apart so work done in one can be readily applied to the other, and helping them make sure that things are kept in sync so this is tractable.
I don't see that fitting in with my LT's plans at all. LSK has not been accounted for at all, other than as a team requiring support on an ad-hoc basis, along with many other teams inside Linaro.
I would actually hope that in producing a stable kernel Linaro will provide a positive example of how this can be done effectively that people can look to when trying to understand what's going on. When we keep our stable trees trackable back to current upstream work we will be showing this process in action which the landing teams will be able to point to when making the case for this approach to their member companies.
The other thing that is often a good example here is going through the process of moving to a new kernel version when you've been tracking upstream - this often brings home the advantages to people. I've seen some companies in this space go from just not caring about upstream to strongly advocating for upstream after going through this process.
linaro-kernel mailing list linaro-kernel@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-kernel
On Thu, Jun 20, 2013 at 06:06:03PM +0100, Ryan Harkin wrote:
On 20 June 2013 17:22, Mark Brown broonie@linaro.org wrote:
So what I think the landing teams need to be doing here is unchanged - you need to be showing people how to develop in such a way that the backport/release code and upstream code don't shift too far apart so work done in one can be readily applied to the other, and helping them make sure that things are kept in sync so this is tractable.
I don't see that fitting in with my LT's plans at all. LSK has not been accounted for at all, other than as a team requiring support on an ad-hoc basis, along with many other teams inside Linaro.
I'm not talking here about the LSK but rather about the productisation trees that the consumer electronics SoC vendors are shipping to their customers. I don't know if that's an issue at all at ARM, given that ARM ship IP not silicon I'd not at all be surprised if they didn't have such things and therefore there's no need to worry about integrating upstream into the development process. What I was discussing with Andy was the challenge that many of the landing teams face when trying to add upstream to a productisation based workflow.
There are some other places where this won't be much of an issue, for example in the enterprise space the focus is strongly on upstream anyway so there aren't really the same productisation kernels,
On 21 June 2013 00:22, Mark Brown broonie@linaro.org wrote:
On Thu, Jun 20, 2013 at 08:02:54PM +0800, Andy Green wrote:
On 20 June 2013 19:16, Mark Brown broonie@sirena.org.uk wrote:
Please don't top post on upstream style lists such as this one.
shrug
It's important to follow good practice here; it helps set a good example for people of upstream processes.
Unless I use my phone, the last 10+ years my mail is all inline reply. This last week I have to use gmail interface for the first time.
This is happening anyway and will continue to happen for the forseeable future in the consumer electronics space, it's just a function of shipping product on something other than -next really. I'd actually hope that our strong policy on using upstream code and backporting it might help a little here, keeping things more in sync. It's not a particular problem to work on a backport for a product so long as the backport and upstream are reasonably in sync.
Linaro works with its members to maintain kernels for them via Landing Teams. Linaro has always had a "strong policy on upstream code" but that is the art of the possible when you work directly with member companies who variously may not want things upstreamed because somebody might learn something about their IP, have a large number of engineers already supposedly upstreaming things or not even understand why upstreaming things is worth the time.
I'm sure you have a good handle on nice clean, sane upstream world but I'm telling you something specific about Linaro and LT experience with members.
So, my previous job was all about shipping into smartphones so I'm more than familiar with the situation here - this is in no way specific to Linaro, it's a pressure felt by everyone shipping into the consumer electronics space. To me part of teaching people about the value of upstreaming is teaching them how that process interacts with and helps the process of stabalisation for release.
This kind of assumes the Landing Teams are in a position where people are interested in their "teachings". More often someone in higher management has realized that the LT can do them a lot of good, but on the ground that's not always understood. If the company in question has established fiefdoms, it can get difficult even reaching agreement on LT necessities like having a rebase tree (the otherwise excellent Felipe Balbi at a member gave me a nice lecture about his certainties we should use history trees, because the reasoning was outside his experience).
We're basically there to advise the right thing but as a kind of consultancy action to help them get what they want done. At the moment in Fujitsu our advice is mostly taken which is great for them and us, but in other LT past and future that's not always the case. If the member insists, we're gong to only work on LTS kernel now that's actually what will happen because that's their business decision to make.
Anyway the point is we're used to people explaining to use what we should do and how to do it when they don't understand the relationship with the member. We are definitely motivated to do the Right Thing if we look glum as we appear to be doing the wrong thing it may be because there's nothing we can do about it.
As I said before this issue isn't necessarily your problem but because the TSC is controlled by members, what you may get asked to do is unbounded in fact. What I'm suggesting is if you see a real piece of madness coming, please push back. Some mixtures of LTS base version and backport request will get asked in order to avoid the work they ought to be doing to base on a later kernel; that work then and the retardation of alignment of the member with upstream can be bad news. Consider for example 3.0 basis and big.LITTLE backport. Maybe if you start from 3.9 or other modern basis and v8 support is already started in there, the worst kind of thing won't happen any more, in which case great.
The investment in old kernels thing that you mention is not really an issue these days thanks to Android, it's got people used to an upgrade cycle. The reason people standardise on v3.4 right now is more things like being more comfortable using whatever Google were using for their lead device and it giving all the component makers a target to aim at so the product people don't have integration hassle.
What that causes is a large number of engineers spending time on "disposable kernels" that are not aligned with, and don't feed upstream. We saw it at TI before many of the "large number of
Sure, it's not ideal and one of the things you guys on the landing teams need to be teaching people is how to do this effectively. The issue
As I explained above, even though the member pays for the LT it does not mean that the various parts of the member which may be under a different management structure than called us in, will listen to us. We also met characters who prioritized upstreaming over servicing the member's customers at all, which is a different form of getting it fatally wrong.
We found running a tracking kernel is a good balance, it forces all their driver assets to be updated and aligned enough to at least work with current kernels, and at the time when Linus releases, we can fork off a stable branch ourselves with hopefully everything working. If later they hop from eg 3.4 to 3.9, well we already have a 3.9 tree that is working from tracking at that time to get started with. In this case, it should be possible to look at rebase or merge of that 3.9 tree with the LTS one in a fairly compatible way. So it all seems workable.
here isn't that people are picking a kernel and stabalising for release on that, it's that they're frequently doing so in an ineffective fashion. What you're talking about here isn't sticking to a single kernel and staying with it for ever more, it's something different to that. More on this below...
engineers" wasting time on that got canned (it didn't stop them just slowed them down ^^). People are still working on 3.0 even and 3.4 now, sometimes without DT, when 3.10 will soon be here, it's actually
Essentially everyone in the Android space will be working on v3.4 up until the release of Key Lime Pie. That's the stable kernel for Jelly Bean, that's what everyone works with. For ICS it was v3.0, though that's pretty much dead at this point for anything except support.
Not in "some" companies: it's still alive and getting new engineering work done on it, despite it being explained it's like giving birth to old men. For the people who think that's OK, I'm worrying the LTS kernel will allow them to argue for that approach.
As far as device tree goes... it's not really solving a problem that people doing phones have. People will use it (and IME do use it) when it's a standard feature in the baseline kernel but there's no real pull to backport it, if anything it's likely to cause problems with third party code as nobody else is using it on that kernel version so there's unlikely to be DT support in their drivers.
Yes, I mention it because it's a sign there's no investment towards alignment with upstream, where DT is required.
However the code we have seen from vendor suppliers for drivers is really nasty in many cases. It's quite normal to embed physical addresses and IRQs direct in the driver code, we even got one where they had put Samsung pre-generic gpio gpio api usage right in there (the driver is nothing Samsung specific). Even Mali has that kind of stuff piled in there. So even for phones DT will be very valuable (and I noticed yesterday, Chromebook kernels have DT attached to kernel binary). It is expensive running around dealing with and maintaining that.
I agree with what you're saying they don't feel that yet and there's no DT backport pressure, it's because they are used to one kernel per product and wasting time running around the code meddling as a one-off, rather than one kernel per member and CONFIG_ARCH_MULTIPLATFORM / DT and do the work once.
not a good situation and not in itself what Linaro is meant to encourage. To the extent an LTS kernel will encourage / enable / legitimize that when people could be on tracking kernels, it's a problem. In other cases LTS usage is legitimate especially if the platform is mature.
In the Landing Teams, it's Linaro's policy to encourage the members to make sure their stuff is tracking Linus' HEAD so they always have a contemporary kernel with all the pieces, reducing the barrier to upstreaming the content.
So what I think the landing teams need to be doing here is unchanged - you need to be showing people how to develop in such a way that the backport/release code and upstream code don't shift too far apart so work done in one can be readily applied to the other, and helping them make sure that things are kept in sync so this is tractable.
Right, I also believe this is already a good way and want to keep it going that way.
I would actually hope that in producing a stable kernel Linaro will provide a positive example of how this can be done effectively that people can look to when trying to understand what's going on. When we keep our stable trees trackable back to current upstream work we will be showing this process in action which the landing teams will be able to point to when making the case for this approach to their member companies.
The other thing that is often a good example here is going through the process of moving to a new kernel version when you've been tracking upstream - this often brings home the advantages to people. I've seen some companies in this space go from just not caring about upstream to strongly advocating for upstream after going through this process.
Yes tracking has opened some eyes in my experience too. There's a lot of disbelief initially that the massive agony of shifting off an old kernel can actually be almost painless when done week by week. However I found it has to be combined with an approach of squashing to single topic patches. At TI we did not do this and ended up dragging 2.5K patches of "internal development history" around with us rebasing it all every time... we never ever cared about that history. We should have refactored and squashed it into a smaller number of "patches approximating what you would send upstream" for the various subsystems, as we have now done at Fujitsu. Then the tree is much easier to maintain and more meaningfully aligned with a potential upstream effort.
I discussed the llct relationship with LTS with Andrey offlist. Actually for tracking LTs don't base on Linus directly but use Andrey's linux-linaro-core-tracking. This has many interesting branches for Linaro merged in already including Androidization, and is recomposed for every -rc. It's worked out very well for us since we get things like current gator via ARM LT we don't otherwise have time to consistently care about coming in with the basis.
He mentioned that the Andriodization patches are in LTS already... what I was initially talking about is how to make sure llct merged content is still there if we later switch our LT kernel to use LTS basis. So if he added a pending series about xyz subsystem to llct for 3.x, it's arguable to expect that the accepted series for xyz subsystem appear in some downstream version of LTS for 3.x, even if it was accepted after 3.x.
As I said that's probably not your problem, it's something Andrey could take care of doing on top of your mainline-only core LTS branch, but there needs to be a discussion about if we even want that, or if content added at llct just stops at llct and your mainline-only core LTS branch is all there is.
-Andy
On Fri, Jun 21, 2013 at 09:40:43AM +0800, Andy Green wrote:
On 21 June 2013 00:22, Mark Brown broonie@linaro.org wrote:
It's important to follow good practice here; it helps set a good example for people of upstream processes.
Unless I use my phone, the last 10+ years my mail is all inline reply. This last week I have to use gmail interface for the first time.
You can actually do inline replies with gmail, it just doesn't encourage it - click on the [...] and it'll expand the quoted mail in an editable format.
So, my previous job was all about shipping into smartphones so I'm more than familiar with the situation here - this is in no way specific to Linaro, it's a pressure felt by everyone shipping into the consumer electronics space. To me part of teaching people about the value of upstreaming is teaching them how that process interacts with and helps the process of stabalisation for release.
This kind of assumes the Landing Teams are in a position where people are interested in their "teachings". More often someone in higher management has realized that the LT can do them a lot of good, but on the ground that's not always understood. If the company in question has established fiefdoms, it can get difficult even reaching agreement on LT necessities like having a rebase tree (the otherwise excellent Felipe Balbi at a member gave me a nice lecture about his certainties we should use history trees, because the reasoning was outside his experience).
Of course, I'm not saying that this is a trivial thing to sell but that doesn't mean we shouldn't be pushing it. If it reall was trivially obvious probably everyone would be doing it already :)
Anyway the point is we're used to people explaining to use what we should do and how to do it when they don't understand the relationship with the member. We are definitely motivated to do the Right Thing if we look glum as we appear to be doing the wrong thing it may be because there's nothing we can do about it.
Sure, and doing and talking are obviously different. One thing that can sometimes be done in that situation is to do the upstreaming thing on anything that's already upstream, though obviously that's not an option in many cases.
As I said before this issue isn't necessarily your problem but because the TSC is controlled by members, what you may get asked to do is unbounded in fact. What I'm suggesting is if you see a real piece of madness coming, please push back. Some mixtures of LTS base version and backport request will get asked in order to avoid the work they ought to be doing to base on a later kernel; that work then and the retardation of alignment of the member with upstream can be bad news. Consider for example 3.0 basis and big.LITTLE backport. Maybe if you start from 3.9 or other modern basis and v8 support is already started in there, the worst kind of thing won't happen any more, in which case great.
I think you know me well enough to realise that I will do that, and certainly the current requests are all for relatively modern kernels. Like I say the pressure generated by Android has been really helpful here, it creates a lot of costs for people trying to avoid moving forwards.
Sure, it's not ideal and one of the things you guys on the landing teams need to be teaching people is how to do this effectively. The issue
As I explained above, even though the member pays for the LT it does not mean that the various parts of the member which may be under a different management structure than called us in, will listen to us. We also met characters who prioritized upstreaming over servicing the member's customers at all, which is a different form of getting it fatally wrong.
I appreciate this and I do think this isn't work solely for the landing teams - there's a non-technical side here as well for example.
Essentially everyone in the Android space will be working on v3.4 up until the release of Key Lime Pie. That's the stable kernel for Jelly Bean, that's what everyone works with. For ICS it was v3.0, though that's pretty much dead at this point for anything except support.
Not in "some" companies: it's still alive and getting new engineering work done on it, despite it being explained it's like giving birth to old men. For the people who think that's OK, I'm worrying the LTS kernel will allow them to argue for that approach.
Our current plan of record is to taper off new features after a year so hopefully there's at least some back pressure there.
As far as device tree goes... it's not really solving a problem that people doing phones have. People will use it (and IME do use it) when
However the code we have seen from vendor suppliers for drivers is really nasty in many cases. It's quite normal to embed physical addresses and IRQs direct in the driver code, we even got one where they had put Samsung pre-generic gpio gpio api usage right in there (the driver is nothing Samsung specific). Even Mali has that kind of stuff piled in there. So even for phones DT will be very valuable (and I noticed yesterday, Chromebook kernels have DT attached to kernel binary). It is expensive running around dealing with and maintaining that.
Note however that the Chromebooks aren't fully supported with only DT yet - we still need to work out how to handle things like the WiFi reset. As far as the code quality thing goes my experience is that DT will just push around the quality issues, it won't stop them (as you'd expect really).
I agree with what you're saying they don't feel that yet and there's no DT backport pressure, it's because they are used to one kernel per product and wasting time running around the code meddling as a one-off, rather than one kernel per member and CONFIG_ARCH_MULTIPLATFORM / DT and do the work once.
That's not really it - the trick is to keep the active parts of the code in sync with each other to minimise redundant development and there's other ways of doing that than doing a single binary for multiple boards that people have already figured out. By the time you get to actually shipping most of the effort is on QA anyway which for thoroughness needs to be on a per system basis.
He mentioned that the Andriodization patches are in LTS already... what I was initially talking about is how to make sure llct merged content is still there if we later switch our LT kernel to use LTS basis. So if he added a pending series about xyz subsystem to llct for 3.x, it's arguable to expect that the accepted series for xyz subsystem appear in some downstream version of LTS for 3.x, even if it was accepted after 3.x.
That's what we'll be going for for for anything which is included in LTS, though that won't be absolutely everything that is being done within Linaro.
As I said that's probably not your problem, it's something Andrey could take care of doing on top of your mainline-only core LTS branch, but there needs to be a discussion about if we even want that, or if content added at llct just stops at llct and your mainline-only core LTS branch is all there is.
I suspect the backports will be too much effort there for anything with non-trivial invasiveness and that people might want the support. From an encouraging people to upstream point of view it does also seem like we should have some barriers on the backports, to a certain extent what
On 21 June 2013 19:21, Mark Brown broonie@linaro.org wrote:
On Fri, Jun 21, 2013 at 09:40:43AM +0800, Andy Green wrote:
On 21 June 2013 00:22, Mark Brown broonie@linaro.org wrote:
So, my previous job was all about shipping into smartphones so I'm more than familiar with the situation here - this is in no way specific to Linaro, it's a pressure felt by everyone shipping into the consumer electronics space. To me part of teaching people about the value of upstreaming is teaching them how that process interacts with and helps the process of stabalisation for release.
This kind of assumes the Landing Teams are in a position where people are interested in their "teachings". More often someone in higher management has realized that the LT can do them a lot of good, but on the ground that's not always understood. If the company in question has established fiefdoms, it can get difficult even reaching agreement on LT necessities like having a rebase tree (the otherwise excellent Felipe Balbi at a member gave me a nice lecture about his certainties we should use history trees, because the reasoning was outside his experience).
Of course, I'm not saying that this is a trivial thing to sell but that doesn't mean we shouldn't be pushing it. If it reall was trivially obvious probably everyone would be doing it already :)
Yes this gets pushed, and we demonstrate by doing it, which is highly effective against people saying it can't be done.
However this LKS kernel will definitely make tracking activity harder for us. The problem will be all the driver integration focus at the member will be at LKS kernel and not tracking. Since LKS has a direct line to product ship date, parallel driver integration at later tracking kernel they're not shipping will seem like a distraction. We could do the driver integration engineering work at tracking and backport to LKS, but this really is needlessly inefficient from product-focused observer POV.
If we don't do something about it for tracking, by default we end up with a single old tree that works well and no forward path for the code.
When LKS and HEAD are closely aligned, it's not going to be that difficult to follow LKS integration work on tracking. However when LKS and HEAD diverge significantly rebasing the LKS-aligned patch to HEAD each time may be a nasty job.
Perhaps if the LT tree based on LKS is kept by rebase, with the "squashed topic patches with history via tags" method, there's a way to port LKS-based driver patches initially to tracking, then squash interdiffs between the last ported tag and new tag on the LT LKS tree to the tracking tree patch for the same function. It would require care but then you only need to do the main uplevel porting once and there's a path to follow LKS-based changes.
Essentially everyone in the Android space will be working on v3.4 up until the release of Key Lime Pie. That's the stable kernel for Jelly Bean, that's what everyone works with. For ICS it was v3.0, though that's pretty much dead at this point for anything except support.
Not in "some" companies: it's still alive and getting new engineering work done on it, despite it being explained it's like giving birth to old men. For the people who think that's OK, I'm worrying the LTS kernel will allow them to argue for that approach.
Our current plan of record is to taper off new features after a year so hopefully there's at least some back pressure there.
Yes it'll stop clinging on to it forever anyway.
As far as device tree goes... it's not really solving a problem that people doing phones have. People will use it (and IME do use it) when
However the code we have seen from vendor suppliers for drivers is really nasty in many cases. It's quite normal to embed physical addresses and IRQs direct in the driver code, we even got one where they had put Samsung pre-generic gpio gpio api usage right in there (the driver is nothing Samsung specific). Even Mali has that kind of stuff piled in there. So even for phones DT will be very valuable (and I noticed yesterday, Chromebook kernels have DT attached to kernel binary). It is expensive running around dealing with and maintaining that.
Note however that the Chromebooks aren't fully supported with only DT yet - we still need to work out how to handle things like the WiFi reset. As far as the code quality thing goes my experience is that DT will just push around the quality issues, it won't stop them (as you'd expect really).
I agree with what you're saying they don't feel that yet and there's no DT backport pressure, it's because they are used to one kernel per product and wasting time running around the code meddling as a one-off, rather than one kernel per member and CONFIG_ARCH_MULTIPLATFORM / DT and do the work once.
That's not really it - the trick is to keep the active parts of the code in sync with each other to minimise redundant development and there's other ways of doing that than doing a single binary for multiple boards that people have already figured out. By the time you get to actually shipping most of the effort is on QA anyway which for thoroughness needs to be on a per system basis.
The QA certainly needs to be done individually on the final system. We'll see the impact (or not) as we go, but at least nothing about DT and ARCH_MULTIPLATFORM stops them having a single source tree and doing it the old way for as long as they like. But I expect we see reasons why one kernel binary will become highly interesting and eliminate a lot of work for people that embrace it.
He mentioned that the Andriodization patches are in LTS already... what I was initially talking about is how to make sure llct merged content is still there if we later switch our LT kernel to use LTS basis. So if he added a pending series about xyz subsystem to llct for 3.x, it's arguable to expect that the accepted series for xyz subsystem appear in some downstream version of LTS for 3.x, even if it was accepted after 3.x.
That's what we'll be going for for for anything which is included in LTS, though that won't be absolutely everything that is being done within Linaro.
Great.
As I said that's probably not your problem, it's something Andrey could take care of doing on top of your mainline-only core LTS branch, but there needs to be a discussion about if we even want that, or if content added at llct just stops at llct and your mainline-only core LTS branch is all there is.
I suspect the backports will be too much effort there for anything with non-trivial invasiveness and that people might want the support. From an encouraging people to upstream point of view it does also seem like we should have some barriers on the backports, to a certain extent what
Yes backports are not good things to encourage.
Generally though the llct idea is adding things before upstream because they're desirable even at the current version. So there should at least be a chance they each might be candidates for backport treatment in LKS.
-Andy
I completely agree that the best place to put code is upstream. Must get some T-shirts made.
I think that the key here is to have a Greg KH like process that takes 'baked' kernel releases and maintains them over time and has strict guidelines as to what code can be added. My view is that the TSC should not treat LSKs as bit buckets and back port at all if it can help it. In other words, I don't want the LSK to turn into LTSI. The LSKs exist as a known good mixture of things at a point in time. So, if 3.10 is blessed as a stable kernel and if the TSC wants to see ARM's big.LITTLE MP patches applied to it, we should apply those patches to 3.10 and freeze it. After that we'd only add bug fixes. Once the maintainers have agreed the frameworks into which those MP patches need to go*, you would not, for example, put newer MP patch sets into the 3.10 LSK, that code would go upstream.
OK, so some may argue that the TSC will always want to add all sorts of things. What will act as a brake here is that adding things costs effort and the TSC would have to move effort from somewhere in order to support their wants. My usual response to back porting costs is that 1 version back is easy, 2 is tricky and 3 is impossible...
Dave
* There's a rather interesting discussion on LKML going on at the moment
On 24 Jun 2013, at 10:08, Andy Green andy.green@linaro.org wrote:
On 21 June 2013 19:21, Mark Brown broonie@linaro.org wrote:
On Fri, Jun 21, 2013 at 09:40:43AM +0800, Andy Green wrote:
On 21 June 2013 00:22, Mark Brown broonie@linaro.org wrote:
So, my previous job was all about shipping into smartphones so I'm more than familiar with the situation here - this is in no way specific to Linaro, it's a pressure felt by everyone shipping into the consumer electronics space. To me part of teaching people about the value of upstreaming is teaching them how that process interacts with and helps the process of stabalisation for release.
This kind of assumes the Landing Teams are in a position where people are interested in their "teachings". More often someone in higher management has realized that the LT can do them a lot of good, but on the ground that's not always understood. If the company in question has established fiefdoms, it can get difficult even reaching agreement on LT necessities like having a rebase tree (the otherwise excellent Felipe Balbi at a member gave me a nice lecture about his certainties we should use history trees, because the reasoning was outside his experience).
Of course, I'm not saying that this is a trivial thing to sell but that doesn't mean we shouldn't be pushing it. If it reall was trivially obvious probably everyone would be doing it already :)
Yes this gets pushed, and we demonstrate by doing it, which is highly effective against people saying it can't be done.
However this LKS kernel will definitely make tracking activity harder for us. The problem will be all the driver integration focus at the member will be at LKS kernel and not tracking. Since LKS has a direct line to product ship date, parallel driver integration at later tracking kernel they're not shipping will seem like a distraction. We could do the driver integration engineering work at tracking and backport to LKS, but this really is needlessly inefficient from product-focused observer POV.
If we don't do something about it for tracking, by default we end up with a single old tree that works well and no forward path for the code.
When LKS and HEAD are closely aligned, it's not going to be that difficult to follow LKS integration work on tracking. However when LKS and HEAD diverge significantly rebasing the LKS-aligned patch to HEAD each time may be a nasty job.
Perhaps if the LT tree based on LKS is kept by rebase, with the "squashed topic patches with history via tags" method, there's a way to port LKS-based driver patches initially to tracking, then squash interdiffs between the last ported tag and new tag on the LT LKS tree to the tracking tree patch for the same function. It would require care but then you only need to do the main uplevel porting once and there's a path to follow LKS-based changes.
Essentially everyone in the Android space will be working on v3.4 up until the release of Key Lime Pie. That's the stable kernel for Jelly Bean, that's what everyone works with. For ICS it was v3.0, though that's pretty much dead at this point for anything except support.
Not in "some" companies: it's still alive and getting new engineering work done on it, despite it being explained it's like giving birth to old men. For the people who think that's OK, I'm worrying the LTS kernel will allow them to argue for that approach.
Our current plan of record is to taper off new features after a year so hopefully there's at least some back pressure there.
Yes it'll stop clinging on to it forever anyway.
As far as device tree goes... it's not really solving a problem that people doing phones have. People will use it (and IME do use it) when
However the code we have seen from vendor suppliers for drivers is really nasty in many cases. It's quite normal to embed physical addresses and IRQs direct in the driver code, we even got one where they had put Samsung pre-generic gpio gpio api usage right in there (the driver is nothing Samsung specific). Even Mali has that kind of stuff piled in there. So even for phones DT will be very valuable (and I noticed yesterday, Chromebook kernels have DT attached to kernel binary). It is expensive running around dealing with and maintaining that.
Note however that the Chromebooks aren't fully supported with only DT yet - we still need to work out how to handle things like the WiFi reset. As far as the code quality thing goes my experience is that DT will just push around the quality issues, it won't stop them (as you'd expect really).
I agree with what you're saying they don't feel that yet and there's no DT backport pressure, it's because they are used to one kernel per product and wasting time running around the code meddling as a one-off, rather than one kernel per member and CONFIG_ARCH_MULTIPLATFORM / DT and do the work once.
That's not really it - the trick is to keep the active parts of the code in sync with each other to minimise redundant development and there's other ways of doing that than doing a single binary for multiple boards that people have already figured out. By the time you get to actually shipping most of the effort is on QA anyway which for thoroughness needs to be on a per system basis.
The QA certainly needs to be done individually on the final system. We'll see the impact (or not) as we go, but at least nothing about DT and ARCH_MULTIPLATFORM stops them having a single source tree and doing it the old way for as long as they like. But I expect we see reasons why one kernel binary will become highly interesting and eliminate a lot of work for people that embrace it.
He mentioned that the Andriodization patches are in LTS already... what I was initially talking about is how to make sure llct merged content is still there if we later switch our LT kernel to use LTS basis. So if he added a pending series about xyz subsystem to llct for 3.x, it's arguable to expect that the accepted series for xyz subsystem appear in some downstream version of LTS for 3.x, even if it was accepted after 3.x.
That's what we'll be going for for for anything which is included in LTS, though that won't be absolutely everything that is being done within Linaro.
Great.
As I said that's probably not your problem, it's something Andrey could take care of doing on top of your mainline-only core LTS branch, but there needs to be a discussion about if we even want that, or if content added at llct just stops at llct and your mainline-only core LTS branch is all there is.
I suspect the backports will be too much effort there for anything with non-trivial invasiveness and that people might want the support. From an encouraging people to upstream point of view it does also seem like we should have some barriers on the backports, to a certain extent what
Yes backports are not good things to encourage.
Generally though the llct idea is adding things before upstream because they're desirable even at the current version. So there should at least be a chance they each might be candidates for backport treatment in LKS.
-Andy
linaro-kernel mailing list linaro-kernel@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-kernel
On Mon, Jun 24, 2013 at 09:08:23AM +0800, Andy Green wrote:
However this LKS kernel will definitely make tracking activity harder for us. The problem will be all the driver integration focus at the member will be at LKS kernel and not tracking. Since LKS has a direct line to product ship date, parallel driver integration at later tracking kernel they're not shipping will seem like a distraction. We could do the driver integration engineering work at tracking and backport to LKS, but this really is needlessly inefficient from product-focused observer POV.
So, as David said this won't cover everything Linaro does so hopefully people will want to track our development stuff to play with those toys.
The other thing I'd point out here is that people are already shipping products based on stable kernels today; the issues you mention are there already and ignoring them doesn't make them go away. If anything us providing some of the backport will hopefully avoid people needing to spend time doing that for themselves which ought to be helpful.
That's not really it - the trick is to keep the active parts of the code in sync with each other to minimise redundant development and there's other ways of doing that than doing a single binary for multiple boards that people have already figured out. By the time you get to actually shipping most of the effort is on QA anyway which for thoroughness needs to be on a per system basis.
The QA certainly needs to be done individually on the final system. We'll see the impact (or not) as we go, but at least nothing about DT and ARCH_MULTIPLATFORM stops them having a single source tree and doing it the old way for as long as they like. But I expect we see reasons why one kernel binary will become highly interesting and eliminate a lot of work for people that embrace it.
It certainly won't hurt but I'd not oversell what it's going to achieve; remember that the kernel is just one piece in the jigsaw here.
Generally though the llct idea is adding things before upstream because they're desirable even at the current version. So there should at least be a chance they each might be candidates for backport treatment in LKS.
There will be, as the process thing I posted at the top of the thread said things like that should have a reference to the current development so we can track them on their way upstream.
On 24 June 2013 19:24, Mark Brown broonie@linaro.org wrote:
On Mon, Jun 24, 2013 at 09:08:23AM +0800, Andy Green wrote:
However this LKS kernel will definitely make tracking activity harder for us. The problem will be all the driver integration focus at the member will be at LKS kernel and not tracking. Since LKS has a direct line to product ship date, parallel driver integration at later tracking kernel they're not shipping will seem like a distraction. We could do the driver integration engineering work at tracking and backport to LKS, but this really is needlessly inefficient from product-focused observer POV.
So, as David said this won't cover everything Linaro does so hopefully people will want to track our development stuff to play with those toys.
Whatever way you look at it this complicates Linaro's story.
There's no upstream path for engineering done on top of the older, stable kernel. I don't mean the work Mark is doing I mean the "BSP" work done on top of LKS to make it useful in actual products.
It may still be a good idea to enter the business of supporting stable product kernels but it has knock-ons; it's much more like a one-off consultancy activity.
The other thing I'd point out here is that people are already shipping products based on stable kernels today; the issues you mention are there already and ignoring them doesn't make them go away. If anything us providing some of the backport will hopefully avoid people needing to spend time doing that for themselves which ought to be helpful.
Yes. But the tree you're making will realistically be a starting point for a bunch of product kernels. In the case of LTs they'll be doing that product driver integration work which previously we did at HEAD on an older kernel. It's not the end of the world but it's philosophically and practically a bit different.
We might at the same time need to think about the general Linaro approach to burning energy on trees that are not going to leave a useful legacy to make sure the result is a bit more balanced.
-Andy
On Mon, Jun 24, 2013 at 08:20:46PM +0800, Andy Green wrote:
Yes. But the tree you're making will realistically be a starting point for a bunch of product kernels. In the case of LTs they'll be
That's one of the ideas, yes.
doing that product driver integration work which previously we did at HEAD on an older kernel. It's not the end of the world but it's philosophically and practically a bit different.
You keep saying this but I'm not sure why - I'm really struggling to see how anything is changing here; you seem to be saying that given the LSK member companies are going to start asking the landing teams to work on their production trees instead of mainline. I'd be very surprised if the fact that their production trees may be based on the LSK makes a difference here, or any particular reason why that would be the case. Those trees exist anyway and are where the money comes from as things stand.
On 26 June 2013 00:54, Mark Brown broonie@linaro.org wrote:
On Mon, Jun 24, 2013 at 08:20:46PM +0800, Andy Green wrote:
Yes. But the tree you're making will realistically be a starting point for a bunch of product kernels. In the case of LTs they'll be
That's one of the ideas, yes.
doing that product driver integration work which previously we did at HEAD on an older kernel. It's not the end of the world but it's philosophically and practically a bit different.
You keep saying this but I'm not sure why - I'm really struggling to see how anything is changing here; you seem to be saying that given the LSK member companies are going to start asking the landing teams to work on their production trees instead of mainline. I'd be very surprised if the fact that their production trees may be based on the LSK makes a difference here, or any particular reason why that would be the case. Those trees exist anyway and are where the money comes from as things stand.
What's different is this is the first time Linaro is going to do engineering on, provide, and maintain an old kernel.
I agree in many cases for LTs it can't be dodged anyway, and in those cases it's better to have a nice, maintained old kernel to work from. So I am not laying the problems at LSK's door particularly.
However, work done on that old kernel will just rot, in the grand tradition of old kernels everywhere. It seems to me it might be good to think about how to mitigate the waste of doing all that disposable engineering on top of it.
-Andy
On Wed, Jun 19, 2013 at 10:02 PM, Mark Brown broonie@linaro.org wrote:
This is a first draft at a process for getting things integrated into the Linaro Stable Kernel releases. Any feedback is appreciated, hopefully this is all fairly boring and uncontroversial so there shouldn't be too many surprises.
The first thing to say here is that all LSK releases will be based off the latest generic Linux stable kernel so the best way to get a change into a Linaro release is to get it into the generic Linux stable kernel using the standard processes. This will maximise the number of people who can use the change and is just generally good practice.
New features
Features for the stable kernel are agreed by the TSC. Once a feature has been agreed by the TSC there should be an owner assigned to deliver a feature branch into the stable kernel and work with the stable kernel team to resolve any integration issues at least up until the feature has been included in a release. This will be done per kernel version.
Each feature request from the TSC may or may not conflict with other features already part of the LSK tree. Members might want to pick and choose what feature branches from LSK they include internally.
Is there a single LSK integrated tip or do you expect to maintain a forest of feature branches and then make several flavours of tip branches available based on member requirements?
These feature branches should be based on the relevant upstream kernel as far as possible (any dependencies on other branches should be discussed with the stable kernel team). Some information about where the code came fromm should be included along with the code, in order of preference:
1. Commit IDs from the standard kernel in the changelogs of the individual patches. 2. A description of how the equivalent change was made upstream or why it isn't required in LSK (eg, explaining that this is taken care of by features not present in the stable kernel). 3. References to where out of tree development is happening including contact information for followup.The code should be sent as a pull request or patches, with review by the stable team following normal kernel process and focusing on backporting and integration issues. Relevant testing infrastructure should also be available in LAVA for QA and the review will also include ensuring that the testsuite passes with the changes integrated.
Once the code has been accepted it will be stored as a branch in the LSK tree and the submission branch can be deleted.
Updating code in the LSK
The LSK can be updated either by replacing an existing topic branch or by submitting incremental patches. Replacement would be most useful in cases where a feature has not yet been merged into the standard kernel and is still being redeveloped there but otherwise incremental updates are preferred. The process for submitting changes is the same as for new features with the exception that incremental updates should be based on the topic branch in the LSK git rather than the standard kernel.
Assuming you keep all feature branches, do you have a versioning policy in mind? e.g. feature_foo_v1, feature_foo_v2, etc.
Regards, Amit
On Thu, Jun 20, 2013 at 05:08:25PM +0530, Amit Kucheria wrote:
On Wed, Jun 19, 2013 at 10:02 PM, Mark Brown broonie@linaro.org wrote:
Features for the stable kernel are agreed by the TSC. Once a feature has been agreed by the TSC there should be an owner assigned to deliver a feature branch into the stable kernel and work with the stable kernel team to resolve any integration issues at least up until the feature has been included in a release. This will be done per kernel version.
Each feature request from the TSC may or may not conflict with other features already part of the LSK tree. Members might want to pick and choose what feature branches from LSK they include internally.
Is there a single LSK integrated tip or do you expect to maintain a forest of feature branches and then make several flavours of tip branches available based on member requirements?
Both - cut the number of kernels down as far as possible but no further, merging as much as we can together and basing as much as possible on a single core kernel. We've already got a separate variant for Android for example. A lot depends on what the features we end up with are and how nicely they play with each other and with being disabled at compile time.
The LSK can be updated either by replacing an existing topic branch or by submitting incremental patches. Replacement would be most useful in cases where a feature has not yet been merged into the standard kernel and is still being redeveloped there but otherwise incremental updates are preferred. The process for submitting changes is the same as for new features with the exception that incremental updates should be based on the topic branch in the LSK git rather than the standard kernel.
Assuming you keep all feature branches, do you have a versioning policy in mind? e.g. feature_foo_v1, feature_foo_v2, etc.
I don't propose to keep older versions of the branches hanging around except as part of the tagged releases, therefore the branches would just be featureX, featureY and so on. I don't think it's worth the hassle of trying to version this stuff especially in the case where we are doing incremental updates on top of an existing feature branch.
If there were multiple LTS kernel versions active they would be per kernel version but that's a bit of a different game.
On Wed, 2013-06-19 at 17:32 +0100, Mark Brown wrote:
This is a first draft at a process for getting things integrated into the Linaro Stable Kernel releases. Any feedback is appreciated, hopefully this is all fairly boring and uncontroversial so there shouldn't be too many surprises.
The first thing to say here is that all LSK releases will be based off the latest generic Linux stable kernel so the best way to get a change into a Linaro release is to get it into the generic Linux stable kernel using the standard processes. This will maximise the number of people who can use the change and is just generally good practice.
New features
Features for the stable kernel are agreed by the TSC. Once a feature has been agreed by the TSC there should be an owner assigned to deliver a feature branch
Who does the assigning, and from what pool does the assignee come?
into the stable kernel and work with the stable kernel team to resolve any integration issues at least up until the feature has been included in a release. This will be done per kernel version.
These feature branches should be based on the relevant upstream kernel as far as possible (any dependencies on other branches should be discussed with the stable kernel team). Some information about where the code came fromm should be included along with the code, in order of preference:
1. Commit IDs from the standard kernel in the changelogs of the individual patches. 2. A description of how the equivalent change was made upstream or why it isn't required in LSK (eg, explaining that this is taken care of by features not present in the stable kernel). 3. References to where out of tree development is happening including contact information for followup.The code should be sent as a pull request or patches, with review by the stable team following normal kernel process and focusing on backporting and integration issues. Relevant testing infrastructure should also be available in LAVA for QA and the review will also include ensuring that the testsuite passes with the changes integrated.
Once the code has been accepted it will be stored as a branch in the LSK tree and the submission branch can be deleted.
Updating code in the LSK
Who decides when and how often the LSK code should be updated, and is the person doing the updating chosen in the same way as the original feature submission?
I'm just trying to work out if my future at Linaro is going to consiste of full time backporting of TC2 and big.LITTLE features...
The LSK can be updated either by replacing an existing topic branch or by submitting incremental patches. Replacement would be most useful in cases where a feature has not yet been merged into the standard kernel and is still being redeveloped there but otherwise incremental updates are preferred. The process for submitting changes is the same as for new features with the exception that incremental updates should be based on the topic branch in the LSK git rather than the standard kernel.
On Thu, Jun 20, 2013 at 01:42:07PM +0100, Jon Medhurst (Tixy) wrote:
On Wed, 2013-06-19 at 17:32 +0100, Mark Brown wrote:
Features for the stable kernel are agreed by the TSC. Once a feature has been agreed by the TSC there should be an owner assigned to deliver a feature branch
Who does the assigning, and from what pool does the assignee come?
Presumably the TSC ultimately in the same way anyone gets assigned to work on anything in Linaro, it'd probably depend on what the feature is and who's avaiable at a given time.
Updating code in the LSK
Who decides when and how often the LSK code should be updated, and is the person doing the updating chosen in the same way as the original feature submission?
I think we need to figure this out on an ongoing and probably feature by feature basis, and of course some of this is going to be triggered by bug reports rather than just a "hey let's update this" thing. The tradeoffs here are also going to be very different for something which is being included as a preview and is still under active development as opposed to something that is pretty much done and dusted.
For now we've never had anyone wanting to update anything at all except for general tracking of kernel.org stable release; I think for now we should just leave it as a "do something sensible" - ultimately it's the TSC again.
I'm just trying to work out if my future at Linaro is going to consiste of full time backporting of TC2 and big.LITTLE features...
Have to talk to your manager about that I guess...
As far as TC2 goes there's no real interest in it for LSK except in that we need platform enablement in order to test the features.
hi,
On Wed, Jun 19, 2013 at 12:32 PM, Mark Brown broonie@linaro.org wrote:
This is a first draft at a process for getting things integrated into the Linaro Stable Kernel releases. Any feedback is appreciated, hopefully this is all fairly boring and uncontroversial so there shouldn't be too many surprises.
The first thing to say here is that all LSK releases will be based off the latest generic Linux stable kernel so the best way to get a change into a Linaro release is to get it into the generic Linux stable kernel using the standard processes. This will maximise the number of people who can use the change and is just generally good practice.
is that documented anywhere, why we (Linaro) are doing LSK at all? especially, i am interested to understand who is actually going to be a downstream of LSK? e.g. who is going to benefits from that work. I am not saying it's wrong to do it, just want to understand why we do it.
i tend to agree with some of the arguments from Andy (not all ;-) and I definitely see and understand the value of the LLCT as it provides a 'baseline' for LTs to 'move' forward, but LSK is not about moving toward upstream.
On Fri, Jun 21, 2013 at 12:24:18AM -0400, Nicolas Dechesne wrote:
is that documented anywhere, why we (Linaro) are doing LSK at all?
Sadly no, this is something I've been reverse engineering myself. I do have some notes but I'd need to scrub them before sharing them.
Broadly speaking it's to provide a basis for product stabalisation as opposed to development, a less moving target.
especially, i am interested to understand who is actually going to be a downstream of LSK? e.g. who is going to benefits from that work. I am not saying it's wrong to do it, just want to understand why we do it.
The main users are expected to be members and community members who want Linaro kernel work in something with a reasonable supported lifespan that they can base their own supported products on.
i tend to agree with some of the arguments from Andy (not all ;-) and I definitely see and understand the value of the LLCT as it provides a 'baseline' for LTs to 'move' forward, but LSK is not about moving toward upstream.
Like I say hopefully that's something that the tie back to upstream is helping to address; providing an example of doing both at the same time.
linaro-kernel@lists.linaro.org
-
Amit Kucheria -
Andy Green -
David Rusling -
Jon Medhurst (Tixy) -
Mark Brown -
Mark Brown -
Nicolas Dechesne -
Ryan Harkin