On 21 June 2013 00:22, Mark Brown broonie@linaro.org wrote:
On Thu, Jun 20, 2013 at 08:02:54PM +0800, Andy Green wrote:
On 20 June 2013 19:16, Mark Brown broonie@sirena.org.uk wrote:
Please don't top post on upstream style lists such as this one.
shrug
It's important to follow good practice here; it helps set a good example for people of upstream processes.
Unless I use my phone, the last 10+ years my mail is all inline reply. This last week I have to use gmail interface for the first time.
This is happening anyway and will continue to happen for the forseeable future in the consumer electronics space, it's just a function of shipping product on something other than -next really. I'd actually hope that our strong policy on using upstream code and backporting it might help a little here, keeping things more in sync. It's not a particular problem to work on a backport for a product so long as the backport and upstream are reasonably in sync.
Linaro works with its members to maintain kernels for them via Landing Teams. Linaro has always had a "strong policy on upstream code" but that is the art of the possible when you work directly with member companies who variously may not want things upstreamed because somebody might learn something about their IP, have a large number of engineers already supposedly upstreaming things or not even understand why upstreaming things is worth the time.
I'm sure you have a good handle on nice clean, sane upstream world but I'm telling you something specific about Linaro and LT experience with members.
So, my previous job was all about shipping into smartphones so I'm more than familiar with the situation here - this is in no way specific to Linaro, it's a pressure felt by everyone shipping into the consumer electronics space. To me part of teaching people about the value of upstreaming is teaching them how that process interacts with and helps the process of stabalisation for release.
This kind of assumes the Landing Teams are in a position where people are interested in their "teachings". More often someone in higher management has realized that the LT can do them a lot of good, but on the ground that's not always understood. If the company in question has established fiefdoms, it can get difficult even reaching agreement on LT necessities like having a rebase tree (the otherwise excellent Felipe Balbi at a member gave me a nice lecture about his certainties we should use history trees, because the reasoning was outside his experience).
We're basically there to advise the right thing but as a kind of consultancy action to help them get what they want done. At the moment in Fujitsu our advice is mostly taken which is great for them and us, but in other LT past and future that's not always the case. If the member insists, we're gong to only work on LTS kernel now that's actually what will happen because that's their business decision to make.
Anyway the point is we're used to people explaining to use what we should do and how to do it when they don't understand the relationship with the member. We are definitely motivated to do the Right Thing if we look glum as we appear to be doing the wrong thing it may be because there's nothing we can do about it.
As I said before this issue isn't necessarily your problem but because the TSC is controlled by members, what you may get asked to do is unbounded in fact. What I'm suggesting is if you see a real piece of madness coming, please push back. Some mixtures of LTS base version and backport request will get asked in order to avoid the work they ought to be doing to base on a later kernel; that work then and the retardation of alignment of the member with upstream can be bad news. Consider for example 3.0 basis and big.LITTLE backport. Maybe if you start from 3.9 or other modern basis and v8 support is already started in there, the worst kind of thing won't happen any more, in which case great.
The investment in old kernels thing that you mention is not really an issue these days thanks to Android, it's got people used to an upgrade cycle. The reason people standardise on v3.4 right now is more things like being more comfortable using whatever Google were using for their lead device and it giving all the component makers a target to aim at so the product people don't have integration hassle.
What that causes is a large number of engineers spending time on "disposable kernels" that are not aligned with, and don't feed upstream. We saw it at TI before many of the "large number of
Sure, it's not ideal and one of the things you guys on the landing teams need to be teaching people is how to do this effectively. The issue
As I explained above, even though the member pays for the LT it does not mean that the various parts of the member which may be under a different management structure than called us in, will listen to us. We also met characters who prioritized upstreaming over servicing the member's customers at all, which is a different form of getting it fatally wrong.
We found running a tracking kernel is a good balance, it forces all their driver assets to be updated and aligned enough to at least work with current kernels, and at the time when Linus releases, we can fork off a stable branch ourselves with hopefully everything working. If later they hop from eg 3.4 to 3.9, well we already have a 3.9 tree that is working from tracking at that time to get started with. In this case, it should be possible to look at rebase or merge of that 3.9 tree with the LTS one in a fairly compatible way. So it all seems workable.
here isn't that people are picking a kernel and stabalising for release on that, it's that they're frequently doing so in an ineffective fashion. What you're talking about here isn't sticking to a single kernel and staying with it for ever more, it's something different to that. More on this below...
engineers" wasting time on that got canned (it didn't stop them just slowed them down ^^). People are still working on 3.0 even and 3.4 now, sometimes without DT, when 3.10 will soon be here, it's actually
Essentially everyone in the Android space will be working on v3.4 up until the release of Key Lime Pie. That's the stable kernel for Jelly Bean, that's what everyone works with. For ICS it was v3.0, though that's pretty much dead at this point for anything except support.
Not in "some" companies: it's still alive and getting new engineering work done on it, despite it being explained it's like giving birth to old men. For the people who think that's OK, I'm worrying the LTS kernel will allow them to argue for that approach.
As far as device tree goes... it's not really solving a problem that people doing phones have. People will use it (and IME do use it) when it's a standard feature in the baseline kernel but there's no real pull to backport it, if anything it's likely to cause problems with third party code as nobody else is using it on that kernel version so there's unlikely to be DT support in their drivers.
Yes, I mention it because it's a sign there's no investment towards alignment with upstream, where DT is required.
However the code we have seen from vendor suppliers for drivers is really nasty in many cases. It's quite normal to embed physical addresses and IRQs direct in the driver code, we even got one where they had put Samsung pre-generic gpio gpio api usage right in there (the driver is nothing Samsung specific). Even Mali has that kind of stuff piled in there. So even for phones DT will be very valuable (and I noticed yesterday, Chromebook kernels have DT attached to kernel binary). It is expensive running around dealing with and maintaining that.
I agree with what you're saying they don't feel that yet and there's no DT backport pressure, it's because they are used to one kernel per product and wasting time running around the code meddling as a one-off, rather than one kernel per member and CONFIG_ARCH_MULTIPLATFORM / DT and do the work once.
not a good situation and not in itself what Linaro is meant to encourage. To the extent an LTS kernel will encourage / enable / legitimize that when people could be on tracking kernels, it's a problem. In other cases LTS usage is legitimate especially if the platform is mature.
In the Landing Teams, it's Linaro's policy to encourage the members to make sure their stuff is tracking Linus' HEAD so they always have a contemporary kernel with all the pieces, reducing the barrier to upstreaming the content.
So what I think the landing teams need to be doing here is unchanged - you need to be showing people how to develop in such a way that the backport/release code and upstream code don't shift too far apart so work done in one can be readily applied to the other, and helping them make sure that things are kept in sync so this is tractable.
Right, I also believe this is already a good way and want to keep it going that way.
I would actually hope that in producing a stable kernel Linaro will provide a positive example of how this can be done effectively that people can look to when trying to understand what's going on. When we keep our stable trees trackable back to current upstream work we will be showing this process in action which the landing teams will be able to point to when making the case for this approach to their member companies.
The other thing that is often a good example here is going through the process of moving to a new kernel version when you've been tracking upstream - this often brings home the advantages to people. I've seen some companies in this space go from just not caring about upstream to strongly advocating for upstream after going through this process.
Yes tracking has opened some eyes in my experience too. There's a lot of disbelief initially that the massive agony of shifting off an old kernel can actually be almost painless when done week by week. However I found it has to be combined with an approach of squashing to single topic patches. At TI we did not do this and ended up dragging 2.5K patches of "internal development history" around with us rebasing it all every time... we never ever cared about that history. We should have refactored and squashed it into a smaller number of "patches approximating what you would send upstream" for the various subsystems, as we have now done at Fujitsu. Then the tree is much easier to maintain and more meaningfully aligned with a potential upstream effort.
I discussed the llct relationship with LTS with Andrey offlist. Actually for tracking LTs don't base on Linus directly but use Andrey's linux-linaro-core-tracking. This has many interesting branches for Linaro merged in already including Androidization, and is recomposed for every -rc. It's worked out very well for us since we get things like current gator via ARM LT we don't otherwise have time to consistently care about coming in with the basis.
He mentioned that the Andriodization patches are in LTS already... what I was initially talking about is how to make sure llct merged content is still there if we later switch our LT kernel to use LTS basis. So if he added a pending series about xyz subsystem to llct for 3.x, it's arguable to expect that the accepted series for xyz subsystem appear in some downstream version of LTS for 3.x, even if it was accepted after 3.x.
As I said that's probably not your problem, it's something Andrey could take care of doing on top of your mainline-only core LTS branch, but there needs to be a discussion about if we even want that, or if content added at llct just stops at llct and your mainline-only core LTS branch is all there is.
-Andy