I finally got to this and wrote up a summary of the contents of the ST-E BSP that we are mirroring on the Linaro git server:
https://wiki.linaro.org/BSP/ST-Ericsson
I did not write up my recommendations of how to proceed, but it should give an idea of the work in front of us/them.
Linus, if you see any significant omissions or mistakes, please correct me.
Arnd
2010/9/24 Arnd Bergmann arnd@arndb.de:
I finally got to this and wrote up a summary of the contents of the ST-E BSP that we are mirroring on the Linaro git server:
https://wiki.linaro.org/BSP/ST-Ericsson
I did not write up my recommendations of how to proceed, but it should give an idea of the work in front of us/them.
Thanks Arnd, we're reading it now...
Linus, if you see any significant omissions or mistakes, please correct me.
None that you could know about. Anyway, here is an insiders' view:
- Our internal codebase is now already revamped a few times. It is looking better now, we only need to push it out. (And a process for keeping the trees in sync.)
Good news:
- Controversial STM DMA engine is rewritten, merged and already hardened a few times in the mainline code with a third round of fixups queued in -next for 2.6.37 in the async_tx tree. (drivers/dma/ste_dma40.c).
The new STEDMA40 DMA driver (merged since 2.6.35) has been brough back internally as well and the old STM DMA driver is deleted.
- CG2900 combochip core and drivers are rewritten extensively and sent for review on LKML this friday (and will need more reviews and iterations, but the process is started).
- The new AB8500 MFD driver from the mainline kernel has been merged back internally and is now in sync.
- The mmc-u8500.c MMC driver is not OK actually, it is duplicating functionality in mmci.c so we have been patching mmci.c so as to arrive on a single driver to be shared by all PL180 derivates.
- The CAIF stuff is nowadays all upstream and we have just had a cleanout round syncing the internal tree to use the mainline code internally as well.
- U5500-specific modem communication and mailboxes have been submitted to the ARM ML and Russells patch trackers last week.
- Phonet extensions are submitted to Davice Miller and getting merged as we speak. (Actual shared mem and mailboxes backing Phonet in U8500 is still to be prepared but need the Phonet changes to be available first.)
- The MCDE display driver is soon ripe for public review.
Special cases:
- The hwmem driver is our way of getting rid of pmem dependencies so we can atleast live without Android patches. IMO it is to be an intermediate step as we move toward a unified CMA (contiguous memory allocator) but we'll see how this develops.
- b2r2 (dubbed "scary" :-) is really unlike any other hardware found in the drivers/video/* hierarchy, and we'll likely need the advice of video4linux developers on how to proceed on it, so we hope to mail this off for review soon enough. However to proceed I believe this driver needs the MCDE display driver to be submitted first (Gantt-type dependency). So we are working on getting that one out for review first and foremost.
Do you want me to update the Wikipage with this or is the review a static piece or to be "owned" by the reviewer?
Yours, Linus Walleij
On Tue, Sep 28, 2010 at 11:29 AM, Linus Walleij linus.walleij@stericsson.com wrote:
Do you want me to update the Wikipage with this or is the review a static piece or to be "owned" by the reviewer?
I cannot really speak for kernel WG, but imo its just a wiki page. So just add yourself to the author/contributor section and edit right away ;)
On Tuesday 28 September 2010, Linus Walleij wrote:
2010/9/24 Arnd Bergmann arnd@arndb.de:
- Our internal codebase is now already revamped a few times. It is looking better now, we only need to push it out. (And a process for keeping the trees in sync.)
Ok. Having real-time access to your internal trees would be extrememly helpful. The tree we have on git.linaro.org is good for reviewing the existing code and I hope it also helps people build usable kernels, but we need something more dynamic to get the most out of both the Linaro and ST-Ericsson development teams.
Good news:
- Controversial STM DMA engine is rewritten, merged and already hardened a few times in the mainline code with a third round of fixups queued in -next for 2.6.37 in the async_tx tree. (drivers/dma/ste_dma40.c).
Yes, that looks pretty good now.
I noticed that this driver still needs to export three symbols on its own, which is presumably an extension to the common dmaengine API. Is that by design or do you expect this will be integrated into the standard API at some point?
The new STEDMA40 DMA driver (merged since 2.6.35) has been brough back internally as well and the old STM DMA driver is deleted.
CG2900 combochip core and drivers are rewritten extensively and sent for review on LKML this friday (and will need more reviews and iterations, but the process is started).
The new AB8500 MFD driver from the mainline kernel has been merged back internally and is now in sync.
The mmc-u8500.c MMC driver is not OK actually, it is duplicating functionality in mmci.c so we have been patching mmci.c so as to arrive on a single driver to be shared by all PL180 derivates.
The CAIF stuff is nowadays all upstream and we have just had a cleanout round syncing the internal tree to use the mainline code internally as well.
U5500-specific modem communication and mailboxes have been submitted to the ARM ML and Russells patch trackers last week.
Phonet extensions are submitted to Davice Miller and getting merged as we speak. (Actual shared mem and mailboxes backing Phonet in U8500 is still to be prepared but need the Phonet changes to be available first.)
The MCDE display driver is soon ripe for public review.
Great work on all of these!
Special cases:
- The hwmem driver is our way of getting rid of pmem dependencies so we can atleast live without Android patches. IMO it is to be an intermediate step as we move toward a unified CMA (contiguous memory allocator) but we'll see how this develops.
Yes, this is a very interesting topic and a number of other people are struggling with it. I'm a bit sceptical about the CMA drivers that are being discussed on the mailing list, but it's clear that some solution is needed here.
- b2r2 (dubbed "scary" :-) is really unlike any other hardware found in the drivers/video/* hierarchy, and we'll likely need the advice of video4linux developers on how to proceed on it, so we hope to mail this off for review soon enough. However to proceed I believe this driver needs the MCDE display driver to be submitted first (Gantt-type dependency). So we are working on getting that one out for review first and foremost.
What does it actually do? I didn't think it had anything to do with video4linux (a.k.a. video input, drivers/media/video), so if this is the 2D video acceleration, it sounds like this should be a DRM driver and go into drivers/gpu/drm/.
Do you want me to update the Wikipage with this or is the review a static piece or to be "owned" by the reviewer?
I'd be happy if you edit it yourself. If you make significant changes, please add your name to the author section.
Arnd
On Tue, 28 Sep 2010, Arnd Bergmann wrote:
On Tuesday 28 September 2010, Linus Walleij wrote:
2010/9/24 Arnd Bergmann arnd@arndb.de:
- Our internal codebase is now already revamped a few times. It is looking better now, we only need to push it out. (And a process for keeping the trees in sync.)
Ok. Having real-time access to your internal trees would be extrememly helpful. The tree we have on git.linaro.org is good for reviewing the existing code and I hope it also helps people build usable kernels, but we need something more dynamic to get the most out of both the Linaro and ST-Ericsson development teams.
Indeed. And that's exactly what I'm looking for in order to produce the linaro-next kernel tree. The linaro-next tree is discarded and rebuilt from scratch by merging together a bunch of development trees on a regular basis. I'd like to add your up-to-date tree to the lot.
Nicolas
Nicolas Pitre wrote:
Indeed. And that's exactly what I'm looking for in order to produce the linaro-next kernel tree. The linaro-next tree is discarded and rebuilt from scratch by merging together a bunch of development trees on a regular basis. I'd like to add your up-to-date tree to the lot.
We currently don't have anything like a git, what we have is this patchstack against Rothwells next-tags: http://userweb.kernel.org/~linusw/st-ericsson/
This consists of the patches we have considered to be in such a good shape that we have sent them off to the different mailing lists and subsystem maintainers.
NB: this is on top of the stuff we already have pending in next, which is currently actually a lot.
Right now we have ~49 patches pending in the latest next tag and ~27 patches in the additional patch stack (i.e. stuff under review).
I'm rebasing this patch stack agains Rothwells tree at irregular intervals.
The reason for not having this in a git is that since we're currently messing in so many subsystems and extending and fixing the subsystems as we go along and since we get so many cross-dependencies between subsystems, I have found it a lot easier to just keep a simple patch stack for the time being.
Maybe I'm just not understanding some easy way out of cross- subsystem dependencies, but the way I'm experiencing it is that we have done the simple drivers already, the stuff we have left is all of the type where we must have some special header file for the driver platform data creating a cross- subsystem dependency to the ARM tree. Any incremental fixes once the driver is in are usually a breeze to merge and maintain.
We are making this even more elaborate by trying to push stuff like DMA into drivers/dma/* and GPIO expanders into drivers/gpio/* instead of just slamming all hairy cases into arch/arm/* like many platforms have done previously...
Any hints on how to get out of this and create a simple git tree or set of trees is welcome, I have yet not found any.
Yours, Linus Walleij
On Wednesday 29 September 2010, Linus Walleij wrote:
Nicolas Pitre wrote:
Indeed. And that's exactly what I'm looking for in order to produce the linaro-next kernel tree. The linaro-next tree is discarded and rebuilt from scratch by merging together a bunch of development trees on a regular basis. I'd like to add your up-to-date tree to the lot.
We currently don't have anything like a git, what we have is this patchstack against Rothwells next-tags: http://userweb.kernel.org/~linusw/st-ericsson/
This consists of the patches we have considered to be in such a good shape that we have sent them off to the different mailing lists and subsystem maintainers.
NB: this is on top of the stuff we already have pending in next, which is currently actually a lot.
Right now we have ~49 patches pending in the latest next tag and ~27 patches in the additional patch stack (i.e. stuff under review).
I'm rebasing this patch stack agains Rothwells tree at irregular intervals.
How many trees are there that have your patches in linux-next? I'm guessing that we have some of them in linaro-next already, so maybe we can add the others as well.
Maybe I'm just not understanding some easy way out of cross- subsystem dependencies, but the way I'm experiencing it is that we have done the simple drivers already, the stuff we have left is all of the type where we must have some special header file for the driver platform data creating a cross- subsystem dependency to the ARM tree. Any incremental fixes once the driver is in are usually a breeze to merge and maintain.
Yes, this is a known problem with the way that devices are handled on ARM. Normally the platform code wouldn't have to known anything about the devices except for essential things like the interrupt controllers.
What we need here is a generic way to find out what devices are present other than encoding it into the platform. The flattened device tree is one solution to this, but it seems that it will take a long time before you can use it.
We are making this even more elaborate by trying to push stuff like DMA into drivers/dma/* and GPIO expanders into drivers/gpio/* instead of just slamming all hairy cases into arch/arm/* like many platforms have done previously...
Yes, that is very good.
Any hints on how to get out of this and create a simple git tree or set of trees is welcome, I have yet not found any.
I think you need to stop basing on top of linux-next and move to basing on top of the smallest possible set of trees other than mainline.
Today's series seems to apply almost cleanly on top of Russell's ARM tree, so as a start, you can create one git tree based on his and apply all your patches on top.
Since the base also in linaro-next, Nicolas can easily merge it. If you have dependencies on other trees, say the MMC tree, then split out the MMC related patches into a separate series and apply them on the mmc-next tree, and put them into a different branch in your git.
Create a master branch in your git than merges all the separate branches for people to test out your code.
Things get a bit hairy if the tree you base on gets rebased. Most maintainers do this very rarely, so you should be fairly safe.
You can also have multiple branches based on Linus Torvald's tree, e.g. for things like i2c or mtd that do not depend on changes in Russell's or anyone else's tree.
Arnd
2010/9/29 Arnd Bergmann arnd@arndb.de:
[Me]
Any hints on how to get out of this and create a simple git tree or set of trees is welcome, I have yet not found any.
I think you need to stop basing on top of linux-next and move to basing on top of the smallest possible set of trees other than mainline.
Right now we have dependencies on this forest: - partitions - doesn't have git tree - ARM - input - SPI - I2C - GPIO - MTD - MFD - async_tx (DMA) - Andrews patch stack
Today's series seems to apply almost cleanly on top of Russell's ARM tree, so as a start, you can create one git tree based on his and apply all your patches on top.
Maybe, the 9 other repos more or less depend on their platform data going into the ARM tree.
Since the base also in linaro-next, Nicolas can easily merge it. If you have dependencies on other trees, say the MMC tree, then split out the MMC related patches into a separate series and apply them on the mmc-next tree, and put them into a different branch in your git.
The MMC tree does not exist, the MMC patches are maintained in Mortons patch stack. We have also had to shovel all our backlight and LED patches into that patch stack in Purdies absence.
But I've seen Mortons stack being used as a pseudo-tree in -next so must be possible to bring that in anyway.
The big hit from using Nicolas tree will rather be that it is 2.6.35-based, while we have a lot of stuff merged for 2.6.36 and a similar pile of stuff stacked in different next trees so I will have to pull all of that out (including some extensions to frameworks we made along the way) in order to get something in shape for a 2.6.35 branch.
But it can probably be done.
Create a master branch in your git than merges all the separate branches for people to test out your code.
Yeah I understand that part atleast, it looks like I'll spindle up something like 9 topic branches per subsystem and bring that into a master branch with an octopus merge, it's more or less what the current patch stack does but with a number of topic branches.
Do you usually split the ARM platform data per-subsystem or do you stack that up in the ARM branch and bring that on top of the others?
Right now we have a lot of collisions in ARM due to a lot of platform data patches, we've tried to limit it by splitting the platform files a bit, but it's still a bit of churn...
Things get a bit hairy if the tree you base on gets rebased. Most maintainers do this very rarely, so you should be fairly safe.
That's not going to be a big problem I think.
The upside of just using a smaller stack on top of next is that once the patches are in respective subsystem next branch I don't have to worry about them anymore.
The proposal here is probably more along the lines of satisfying the Linaro needs of keeping a (currently) 2.6.35-based tree with all members stuff on top.
It seems like pretty much a full time integration effort if it is to be maintained so I'll need to be assigned to do it first I believe, I still need to get my official Linaro assignment, you never know, maybe I will be assigned to clean the coffee machine in Taipei instead ;-)
Yours, Linus Walleij
On Wed, 29 Sep 2010, Linus Walleij wrote:
The big hit from using Nicolas tree will rather be that it is 2.6.35-based, while we have a lot of stuff merged for 2.6.36 and a similar pile of stuff stacked in different next trees so I will have to pull all of that out (including some extensions to frameworks we made along the way) in order to get something in shape for a 2.6.35 branch.
Note that I maintain 2 trees: the Linaro stable tree based on 2.6.35 currently, and the linaro-next tree which is intended to closely follow the latest upstream and periodically remerge a bunch of development trees. I say "remerge" because the linaro-next tree is a throw-away one as it is constantly rebased/rebuilt just like linux-next. I think that your patches would be more for the later tree.
Nicolas
On Wednesday 29 September 2010, Linus Walleij wrote:
2010/9/29 Arnd Bergmann arnd@arndb.de:
[Me]
Any hints on how to get out of this and create a simple git tree or set of trees is welcome, I have yet not found any.
I think you need to stop basing on top of linux-next and move to basing on top of the smallest possible set of trees other than mainline.
Right now we have dependencies on this forest:
- partitions - doesn't have git tree
- ARM
- input
- SPI
- I2C
- GPIO
- MTD
- MFD
- async_tx (DMA)
- Andrews patch stack
Ok, I see. That's roughly the same list I had come up with when I looked at yesterday's series of patches.
However, I guess for most of these, you don't have *both* patches merged to the subsystem tree and unmerged patches on top, right?
The easy cases are those where you have a series of patches for one subsystem and they all get merged at once.
Just create one branch that contains your patches if they are not merged and ask the maintainer to pull them. Before they are merged, your own branch then has no dependencies on the main one. After they are merged, you can pull the maintainer tree into your own branch.
If linaro-next contains the subsystem tree and your branch, and they contain identical commits (because one of them got pulled into the other), the merge will be trivial.
Today's series seems to apply almost cleanly on top of Russell's ARM tree, so as a start, you can create one git tree based on his and apply all your patches on top.
Maybe, the 9 other repos more or less depend on their platform data going into the ARM tree.
This is just a dependency for being able to use it, right? AFAICT nothing should break if you have just the subsystem changes or just the architecture changes, and in order to use the new code you just need to pull both branches.
Since the base also in linaro-next, Nicolas can easily merge it. If you have dependencies on other trees, say the MMC tree, then split out the MMC related patches into a separate series and apply them on the mmc-next tree, and put them into a different branch in your git.
The MMC tree does not exist, the MMC patches are maintained in Mortons patch stack. We have also had to shovel all our backlight and LED patches into that patch stack in Purdies absence.
But I've seen Mortons stack being used as a pseudo-tree in -next so must be possible to bring that in anyway.
No, it actually works the other way round. Andrew bases his quilt on top of -next.
linux-next contains an mmc tree pulled from git://git.kernel.org/pub/scm/linux/kernel/git/cjb/mmc.git#mmc-next
You should try to get Chris Ball to take all the patches that are currently in Andrew's -mm tree. This is how Andrew normally expects things to flow when there is a maintainer.
The big hit from using Nicolas tree will rather be that it is 2.6.35-based, while we have a lot of stuff merged for 2.6.36 and a similar pile of stuff stacked in different next trees so I will have to pull all of that out (including some extensions to frameworks we made along the way) in order to get something in shape for a 2.6.35 branch.
But it can probably be done.
Remember you should not "use" Nicolas' tree, you should feed into it. As he already explained, there are two trees, so you need to decide if you want to have stuff in both of them or just the one following mainline.
Create a master branch in your git than merges all the separate branches for people to test out your code.
Yeah I understand that part atleast, it looks like I'll spindle up something like 9 topic branches per subsystem and bring that into a master branch with an octopus merge, it's more or less what the current patch stack does but with a number of topic branches.
Right.
Do you usually split the ARM platform data per-subsystem or do you stack that up in the ARM branch and bring that on top of the others?
Right now we have a lot of collisions in ARM due to a lot of platform data patches, we've tried to limit it by splitting the platform files a bit, but it's still a bit of churn...
I've seen both done before. It generally depends how hard the octopus merge gets. If it's just different branches adding new code to the same place, the merge is not that hard and you can try to keep the platform data together with the drivers.
Things get a bit hairy if the tree you base on gets rebased. Most maintainers do this very rarely, so you should be fairly safe.
That's not going to be a big problem I think.
The upside of just using a smaller stack on top of next is that once the patches are in respective subsystem next branch I don't have to worry about them anymore.
The proposal here is probably more along the lines of satisfying the Linaro needs of keeping a (currently) 2.6.35-based tree with all members stuff on top.
No, don't worry about the 2.6.35 tree too much. The first priority is to help you get your stuff upstream, the second priority is to increase the amount of testing before it gets there.
Arnd
On Wed, Sep 29, 2010 at 11:38:15PM +0200, Linus Walleij wrote:
The big hit from using Nicolas tree will rather be that it is 2.6.35-based, while we have a lot of stuff merged for 2.6.36 and a similar pile of stuff stacked in different next trees so I will have to pull all of that out (including some extensions to frameworks we made along the way) in order to get something in shape for a 2.6.35 branch.
As Nico said, you should just worry about targeting linaro-next; the 2.6.35 tree that will be used in the release is in stabilization freeze right now anyway. For ux500 support that means that its hardware pack will be a BSP-tree-based hardware pack -- and that's not really a big deal, since AIUI it should mostly work with the Linaro images anyway.
We are not going to be asking you to spend time backporting patches ;-)
2010/9/29 Arnd Bergmann arnd@arndb.de:
I think you need to stop basing on top of linux-next and move to basing on top of the smallest possible set of trees other than mainline.
Sorry for learning and fixing delays...
I have now created a tree based on Torvalds' clean v2.6.37-rc1 that is available straight off kernel.org:
git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-stericsson.git
NOTE: I am no Jedi yet. I still have to learn a few things about git I believe, right now this is really a -next type tree where I rebase the topic branches partly against each other, so it is being pushed out with forced updates, sometimes changing a lot of stuff inside the branches.
Today's series seems to apply almost cleanly on top of Russell's ARM tree, so as a start, you can create one git tree based on his and apply all your patches on top.
Right now this is just plain old v2.6.37-rc1 since Russell merged a lot of the stuff we wanted into that RC.
But if desired by the other members I can sure move over to Russells tree as a baseline.
Create a master branch in your git than merges all the separate branches for people to test out your code.
Exploring the remote branches on that git you will find a "next" tag now..
You can also have multiple branches based on Linus Torvald's tree, e.g. for things like i2c or mtd that do not depend on changes in Russell's or anyone else's tree.
Right now this is what I have.
Now we only need to mainline a few thousand patches of platform support :-)
Yours, Linus Walleij
On Friday 12 November 2010, Linus Walleij wrote:
2010/9/29 Arnd Bergmann arnd@arndb.de:
I think you need to stop basing on top of linux-next and move to basing on top of the smallest possible set of trees other than mainline.
Sorry for learning and fixing delays...
No problem.
Welcome to the Linaro team by the way!
I have now created a tree based on Torvalds' clean v2.6.37-rc1 that is available straight off kernel.org:
git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-stericsson.git
NOTE: I am no Jedi yet. I still have to learn a few things about git I believe, right now this is really a -next type tree where I rebase the topic branches partly against each other, so it is being pushed out with forced updates, sometimes changing a lot of stuff inside the branches.
Ok, that sounds good. If you have trouble with git, feel free to ask us any time on IRC or jabber.
Basing multiple branches on top of one another is a bit unusual, but it's fine as long as it works for you.
What other people have found useful there is to strictly separate topic branches where possible and only pull in common branches to resolve the hardest conflicts, e.g. have one main branch with all the changes that multiple other branches depend on, but then never rebase this branch, so you can avoid having to rebase other branches as a result.
It would be nice if you could mark branches that you expect to rebase frequently (e.g. your next branch) as opposed to those that only get stuff on top. This helps other people that may want to build on top of your work.
You might also want to look at "git rerere", which makes it easier for you to track merge conflicts between otherwise independent topic branches.
Today's series seems to apply almost cleanly on top of Russell's ARM tree, so as a start, you can create one git tree based on his and apply all your patches on top.
Right now this is just plain old v2.6.37-rc1 since Russell merged a lot of the stuff we wanted into that RC.
But if desired by the other members I can sure move over to Russells tree as a baseline.
It's not a big difference. As long you don't have dependencies on Russell's branch, staying on a mainline base should make it slightly easier for you.
If you decide to base on top of Russell's tree, make to use a non-rebasing tree as a base.
Now we only need to mainline a few thousand patches of platform support :-)
Just to make sure I understood you correctly: Do you mean more patches that you need to add to your tree, or getting everything from your tree upstream?
Arnd
On Fri, 12 Nov 2010, Linus Walleij wrote:
I have now created a tree based on Torvalds' clean v2.6.37-rc1 that is available straight off kernel.org:
git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-stericsson.git
Excellent!
NOTE: I am no Jedi yet. I still have to learn a few things about git I believe, right now this is really a -next type tree where I rebase the topic branches partly against each other, so it is being pushed out with forced updates, sometimes changing a lot of stuff inside the branches.
And that's _fine_ when it is clear that your tree is not a stable tree.
Today's series seems to apply almost cleanly on top of Russell's ARM tree, so as a start, you can create one git tree based on his and apply all your patches on top.
Right now this is just plain old v2.6.37-rc1 since Russell merged a lot of the stuff we wanted into that RC.
But if desired by the other members I can sure move over to Russells tree as a baseline.
You can also move to the linux-linaro-2.6.36 too. It is an hibrid of the latest ARM stuff from 2.6.37-rc1 (actually most of the latest ARM stuff was itself based on a pre-2.6.36 tree so that's easy) but with the core kernel code unchanged for stability.
But for upstream submission you will still have to base your branches onto well-known stable upstream branches.
And FYI Russell's tree, especially the "devel" branch, is _not_ such a stable branch. Don't ever attempt a pull request based on his devel branch.
Now we only need to mainline a few thousand patches of platform support :-)
Heh, that's no problem. When properly reviewed, you can put them all in a branch that you can ask RMK to pull. No need to stuff them all in his patch tracking system then, especially if they don't touch any of the core ARM code. The biggest problem is usually to make those few thousand patches in good shape.
Nicolas
-
Alexander Sack -
Arnd Bergmann -
Christian Robottom Reis -
Linus Walleij -
Linus Walleij -
Nicolas Pitre