Re: Config fragment for Versatile Express [ was: [RFC] First pass config fragment breakout for linaro kernel]
On 03/29/2012 02:22 AM, Jon Medhurst (Tixy) wrote:
On Mon, 2012-03-26 at 12:20 -0700, John Stultz wrote:
So after talking about it at the last Linaro Connect, I've finally gotten around to making a first pass at providing config fragments for the linaro kernel. I'd like to propose merging this for 12.04, and doing so early so we can make sure that all the desired config options are present in the fragments and to allow the vairous linaro build systems to begin migrating their config generation over.
The current tree is here:
http://git.linaro.org/gitweb?p=people/jstultz/android.git;a=shortlog;h=refs/heads/linaro-configs-3.3[...]
I'd ask Landing teams to take a look at this, and report any missing config options or fragment chunks they'd like to see.
John, I've attached a config fragment for Versatile Express.
Great! I've merged that in! There's a few warnings though:
Value requested for CONFIG_ARCH_VEXPRESS_DT not in final .config Requested value: CONFIG_ARCH_VEXPRESS_DT=y Actual value:
Value requested for CONFIG_FB_ARMHDLCD not in final .config Requested value: CONFIG_FB_ARMHDLCD=y Actual value:
I'm guessing these are features not in the base 3.3 tree? If so you might want to break those out and re-add them with those patches.
This includes loadable module support because one of our topic branches adds the gator module with a default config of 'm'. I did this because Linaro kernels are expected to have this module available but I didn't see any reason for it to be built-in, and as there may be versioning issues between it and the separate user side daemon, I thought it wise to keep the door open for loading an alternate module obtained from elsewhere. That decision does mean that all Linaro kernels would need loadable module support built in, but I don't think that is a bad idea.
Tushar had similar request, but I don't think the android configs (at least the ones I've managed) use modules, so I've added the MODULES=y to the common ubuntu.conf file.
If this is still objectionable, it can be changed and we can push it down to the linaro-base.conf, but I want to make sure the Android tree doesn't run into trouble.
thanks -john
On Thu, 2012-03-29 at 11:00 -0700, John Stultz wrote:
On 03/29/2012 02:22 AM, Jon Medhurst (Tixy) wrote:
John, I've attached a config fragment for Versatile Express.
Great! I've merged that in! There's a few warnings though:
Value requested for CONFIG_ARCH_VEXPRESS_DT not in final .config Requested value: CONFIG_ARCH_VEXPRESS_DT=y Actual value:
Value requested for CONFIG_FB_ARMHDLCD not in final .config Requested value: CONFIG_FB_ARMHDLCD=y Actual value:
I'm guessing these are features not in the base 3.3 tree? If so you might want to break those out and re-add them with those patches.
To do that the vexpress config fragment will need to be a topic branch on the ARM Landing Teams git, and every topic which changes a config needs to be stacked on top of that. Is that what is expected?
Currently I have two separate topics with monolithic Android and Ubuntu configs, so if we are moving over to config fragments I can delete these?
This includes loadable module support because one of our topic branches adds the gator module with a default config of 'm'. I did this because Linaro kernels are expected to have this module available but I didn't see any reason for it to be built-in, and as there may be versioning issues between it and the separate user side daemon, I thought it wise to keep the door open for loading an alternate module obtained from elsewhere. That decision does mean that all Linaro kernels would need loadable module support built in, but I don't think that is a bad idea.
Tushar had similar request, but I don't think the android configs (at least the ones I've managed) use modules, so I've added the MODULES=y to the common ubuntu.conf file.
That sounds fair enough, and meets the current requirements. We can deal with other requirements as the become apparent.
On 03/30/2012 01:19 AM, Jon Medhurst (Tixy) wrote:
On Thu, 2012-03-29 at 11:00 -0700, John Stultz wrote:
On 03/29/2012 02:22 AM, Jon Medhurst (Tixy) wrote:
John, I've attached a config fragment for Versatile Express.
Great! I've merged that in! There's a few warnings though:
Value requested for CONFIG_ARCH_VEXPRESS_DT not in final .config Requested value: CONFIG_ARCH_VEXPRESS_DT=y Actual value:
Value requested for CONFIG_FB_ARMHDLCD not in final .config Requested value: CONFIG_FB_ARMHDLCD=y Actual value:
I'm guessing these are features not in the base 3.3 tree? If so you might want to break those out and re-add them with those patches.
To do that the vexpress config fragment will need to be a topic branch on the ARM Landing Teams git, and every topic which changes a config needs to be stacked on top of that. Is that what is expected?
I'm not sure I'm following you here. I'm hoping to have the base configs added to the lianro-android tree, then as each topic gets merged, the topics which require an option, enable them in the fragments as well. Then as topic branch features are merged upstream, those config enablement patches get merged into the base config tree.
However, if that sort of fine-grained management is too onerous, I'm ok with dealing with the warnings. I realize that managing the patch queues can be hard enough work, so I don't mean to add more work if it doesn't provide much benefit.
Currently I have two separate topics with monolithic Android and Ubuntu configs, so if we are moving over to config fragments I can delete these?
Well, you might hang on to them somewhere, just to make sure no configs get moved or dropped and then cause problems later on.
thanks -john
On Fri, 2012-03-30 at 09:33 -0700, John Stultz wrote:
On 03/30/2012 01:19 AM, Jon Medhurst (Tixy) wrote:
To do that the vexpress config fragment will need to be a topic branch on the ARM Landing Teams git, and every topic which changes a config needs to be stacked on top of that. Is that what is expected?
I'm not sure I'm following you here. I'm hoping to have the base configs added to the lianro-android tree, then as each topic gets merged, the topics which require an option, enable them in the fragments as well.
I'm not sure I follow you either :-)
Our topic branches are based on mainline Linux. If those topics require config changes, do you suggest they add a separate config fragment? Or patch an existing one? If the latter, where does this fragment come from? It will have to exist into our tree and our topics based on top of it. I was saying that in the case of the vexpress fragment, this would live in our tree as it's own topic branch and be pulled into linux-linaro. Which seems to make sense, as our tree exists to provide enablement for vexpress.
On 03/30/2012 10:07 AM, Jon Medhurst (Tixy) wrote:
On Fri, 2012-03-30 at 09:33 -0700, John Stultz wrote:
On 03/30/2012 01:19 AM, Jon Medhurst (Tixy) wrote:
To do that the vexpress config fragment will need to be a topic branch on the ARM Landing Teams git, and every topic which changes a config needs to be stacked on top of that. Is that what is expected?
I'm not sure I'm following you here. I'm hoping to have the base configs added to the lianro-android tree, then as each topic gets merged, the topics which require an option, enable them in the fragments as well.
I'm not sure I follow you either :-)
Our topic branches are based on mainline Linux. If those topics require config changes, do you suggest they add a separate config fragment? Or patch an existing one? If the latter, where does this fragment come from? It will have to exist into our tree and our topics based on top of it. I was saying that in the case of the vexpress fragment, this would live in our tree as it's own topic branch and be pulled into linux-linaro. Which seems to make sense, as our tree exists to provide enablement for vexpress.
Right right right. I forgot with the new topic branch method, everything based on mainline and not a tree Andrey maintains, so you don't have a reference to the config tree.
In that case, just go ahead and push the full config to the config tree. If we need to do have fullly-enabled vs upstream builds we can deal with the warnings in the latter case (or maybe further split the board configs into -upstream and -lt ?).
The only hard part is that I have to somewhat blindly trust the configs being sent to me, as the tree I'm building/testing with doesn't necessarily have all of the features requested. But I'll try to get the build folks to keep me in the loop on what warnings they see.
thanks -john
On Fri, 2012-03-30 at 10:15 -0700, John Stultz wrote:
Right right right. I forgot with the new topic branch method, everything based on mainline and not a tree Andrey maintains, so you don't have a reference to the config tree.
Yes, Andrey's tree is a merge of all the LT and working group topics.
In that case, just go ahead and push the full config to the config tree. If we need to do have fullly-enabled vs upstream builds we can deal with the warnings in the latter case (or maybe further split the board configs into -upstream and -lt ?).
So this means Landing Teams should host the configs for their boards and you will host the linaro-base, ubuntu and android fragments?
We almost certainly need board specific android and ubuntu fragments as well, so I'll add vexpress-android.conf and vexpress-ubuntu.conf as well. (Unless there is some magic to have conditional config options in a fragment?)
On 03/31/2012 01:17 PM, Jon Medhurst (Tixy) wrote:
On Fri, 2012-03-30 at 10:15 -0700, John Stultz wrote:
Right right right. I forgot with the new topic branch method, everything based on mainline and not a tree Andrey maintains, so you don't have a reference to the config tree.
Yes, Andrey's tree is a merge of all the LT and working group topics.
In that case, just go ahead and push the full config to the config tree. If we need to do have fullly-enabled vs upstream builds we can deal with the warnings in the latter case (or maybe further split the board configs into -upstream and -lt ?).
So this means Landing Teams should host the configs for their boards and you will host the linaro-base, ubuntu and android fragments?
We could have a separate topic branch for the linaro-base and ubuntu and fragments (not board specific), as there is no linaro-base or ubuntu topic in linux-linaro. Otherwise the generic features (not board specific) should also add the config fragments to their topic branches. So the android fragment could live in the android topic as well.
We almost certainly need board specific android and ubuntu fragments as well, so I'll add vexpress-android.conf and vexpress-ubuntu.conf as well. (Unless there is some magic to have conditional config options in a fragment?)
On 04/02/2012 12:59 AM, Somebody in the thread at some point said:
On 03/31/2012 01:17 PM, Jon Medhurst (Tixy) wrote:
On Fri, 2012-03-30 at 10:15 -0700, John Stultz wrote:
Right right right. I forgot with the new topic branch method, everything based on mainline and not a tree Andrey maintains, so you don't have a reference to the config tree.
Yes, Andrey's tree is a merge of all the LT and working group topics.
In that case, just go ahead and push the full config to the config tree. If we need to do have fullly-enabled vs upstream builds we can deal with the warnings in the latter case (or maybe further split the board configs into -upstream and -lt ?).
So this means Landing Teams should host the configs for their boards and you will host the linaro-base, ubuntu and android fragments?
We could have a separate topic branch for the linaro-base and ubuntu and fragments (not board specific), as there is no linaro-base or ubuntu topic in linux-linaro. Otherwise the generic features (not board specific) should also add the config fragments to their topic branches. So the android fragment could live in the android topic as well.
This is definitely the right way.
-Andy
On Sun, 2012-04-01 at 20:59 +0400, Andrey Konovalov wrote:
On 03/31/2012 01:17 PM, Jon Medhurst (Tixy) wrote:
On Fri, 2012-03-30 at 10:15 -0700, John Stultz wrote:
In that case, just go ahead and push the full config to the config tree. If we need to do have fullly-enabled vs upstream builds we can deal with the warnings in the latter case (or maybe further split the board configs into -upstream and -lt ?).
So this means Landing Teams should host the configs for their boards and you will host the linaro-base, ubuntu and android fragments?
We could have a separate topic branch for the linaro-base and ubuntu and fragments (not board specific), as there is no linaro-base or ubuntu topic in linux-linaro. Otherwise the generic features (not board specific) should also add the config fragments to their topic branches. So the android fragment could live in the android topic as well.
I'm not sure I follow this, can you give some examples of what files live in what repos in what branches?
The things we need to know is: - What config fragemnts is the LT the origin of? - How these should be organised. - Where I get the rest of the config fragments from which I need to build and test the LT tree for Ubuntu and Android.
One suggestion...
Have a directory structure like
configs/linaro/base/ configs/linaro/android/ configs/linaro/ubuntu/
configs/$BOARD/base/ configs/$BOARD/android/ configs/$BOARD/ubuntu/
Then the configs for Android on $BOARD would be...
Get and sort alpha-numerically: configs/linaro/base/* configs/$BOARD/base/* then get and sort alpha-numerically configs configs/linaro/android/* configs/$BOARD/android/*
And if the 'linaro' directory used configs prefixed with numbers say from 30..69, and $BOARD could use 00..29 and 70..99 (the latter to override linaro configs which cause problems.
Then topic branches which add new generic feature can add a file like
configs/linaro/base/60-new-feature.conf
and for say a board specific feature:
configs/$BOARD/base/20-new-feature.conf
and a board workaround for an Android problem
configs/$BOARD/android/99-disable-problematic-config.conf
The current set of generic configs we have would become
configs/linaro/base/50-general.conf configs/linaro/android/50-general.conf configs/linaro/ubuntu/50-general.conf
but they should be split into smaller files eventually, e.g. 50-general.conf would currently have a bunch of trace and profiling options that Gator requires, if these were moved to 51-gator.conf then people would know what and why these are there.
Anyway, that was just my proposal because I haven't seen it explained anywhere how this stuff is meant to work.
On 04/02/2012 05:31 PM, Somebody in the thread at some point said:
On Sun, 2012-04-01 at 20:59 +0400, Andrey Konovalov wrote:
On 03/31/2012 01:17 PM, Jon Medhurst (Tixy) wrote:
On Fri, 2012-03-30 at 10:15 -0700, John Stultz wrote:
In that case, just go ahead and push the full config to the config tree. If we need to do have fullly-enabled vs upstream builds we can deal with the warnings in the latter case (or maybe further split the board configs into -upstream and -lt ?).
So this means Landing Teams should host the configs for their boards and you will host the linaro-base, ubuntu and android fragments?
We could have a separate topic branch for the linaro-base and ubuntu and fragments (not board specific), as there is no linaro-base or ubuntu topic in linux-linaro. Otherwise the generic features (not board specific) should also add the config fragments to their topic branches. So the android fragment could live in the android topic as well.
I'm not sure I follow this, can you give some examples of what files live in what repos in what branches?
This is all being made up as we go along, nobody is using this new flow yet.
The things we need to know is:
- What config fragemnts is the LT the origin of?
I think we have to provide a minimal, working, defconfig like we do now and that is the starting point for everything else piled on top.
- How these should be organised.
- Where I get the rest of the config fragments from which I need to build and test the LT tree for Ubuntu and Android.
One suggestion...
Have a directory structure like
configs/linaro/base/ configs/linaro/android/ configs/linaro/ubuntu/
I don't think this is a good way. There are two things we found having already being doing "config fragments" for about a year in TI LT repo.
- having multiple defconfigs is a mistake, they will diverge
- the fragments themselves rot quickly from changes in mainline, both by way of defaults changing and diffing the defconfig not being a perfect fit for what it represents (the defconfig is an output of another process out of sight with its own inputs, so the patches in the tree changing it are not the only things touching it). In particular it's almost impossible to hold the line with multiple finegrained config changes in one topic, we now squash everything into one config patch per topic.
By way of example, previously we ran a different defconfig for android and vanilla, now we patch the one defconfig when we add Androidization patches. That means the Android build always gets the latest and greatest stuff same as vanilla, plus whatever it needs specially.
Don't forget we won't be basing on linux-linaro-tracking, but vanilla. So that means we'll be producing linux-linaro-tracking "flavour branches" which can apply the "linaro house rules" for defconfigs such as being more like allmodules.
-Andy
On Mon, 2012-04-02 at 17:45 +0800, Andy Green wrote:
On 04/02/2012 05:31 PM, Somebody in the thread at some point said:
On Sun, 2012-04-01 at 20:59 +0400, Andrey Konovalov wrote:
We could have a separate topic branch for the linaro-base and ubuntu and fragments (not board specific), as there is no linaro-base or ubuntu topic in linux-linaro. Otherwise the generic features (not board specific) should also add the config fragments to their topic branches. So the android fragment could live in the android topic as well.
I'm not sure I follow this, can you give some examples of what files live in what repos in what branches?
This is all being made up as we go along, nobody is using this new flow yet.
I believe we (ARM LT) are expecting to use it imminently, so that it why I'm trying to work out what to do.
[snipped my suggestion about organising lots of config fragment]
I don't think this is a good way. There are two things we found having already being doing "config fragments" for about a year in TI LT repo.
having multiple defconfigs is a mistake, they will diverge
the fragments themselves rot quickly from changes in mainline, both
by way of defaults changing and diffing the defconfig not being a perfect fit for what it represents (the defconfig is an output of another process out of sight with its own inputs, so the patches in the tree changing it are not the only things touching it). In particular it's almost impossible to hold the line with multiple finegrained config changes in one topic, we now squash everything into one config patch per topic.
I can see that lot's of fragments might be a problem, but I think we need some middle ground.
On Mon, 2012-04-02 at 11:31 +0100, Jon Medhurst (Tixy) wrote:
On Mon, 2012-04-02 at 17:45 +0800, Andy Green wrote:
On 04/02/2012 05:31 PM, Somebody in the thread at some point said:
[snipped my suggestion about organising lots of config fragment]
I don't think this is a good way. There are two things we found having already being doing "config fragments" for about a year in TI LT repo.
having multiple defconfigs is a mistake, they will diverge
the fragments themselves rot quickly from changes in mainline, both
by way of defaults changing and diffing the defconfig not being a perfect fit for what it represents (the defconfig is an output of another process out of sight with its own inputs, so the patches in the tree changing it are not the only things touching it). In particular it's almost impossible to hold the line with multiple finegrained config changes in one topic, we now squash everything into one config patch per topic.
I can see that lot's of fragments might be a problem, but I think we need some middle ground.
I accidentally sent my previous reply whilst still editing it, and then couldn't think of what I wanted to say.
One problem I'm worried about is bloat and rot caused by a config that just constantly expanded and patched. Currently, our Ubuntu kernels use a config got from the pre-existing vexpress builds which seemed to include a lot of OMAP devices and various miscellaneous crap.
Another issue is we occasionally get requirements or bugs which require config option to be applied across all kernels.
I guess I think the current split with board/base/{linaro|android} is about right. At least if the common bits have a permanent home in a git repo, then we have a single place to apply system wide changes and the git log can explain the history of configs change.
One issue I think we will have is if there are any board specific platform level hacks or workarounds. E.g. how do do an Android specific config change for vexpress when android.conf doesn't live in our tree? I guess that if the board conf settings always override the generic one, then we can just have and android topic which patches our board conf.
On 04/02/2012 07:40 PM, Somebody in the thread at some point said:
On Mon, 2012-04-02 at 11:31 +0100, Jon Medhurst (Tixy) wrote:
On Mon, 2012-04-02 at 17:45 +0800, Andy Green wrote:
On 04/02/2012 05:31 PM, Somebody in the thread at some point said:
[snipped my suggestion about organising lots of config fragment]
I don't think this is a good way. There are two things we found having already being doing "config fragments" for about a year in TI LT repo.
having multiple defconfigs is a mistake, they will diverge
the fragments themselves rot quickly from changes in mainline, both
by way of defaults changing and diffing the defconfig not being a perfect fit for what it represents (the defconfig is an output of another process out of sight with its own inputs, so the patches in the tree changing it are not the only things touching it). In particular it's almost impossible to hold the line with multiple finegrained config changes in one topic, we now squash everything into one config patch per topic.
I can see that lot's of fragments might be a problem, but I think we need some middle ground.
I accidentally sent my previous reply whilst still editing it, and then couldn't think of what I wanted to say.
One problem I'm worried about is bloat and rot caused by a config that just constantly expanded and patched. Currently, our Ubuntu kernels use a config got from the pre-existing vexpress builds which seemed to include a lot of OMAP devices and various miscellaneous crap.
Another issue is we occasionally get requirements or bugs which require config option to be applied across all kernels.
I guess I think the current split with board/base/{linaro|android} is about right. At least if the common bits have a permanent home in a git repo, then we have a single place to apply system wide changes and the git log can explain the history of configs change.
Something to bear in mind is what happens to the "single defconfig" is spread over multiple output trees / branches each with a different purpose and context.
So patching CONFIG_ANDROID etc on to the common defconfig is not that scary and prone to rot when you consider it happens only on the branch you flavoured with linaro-androidization-tracking, by one patch on that flavouring branch used by all the LTs the same. Ie, you end up only seeing CONFIG_ANDROID on top of your minimal config *in the android branch*, which is what you want.
Your source tree that was used as an ingredient in the flavouring is "read-only" for that purpose and his defconfig stays the same -- that's the came for other "flavourings" like linux-linaro as well.
So as long as your minimal defconfig in your LT tree is in good shape for your board, you can transform it many different ways in the flavouring trees without adding risk and keeping the advantage that the flavoured versions inherit your definitive minimal config set relevant to the board.
One issue I think we will have is if there are any board specific platform level hacks or workarounds. E.g. how do do an Android specific config change for vexpress when android.conf doesn't live in our tree? I guess that if the board conf settings always override the generic one, then we can just have and android topic which patches our board conf.
Yes that was how we did our Android tree before, like this
- vanilla tree - Panda-specific Android features (SGX) + android defconfig - androidization patches
Now we will do it slightly differently, not just to conform to a common config fragment but because we learned it's a better flow
- vanilla tree - generic androidization + androidize our vanilla defconfig - Panda-specific Android features (SGX) + additional defconfig bits
In TI LT we script and manage these flows across multiple trees so very little is by hand and not reproducible; you don't have to use our same scheme but you'll need something to automate it, especially when more flavours appear. We also use rebases so we know what is where (and our trees are fully serialized) when other people think merges are better when combining flavours.
-Andy
On Mon, 2012-04-02 at 12:40 +0100, Jon Medhurst (Tixy) wrote:
I guess I think the current split with board/base/{linaro|android} is about right. At least if the common bits have a permanent home in a git repo, then we have a single place to apply system wide changes and the git log can explain the history of configs change.
On third thoughts ;-) it seems a lot nicer if topics which introduce a feature can contain a fragment to enable that feature, rather than have to base all topics branches on top of a config topic branch just so they can patch the config.
I'll go with my first suggestion, having multiple files in directories seems cleaner and if people want to manage it as a single monolithic config they can just put one file in that directory.
So, to get the Android config for $BOARD we would have...
merge_config.sh configs/linaro/base/* configs/$BOARD/base/* \ configs/linaro/android/* configs/$BOARD/android/*
I believe that globbing will always sort alphanumeric ASCII based names in a consistent order, or would we need to prefix it with LC_COLLATE=C ?
On 04/02/2012 08:42 PM, Somebody in the thread at some point said:
On Mon, 2012-04-02 at 12:40 +0100, Jon Medhurst (Tixy) wrote:
I guess I think the current split with board/base/{linaro|android} is about right. At least if the common bits have a permanent home in a git repo, then we have a single place to apply system wide changes and the git log can explain the history of configs change.
On third thoughts ;-) it seems a lot nicer if topics which introduce a feature can contain a fragment to enable that feature, rather than have to base all topics branches on top of a config topic branch just so they can patch the config.
... which is what we have been doing for a year or so.
But the patches on the defconfig for particular features related to the topic live in the topic branch implementing the features; we haven't found they need any directory structure just operate direct on the defconfig.
If you want to do it with this complex directory scheme, please don't so anything to the definitive sources that makes it mandatory.
-Andy
On Mon, 2012-04-02 at 21:10 +0800, Andy Green wrote:
If you want to do it with this complex directory scheme, please don't so anything to the definitive sources that makes it mandatory.
Just so I understand properly, are you saying that for the TI kernels you want to just supply a single fully featured defconfig file and have that used to produce builds, rather than having one built up from OMAP bits supplied by you and other android/ubuntu/linaro bits obtained from another source?
On 04/02/2012 09:40 PM, Somebody in the thread at some point said:
On Mon, 2012-04-02 at 21:10 +0800, Andy Green wrote:
If you want to do it with this complex directory scheme, please don't so anything to the definitive sources that makes it mandatory.
Just so I understand properly, are you saying that for the TI kernels you want to just supply a single fully featured defconfig file and have that used to produce builds, rather than having one built up from OMAP bits supplied by you and other android/ubuntu/linaro bits obtained from another source?
Not at all.
I don't want to sound like a broken record but we have been doing this layered config stuff for a long time. It's a very good wheeze and centralizing some of it will give good results if we can do it in a good way.
Here's an example from our repo.
Currently our vanilla tree ends at
http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git%3Ba=short...
that's made up of many topics each of which add to the single defconfig omap_5430evm_defconfig so it builds and configures for the topic content. At this end point, it has all the topic patches in and all the topic config enabled.
An example of what we have been talking about can be found at the "flavour" branch lat-3.3.
http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git%3Ba=short...
This is based on the vanilla tree from tracking-topic-omapdrm, it contains the generic androidization patches.
At the end of lat-3.3, I have a patch "config OMAP5 Android" which adapts the single defconfig that came in from vanilla, omap_5430evm_defconfig, to have generic Androidization options
http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git%3Ba=commi...
So when you check out lat-3.3 or one of the tags pointing at it, you get always up-to-date, androidized, omap4+omap5 defconfig with the same name as vanilla.
The defconfig you check out at the vanilla tree back at tracking-topic-omapdrm doesn't have to know or take care or be affected by any of that.
When it's available and we're able to, we'll also start putting out a new flavour branch which has linux-linaro-tracking "flavoured" on to it, and just like in lat-3.3 case we will modify the vanilla defconfig according to what that flavour needs.
All we require is that the flavour tree, be it linux-androidization-tracking, or linux-linaro-tracking, comes with a patch containing the defconfig delta that's meaningful for it.
If that makes sense, you can see there's no need for directory structures, the various flavour and result trees can contain all the deltas and variant defconfigs.
-Andy
On Mon, 2012-04-02 at 21:58 +0800, Andy Green wrote:
I don't want to sound like a broken record but we have been doing this layered config stuff for a long time. It's a very good wheeze and centralizing some of it will give good results if we can do it in a good way.
Here's an example from our repo.
<snip>
When it's available and we're able to, we'll also start putting out a new flavour branch which has linux-linaro-tracking "flavoured" on to it, and just like in lat-3.3 case we will modify the vanilla defconfig according to what that flavour needs.
All we require is that the flavour tree, be it linux-androidization-tracking, or linux-linaro-tracking, comes with a patch containing the defconfig delta that's meaningful for it.
Those methods are only really applicable if you have all the topics stacked on top of each other. Some other teams won't be doing that.
Also, the 'flavour' added onto a basic board config would include topics from working groups as well as the platform specific settings (Ubuntu/Android) and Linaro policy settings. These can't be practically supplied as patches to each separate boards defconfig so they will have to be some mechanism like config fragments and a tool to merge these with the board config.
I guess I'm arguing that such a config merge tool be available for use with for board specific configuration, for those teams without stacked topics. And that the inputs to the tool (config fragments) be managed and stored in a way that was intuitively used, e.g.
get linaro-bits android-bits board-bits
where it doesn't need to reference something outside of the git tree to know what are in the 'bits'.
On 04/03/2012 02:39 AM, Somebody in the thread at some point said:
On Mon, 2012-04-02 at 21:58 +0800, Andy Green wrote:
I don't want to sound like a broken record but we have been doing this layered config stuff for a long time. It's a very good wheeze and centralizing some of it will give good results if we can do it in a good way.
Here's an example from our repo.
<snip> > When it's available and we're able to, we'll also start putting out a > new flavour branch which has linux-linaro-tracking "flavoured" on to > it, > and just like in lat-3.3 case we will modify the vanilla defconfig > according to what that flavour needs. > > All we require is that the flavour tree, be it > linux-androidization-tracking, or linux-linaro-tracking, comes with a > patch containing the defconfig delta that's meaningful for it.
Those methods are only really applicable if you have all the topics stacked on top of each other. Some other teams won't be doing that.
Right, that is the source of the different thinking and approaches. Merge-based process is not equivalent to serialized rebases. However as seen where merge-based process has to fall back to being serialized rebases due to topic dependencies ^^ they are not a million miles from each other either.
Also, the 'flavour' added onto a basic board config would include topics
Well, "some flavours" rather than "the flavour", but yes.
from working groups as well as the platform specific settings (Ubuntu/Android) and Linaro policy settings. These can't be practically supplied as patches to each separate boards defconfig so they will have to be some mechanism like config fragments and a tool to merge these with the board config.
Yes I was showing how we managed it until now. The new system AIUI will give a new power to "assert config deltas" using this new script (so you can graft on like mostly-allmodules for distros easily). If it's workable for us we'll probably move to using the same script and config delta files, which is why I am engaging on this thread.
I guess I'm arguing that such a config merge tool be available for use with for board specific configuration, for those teams without stacked topics. And that the inputs to the tool (config fragments) be managed and stored in a way that was intuitively used, e.g.
get linaro-bits android-bits board-bits
where it doesn't need to reference something outside of the git tree to know what are in the 'bits'.
As I see it the config deltas need to come from the flavouring branch. They won't be a patch on the LT-specific defconfig like I showed we do now, the flavouring branches should each contain a config delta file like ./somewhere/linaro-androidization-tracking.config, eg ./somewhere/<original branchname>.config.
That way we avoid needing a special central config repo and the guy who works on the flavouring tree can ensure his config delta is always in sync with whatever version of his tree you pulled. Scripts who know which branch they're merging or rebasing can easily find the correct, contemporary config delta needed by the branch (doing it with separate central config repo would suck if you want to get config delta out of it to match random old version, from a tag say, of flavouring tree. If the config delta is in the flavouring tree, that just works).
AIUI all I need to do then is adapt my topic management scripts to check for such a file and create a patch to capture the changes from running the "assert config deltas" script for what's being as part of the rebase action on my specific defconfig, and it will be transparent.
-Andy
On 04/02/2012 06:58 AM, Andy Green wrote:
On 04/02/2012 09:40 PM, Somebody in the thread at some point said:
On Mon, 2012-04-02 at 21:10 +0800, Andy Green wrote:
If you want to do it with this complex directory scheme, please don't so anything to the definitive sources that makes it mandatory.
Just so I understand properly, are you saying that for the TI kernels you want to just supply a single fully featured defconfig file and have that used to produce builds, rather than having one built up from OMAP bits supplied by you and other android/ubuntu/linaro bits obtained from another source?
Not at all.
I don't want to sound like a broken record but we have been doing this layered config stuff for a long time. It's a very good wheeze and centralizing some of it will give good results if we can do it in a good way.
Here's an example from our repo.
Currently our vanilla tree ends at
http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git%3Ba=short...
that's made up of many topics each of which add to the single defconfig omap_5430evm_defconfig so it builds and configures for the topic content. At this end point, it has all the topic patches in and all the topic config enabled.
An example of what we have been talking about can be found at the "flavour" branch lat-3.3.
http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git%3Ba=short...
This is based on the vanilla tree from tracking-topic-omapdrm, it contains the generic androidization patches.
At the end of lat-3.3, I have a patch "config OMAP5 Android" which adapts the single defconfig that came in from vanilla, omap_5430evm_defconfig, to have generic Androidization options
http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git%3Ba=commi...
So when you check out lat-3.3 or one of the tags pointing at it, you get always up-to-date, androidized, omap4+omap5 defconfig with the same name as vanilla.
The defconfig you check out at the vanilla tree back at tracking-topic-omapdrm doesn't have to know or take care or be affected by any of that.
So, to make sure I'm following properly here, would it be fair to call this somewhat of an "additive", or "constructive" style config management, where you have your board defconfig, which you add to the end of your base tree, then any branches based off of that base tree will also contain patches to the board defconfig that enable the new features found in that branch?
In this way you have one config, that each branch adds to as needed.
When it's available and we're able to, we'll also start putting out a new flavour branch which has linux-linaro-tracking "flavoured" on to it, and just like in lat-3.3 case we will modify the vanilla defconfig according to what that flavour needs.
All we require is that the flavour tree, be it linux-androidization-tracking, or linux-linaro-tracking, comes with a patch containing the defconfig delta that's meaningful for it.
If that makes sense, you can see there's no need for directory structures, the various flavour and result trees can contain all the deltas and variant defconfigs.
So part of the config fragment work is motivated by the need to have consistent configs across a number of boards.
While the method you described above works well for managing board-specific config features enabled by LT branches, the config fragment work is trying to find a way to help simplify config management for features that are generic.
An example is the SCHED_MC item that I know Amit had to chase a few times in order to make sure it was enabled in all the various build configs, and that it stayed enabled as folks rebased forward to new kernel versions.
Initially I was hoping to provide a basic set of configs split up by base/distro/board. Then have folks use the same additive method you describe above to enable features per branch (in the appropriate config so LT's branches modifying the board config fragment, while power-management team's branches can modify the base fragment).
The difficulty is that as Tixy earlier pointed out, are that the LT kernel trees are mainline based, and thus aren't based off of something that would contain the base/distro/board config fragments.
One approach we might be able to use is if the board defconfigs really are minimal, and restricted in scope to only the board options, we could switch the merge order (board/distro/base). This way the LT's "additive" defconfig can be used (from arch/arm/configs/ ) and we can still also get consistent generic options as well.
The only issue with this idea is that it goes specific->generic it doesn't allow us to add board-specific hacks like Tixy's IPv6 example, since the last fragment wins.
Any other ideas?
thanks -john
On 04/03/2012 03:30 AM, Somebody in the thread at some point said:
On 04/02/2012 06:58 AM, Andy Green wrote:
On 04/02/2012 09:40 PM, Somebody in the thread at some point said:
On Mon, 2012-04-02 at 21:10 +0800, Andy Green wrote:
If you want to do it with this complex directory scheme, please don't so anything to the definitive sources that makes it mandatory.
Just so I understand properly, are you saying that for the TI kernels you want to just supply a single fully featured defconfig file and have that used to produce builds, rather than having one built up from OMAP bits supplied by you and other android/ubuntu/linaro bits obtained from another source?
Not at all.
I don't want to sound like a broken record but we have been doing this layered config stuff for a long time. It's a very good wheeze and centralizing some of it will give good results if we can do it in a good way.
Here's an example from our repo.
Currently our vanilla tree ends at
http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git%3Ba=short...
that's made up of many topics each of which add to the single defconfig omap_5430evm_defconfig so it builds and configures for the topic content. At this end point, it has all the topic patches in and all the topic config enabled.
An example of what we have been talking about can be found at the "flavour" branch lat-3.3.
http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git%3Ba=short...
This is based on the vanilla tree from tracking-topic-omapdrm, it contains the generic androidization patches.
At the end of lat-3.3, I have a patch "config OMAP5 Android" which adapts the single defconfig that came in from vanilla, omap_5430evm_defconfig, to have generic Androidization options
http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git%3Ba=commi...
So when you check out lat-3.3 or one of the tags pointing at it, you get always up-to-date, androidized, omap4+omap5 defconfig with the same name as vanilla.
The defconfig you check out at the vanilla tree back at tracking-topic-omapdrm doesn't have to know or take care or be affected by any of that.
So, to make sure I'm following properly here, would it be fair to call this somewhat of an "additive", or "constructive" style config management, where you have your board defconfig, which you add to the end of your base tree, then any branches based off of that base tree will also contain patches to the board defconfig that enable the new features found in that branch?
In this way you have one config, that each branch adds to as needed.
Right.
When it's available and we're able to, we'll also start putting out a new flavour branch which has linux-linaro-tracking "flavoured" on to it, and just like in lat-3.3 case we will modify the vanilla defconfig according to what that flavour needs.
All we require is that the flavour tree, be it linux-androidization-tracking, or linux-linaro-tracking, comes with a patch containing the defconfig delta that's meaningful for it.
If that makes sense, you can see there's no need for directory structures, the various flavour and result trees can contain all the deltas and variant defconfigs.
So part of the config fragment work is motivated by the need to have consistent configs across a number of boards.
While the method you described above works well for managing board-specific config features enabled by LT branches, the config fragment work is trying to find a way to help simplify config management for features that are generic.
An example is the SCHED_MC item that I know Amit had to chase a few times in order to make sure it was enabled in all the various build configs, and that it stayed enabled as folks rebased forward to new kernel versions.
Initially I was hoping to provide a basic set of configs split up by base/distro/board. Then have folks use the same additive method you describe above to enable features per branch (in the appropriate config so LT's branches modifying the board config fragment, while power-management team's branches can modify the base fragment).
The difficulty is that as Tixy earlier pointed out, are that the LT kernel trees are mainline based, and thus aren't based off of something that would contain the base/distro/board config fragments.
One approach we might be able to use is if the board defconfigs really are minimal, and restricted in scope to only the board options, we could switch the merge order (board/distro/base). This way the LT's "additive" defconfig can be used (from arch/arm/configs/ ) and we can still also get consistent generic options as well.
The LTs are strongly motivated by build time to maintain a really stripped-down defconfig for their board, because they sit there building it all day. However even with that desire for minimum build time, the config has to represent the minimum necessary to build all workable feature support for the boards the kernel targets, otherwise stuff will go backwards very quickly if we don't always build and test everything.
So the common config on our new tree does enable everything for all supported boards, generate some modules, but it does that because it's required so that conflicting or different support for different supported arch boards can coexist (for example, there is different WLAN module on Omap5 stuff, we have to build both Omap4 and Omap5 one as a minimum so we are workable on all the boards with one kernel binary).
I think in effect, although a true minimum config would have no display support, no audio etc, this is effectively a correct minimum basis that supports everything we have working on all boards, and anything less is not legit for downstream branches.
It also feels correct that LT control that since they know what's needed for their boards very well, correct that it is inside LT tree so if you go to earlier version that moves with it, etc.
The only issue with this idea is that it goes specific->generic it doesn't allow us to add board-specific hacks like Tixy's IPv6 example, since the last fragment wins.
Any other ideas?
Yes to be clear we know how we have been doing it isn't enough for what you're trying to do. It is useful to demonstrate general flow that works privately, so we all talk about same things.
As Tixy said, when it's just private the patches can "know" which defconfig(s) to patch, but that won't fly for this unified source scheme where all the LTs want to pull the same generic flavouring branch and have it process the defconfigs they happen to care about. So that's understood.
What I hope we end up with is the flavouring trees carrying a file with "defconfig assertions" in them, which this new script "makes so" on top of the inherited deconfig(s) from LT tree. This is not patching the defconfig any more but processing it so the assertions needed by the flavour branch are true, eg, CONFIG_ANDROID=y is easy enough but it also needs to be able to describe like - CONFIG_KILLS_ANDROID which will do what's needed, if anything, to the inherited defconfig to nuke that option.
As for last guy in winning, this is correct behaviour I think. If you consider Panda android flow
- vanilla - androidization - Panda-specific Android feature (CONFIG_SGX_540 etc)
the androidization should assert the config bits needed by generic androidization, and normally nothing in the Panda specific one would argue with that. In the case android logging code kills Panda for some reason though, that last flavouring branch should have the chance to override what the generic androidization assertions did about that with its own assertions... it's not conflicting but overriding.
-Andy
On Mon, 2012-04-02 at 12:30 -0700, John Stultz wrote:
The difficulty is that as Tixy earlier pointed out, are that the LT kernel trees are mainline based, and thus aren't based off of something that would contain the base/distro/board config fragments.
One approach we might be able to use is if the board defconfigs really are minimal, and restricted in scope to only the board options, we could switch the merge order (board/distro/base). This way the LT's "additive" defconfig can be used (from arch/arm/configs/ ) and we can still also get consistent generic options as well.
The mainline defconfigs aren't minimal in the sense we want, they include things like file-systems and networking options so somebody can build a usable kernel, and I think it's sensible to keep it that way.
For Linaro's purposes we would need a new board config. We could make that minimal, but we get back to the idea that topic branches which change configs would need to sit on top of the topic with this config. Perhaps that is something we can live with if a directory-of-config-fragments approach is deemed undesirable.
It's a pity that the title of the thread possibly means no one from other LTs are reading. (Probably a bit late to change it now and get noticed.)
On 04/03/2012 01:43 PM, Jon Medhurst (Tixy) wrote:
On Mon, 2012-04-02 at 12:30 -0700, John Stultz wrote:
The difficulty is that as Tixy earlier pointed out, are that the LT kernel trees are mainline based, and thus aren't based off of something that would contain the base/distro/board config fragments.
One approach we might be able to use is if the board defconfigs really are minimal, and restricted in scope to only the board options, we could switch the merge order (board/distro/base). This way the LT's "additive" defconfig can be used (from arch/arm/configs/ ) and we can still also get consistent generic options as well.
The mainline defconfigs aren't minimal in the sense we want, they include things like file-systems and networking options so somebody can build a usable kernel, and I think it's sensible to keep it that way.
For Linaro's purposes we would need a new board config. We could make that minimal, but we get back to the idea that topic branches which change configs would need to sit on top of the topic with this config. Perhaps that is something we can live with if a directory-of-config-fragments approach is deemed undesirable.
It's a pity that the title of the thread possibly means no one from other LTs are reading. (Probably a bit late to change it now and get noticed.)
I hope not.
For Samsung LT kernel, we have followed an approach where in the commits in John's linaro_config_3.3 branch are taken to be stable commits and we have put those commits as the very first set of commits on LT kernel. Our LT kernel being a _serialized_ kernel where topic branches sit one over the other, the related config fragments are made part of the topic branches.
A sample view of the same is posted at [1].
[1] git://git.linaro.org/landing-teams/working/samsung/kernel.git (lt/next)
On Tue, 2012-04-03 at 14:51 +0530, Tushar Behera wrote:
For Samsung LT kernel, we have followed an approach where in the commits in John's linaro_config_3.3 branch are taken to be stable commits and we have put those commits as the very first set of commits on LT kernel. Our LT kernel being a _serialized_ kernel where topic branches sit one over the other, the related config fragments are made part of the topic branches.
A sample view of the same is posted at [1].
[1] git://git.linaro.org/landing-teams/working/samsung/kernel.git (lt/next)
The Samsung tree has the non samsung config files, like linaro-base.conf. I was wondering if that would cause merge problems if every LT had them and at possibly different versions. I guess it doesn't matter so long as all LTs got them from the same definitive 'upstream' source, and that they didn't edit them.
I think it makes sense if this 'upstream' doesn't include board files though, they should come from LT trees.
We also need a better upstream than John's personal Android tree :-)
On 04/03/2012 06:29 PM, Somebody in the thread at some point said:
On Tue, 2012-04-03 at 14:51 +0530, Tushar Behera wrote:
For Samsung LT kernel, we have followed an approach where in the commits in John's linaro_config_3.3 branch are taken to be stable commits and we have put those commits as the very first set of commits on LT kernel. Our LT kernel being a _serialized_ kernel where topic branches sit one over the other, the related config fragments are made part of the topic branches.
A sample view of the same is posted at [1].
[1] git://git.linaro.org/landing-teams/working/samsung/kernel.git (lt/next)
The Samsung tree has the non samsung config files, like linaro-base.conf. I was wondering if that would cause merge problems if every LT had them and at possibly different versions. I guess it doesn't matter so long as all LTs got them from the same definitive 'upstream' source, and that they didn't edit them.
Squidging them all together will anyway require a lot of automated conflict resolution happening, favouring later version of this common file would probably not be noticable.
I'll have a go at Tushar's method tomorrow it looks like a good plan.
I think it makes sense if this 'upstream' doesn't include board files though, they should come from LT trees.
Normally "board files" means mach-xyz/board*.c for me I am guessing you mean board-specific defconfigs ^^
-Andy
On Tue, 2012-04-03 at 18:43 +0800, Andy Green wrote:
On 04/03/2012 06:29 PM, Somebody in the thread at some point said:
I think it makes sense if this 'upstream' doesn't include board files though, they should come from LT trees.
Normally "board files" means mach-xyz/board*.c for me I am guessing you mean board-specific defconfigs ^^
Indeed I did.
On 04/03/2012 03:59 PM, Jon Medhurst (Tixy) wrote:
On Tue, 2012-04-03 at 14:51 +0530, Tushar Behera wrote:
For Samsung LT kernel, we have followed an approach where in the commits in John's linaro_config_3.3 branch are taken to be stable commits and we have put those commits as the very first set of commits on LT kernel. Our LT kernel being a _serialized_ kernel where topic branches sit one over the other, the related config fragments are made part of the topic branches.
A sample view of the same is posted at [1].
[1] git://git.linaro.org/landing-teams/working/samsung/kernel.git (lt/next)
The Samsung tree has the non samsung config files, like linaro-base.conf. I was wondering if that would cause merge problems if every LT had them and at possibly different versions. I guess it doesn't
This method works well if the common config fragments, once committed to some upstream tree, are stable. Then only it makes sense to base other topics on top of this.
matter so long as all LTs got them from the same definitive 'upstream' source, and that they didn't edit them.
I think it makes sense if this 'upstream' doesn't include board files though, they should come from LT trees.
As for the board specific fragments, I feel it would be better if the config-upstream tree has the initial fragment (which I suppose is minimal enough to compile the common kernel). That way anyone taking linux-linaro-tracking can have at least a working setup.
As and when some topics get merged to mainline, we can move those config-fragments to upstream tree.
We also need a better upstream than John's personal Android tree :-)
I suppose, it would soon get a place in linux-linaro-tracking.
On Wed, 2012-04-04 at 10:05 +0530, Tushar Behera wrote:
On 04/03/2012 03:59 PM, Jon Medhurst (Tixy) wrote:
I think it makes sense if this 'upstream' doesn't include board files though, they should come from LT trees.
As for the board specific fragments, I feel it would be better if the config-upstream tree has the initial fragment (which I suppose is minimal enough to compile the common kernel). That way anyone taking linux-linaro-tracking can have at least a working setup.
I thought linux-linaro-tracking was going to operate a bit like linux-next, i.e. just a merge of all the topics from LTs and working groups, and from which Linaro produces it's releases and does some CI build+test. If that's true, then anyone using that tree will always have the LT code.
Though I guess if LT code breaks other boards and has to get temporarily dropped, and if that LT code was pulled in as one monolithic topic like TI and Samsung trees, then it would probably mean dropping the board config too if we had that coming from the LT tree. But in that case, does it matter that the broken board can't be built from linux-linaro-tracking?
On 04/04/2012 01:02 PM, Jon Medhurst (Tixy) wrote:
On Wed, 2012-04-04 at 10:05 +0530, Tushar Behera wrote:
On 04/03/2012 03:59 PM, Jon Medhurst (Tixy) wrote:
I think it makes sense if this 'upstream' doesn't include board files though, they should come from LT trees.
As for the board specific fragments, I feel it would be better if the config-upstream tree has the initial fragment (which I suppose is minimal enough to compile the common kernel). That way anyone taking linux-linaro-tracking can have at least a working setup.
I thought linux-linaro-tracking was going to operate a bit like linux-next, i.e. just a merge of all the topics from LTs and working groups, and from which Linaro produces it's releases and does some CI build+test. If that's true, then anyone using that tree will always have the LT code.
In some cases, the mainline kernel cannot boot a specific board using the default config file. If we have a config fragment that adds the necessary changes, people can use that specific board even if LT has no other enablement patches merged to linux-linaro-tracking.
An example would be the thermal patches, which got merged to linux-linaro-tracking through PM WG. But for someone to validate those changes, they won't have a definite config to build linux-linaro-tracking and run on Origen board. Hence I feel that having a minimal board fragment, that boots the kernel till console, in linux-linaro-samsung would be helpful.
As long as we have same set of commits in our tree, we won't have merge conflicts too.
Though I guess if LT code breaks other boards and has to get temporarily dropped, and if that LT code was pulled in as one monolithic topic like TI and Samsung trees, then it would probably mean dropping the board config too if we had that coming from the LT tree. But in that case, does it matter that the broken board can't be built from linux-linaro-tracking?
That is one concern from my side too, as to how do we build linux-linaro-tracking kernel and what all things goes into that.
On 04/04/2012 06:20 PM, Somebody in the thread at some point said:
On 04/04/2012 01:02 PM, Jon Medhurst (Tixy) wrote:
On Wed, 2012-04-04 at 10:05 +0530, Tushar Behera wrote:
On 04/03/2012 03:59 PM, Jon Medhurst (Tixy) wrote:
I think it makes sense if this 'upstream' doesn't include board files though, they should come from LT trees.
As for the board specific fragments, I feel it would be better if the config-upstream tree has the initial fragment (which I suppose is minimal enough to compile the common kernel). That way anyone taking linux-linaro-tracking can have at least a working setup.
I thought linux-linaro-tracking was going to operate a bit like linux-next, i.e. just a merge of all the topics from LTs and working groups, and from which Linaro produces it's releases and does some CI build+test. If that's true, then anyone using that tree will always have the LT code.
In some cases, the mainline kernel cannot boot a specific board using the default config file. If we have a config fragment that adds the necessary changes, people can use that specific board even if LT has no other enablement patches merged to linux-linaro-tracking.
An example would be the thermal patches, which got merged to linux-linaro-tracking through PM WG. But for someone to validate those changes, they won't have a definite config to build linux-linaro-tracking and run on Origen board. Hence I feel that having a minimal board fragment, that boots the kernel till console, in linux-linaro-samsung would be helpful.
As long as we have same set of commits in our tree, we won't have merge conflicts too.
Though I guess if LT code breaks other boards and has to get temporarily dropped, and if that LT code was pulled in as one monolithic topic like TI and Samsung trees, then it would probably mean dropping the board config too if we had that coming from the LT tree. But in that case, does it matter that the broken board can't be built from linux-linaro-tracking?
That is one concern from my side too, as to how do we build linux-linaro-tracking kernel and what all things goes into that.
That's a slightly different issue. If stuff that's too avant garde is put into linux-linaro-tracking it'll break one or more lt trees in a way that can't be quickly fixed.
This was seen in January where I tried to base on it, it broke my tree, called for help, got ignored for weeks until it was too late to care about that release because tracking had moved on.
In the end LTs can only help solve issues that fall into their areas of competence, which is far from infinite. If enough next-y stuff is shovelled into linux-linaro-tracking and we're left to it, it'll break all the LT trees. But I think that's understood and there's some attempt to keep it sane and hopefully now take care of any damage it causes.
The big problem in front of us today is "how well do the LTs trees fit together right now as a starting point?". We don't seem to be trying to bind the available trees at v3.3 together at the moment so we can understand our situation for some reason. Instead we're trying to use a single LT tree as an exemplar for a process that is all about binding all the trees.
Maybe we should take some time to do a full scope trial run and start making the small improvements that will be needed for them to slot together well ongoing.
-Andy
Hi,
On 2012-04-02, at 5:45 AM, Andy Green wrote:
On 04/02/2012 05:31 PM, Somebody in the thread at some point said:
On Sun, 2012-04-01 at 20:59 +0400, Andrey Konovalov wrote:
On 03/31/2012 01:17 PM, Jon Medhurst (Tixy) wrote:
On Fri, 2012-03-30 at 10:15 -0700, John Stultz wrote:
In that case, just go ahead and push the full config to the config tree. If we need to do have fullly-enabled vs upstream builds we can deal with the warnings in the latter case (or maybe further split the board configs into -upstream and -lt ?).
So this means Landing Teams should host the configs for their boards and you will host the linaro-base, ubuntu and android fragments?
We could have a separate topic branch for the linaro-base and ubuntu and fragments (not board specific), as there is no linaro-base or ubuntu topic in linux-linaro. Otherwise the generic features (not board specific) should also add the config fragments to their topic branches. So the android fragment could live in the android topic as well.
I'm not sure I follow this, can you give some examples of what files live in what repos in what branches?
This is all being made up as we go along, nobody is using this new flow yet.
The things we need to know is:
- What config fragemnts is the LT the origin of?
I think we have to provide a minimal, working, defconfig like we do now and that is the starting point for everything else piled on top.
- How these should be organised.
- Where I get the rest of the config fragments from which I need to
build and test the LT tree for Ubuntu and Android.
One suggestion...
Have a directory structure like
configs/linaro/base/ configs/linaro/android/ configs/linaro/ubuntu/
I don't think this is a good way. There are two things we found having already being doing "config fragments" for about a year in TI LT repo.
having multiple defconfigs is a mistake, they will diverge
the fragments themselves rot quickly from changes in mainline, both
by way of defaults changing and diffing the defconfig not being a perfect fit for what it represents (the defconfig is an output of another process out of sight with its own inputs, so the patches in the tree changing it are not the only things touching it). In particular it's almost impossible to hold the line with multiple finegrained config changes in one topic, we now squash everything into one config patch per topic.
By way of example, previously we ran a different defconfig for android and vanilla, now we patch the one defconfig when we add Androidization patches. That means the Android build always gets the latest and greatest stuff same as vanilla, plus whatever it needs specially.
Don't forget we won't be basing on linux-linaro-tracking, but vanilla. So that means we'll be producing linux-linaro-tracking "flavour branches" which can apply the "linaro house rules" for defconfigs such as being more like all modules.
Just as a proposition for the management of all these config files that you may already have discuss...:
Why not keeping one defconfig by board with all the latest dev in it and managing a script that fulfil the needs of dev and build server by adding the default config of linaro, android and ubuntu? Why not having a centralized place (git tree config) that keep the default linaro, android and ubuntu config files?
// Script that use merge_config to validate that defconfig contains config set and add them if not: linaroConfigAdder.sh -android defconfig linaroConfigAdder.sh -ubuntu defconfig linaroConfigAdder.sh -linaro defconfig
Question, why are you not adding the linaro config in the defconfig by default?
thx
KA
-Andy
-- Andy Green | TI Landing Team Leader Linaro.org │ Open source software for ARM SoCs | Follow Linaro http://facebook.com/pages/Linaro/155974581091106 - http://twitter.com/#%21/linaroorg - http://linaro.org/linaro-blog
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
On 2012-04-04, at 10:07 AM, Kevyn-Alexandre Paré wrote:
Hi,
On 2012-04-02, at 5:45 AM, Andy Green wrote:
On 04/02/2012 05:31 PM, Somebody in the thread at some point said:
On Sun, 2012-04-01 at 20:59 +0400, Andrey Konovalov wrote:
On 03/31/2012 01:17 PM, Jon Medhurst (Tixy) wrote:
On Fri, 2012-03-30 at 10:15 -0700, John Stultz wrote:
In that case, just go ahead and push the full config to the config tree. If we need to do have fullly-enabled vs upstream builds we can deal with the warnings in the latter case (or maybe further split the board configs into -upstream and -lt ?).
So this means Landing Teams should host the configs for their boards and you will host the linaro-base, ubuntu and android fragments?
We could have a separate topic branch for the linaro-base and ubuntu and fragments (not board specific), as there is no linaro-base or ubuntu topic in linux-linaro. Otherwise the generic features (not board specific) should also add the config fragments to their topic branches. So the android fragment could live in the android topic as well.
I'm not sure I follow this, can you give some examples of what files live in what repos in what branches?
This is all being made up as we go along, nobody is using this new flow yet.
The things we need to know is:
- What config fragemnts is the LT the origin of?
I think we have to provide a minimal, working, defconfig like we do now and that is the starting point for everything else piled on top.
- How these should be organised.
- Where I get the rest of the config fragments from which I need to
build and test the LT tree for Ubuntu and Android.
One suggestion...
Have a directory structure like
configs/linaro/base/ configs/linaro/android/ configs/linaro/ubuntu/
I don't think this is a good way. There are two things we found having already being doing "config fragments" for about a year in TI LT repo.
having multiple defconfigs is a mistake, they will diverge
the fragments themselves rot quickly from changes in mainline, both
by way of defaults changing and diffing the defconfig not being a perfect fit for what it represents (the defconfig is an output of another process out of sight with its own inputs, so the patches in the tree changing it are not the only things touching it). In particular it's almost impossible to hold the line with multiple finegrained config changes in one topic, we now squash everything into one config patch per topic.
By way of example, previously we ran a different defconfig for android and vanilla, now we patch the one defconfig when we add Androidization patches. That means the Android build always gets the latest and greatest stuff same as vanilla, plus whatever it needs specially.
Don't forget we won't be basing on linux-linaro-tracking, but vanilla. So that means we'll be producing linux-linaro-tracking "flavour branches" which can apply the "linaro house rules" for defconfigs such as being more like all modules.
Just as a proposition for the management of all these config files that you may already have discuss...:
Why not keeping one defconfig by board with all the latest dev in it and managing a script that fulfil the needs of dev and build server by adding the default config of linaro, android and ubuntu? Why not having a centralized place (git tree config) that keep the default linaro, android and ubuntu config files?
// Script that use merge_config to validate that defconfig contains config set and add them if not: linaroConfigAdder.sh -android defconfig linaroConfigAdder.sh -ubuntu defconfig linaroConfigAdder.sh -linaro defconfig
Question, why are you not adding the linaro config in the defconfig by default?
thx
KA
Further reading just respond to my answer sorry for that noise.
But for an outsider understanding Linaro workflow is really hard to understand and also the fact that all your workflow are different and requires different needs add to this complexity.
Wiki should be nice to explain your final work flow at the end with which specific requirements it fulfil?!
thx
KA
-Andy
-- Andy Green | TI Landing Team Leader Linaro.org │ Open source software for ARM SoCs | Follow Linaro http://facebook.com/pages/Linaro/155974581091106 - http://twitter.com/#%21/linaroorg - http://linaro.org/linaro-blog
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
On 03/31/2012 02:17 AM, Jon Medhurst (Tixy) wrote:
On Fri, 2012-03-30 at 10:15 -0700, John Stultz wrote:
In that case, just go ahead and push the full config to the config tree. If we need to do have fullly-enabled vs upstream builds we can deal with the warnings in the latter case (or maybe further split the board configs into -upstream and -lt ?).
So this means Landing Teams should host the configs for their boards and you will host the linaro-base, ubuntu and android fragments?
I guess it depends on whats easiest. I'm fine hosting all the configs, if landing teams want to provide them to me. But if landing teams want to host them, that's fine too.
We almost certainly need board specific android and ubuntu fragments as well, so I'll add vexpress-android.conf and vexpress-ubuntu.conf as well. (Unless there is some magic to have conditional config options in a fragment?)
No, the config fragments are pretty simple, so there's no conditional logic in them. Your idea for a board-type.conf style split sounds like a fair idea. Although, I'm curious what would be an example of this?
thanks -john
On Mon, 2012-04-02 at 08:37 -0700, John Stultz wrote:
On 03/31/2012 02:17 AM, Jon Medhurst (Tixy) wrote:
We almost certainly need board specific android and ubuntu fragments as well, so I'll add vexpress-android.conf and vexpress-ubuntu.conf as well. (Unless there is some magic to have conditional config options in a fragment?)
No, the config fragments are pretty simple, so there's no conditional logic in them. Your idea for a board-type.conf style split sounds like a fair idea. Although, I'm curious what would be an example of this?
There's often let's-just-get-it-working hacks produced, and sometimes you don't want to risk breaking other boards by putting things into a common config.
My example of this is http://android.git.linaro.org/gitweb?p=kernel/vexpress-a9.git%3Ba=commit%3Bh...
Of course, this could be an argument for not enabling such things. ;-)
On 04/02/2012 10:29 AM, Jon Medhurst (Tixy) wrote:
On Mon, 2012-04-02 at 08:37 -0700, John Stultz wrote:
On 03/31/2012 02:17 AM, Jon Medhurst (Tixy) wrote:
We almost certainly need board specific android and ubuntu fragments as well, so I'll add vexpress-android.conf and vexpress-ubuntu.conf as well. (Unless there is some magic to have conditional config options in a fragment?)
No, the config fragments are pretty simple, so there's no conditional logic in them. Your idea for a board-type.conf style split sounds like a fair idea. Although, I'm curious what would be an example of this?
There's often let's-just-get-it-working hacks produced, and sometimes you don't want to risk breaking other boards by putting things into a common config.
My example of this is http://android.git.linaro.org/gitweb?p=kernel/vexpress-a9.git%3Ba=commit%3Bh...
Of course, this could be an argument for not enabling such things. ;-)
Ok. Interesting. I can see how this sort of flexibility is useful. So I'm fine if folks want to add board-distro specific tweaks. Trying to find some more unified way of building might be necessary, so folks aren't having to customize things too drastically. Your directory method seems like would solve this, but I need to spend some more time understanding Andy's suggestion and understanding how it works with Andrey's centralized tree.
I recognize everyone has a different workflow here, and I do want to allow for flexibility. So it might be reasonable to start w/ the config dir that Tixy is suggesting, and only convert the build scripts for those boards using it. Leaving the others to use their own methods.
Ricardo, Zach: Do either of you have any thoughts feedback here?
thanks -john
On Mon, 2012-04-02 at 11:18 -0700, John Stultz wrote:
On 04/02/2012 10:29 AM, Jon Medhurst (Tixy) wrote:
On Mon, 2012-04-02 at 08:37 -0700, John Stultz wrote:
On 03/31/2012 02:17 AM, Jon Medhurst (Tixy) wrote:
We almost certainly need board specific android and ubuntu fragments as well, so I'll add vexpress-android.conf and vexpress-ubuntu.conf as well. (Unless there is some magic to have conditional config options in a fragment?)
No, the config fragments are pretty simple, so there's no conditional logic in them. Your idea for a board-type.conf style split sounds like a fair idea. Although, I'm curious what would be an example of this?
There's often let's-just-get-it-working hacks produced, and sometimes you don't want to risk breaking other boards by putting things into a common config.
My example of this is http://android.git.linaro.org/gitweb?p=kernel/vexpress-a9.git%3Ba=commit%3Bh...
Of course, this could be an argument for not enabling such things. ;-)
Ok. Interesting. I can see how this sort of flexibility is useful. So I'm fine if folks want to add board-distro specific tweaks. Trying to find some more unified way of building might be necessary, so folks aren't having to customize things too drastically. Your directory method seems like would solve this, but I need to spend some more time understanding Andy's suggestion and understanding how it works with Andrey's centralized tree.
Possibly if we just had a symlink
configs/omap_5430evm/base/main.conf --> arch/arm/configs/omap_5430evm_defconfig
but then that defconfig has some non board specific stuff, like EXT file-systems configured. And Andrey's Android branch would have TI's Android topics, so that 'base' defconfig file would also gain TI's Android flavourings. (Neither of these issues might be a problem in practice.)
On 04/02/2012 11:58 AM, Jon Medhurst (Tixy) wrote:
On Mon, 2012-04-02 at 11:18 -0700, John Stultz wrote:
On 04/02/2012 10:29 AM, Jon Medhurst (Tixy) wrote:
On Mon, 2012-04-02 at 08:37 -0700, John Stultz wrote:
On 03/31/2012 02:17 AM, Jon Medhurst (Tixy) wrote:
We almost certainly need board specific android and ubuntu fragments as well, so I'll add vexpress-android.conf and vexpress-ubuntu.conf as well. (Unless there is some magic to have conditional config options in a fragment?)
No, the config fragments are pretty simple, so there's no conditional logic in them. Your idea for a board-type.conf style split sounds like a fair idea. Although, I'm curious what would be an example of this?
There's often let's-just-get-it-working hacks produced, and sometimes you don't want to risk breaking other boards by putting things into a common config.
My example of this is http://android.git.linaro.org/gitweb?p=kernel/vexpress-a9.git%3Ba=commit%3Bh...
Of course, this could be an argument for not enabling such things. ;-)
Ok. Interesting. I can see how this sort of flexibility is useful. So I'm fine if folks want to add board-distro specific tweaks. Trying to find some more unified way of building might be necessary, so folks aren't having to customize things too drastically. Your directory method seems like would solve this, but I need to spend some more time understanding Andy's suggestion and understanding how it works with Andrey's centralized tree.
Possibly if we just had a symlink
configs/omap_5430evm/base/main.conf --> arch/arm/configs/omap_5430evm_defconfig
That could be possible. Although the file location isn't critical for the merge_config.sh tool, so the sym link is only necessary if we are trying to come with with a more generic config generation script in the builder.
In other words, if each build script has to have a custom merge_config line (much as it currently has a custom make X_defconfig line), we can just add any extra tweaks or hacks there.
but then that defconfig has some non board specific stuff, like EXT file-systems configured. And Andrey's Android branch would have TI's Android topics, so that 'base' defconfig file would also gain TI's Android flavourings. (Neither of these issues might be a problem in practice.)
Indeed. The non-board specific items in the defconfigs would likely be an issue, but I guess we could push the LT's to cleanup where those issues were noticed.
thanks -john
On 04/03/2012 02:58 AM, Somebody in the thread at some point said:
On Mon, 2012-04-02 at 11:18 -0700, John Stultz wrote:
On 04/02/2012 10:29 AM, Jon Medhurst (Tixy) wrote:
On Mon, 2012-04-02 at 08:37 -0700, John Stultz wrote:
On 03/31/2012 02:17 AM, Jon Medhurst (Tixy) wrote:
We almost certainly need board specific android and ubuntu fragments as well, so I'll add vexpress-android.conf and vexpress-ubuntu.conf as well. (Unless there is some magic to have conditional config options in a fragment?)
No, the config fragments are pretty simple, so there's no conditional logic in them. Your idea for a board-type.conf style split sounds like a fair idea. Although, I'm curious what would be an example of this?
There's often let's-just-get-it-working hacks produced, and sometimes you don't want to risk breaking other boards by putting things into a common config.
My example of this is http://android.git.linaro.org/gitweb?p=kernel/vexpress-a9.git%3Ba=commit%3Bh...
Of course, this could be an argument for not enabling such things. ;-)
Ok. Interesting. I can see how this sort of flexibility is useful. So I'm fine if folks want to add board-distro specific tweaks. Trying to find some more unified way of building might be necessary, so folks aren't having to customize things too drastically. Your directory method seems like would solve this, but I need to spend some more time understanding Andy's suggestion and understanding how it works with Andrey's centralized tree.
Possibly if we just had a symlink
configs/omap_5430evm/base/main.conf --> arch/arm/configs/omap_5430evm_defconfigbut then that defconfig has some non board specific stuff, like EXT file-systems configured. And Andrey's Android branch would have TI's Android topics, so that 'base' defconfig file would also gain TI's Android flavourings. (Neither of these issues might be a problem in practice.)
It's really preferable to keep to one defconfig for everything but sometimes that won't be possible (you were saying you might need more than one for presumably very basic reasons). In that case the config deltas will need to be applied to n defconfigs as part of the rebase / merge process creating the output tree.
So it might be simpler instead of using multiple symlinks if the LT base tree creates and maintains a text file that is a list of "defconfigs this tree maintains" which will all get processed by the topic management scripts in turn for each config delta when the flavourings are added.
-Andy
On Tue, 2012-04-03 at 09:13 +0800, Andy Green wrote:
On 04/03/2012 02:58 AM, Somebody in the thread at some point said:
On Mon, 2012-04-02 at 11:18 -0700, John Stultz wrote:
On 04/02/2012 10:29 AM, Jon Medhurst (Tixy) wrote:
On Mon, 2012-04-02 at 08:37 -0700, John Stultz wrote:
On 03/31/2012 02:17 AM, Jon Medhurst (Tixy) wrote:
We almost certainly need board specific android and ubuntu fragments as well, so I'll add vexpress-android.conf and vexpress-ubuntu.conf as well. (Unless there is some magic to have conditional config options in a fragment?)
No, the config fragments are pretty simple, so there's no conditional logic in them. Your idea for a board-type.conf style split sounds like a fair idea. Although, I'm curious what would be an example of this?
There's often let's-just-get-it-working hacks produced, and sometimes you don't want to risk breaking other boards by putting things into a common config.
My example of this is http://android.git.linaro.org/gitweb?p=kernel/vexpress-a9.git%3Ba=commit%3Bh...
Of course, this could be an argument for not enabling such things. ;-)
Ok. Interesting. I can see how this sort of flexibility is useful. So I'm fine if folks want to add board-distro specific tweaks. Trying to find some more unified way of building might be necessary, so folks aren't having to customize things too drastically. Your directory method seems like would solve this, but I need to spend some more time understanding Andy's suggestion and understanding how it works with Andrey's centralized tree.
Possibly if we just had a symlink
configs/omap_5430evm/base/main.conf --> arch/arm/configs/omap_5430evm_defconfigbut then that defconfig has some non board specific stuff, like EXT file-systems configured. And Andrey's Android branch would have TI's Android topics, so that 'base' defconfig file would also gain TI's Android flavourings. (Neither of these issues might be a problem in practice.)
It's really preferable to keep to one defconfig for everything but sometimes that won't be possible (you were saying you might need more than one for presumably very basic reasons).
It depends what more than one defconfig means. I was suggesting that we have multiple fragments so topics could have there own fragment if necessary, for when people don't have stacked topics to modify a single defconfig. I also suggested it might be wise for a get-out-of-jail method so a board could override distro options. (Perhaps the later shouldn't be made part of the normal workflow and instead be a manual hack applied by distro maintainer in exceptional circumstances.)
In that case the config deltas will need to be applied to n defconfigs as part of the rebase / merge process creating the output tree.
So it might be simpler instead of using multiple symlinks if the LT base tree creates and maintains a text file that is a list of "defconfigs this tree maintains" which will all get processed by the topic management scripts in turn for each config delta when the flavourings are added.
I though the distro building would be using merge_config.sh to generate the .config for the build, therefore this would all just be as simple as having it use the argument "my-board-config-directory/*" instead of "my-board-config-file".
For most cases I would guess even "cat my-board-config-directory/*
board.conf" would work. So perhaps we could allow each board to specify
a shell script to generate a config rather than coming up with a mechanism we all agree on? We could also pass the distro as an argument so the script could apply hacks as appropriate.
-
Andrey Konovalov -
Andy Green -
John Stultz -
Jon Medhurst (Tixy) -
Kevyn-Alexandre Paré -
Tushar Behera