On Mon, May 20, 2013 at 12:20 PM, Fathi Boudra fathi.boudra@linaro.orgwrote:

Hi,

On 20 May 2013 13:00, Nicolas Dechesne nicolas.dechesne@linaro.org wrote:

...

hi there,

i am aware and familiar with hwpacks that were initially designed and

used

...

to generate ubuntu/android Linaro images.

however i don't know if hwpacks are being used for OE images nowadays?

Yes, they are. To create an OE pre-built image, we use l-m-c in combination of an OE rootfs+hwpack.

ok.

...

OE has intrinsic mechanism to generate/construct full images which

include

...

machine dependent binaries as well as generic components. So are we still using hwpacks for building OE images somehow or not? Can someone point

me to

...

relevant documentation?

You can look at the following CI jobs:

https://ci.linaro.org/jenkins/job/openembedded-armv7a-pre-built-images-panda

https://ci.linaro.org/jenkins/job/openembedded-armv8-pre-built-images-vexpre...

the following folder seems to be empty:

http://snapshots.linaro.org/openembedded/images/lamp-armv7-gcc-4.7

so, from that job, i can see that Panda OE prebuilt images are made by re-using the 'ubuntu' hwpacks which I believe is generated from this job:

https://ci.linaro.org/jenkins/view/engineering-builds/job/ubuntu-armhf-hwpac...

And that 'versatile' OE images are built using OE hwpacks (it's about the -pw argument).

So couple of more questions: - why are there raring hwpacks for PAnda and OE hwpacks for versatile? is that what hwpack v4 is about? do we intend to get a single hwpack for all distro that we care about? - i don't find the job that creates the versatile hwpacks in Jenkins, can you give me the link? i'd like to understand how it's built.

Well, i find all these things a little bit convoluted, i hope this is because i am new to it ;-) and having the scripts spread across multiple repo (git, lp, and part of the scripts in Jenkins job definition) doesn't really help.

For historical reason, linaro-image-tools is making a few assumptions (ubuntu and deb packages). Since our engineering builds have evolved (android, fedora, openembedded, ubuntu), we're slowly fixing those assumptions and going to a distro agnostic approach.

does that mean moving all scripts/tools away from LP? I can see a few things from LP bazaar, i think having all our tools on git.linaro.org would be nice. no rant here.. just consistency ;-)

There's a session planned to talk about this topic: hwpackv4.

i will attend for sure. is there any wiki/doc already available on what might change?

i remember the very first discussions about hwpacks couple of years ago, and i think i understand the main motivations for them, however with OE i tend to believe that hwpacks are much less needed and more a source of pain than something useful. The thing is that most of the problems that hwpacks want to solve are solved already in OE, unlike with Ubuntu and Android which were the 2 distro used when hwpacks were designed.

so from the perspective of a Linaro user, or a customer who will care only about OE, and there will be such cases, we need to make sure that we don't create a tool that is going to be counter-productive vs a 'vanilla' OE solution.

...

in the short future i will have to build OE images for a couple of

different

...

h/w boards, so i am trying to understand the process to do that a-la

Linaro.

...

thanks

nicolas

Cheers, Fathi

On Mon, May 20, 2013 at 5:05 PM, Fathi Boudra fathi.boudra@linaro.orgwrote:

On 20 May 2013 17:22, Nicolas Dechesne nicolas.dechesne@linaro.org wrote:

...

On Mon, May 20, 2013 at 12:20 PM, Fathi Boudra fathi.boudra@linaro.org wrote:

...

Hi,

On 20 May 2013 13:00, Nicolas Dechesne nicolas.dechesne@linaro.org wrote:

...

hi there,

i am aware and familiar with hwpacks that were initially designed and used to generate ubuntu/android Linaro images.

however i don't know if hwpacks are being used for OE images nowadays?

Yes, they are. To create an OE pre-built image, we use l-m-c in combination of an OE rootfs+hwpack.

ok.

...

...

OE has intrinsic mechanism to generate/construct full images which include machine dependent binaries as well as generic components. So are we still using hwpacks for building OE images somehow or not? Can someone point me to relevant documentation?

You can look at the following CI jobs:

https://ci.linaro.org/jenkins/job/openembedded-armv7a-pre-built-images-panda

...

...

https://ci.linaro.org/jenkins/job/openembedded-armv8-pre-built-images-vexpre...

...

...

the following folder seems to be empty:

http://snapshots.linaro.org/openembedded/images/lamp-armv7-gcc-4.7

You're looking at the wrong directory: http://snapshots.linaro.org/openembedded/pre-built/panda/latest

ok, i was confused by the output directory name for vexpress.

so currently, iirc, the process of 'building' a hwpack requires to 1) push the appropriate source packages in a dedicated PPA, 2) let l-h-c pull the binary and create the hwpack.

well, we seem to be doing something different for vexpress64 which i don't really understand, but i guess is something that will be streamlined with hwpacks v4...

...

so, from that job, i can see that Panda OE prebuilt images are made by re-using the 'ubuntu' hwpacks which I believe is generated from this job:

https://ci.linaro.org/jenkins/view/engineering-builds/job/ubuntu-armhf-hwpac...

That's right.

...

And that 'versatile' OE images are built using OE hwpacks (it's about the -pw argument).

I don't follow here. What's an OE hwpack? there isn't such thing. The -pw argument is used to specify the platform when you create the pre-built image. The code path is slightly different depending on the platform (ubuntu vs oe).

what I meant is that vexpress64 hwpacks are stored here:

http://snapshots.linaro.org/openembedded/hwpacks/vexpress64

while other hwpackas are stored here:

http://snapshots.linaro.org/raring/hwpacks

that's what triggered my question (and my confusion)

...

So couple of more questions:

• why are there raring hwpacks for PAnda and OE hwpacks for versatile?

is

...

that what hwpack v4 is about? do we intend to get a single hwpack for all distro that we care about?

See http://cards.linaro.org/browse/CARD-481

We want to consolidate our approaches into a single approach that scales across distributions.

...

• i don't find the job that creates the versatile hwpacks in Jenkins,

can

...

you give me the link? i'd like to understand how it's built.

Same as other hwpacks, see https://ci.linaro.org/jenkins/job/ubuntu-armhf-hwpacks/

...

Well, i find all these things a little bit convoluted, i hope this is because i am new to it ;-) and having the scripts spread across multiple repo (git, lp, and part of the scripts in Jenkins job definition) doesn't really help.

...

For historical reason, linaro-image-tools is making a few assumptions (ubuntu and deb packages). Since our engineering builds have evolved (android, fedora, openembedded, ubuntu), we're slowly fixing those assumptions and going to a distro agnostic approach.

does that mean moving all scripts/tools away from LP? I can see a few

things

...

from LP bazaar, i think having all our tools on git.linaro.org would be nice. no rant here.. just consistency ;-)

Same here, historically reasons. I moved some bits to git but there's still some things on LP. Rome Wasn't Built in a Day :)

sure, that i understand ;-) but good to know that it's on-going!

...

...

There's a session planned to talk about this topic: hwpackv4.

i will attend for sure. is there any wiki/doc already available on what might change?

Not yet.

...

i remember the very first discussions about hwpacks couple of years ago,

and

...

i think i understand the main motivations for them, however with OE i

tend

...

to believe that hwpacks are much less needed and more a source of pain

than

...

something useful. The thing is that most of the problems that hwpacks

want

...

to solve are solved already in OE, unlike with Ubuntu and Android which

were

...

the 2 distro used when hwpacks were designed.

hwpacks were never designed for Android. The Android hwpack is pretty new. It was designed for Ubuntu only and that was a fair assumption at that time.

...

so from the perspective of a Linaro user, or a customer who will care

only

...

about OE, and there will be such cases, we need to make sure that we

don't

...

create a tool that is going to be counter-productive vs a 'vanilla' OE solution.

That's something to discuss... We don't have "customers". The engineering builds are done to support our engineering effort, in the fastest and cost effective way The hwpack concept is still valid but not as it used to be (hardware specific abstraction). We still want to have a single hwpack that works for the supported distributions.

There's pros/cons for both solutions. I see it as: common approach (hwpack+rootfs) vs native/distro specific approach. Talk to you at the connect session :)

...

...

...

in the short future i will have to build OE images for a couple of different h/w boards, so i am trying to understand the process to do that a-la Linaro.

thanks

nicolas

Cheers, Fathi

Cheers,

Fathi Boudra Builds and Baselines Manager | Release Manager Linaro.org | Open source software for ARM SoCs

On 21 May 2013 10:50, Riku Voipio riku.voipio@linaro.org wrote:

On 20 May 2013 18:05, Fathi Boudra fathi.boudra@linaro.org wrote:

...

See http://cards.linaro.org/browse/CARD-481

...

We want to consolidate our approaches into a single approach that scales across distributions.

...

...

so from the perspective of a Linaro user, or a customer who will care only about OE, and there will be such cases, we need to make sure that we don't create a tool that is going to be counter-productive vs a 'vanilla' OE solution.

That's something to discuss... We don't have "customers". The engineering builds are done to support our engineering effort, in the fastest and cost effective way The hwpack concept is still valid but not as it used to be (hardware specific abstraction). We still want to have a single hwpack that works for the supported distributions.

"We still want" - Well I'd like to see the list of "we" ;)

As you know, I'm in the list and I'm also the one doing the work so I guess my vote counts ;)

hwpacks were a good idea for ubuntu, but even they no longer use the concept on ubuntu touch. From the OE side, burying hwpacks and using OE images directly would make our engineering effort greatly faster, simpler and more cost effective.

That's your claim. Re-using an already built hwpack, re-usable across all distros, seems faster and more cost effective to me.

I don't think hwpacks are requested by fedora engineers either.

or ubuntu engineers, or $distro engineer. the hwpacks are linaro specific like many things...

...

There's pros/cons for both solutions. I see it as: common approach (hwpack+rootfs) vs native/distro specific approach. Talk to you at the connect session :)

The "common approach" is more like "linaro specific approach".

That's right and I'm fine with it.

If we do linaro-specific tools, we might save some of our own time - but the distributions and endusers will have to duplicate our work in their own setups.

I'm fine with it as well. Distributions have to do their work and do their own integration. They won't use our solution anyway. If we take their own approaches, they'll redo the work anyway (as proven in the past).

If we work with native/distro specific tools and contribute back, our changes ripple back to the actual end users. The ARM ecostystem as whole benefits.

I tend to disagree. I'm looking at the past 2 years, we've worked with native/distro specific tools and I don't see our changes reaching the end users. There's only one way to reach end users and distributions, it's upstreaming.

Yes discuss at connect makes sense, just food of thought... Riku

On 23 May 2013 11:43, Nicolas Dechesne nicolas.dechesne@linaro.org wrote:

On Tue, May 21, 2013 at 10:41 AM, Fathi Boudra fathi.boudra@linaro.org wrote:

...

...

...

That's something to discuss... We don't have "customers". The engineering builds are done to support our engineering effort, in the fastest and cost effective way The hwpack concept is still valid but not as it used to be (hardware specific abstraction). We still want to have a single hwpack that works for the supported distributions.

"We still want" - Well I'd like to see the list of "we" ;)

As you know, I'm in the list and I'm also the one doing the work so I guess my vote counts ;)

well, as far as I am concerned and for my project i don't think that using hwpack will make sense, and would prefer to go with an OE native approach, even though we will support multiple BSP/SoC, that will be managed with dedicated BSP layers. I will participate in the hwpacks v4 discussions at connect though to align with you guys.

Your call (not only ;) I suspect your project has some stakeholders requirements around OE native approach). I'm guessing you'll have to make some adjustments if you want to submit your pre-built images jobs to LAVA.

On 23 May 2013 12:53, Nicolas Dechesne nicolas.dechesne@linaro.org wrote:

On Thu, May 23, 2013 at 11:02 AM, Fathi Boudra fathi.boudra@linaro.org wrote:

...

Your call (not only ;) I suspect your project has some stakeholders requirements around OE native approach). I'm guessing you'll have to make some adjustments if you want to submit your pre-built images jobs to LAVA.

yes, guess so. and that's what's worries me a little bit too. and that's why we need to talk about that at Connect!

It's not a big deal since LAVA started supporting prebuilt images. For ARMv8 the only problem I had is that LAVA digs some files from the rootfs partition, and due to hwpack/linaro-media-create LAVA expects the rootfs to be the second partition, while the OE image doesn't have a partition table.

Riku

On Thu, May 23, 2013 at 12:16 PM, Fathi Boudra fathi.boudra@linaro.orgwrote:

...

It's not a big deal since LAVA started supporting prebuilt images. For ARMv8 the only problem I had is that LAVA digs some files from the rootfs partition, and due to hwpack/linaro-media-create LAVA expects the rootfs to be the second partition, while the OE image doesn't have a partition table.

there's always "the only problem", and then you have another one and so on. "greatly faster, simpler and more cost effective" that's what you said? afaics, you're wasting your time (and probably some LAVA guys time) on something already working... what's the benefit?

well, in my case, we might not have the choice, as you guessed it right, there are other stakeholders, and i might not be able to justify an hwpack solution as opposed to a native OE solution. i will get a better (complete) picture by Connect, and we can discuss more then.

Chiming in.

We've been working to put together a linaro image with a graphics stack that is based on xfce for OE.

I really like the idea that OE should output an axf and img that both the Foundation and RTSM models can utilize straight away without a hwpack or linaro-media-create. Extra steps are a drag.

Out of curiosity where is the code that outputs the root file system for a tarball or ext3 img?

On Thu, May 23, 2013 at 5:34 AM, Fathi Boudra fathi.boudra@linaro.org wrote:

On 23 May 2013 13:20, Nicolas Dechesne nicolas.dechesne@linaro.org wrote:

...

On Thu, May 23, 2013 at 12:16 PM, Fathi Boudra fathi.boudra@linaro.org wrote:

...

...

It's not a big deal since LAVA started supporting prebuilt images. For ARMv8 the only problem I had is that LAVA digs some files from the rootfs partition, and due to hwpack/linaro-media-create LAVA expects the rootfs to be the second partition, while the OE image doesn't have a partition table.

there's always "the only problem", and then you have another one and so on. "greatly faster, simpler and more cost effective" that's what you said? afaics, you're wasting your time (and probably some LAVA guys time) on something already working... what's the benefit?

well, in my case, we might not have the choice, as you guessed it right, there are other stakeholders, and i might not be able to justify an hwpack solution as opposed to a native OE solution. i will get a better (complete) picture by Connect, and we can discuss more then.

Nico, I know your use case and I'm definitely not pushing on the hwpack story specifically (even more for your project).

In other words, I don't think linaro specific approach is perfect and I don't want to maintain the status quo (that's the goal of hwpackv4). I want to move forward to improve the situation and be more efficient but I'm not convinced the native approach is perfect either. As I said previously, there's pros/cons for both solutions.

I'm glad there's interest on the topic and see you at Connect to have a constructive discussion :)

---

linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev

On Thu, May 23, 2013 at 6:26 PM, Christopher Covington cov@codeaurora.orgwrote:

...

Out of curiosity where is the code that outputs the root file system for a tarball or ext3 img?

The decision to generate these images is made in conf/site.conf which is generated by openembedded/meta-aarch64/scripts/init.sh. (In my opinion, it'd be nice if site.conf could be put under revision control itself.) Anyhow, openembedded-core/meta/classes/image_types.bbclass has the filesystem generation commands. Ext2 support, for example:

http://git.openembedded.org/openembedded-core/tree/meta/classes/image_types....

note that if you *just* need to alter/tweak the generated ext3 (or any other image type) that is already generated by OE, it might be simple to 'post process' the image using something along these lines:

http://git.yoctoproject.org/cgit/cgit.cgi/meta-mentor/tree/classes/deploy-li...

basically, the IMAGE_POSTPROCESS_COMMAND is a list of python scripts that is executed immediately after the 'image' is generated, so at that point you can prepare it for the Foundations model.