On Wed, Jan 19, 2011, Amit Kucheria wrote:

Am I correct in my understanding then, that this will address some of the issues I raised in https://wiki.linaro.org/Platform/Infrastructure/Specs/TestDrive ? Basically l-m-c won't have to be touched everytime we add new board support?

Yes, I think one can say this. Unless the new board needs something completely different than the older ones, e.g. a specific partition layout or a new media type etc.

At the risk of sounding like a broken record, I still believe that another tool on top of l-m-c that takes care of the selection and download of correct image & hwpack and the combines & writes the image to SD is what we should strive for to increase the attractiveness of linaro images.

I agree, and I would also hope we gain a GUI someday, but I don't think it relates to changing the hwpack format?

On Wed, 2011-01-19 at 13:22 +0200, Amit Kucheria wrote:

On Wed, Jan 19, 2011 at 3:02 AM, Loïc Minier loic.minier@linaro.org wrote:

...

   Hey there

As a result of a series of bugs around linaro-image-tools and daily images, it seemed a sensible approach to solve this class of issues would be to move more data into hardware packs rather than hardcoding stuff in linaro-image-tools.

Guilherme, Steve Langasek, and Michael Hudson were all interested in the idea, so Michael convinced me to start a wiki page with some notes: https://wiki.linaro.org/Platform/Specs/11.05/HardwarePacksV2

I'm sending this for others to get the chance to raise issues with hwpacks since we don't want to change the format too frequently.

Am I correct in my understanding then, that this will address some of the issues I raised in https://wiki.linaro.org/Platform/Infrastructure/Specs/TestDrive ? Basically l-m-c won't have to be touched everytime we add new board support?

Not specifically, no. But the current rewrite of l-m-c just completed by the infrastructure team does.

At the risk of sounding like a broken record, I still believe that another tool on top of l-m-c that takes care of the selection and download of correct image & hwpack and the combines & writes the image to SD is what we should strive for to increase the attractiveness of linaro images.

This is still a good idea to simplify things for novice users IMHO. However this should be a new tool that used l-m-c to accomplish its tasks.

Scott

On Wed, 2011-01-19 at 12:39 +0100, Alexander Sack wrote:

On Wed, Jan 19, 2011 at 2:02 AM, Loïc Minier loic.minier@linaro.org wrote:

...

   Hey there

As a result of a series of bugs around linaro-image-tools and daily images, it seemed a sensible approach to solve this class of issues would be to move more data into hardware packs rather than hardcoding stuff in linaro-image-tools.

yes, the idea was always to have all hw dependent stuff shipped inside the hwpacks. So moving things that are board specific from l-m-c scripts to hwpacks meta data/configs is in line with that. So +1 from me on moving forward on these.

Please try to preserve the ability to use old hwpacks though.

Definitely; this is why the hwpacks are versioned. It will require some extra code in l-m-c to handle them differently, but I wasn't considering breaking l-m-c for old hwpacks.

On Wed, Jan 19, 2011, Guilherme Salgado wrote:

This looks good to me. The only thing I can think of right now is to also add the board name to the hwpack meta-data and consider dropping the --dev option (making it optional, in fact, so that it keeps working with hwpacks in the current format) which should no longer be needed once all the board-specific options are in the hwpack.

What would we do with the board name?

On Wed, Jan 19, 2011, Guilherme Salgado wrote:

Right now I don't have a use-case for it, but at the same time that I want to make the --dev argument not needed (after all, the user already specifies the hwpack for a specific board, so there's no point in forcing them to specify that yet again), I don't want to lose that information, and it kind of makes sense to me to have hwpacks specify the boards they're made for in a structured format rather than just on their file names.

Ok, I'm fine keeping the information available, but I wanted to double-check that we would NOT be using it, because if we had to, that would mean we miss some other information in the hwpack metadata! :-)

On Wed, Jan 19, 2011, James Westby wrote:

Would in general be nice to deal with other image types like Android and ChromeOS and avoid .debs unless targetting Ubuntu images.

I think this is the wrong aim to be putting in the document. I think that the aim should be to be able to produce hardware packs that can be shared between Ubuntu, Android and ChromeOS images, and leave the implementation of that as a separate question.

Hmm maybe the wording was poor, but it's definitely the intent that the hwpacks be kept as portable across image types as possible.

cmdline

Is this going to be a static string, or will it need to substitute variables in to it as it is used?

Good idea; I don't know; maybe we want linaro-image-tools to substitute rootfs partition number or so.

omap_mlo

I think it's generally better to not put strings like "omap" in the name, and rather have a list of files that will have the same treatment (in this case copy to the boot disk)

I think it can be either way; I don't care too strongly. We could say "extra_boot_files", but then we might not know it's a x-loader anymore. Imagine that we'd use the hwpack as a source to boot a board over a serial line, we might have to send the MLO first, and then u-boot. If we move to extra_boot_files we can't do that anymore. It's a judgment call really.

(I prefixed the name with omap_ as to not clutter the namespace with OMAP specific settings)

igep_uboot_ini

Similar for this (list of files to create)

I don't like the name of that field, but it's basically a flag whether or not we need a boot.ini; I actually think we could do without this entirely ("do we need this" reminder I had put for myself there). This is basically a requirement as long as we miss a x-loader. Once we have one, we don't care about the default bootloader configuration in flash which requires a boot.ini (IIRC).

fdt

What would we do with this if we found it in a hwpack?

I don't know; I need more handson experience with DT to tell. It might be that we don't need this this cycle because the DT will be embedded in the zImage. Otherwise, we'd have to mkimage it along uImage and uInitrd, probably in a uFdt or something like that, and then change the boot script to pass it to the kernel.

linux_image

I don't think there's any point in ignoring this now, and then doing something with it later. It should have a hwpack format bump so that users can be told that they need a newer l-m-c, otherwise when we first release Android hwpacks people will generate unbootable images.

So "old" l-i-t will bail out against new hwpacks again? If we could allow them to continue working, that would be nicer IMO

We should only aim for forward compatibility where we know what we will do with the option when it exists. I have no problem not using all options, but unless we know what to do with an option we should not put it in the format, as we will need a format bump when we do know what to do with it anyway.

Granted we don't have a full plan for Cros and Android, but I was hoping we could have a tentative one including this linux_image field. In the worst case, we indeed move to an incompatible hwpack format. But if it's good enough, it means natty's l-i-t will be able to use natty+1 hwpacks.

On Wednesday 19 January 2011, Nicolas Pitre wrote:

On Wed, 19 Jan 2011, Loïc Minier wrote:

...

On Wed, Jan 19, 2011, James Westby wrote:

...

fdt

What would we do with this if we found it in a hwpack?

I don't know; I need more handson experience with DT to tell. It might be that we don't need this this cycle because the DT will be embedded in the zImage. Otherwise, we'd have to mkimage it along uImage and uInitrd, probably in a uFdt or something like that, and then change the boot script to pass it to the kernel.

I'd prefer if we could keep the kernel and fdt images separate as much as possible. In theory, the fdt should be updated far less often than the kernel. If we really need to bundle the kernel and fdt together then there is generic support for that in the kernel build system already.

More importantly, you might want to update the fdt files on a different cycle than the kernel. If you have a new slightly different configuration in a new machine you want to support, it may be easier to add a new file somewhere than doing a respin of all the kernel packages when they contain the dtc source. I would not see much of a problem with always shipping all trees with the kernel package, even if they never change.

Arnd

On Wed, 19 Jan 2011 15:58:34 +0100, Loïc Minier loic.minier@linaro.org wrote:

Hmm maybe the wording was poor, but it's definitely the intent that the hwpacks be kept as portable across image types as possible.

Right, I agree with the goal. My comment is just the wording, talk about the aim, not about "avoiding .debs" or something.

I think it can be either way; I don't care too strongly. We could say "extra_boot_files", but then we might not know it's a x-loader anymore. Imagine that we'd use the hwpack as a source to boot a board over a serial line, we might have to send the MLO first, and then u-boot. If we move to extra_boot_files we can't do that anymore. It's a judgment call really.

That's a reasonable justification.

...

fdt

What would we do with this if we found it in a hwpack?

I don't know; I need more handson experience with DT to tell. It might be that we don't need this this cycle because the DT will be embedded in the zImage. Otherwise, we'd have to mkimage it along uImage and uInitrd, probably in a uFdt or something like that, and then change the boot script to pass it to the kernel.

Ok. If we can get a definite list of steps then we can include it in v2, if not then we leave it out.

...

linux_image

I don't think there's any point in ignoring this now, and then doing something with it later. It should have a hwpack format bump so that users can be told that they need a newer l-m-c, otherwise when we first release Android hwpacks people will generate unbootable images.

So "old" l-i-t will bail out against new hwpacks again? If we could allow them to continue working, that would be nicer IMO

Right, but what would they do? That's my point.

If you really want to push Andoid in to v2 then we can write code to identify/specify image type, then defer Android/linux_image combination with a specific error message.

The point of a format specifier is such that l-i-t won't try to act on things that are added after that version was released. If the old code won't do the wrong thing then we don't need a format bump.

Thanks,

James

On Wed, 19 Jan 2011 17:58:04 +0100, Loïc Minier loic.minier@linaro.org wrote:

That's exactly my point: have the next version of the code try to do the right thing. Maybe I actually have broken expectations: I expect l-i-t would reject hwpacks with unknown fields. That's the failure I'm trying to avoid if we know we're going to introduce a linux_image field and that it can be safely ignored.

It's a bit like when Debian introduced Breaks, by adding support to dpkg to simply treat the field in a dumb way, and then adding full support the next cycle, as to allow using old dpkg for upgrades.

Good example. We can certainly do this, we just need to have a plan like they did.

We can't just add every field that we think we may one day want and ignore them all, as that will often lead to a bad user experience when we use the field, or having to have a format bump later anyway before we can use it.

An illustration of what I mean: if we add linux_image and ignore it, and then use it within Android hwpacks, someone with the old code will try and use one of those new hwpacks, and get an unbootable Android image. When that happens someone will say "we should have the tool warn people when it can't do what they ask", which we could have done now with a format bump, or by having a more complete plan than "ignore the field".

The crux of my point is that every field we add has to have clear semantics. It's rather an obvious statement, but one we should stick to.

Thanks,

James

On 20 January 2011 05:15, Scott Bambrough scott.bambrough@linaro.orgwrote:

On Wed, 2011-01-19 at 15:02 -0600, James Westby wrote:

...

An illustration of what I mean: if we add linux_image and ignore it, and then use it within Android hwpacks, someone with the old code will try and use one of those new hwpacks, and get an unbootable Android image. When that happens someone will say "we should have the tool warn people when it can't do what they ask", which we could have done now with a format bump, or by having a more complete plan than "ignore the field".

I have to jump in here. Hardware packs for Android or ChromeOS seem ridiculous to me. If we want to be accepted by mainstream developers for these OS, then we need to conform to the accepted norm for that community. Imposing hardware packs in these two projects is just likely to get our efforts ignored.

Scott

I agree with Scott.

There is no such thing in Android. The Android build system creates a number of images. A boot image (depending of configuration), a system image and a user data image. A hardware pack would probably be a subset of the boot and system image. It would be hard to introduce a hardware pack for Android without doing major changes. We should not fork Android and become a new Android distribution. We should try to be as close to AOSP as possible.

/Patrik

-- Scott Bambrough Technical Director, Landing Teams

---

linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev

Sorry for entering late here. Here are my questions:

How does l-m-c know about the boot partition convention? Is the fact that omap wants a dos partition with some files on it but i.MX just needs the raw bits at a fixed location on the card embedded in l-m-c? If a new platform pops up with a completely different convention does l-m-c need to be modified or could we put a script in the hwpack to do that?

For map you could call a script with an argument pointing to the blown out hwpack and and second argument pointing at the mounted boot partition. For mx first argument is the same second is a pointer to raw device. So the hwpack would need a field to say if the scipt needs a raw device or mounted dos partition and.. another field with the script name.

Is this over engineering?

Sorry for the noise, John

On Wed, Jan 19, 2011 at 9:58 AM, Loïc Minier loic.minier@linaro.org wrote:

On Wed, Jan 19, 2011, James Westby wrote:

...

Right, but what would they do? That's my point.

If you really want to push Andoid in to v2 then we can write code to identify/specify image type, then defer Android/linux_image combination with a specific error message.

The point of a format specifier is such that l-i-t won't try to act on things that are added after that version was released. If the old code won't do the wrong thing then we don't need a format bump.

That's exactly my point: have the next version of the code try to do  the right thing.  Maybe I actually have broken expectations: I expect  l-i-t would reject hwpacks with unknown fields.  That's the failure  I'm trying to avoid if we know we're going to introduce a linux_image  field and that it can be safely ignored.

It's a bit like when Debian introduced Breaks, by adding support to  dpkg to simply treat the field in a dumb way, and then adding full  support the next cycle, as to allow using old dpkg for upgrades.

-- Loïc Minier

---

linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev

On Wed, Jan 19, 2011, John Rigby wrote:

How does l-m-c know about the boot partition convention?

I've just added to the wiki page; this came up yesterday on IRC when we started taking imx51 into account.

We would have a partition_layout field which commands which partition layout type we want to use for this hardware, either omap-style or mx5-style.

If a new platform pops up with a completely different convention does l-m-c need to be modified or could we put a script in the hwpack to do that?

I'm all against script; we basically lose control and surrender to what the hwpacks runs, this is fragile and dangerous. I'd rather keep hwpacks as data and linaro-media-create as logic.

For map you could call a script with an argument pointing to the blown out hwpack and and second argument pointing at the mounted boot partition. For mx first argument is the same second is a pointer to raw device. So the hwpack would need a field to say if the scipt needs a raw device or mounted dos partition and.. another field with the script name.

Well, if we ever want to support fancier partition layouts, we could go as far as having a description of the layout; e.g.: partition_layout_data: 0: vfat@4MiB, 1: rootfs or something like that, but I think we don't need such complexity in the near term. There are formats we could reuse like the one used in debian-installer for preseeding if we ever want to do this, but I agree this would be overengineered for now.