Recently my angry post on Google+ [1] got so many comments that it was clear that it would be better to move to some mailing list with discussion.
As it is about boot loaders and Linaro has engineers from most of SoC vendor companies I thought that this will be best one.
1. https://plus.google.com/u/0/+MarcinJuszkiewicz/posts/J79qhndV6FY
All started when I got Pine64 board (based on Allwinner A64 SoC) and had same issue as on several boards in past - boot loader written in some random place on SD card.
Days where people used Texas Instruments SoC chips were great - in-cpu boot loader knew how to read MBR partition table and was able to load 1st stage boot loader (called MLO) from it as long it was FAT filesystem.
GPU used by Raspberry/Pi is able to read MBR, finds 1st partition and reads firmware files from there as long it is FAT.
Chromebooks have some SPI flash to keep boot loaders and use GPT partitioning to find where from load kernel (or another boot loader).
And then we have all those boards where vendors decided that SPI flash for boot loader is too expensive so it will be read from SD card instead. From any random place of course...
Then we have distributions. And instead of generating bunch of images per board they want to make one clean image which will be able to handle as much as possible.
If there are UEFI machines on a list of supported ones then GPT partitioning will be used, boot loader will be stored in "EFI system area" and it boots. This is how AArch64 SBSA/SBBR machines work.
But there are also all those U-Boot (or fastboot/redboot/whateverboot) ones. They are usually handled by taking image from previous stage and adding boot loader(s) by separate script. And this is where "fun" starts...
GPT takes first 17KB of storage media as it allow to store information about 128 partitions. Sure, no one is using so many on such devices but still space is reserved.
But most of chips expects boot loader(s) to be stored:
- right after MBR - from 1KB - from 8KB - any other random place
So scripts start to be sets of magic written to handle all those SoCs...
Solution for existing SoCs is usually adding 1MB of SPI flash during design phase of device and store boot loader(s) there. But it is so expensive someone would say when it is in 10-30 cents range...
Even 96boards CE specification totally ignored that fact while it could be a way of showing how to make popular board. Instead it became yet-another-board-to-laugh (EE spec did not improve much).
Is there a way to get it improved? At least for new designs?
Hi,
On 5 May 2016 at 05:45, Marcin Juszkiewicz marcin.juszkiewicz@linaro.org wrote:
Recently my angry post on Google+ [1] got so many comments that it was clear that it would be better to move to some mailing list with discussion.
As it is about boot loaders and Linaro has engineers from most of SoC vendor companies I thought that this will be best one.
All started when I got Pine64 board (based on Allwinner A64 SoC) and had same issue as on several boards in past - boot loader written in some random place on SD card.
Days where people used Texas Instruments SoC chips were great - in-cpu boot loader knew how to read MBR partition table and was able to load 1st stage boot loader (called MLO) from it as long it was FAT filesystem.
GPU used by Raspberry/Pi is able to read MBR, finds 1st partition and reads firmware files from there as long it is FAT.
Chromebooks have some SPI flash to keep boot loaders and use GPT partitioning to find where from load kernel (or another boot loader).
And then we have all those boards where vendors decided that SPI flash for boot loader is too expensive so it will be read from SD card instead. From any random place of course...
Then we have distributions. And instead of generating bunch of images per board they want to make one clean image which will be able to handle as much as possible.
If there are UEFI machines on a list of supported ones then GPT partitioning will be used, boot loader will be stored in "EFI system area" and it boots. This is how AArch64 SBSA/SBBR machines work.
But there are also all those U-Boot (or fastboot/redboot/whateverboot) ones. They are usually handled by taking image from previous stage and adding boot loader(s) by separate script. And this is where "fun" starts...
GPT takes first 17KB of storage media as it allow to store information about 128 partitions. Sure, no one is using so many on such devices but still space is reserved.
But most of chips expects boot loader(s) to be stored:
- right after MBR
- from 1KB
- from 8KB
- any other random place
So scripts start to be sets of magic written to handle all those SoCs...
Solution for existing SoCs is usually adding 1MB of SPI flash during design phase of device and store boot loader(s) there. But it is so expensive someone would say when it is in 10-30 cents range...
Even 96boards CE specification totally ignored that fact while it could be a way of showing how to make popular board. Instead it became yet-another-board-to-laugh (EE spec did not improve much).
Is there a way to get it improved? At least for new designs?
This has bugged me for a long time. I see that Rockchip stores its boot information 64 blocks into the device, allowing room for the GPT. This seems like the lowest common denominator for the case you raise. Perhaps ARM could create a set of guidelines here?
I don't think the media type is the problem. Obviously different manufacturers are going to choose media depending on their preferences, cost, etc. But it shouldn't be too hard to describe the boot architecture of a machine in some sort of simple human-readable file, and provide a software library to deal with it.
Regards, Simon
W dniu 05.05.2016 o 15:26, Simon Glass pisze:
This has bugged me for a long time. I see that Rockchip stores its boot information 64 blocks into the device, allowing room for the GPT. This seems like the lowest common denominator for the case you raise. Perhaps ARM could create a set of guidelines here?
I don't think the media type is the problem.
When it comes to media all was about split between "firmware device" which can be 1-2MB of SPI flash with just boot loaders + "system device" which can be SD/SATA/whatever and contain system and user data.
On Thu, May 5, 2016 at 12:45 PM, Marcin Juszkiewicz marcin.juszkiewicz@linaro.org wrote:
Recently my angry post on Google+ [1] got so many comments that it was clear that it would be better to move to some mailing list with discussion.
As it is about boot loaders and Linaro has engineers from most of SoC vendor companies I thought that this will be best one.
All started when I got Pine64 board (based on Allwinner A64 SoC) and had same issue as on several boards in past - boot loader written in some random place on SD card.
Days where people used Texas Instruments SoC chips were great - in-cpu boot loader knew how to read MBR partition table and was able to load 1st stage boot loader (called MLO) from it as long it was FAT filesystem.
GPU used by Raspberry/Pi is able to read MBR, finds 1st partition and reads firmware files from there as long it is FAT.
Chromebooks have some SPI flash to keep boot loaders and use GPT partitioning to find where from load kernel (or another boot loader).
And then we have all those boards where vendors decided that SPI flash for boot loader is too expensive so it will be read from SD card instead. From any random place of course...
Then we have distributions. And instead of generating bunch of images per board they want to make one clean image which will be able to handle as much as possible.
If there are UEFI machines on a list of supported ones then GPT partitioning will be used, boot loader will be stored in "EFI system area" and it boots. This is how AArch64 SBSA/SBBR machines work.
But there are also all those U-Boot (or fastboot/redboot/whateverboot) ones. They are usually handled by taking image from previous stage and adding boot loader(s) by separate script. And this is where "fun" starts...
GPT takes first 17KB of storage media as it allow to store information about 128 partitions. Sure, no one is using so many on such devices but still space is reserved.
But most of chips expects boot loader(s) to be stored:
- right after MBR
- from 1KB
- from 8KB
- any other random place
So scripts start to be sets of magic written to handle all those SoCs...
Solution for existing SoCs is usually adding 1MB of SPI flash during design phase of device and store boot loader(s) there. But it is so expensive someone would say when it is in 10-30 cents range...
To try and summarize, what you're asking for is to define the usage model for eMMC/SD when both the firmware* and OS are stored on the same media. Some argue that these things should always be on separate devices, but while the debate is interesting, it doesn't match the reality of how hardware is being built. In which case, the derived requirements are:
1) Co-exist with MBR partitioning 2) Co-exist with GPT partitioning 3) Be detectable --- partitioning tools must respect it 4) Be speced. Write it down so that tool and SoC developers can see it as a requirement 5) Be usable regardless of firmware type (UEFI, U-Boot, Little Kernel, etc) 6) Support some form of firmware non-volatile storage (variable storage)
It would be really nice if we could also have: 7) Support SoCs that hardcode boot code to specific locations (after-MBR, 1K, 8K, random) - May not be able to support all variants, but it is a worthy design goal.
Agreed?
* I'm ignoring eMMC's separate boot area because that solution has firmware and OS logically separated. Strong recommendation is for SoCs to boot from boot area. Then normal GPT/MBR partitioning works just fine. The rest of this discussion only applies If the SoC cannot do that
(For the following discussion, I refer to the UEFI spec because that is where GPT is defined, but the expectation is that anything described here can equally be used by non-UEFI platforms)
I've just read through the UEFI GPT spec, and here are the constraints: - MBR must be at the start of LBA0 (0 - 0.5k) - Primary GPT must be at the start of LBA1 (0.5k to 4k, but may collide with fw), - It /seems/ like the GPT Header and GPT table can be separated by some blocks. The GPT header has a PartitionEntryLBA field which describes where the actual table of partitions starts. - GPTHeader is only 92 bytes. - It should be possible to have: GPTHeader @ start of LBA1 and GPTPartitionTable @ an LBA that doesn't conflict with firmware.
I think we have everything we need to work around the location of the FW boot image without breaking the UEFI spec. The biggest problem is making sure partitioning tools don't go stomping over required firmware data and rendering systems unbootable. I *think* we can solve that problem by extending the MBR definition to block out a required region and then work around that. Tools can generically see the special region in the MBR and work around it accordingly.
So, let me try to itemize the use cases:
a. SoC boots from address immediately after MBR. - We're hooped. - Completely incompatible with current GPT spec. - Options: - Use MBR partitioning - move GPT to LBA > 1 - this creates a new question of how does the OS find the GPT? - I would prefer to treat this hardware as legacy and keep GPT at LBA1
b. SoC boots from fixed address > 1k and < 4k(?) - Place protective MBR at LBA0 - describe firmware block in MBR + protective GPT region - Modify spec to define this new format - Place GPT Header at LBA1 - Place GPT Partition Table > end of firmware LBA - Teach tools to understand & respect protective MBA when formatting disk
c. SoC boots from special MBR partition - place protective MBA at LBA0 (same as b.) - Describe FW partition in MBR - Perhaps mark as movable? - Vendor tools must place FW partition > 4k - Modify spec to define this new format (same format as b.) - Place GPT Header and partition table normally. - Tools should notice FW partition isn't in the way - Generate the GPT in the normal way - Either: - Add FW partition to the GPT table, or - set FirstUsableLBA to be immediately after the FW partition (safest?)
d. SoC boots from GPT partition. - Add new partition type that tools can recognize and refuse to delete unless the user forces it. - Define the new partition type in the UEFI spec, along with the required behaviour
I think that covers it, but I haven't yet talked through what to do with firmware non-volatile storage. It could be handled as another special firmware volume, or it could be treated as appended to the fw image. Need a bit more thought here.
If this looks good, then I can write up a proposed change to the spec and everyone can go look at the impact on firmware, the kernel, and tools.
g.
On Thu, May 05, 2016 at 04:21:59PM +0100, Grant Likely wrote:
I think we have everything we need to work around the location of the FW boot image without breaking the UEFI spec. The biggest problem is making sure partitioning tools don't go stomping over required firmware data and rendering systems unbootable. I *think* we can solve that problem by extending the MBR definition to block out a required region and then work around that. Tools can generically see the special region in the MBR and work around it accordingly.
So, let me try to itemize the use cases:
I think there's one other slightly different angle on this which we should address at the same time, creating fresh boot media for a device ("I just unpacked my board and want to write a system image to a SD card"). If we can come up with a standard way of describing the requirements of boards then we could provide a shared database of this information that tools could use. This might also be useful for the less helpful requirements where it's hard to figure out what's going on from the media itself.
On Thu, May 5, 2016 at 5:53 PM, Mark Brown broonie@kernel.org wrote:
On Thu, May 05, 2016 at 04:21:59PM +0100, Grant Likely wrote:
I think we have everything we need to work around the location of the FW boot image without breaking the UEFI spec. The biggest problem is making sure partitioning tools don't go stomping over required firmware data and rendering systems unbootable. I *think* we can solve that problem by extending the MBR definition to block out a required region and then work around that. Tools can generically see the special region in the MBR and work around it accordingly.
So, let me try to itemize the use cases:
I think there's one other slightly different angle on this which we should address at the same time, creating fresh boot media for a device ("I just unpacked my board and want to write a system image to a SD card"). If we can come up with a standard way of describing the requirements of boards then we could provide a shared database of this information that tools could use. This might also be useful for the less helpful requirements where it's hard to figure out what's going on from the media itself.
Are you talking about for boards that don't have on-board media? For the boards with eMMC, I think the normal use-cases should always be to boot from eMMC (regardless of where the OS is installed). Booting from an SD shouldn't require anything special. The only time to boot firmware from SD is when upgrading the platform firmware, or doing firmware development.
For systems without on-board storage, yes I agree it would be good to have a standard way of describing the layout because the images will always need to be tailored for the board.
g.
On Thu, May 05, 2016 at 06:03:40PM +0100, Grant Likely wrote:
On Thu, May 5, 2016 at 5:53 PM, Mark Brown broonie@kernel.org wrote:
I think there's one other slightly different angle on this which we should address at the same time, creating fresh boot media for a device ("I just unpacked my board and want to write a system image to a SD card"). If we can come up with a standard way of describing the requirements of boards then we could provide a shared database of this information that tools could use. This might also be useful for the less helpful requirements where it's hard to figure out what's going on from the media itself.
Are you talking about for boards that don't have on-board media? For the boards with eMMC, I think the normal use-cases should always be to boot from eMMC (regardless of where the OS is installed). Booting from an SD shouldn't require anything special. The only time to boot firmware from SD is when upgrading the platform firmware, or doing firmware development.
For systems without on-board storage, yes I agree it would be good to have a standard way of describing the layout because the images will always need to be tailored for the board.
Yes, boards that don't have on board media are the big issue here - they're typically the lowest cost and hence quite widely used. The same infrastructure might also help deal with some of the oddballs that have unhelpful on board storage but that's much less important if it's even useful at all.
On 05.05.16 17:21, Grant Likely wrote:
On Thu, May 5, 2016 at 12:45 PM, Marcin Juszkiewicz marcin.juszkiewicz@linaro.org wrote:
Recently my angry post on Google+ [1] got so many comments that it was clear that it would be better to move to some mailing list with discussion.
As it is about boot loaders and Linaro has engineers from most of SoC vendor companies I thought that this will be best one.
All started when I got Pine64 board (based on Allwinner A64 SoC) and had same issue as on several boards in past - boot loader written in some random place on SD card.
Days where people used Texas Instruments SoC chips were great - in-cpu boot loader knew how to read MBR partition table and was able to load 1st stage boot loader (called MLO) from it as long it was FAT filesystem.
GPU used by Raspberry/Pi is able to read MBR, finds 1st partition and reads firmware files from there as long it is FAT.
Chromebooks have some SPI flash to keep boot loaders and use GPT partitioning to find where from load kernel (or another boot loader).
And then we have all those boards where vendors decided that SPI flash for boot loader is too expensive so it will be read from SD card instead. From any random place of course...
Then we have distributions. And instead of generating bunch of images per board they want to make one clean image which will be able to handle as much as possible.
If there are UEFI machines on a list of supported ones then GPT partitioning will be used, boot loader will be stored in "EFI system area" and it boots. This is how AArch64 SBSA/SBBR machines work.
But there are also all those U-Boot (or fastboot/redboot/whateverboot) ones. They are usually handled by taking image from previous stage and adding boot loader(s) by separate script. And this is where "fun" starts...
GPT takes first 17KB of storage media as it allow to store information about 128 partitions. Sure, no one is using so many on such devices but still space is reserved.
But most of chips expects boot loader(s) to be stored:
- right after MBR
- from 1KB
- from 8KB
- any other random place
So scripts start to be sets of magic written to handle all those SoCs...
Solution for existing SoCs is usually adding 1MB of SPI flash during design phase of device and store boot loader(s) there. But it is so expensive someone would say when it is in 10-30 cents range...
To try and summarize, what you're asking for is to define the usage model for eMMC/SD when both the firmware* and OS are stored on the same media. Some argue that these things should always be on separate devices, but while the debate is interesting, it doesn't match the reality of how hardware is being built. In which case, the derived requirements are:
- Co-exist with MBR partitioning
- Co-exist with GPT partitioning
- Be detectable --- partitioning tools must respect it
- Be speced. Write it down so that tool and SoC developers can see it
as a requirement 5) Be usable regardless of firmware type (UEFI, U-Boot, Little Kernel, etc) 6) Support some form of firmware non-volatile storage (variable storage)
It would be really nice if we could also have: 7) Support SoCs that hardcode boot code to specific locations (after-MBR, 1K, 8K, random)
- May not be able to support all variants, but it is a worthy design goal.
Agreed?
- I'm ignoring eMMC's separate boot area because that solution has
firmware and OS logically separated. Strong recommendation is for SoCs to boot from boot area. Then normal GPT/MBR partitioning works just fine. The rest of this discussion only applies If the SoC cannot do that
(For the following discussion, I refer to the UEFI spec because that is where GPT is defined, but the expectation is that anything described here can equally be used by non-UEFI platforms)
I've just read through the UEFI GPT spec, and here are the constraints:
- MBR must be at the start of LBA0 (0 - 0.5k)
- Primary GPT must be at the start of LBA1 (0.5k to 4k, but may
collide with fw),
- It /seems/ like the GPT Header and GPT table can be separated by
some blocks. The GPT header has a PartitionEntryLBA field which describes where the actual table of partitions starts.
- GPTHeader is only 92 bytes.
- It should be possible to have: GPTHeader @ start of LBA1 and
GPTPartitionTable @ an LBA that doesn't conflict with firmware.
I think we have everything we need to work around the location of the FW boot image without breaking the UEFI spec. The biggest problem is making sure partitioning tools don't go stomping over required firmware data and rendering systems unbootable. I *think* we can solve that problem by extending the MBR definition to block out a required region and then work around that. Tools can generically see the special region in the MBR and work around it accordingly.
So what's the goal here? Are we trying to force GPT on systems whose vendors never intended them to run with GPT?
It really shouldn't matter at the end of the day whether we use GPT or MBR. All uEFI firmware implementations I'm aware of support both. So if you have a device whose bootloader collides with the GPT, just use MBR.
As for the "protection" mechanism, I don't think it's a problem either. IIRC parted starts to create partitions with a sensible alignment (1 or 2MB). Most boot loaders fit perfectly fine within that gap.
So this really boils down to
- use GPT for systems that were designed for it or - use MBR with alignment gap to first partition
end of story. There shouldn't be any middle ground. At least I haven't seen any so far :).
Alex
On Thu, May 5, 2016 at 5:15 PM, Marcin Juszkiewicz marcin.juszkiewicz@linaro.org wrote:
<snip>
Solution for existing SoCs is usually adding 1MB of SPI flash during design phase of device and store boot loader(s) there. But it is so expensive someone would say when it is in 10-30 cents range...
Even 96boards CE specification totally ignored that fact while it could be a way of showing how to make popular board. Instead it became yet-another-board-to-laugh (EE spec did not improve much).
Is there a way to get it improved? At least for new designs?
Yes! I've added this suggestion to a list of suggestions for evolution of the 96boards spec.
On Thu, May 05, 2016 at 09:01:05PM +0530, Amit Kucheria wrote:
On Thu, May 5, 2016 at 5:15 PM, Marcin Juszkiewicz
Solution for existing SoCs is usually adding 1MB of SPI flash during design phase of device and store boot loader(s) there. But it is so expensive someone would say when it is in 10-30 cents range...
Even 96boards CE specification totally ignored that fact while it could be a way of showing how to make popular board. Instead it became yet-another-board-to-laugh (EE spec did not improve much).
Is there a way to get it improved? At least for new designs?
Yes! I've added this suggestion to a list of suggestions for evolution of the 96boards spec.
We already went round the houses repeatedly on that one :(
On Thu, May 5, 2016 at 4:59 PM, Mark Brown broonie@kernel.org wrote:
On Thu, May 05, 2016 at 09:01:05PM +0530, Amit Kucheria wrote:
On Thu, May 5, 2016 at 5:15 PM, Marcin Juszkiewicz
Solution for existing SoCs is usually adding 1MB of SPI flash during design phase of device and store boot loader(s) there. But it is so expensive someone would say when it is in 10-30 cents range...
Even 96boards CE specification totally ignored that fact while it could be a way of showing how to make popular board. Instead it became yet-another-board-to-laugh (EE spec did not improve much).
Is there a way to get it improved? At least for new designs?
Yes! I've added this suggestion to a list of suggestions for evolution of the 96boards spec.
We already went round the houses repeatedly on that one :(
It's not so dismal as all that! With the board size and price point the CE spec aims at, it is very unlikely that vendors will build boards with a separate storage device for FW.* However, by and large these platforms will be built with eMMC and eMMC has a separate boot area that is logically separate from the main pool. We can add a requirement for future versions to boot from the boot area when booting from eMMC.
While on the topic of the spec, we need to handle "optional" items in a better way. Rather than having toothless "optional" requirements. How about we have 2 levels of compliance. Something like "Silver" compliance which covers all the absolutely required items, and "Gold" which also implements the optional items. I bring this up because there are several SoCs that vendors want to build board for, but they many not be able to make the business case for respining the chip to meet an item of the spec. Such as device the SoC boot from.
* although anyone who really wants to treat it as separate storage can happily insert an SD card and dedicate the eMMC to firmware -- really, how is that different?.
On Thu, May 5, 2016 at 9:29 PM, Mark Brown broonie@kernel.org wrote:
On Thu, May 05, 2016 at 09:01:05PM +0530, Amit Kucheria wrote:
On Thu, May 5, 2016 at 5:15 PM, Marcin Juszkiewicz
Solution for existing SoCs is usually adding 1MB of SPI flash during design phase of device and store boot loader(s) there. But it is so expensive someone would say when it is in 10-30 cents range...
Even 96boards CE specification totally ignored that fact while it could be a way of showing how to make popular board. Instead it became yet-another-board-to-laugh (EE spec did not improve much).
Is there a way to get it improved? At least for new designs?
Yes! I've added this suggestion to a list of suggestions for evolution of the 96boards spec.
We already went round the houses repeatedly on that one :(
Yes, but it wasn't in the list. So if we want to reject something (or make it optional), I'd like us to document why we reached that decision for posterity.
As Grant mentioned elsewhere, the single-storage device configuration seems to be where a lot of the vendors seem to be focusing on.
I would add, that you need to draw a line in the sand between the 'consumer' (don't flame me, I am just struggling to find a better term) boards and those that are positioned for production/enterprise. We can't state that there is only one way that is the right way, that's not fair to anyone, and counter-culture to the opensource view. We can recommend a best practice, and I think this thread is a really good start at that, but it will always be a view that one can offer to the community. Adoption is a completely other thing.
Specifically for the 96boards, the spec is a recommended view, but its not meant to be constraining, however it does allow one to then show a best practice, that others can adopt. That's where the RPB comes in to play, again to demonstrate and not restrict.
On 5 May 2016 at 19:34, Amit Kucheria amit.kucheria@linaro.org wrote:
On Thu, May 5, 2016 at 9:29 PM, Mark Brown broonie@kernel.org wrote:
On Thu, May 05, 2016 at 09:01:05PM +0530, Amit Kucheria wrote:
On Thu, May 5, 2016 at 5:15 PM, Marcin Juszkiewicz
Solution for existing SoCs is usually adding 1MB of SPI flash during
design
phase of device and store boot loader(s) there. But it is so expensive someone would say when it is in 10-30 cents range...
Even 96boards CE specification totally ignored that fact while it
could be a
way of showing how to make popular board. Instead it became yet-another-board-to-laugh (EE spec did not improve much).
Is there a way to get it improved? At least for new designs?
Yes! I've added this suggestion to a list of suggestions for evolution of the 96boards spec.
We already went round the houses repeatedly on that one :(
Yes, but it wasn't in the list. So if we want to reject something (or make it optional), I'd like us to document why we reached that decision for posterity.
As Grant mentioned elsewhere, the single-storage device configuration seems to be where a lot of the vendors seem to be focusing on. _______________________________________________ cross-distro mailing list cross-distro@lists.linaro.org https://lists.linaro.org/mailman/listinfo/cross-distro
On Thu, May 05, 2016 at 07:47:41PM +0100, Martin Stadtler wrote:
Specifically for the 96boards, the spec is a recommended view, but its not meant to be constraining, however it does allow one to then show a best practice, that others can adopt. That's where the RPB comes in to play, again to demonstrate and not restrict.
Note that this is also going to the cross-distro list, we're probably getting a bit off topic with Linaro/96boards internal discussions like this.
-
Alexander Graf -
Amit Kucheria -
Grant Likely -
Marcin Juszkiewicz -
Mark Brown -
Martin Stadtler -
Simon Glass