Hi,
We (MAAS upstream) are having some issues figuring out how to effectively detect the architecture of a system booting using U-Boot pxelinux emulation in order to send back an appropriate kernel and initrd.
In short, we'd like a mechanism that is separate from the DHCP vendor option which I believe is the only way to differentiate at the moment.
I proposed an alternative mechanism in https://launchpad.net/bugs/1041092 (description copied below) and it was suggested to me that I could get feedback from this list.
If this mechanism is appropriate and we can get vendor consensus on it, then I can continue and implement the other end of it in MAAS. If it is not suitable then I'd love to hear about alternatives. Either way, I'd really like to move within a couple of weeks so that I can make our 12.10 release in October.
Without a non-DHCP mechanism for architecture detection, MAAS will be stuck having to be configured for homogeneous architecture clusters only, which would be unfortunte.
What do you think? The full text of the bug is below.
Thanks,
Robie
An outcome of a recent MAAS-related sprint is the conclusion that MAAS needs to be able to operate in an environment where it cannot control the DHCP server.
This means that we cannot rely on the ability to differentiate architectures based on ISC dhcpd.conf conditional statements on vendor-class-identifier setting alternate next-server filenames as we are doing at the moment (see bug 927781 for details of this mechanism).
Instead we need some other way for MAAS to detect the architecture of the booting system in order to send it the correct architecture kernel/initrd.
If this doesn't happen, then MAAS will have to resort to guessing based on MAC addresses supplied by vendors, or the user entering in the information on a per-MAC basis manually, or the user nominating an entire MAAS cluster as homogeneous on a particular architecture. None of these options are ideal.
Instead, I propose the following minor change to U-Boot to enable architecture detection outside of DHCP.
The original (Intel) pxelinux.0 falls back through MAC addresses, IP address and subnets and then to "default". Prior to U-Boot pxelinux emulation, a TFTP server could assume that sending an i386 kernel would always work.
pxelinux emulation on other architectures breaks this assumption. So I propose that all alternative architecture pxelinux emulators (eg. U-Boot) fall back to "default.<arch>-<subarch>" and then "default.<arch>" before further falling back to "default" as usual.
<arch> and <subarch> must be defined in a new pxelinux emulator namespace, and MAAS will consequently need to map this namespace to Ubuntu's architecture and kernel flavour names. But to start with, they'll probably match 1-1.
So: default.arm-highbank default.arm-armadaxp default.arm-omap4
For this proposed change to work, we need all vendors to take this up. I think the change would be trivial, but I would appreciate feedback from U-Boot maintainers and vendors before we commit to this direction.
On Thu, Aug 30, 2012 at 07:42:01AM +0100, Robie Basak wrote:
Hi,
We (MAAS upstream) are having some issues figuring out how to effectively detect the architecture of a system booting using U-Boot pxelinux emulation in order to send back an appropriate kernel and initrd.
In short, we'd like a mechanism that is separate from the DHCP vendor option which I believe is the only way to differentiate at the moment.
I proposed an alternative mechanism in https://launchpad.net/bugs/1041092 (description copied below) and it was suggested to me that I could get feedback from this list.
If this mechanism is appropriate and we can get vendor consensus on it, then I can continue and implement the other end of it in MAAS. If it is not suitable then I'd love to hear about alternatives. Either way, I'd really like to move within a couple of weeks so that I can make our 12.10 release in October.
Without a non-DHCP mechanism for architecture detection, MAAS will be stuck having to be configured for homogeneous architecture clusters only, which would be unfortunte.
What do you think? The full text of the bug is below.
Thanks,
Robie
An outcome of a recent MAAS-related sprint is the conclusion that MAAS needs to be able to operate in an environment where it cannot control the DHCP server.
This means that we cannot rely on the ability to differentiate architectures based on ISC dhcpd.conf conditional statements on vendor-class-identifier setting alternate next-server filenames as we are doing at the moment (see bug 927781 for details of this mechanism).
fyi, here's a link to that bug: https://bugs.launchpad.net/ubuntu/+source/u-boot-linaro/+bug/927781
-dann
Instead we need some other way for MAAS to detect the architecture of the booting system in order to send it the correct architecture kernel/initrd.
If this doesn't happen, then MAAS will have to resort to guessing based on MAC addresses supplied by vendors, or the user entering in the information on a per-MAC basis manually, or the user nominating an entire MAAS cluster as homogeneous on a particular architecture. None of these options are ideal.
Instead, I propose the following minor change to U-Boot to enable architecture detection outside of DHCP.
The original (Intel) pxelinux.0 falls back through MAC addresses, IP address and subnets and then to "default". Prior to U-Boot pxelinux emulation, a TFTP server could assume that sending an i386 kernel would always work.
pxelinux emulation on other architectures breaks this assumption. So I propose that all alternative architecture pxelinux emulators (eg. U-Boot) fall back to "default.<arch>-<subarch>" and then "default.<arch>" before further falling back to "default" as usual.
<arch> and <subarch> must be defined in a new pxelinux emulator namespace, and MAAS will consequently need to map this namespace to Ubuntu's architecture and kernel flavour names. But to start with, they'll probably match 1-1.
So: default.arm-highbank default.arm-armadaxp default.arm-omap4
For this proposed change to work, we need all vendors to take this up. I think the change would be trivial, but I would appreciate feedback from U-Boot maintainers and vendors before we commit to this direction.
cross-distro mailing list cross-distro@lists.linaro.org http://lists.linaro.org/mailman/listinfo/cross-distro
On 08/30/2012 01:42 AM, Robie Basak wrote:
Hi,
We (MAAS upstream) are having some issues figuring out how to effectively detect the architecture of a system booting using U-Boot pxelinux emulation in order to send back an appropriate kernel and initrd.
In short, we'd like a mechanism that is separate from the DHCP vendor option which I believe is the only way to differentiate at the moment.
I proposed an alternative mechanism in https://launchpad.net/bugs/1041092 (description copied below) and it was suggested to me that I could get feedback from this list.
If this mechanism is appropriate and we can get vendor consensus on it, then I can continue and implement the other end of it in MAAS. If it is not suitable then I'd love to hear about alternatives. Either way, I'd really like to move within a couple of weeks so that I can make our 12.10 release in October.
Without a non-DHCP mechanism for architecture detection, MAAS will be stuck having to be configured for homogeneous architecture clusters only, which would be unfortunte.
What do you think? The full text of the bug is below.
Thanks,
Robie
An outcome of a recent MAAS-related sprint is the conclusion that MAAS needs to be able to operate in an environment where it cannot control the DHCP server.
This means that we cannot rely on the ability to differentiate architectures based on ISC dhcpd.conf conditional statements on vendor-class-identifier setting alternate next-server filenames as we are doing at the moment (see bug 927781 for details of this mechanism).
Instead we need some other way for MAAS to detect the architecture of the booting system in order to send it the correct architecture kernel/initrd.
If this doesn't happen, then MAAS will have to resort to guessing based on MAC addresses supplied by vendors, or the user entering in the information on a per-MAC basis manually, or the user nominating an entire MAAS cluster as homogeneous on a particular architecture. None of these options are ideal.
Instead, I propose the following minor change to U-Boot to enable architecture detection outside of DHCP.
The original (Intel) pxelinux.0 falls back through MAC addresses, IP address and subnets and then to "default". Prior to U-Boot pxelinux emulation, a TFTP server could assume that sending an i386 kernel would always work.
pxelinux emulation on other architectures breaks this assumption. So I propose that all alternative architecture pxelinux emulators (eg. U-Boot) fall back to "default.<arch>-<subarch>" and then "default.<arch>" before further falling back to "default" as usual.
<arch> and <subarch> must be defined in a new pxelinux emulator namespace, and MAAS will consequently need to map this namespace to Ubuntu's architecture and kernel flavour names. But to start with, they'll probably match 1-1.
So: default.arm-highbank default.arm-armadaxp default.arm-omap4
For the DHCP string, we're currently using armv7 as part of it. Is arm too generic? With "single" kernels, we will have armv4/5, armv6/7, armv7, armv7-pae and armv8 kernels. Or will subarch just become v4, v7, v8, etc.?
For this proposed change to work, we need all vendors to take this up. I think the change would be trivial, but I would appreciate feedback from U-Boot maintainers and vendors before we commit to this direction.
In general, I'm fine with this change, and it is fairly trivial.
Rob
cross-distro mailing list cross-distro@lists.linaro.org http://lists.linaro.org/mailman/listinfo/cross-distro
On 08/29/2012 11:42 PM, Robie Basak wrote:
Hi,
We (MAAS upstream) are having some issues figuring out how to effectively detect the architecture of a system booting using U-Boot pxelinux emulation in order to send back an appropriate kernel and initrd.
In short, we'd like a mechanism that is separate from the DHCP vendor option which I believe is the only way to differentiate at the moment.
I proposed an alternative mechanism in https://launchpad.net/bugs/1041092 (description copied below) and it was suggested to me that I could get feedback from this list.
...
pxelinux emulation on other architectures breaks this assumption. So I propose that all alternative architecture pxelinux emulators (eg. U-Boot) fall back to "default.<arch>-<subarch>" and then "default.<arch>" before further falling back to "default" as usual.
<arch> and <subarch> must be defined in a new pxelinux emulator namespace, and MAAS will consequently need to map this namespace to Ubuntu's architecture and kernel flavour names. But to start with, they'll probably match 1-1.
For the values of <arch> and <subarch>, does it make sense to use the U-Boot variables from boards.cfg, which are also exposed as #defines CONFIG_SYS_ARCH, CONFIG_SYS_CPU, & CONFIG_SYS_SOC, and also as shell environment variables arch, cpu, & soc (if CONFIG_ENV_VARS_UBOOT_CONFIG is defined)? I wonder if a 3-level fallback might not be useful (i.e. involving all of those variables)?
Responding to both Rob's and Stephen's comments at once, since I think they're essentially the same issue.
On Fri, Aug 31, 2012 at 07:51:21PM -0500, Rob Herring wrote:
On 08/30/2012 01:42 AM, Robie Basak wrote:
<arch> and <subarch> must be defined in a new pxelinux emulator namespace, and MAAS will consequently need to map this namespace to Ubuntu's architecture and kernel flavour names. But to start with, they'll probably match 1-1.
So: default.arm-highbank default.arm-armadaxp default.arm-omap4
For the DHCP string, we're currently using armv7 as part of it. Is arm too generic? With "single" kernels, we will have armv4/5, armv6/7, armv7, armv7-pae and armv8 kernels. Or will subarch just become v4, v7, v8, etc.?
On Sat, Sep 01, 2012 at 07:12:19PM -0700, Stephen Warren wrote:
For the values of <arch> and <subarch>, does it make sense to use the U-Boot variables from boards.cfg, which are also exposed as #defines CONFIG_SYS_ARCH, CONFIG_SYS_CPU, & CONFIG_SYS_SOC, and also as shell environment variables arch, cpu, & soc (if CONFIG_ENV_VARS_UBOOT_CONFIG is defined)? I wonder if a 3-level fallback might not be useful (i.e. involving all of those variables)?
I think this is a bigger question than I first imagined. I don't really mind what the exact outcome is provided that we all agree.
I reached "arm" because I was thinking about the Debian architectures armel and armhf, but dropped off the el and hf as they {aren't relevant/are equivalent} at the bootloader stage. I think OS architecture definitions are relevant here because the OS kernels available will always belong to an OS-defined architecture.
I got to "highbank" because we need to distinguish the kernels and "highbank", "armadaxp" etc. are the existing Ubuntu kernel flavour names.
I assumed that the ISA distinctions would be covered with just "arm" and "aarch64", with further distinctions within <subarch> since the SoC decides which ISA features are implemented and each one will need a different kernel anyway. So default.armv7.highbank would be a bit redundant because I would expect the "highbank" part of the name to change for a future v7-pae SoC, and again for a v8 SoC. In other words, "highbank" implies v8 (or does it not?) and thus also encoding this in <arch> would be redundant.
But whichever way we have it, the important thing for me is that I can distinguish between the current existing SoCs right now. I need to know that this is the way forward ASAP so that I can implement this inside MAAS. From my point of view the exact strings aren't hard to change before release provided that the general mechanism is agreed upon (and it sounds like it is).
So with that, may I step out of the loop and let Rob, Stephen and anyone else relevant decide what the exact names should look like? I'll happily accept whatever you come up with, provided it leads to a different default.<...> fallback per required kernel/initrd.
Thanks,
Robie
On 09/03/2012 04:44 AM, Robie Basak wrote:
Responding to both Rob's and Stephen's comments at once, since I think they're essentially the same issue.
On Fri, Aug 31, 2012 at 07:51:21PM -0500, Rob Herring wrote:
On 08/30/2012 01:42 AM, Robie Basak wrote:
<arch> and <subarch> must be defined in a new pxelinux emulator namespace, and MAAS will consequently need to map this namespace to Ubuntu's architecture and kernel flavour names. But to start with, they'll probably match 1-1.
So: default.arm-highbank default.arm-armadaxp default.arm-omap4
For the DHCP string, we're currently using armv7 as part of it. Is arm too generic? With "single" kernels, we will have armv4/5, armv6/7, armv7, armv7-pae and armv8 kernels. Or will subarch just become v4, v7, v8, etc.?
On Sat, Sep 01, 2012 at 07:12:19PM -0700, Stephen Warren wrote:
For the values of <arch> and <subarch>, does it make sense to use the U-Boot variables from boards.cfg, which are also exposed as #defines CONFIG_SYS_ARCH, CONFIG_SYS_CPU, & CONFIG_SYS_SOC, and also as shell environment variables arch, cpu, & soc (if CONFIG_ENV_VARS_UBOOT_CONFIG is defined)? I wonder if a 3-level fallback might not be useful (i.e. involving all of those variables)?
I think this is a bigger question than I first imagined. I don't really mind what the exact outcome is provided that we all agree.
I reached "arm" because I was thinking about the Debian architectures armel and armhf, but dropped off the el and hf as they {aren't relevant/are equivalent} at the bootloader stage. I think OS architecture definitions are relevant here because the OS kernels available will always belong to an OS-defined architecture.
I got to "highbank" because we need to distinguish the kernels and "highbank", "armadaxp" etc. are the existing Ubuntu kernel flavour names.
That's a very short sighted view. In 3.7, we will likely multi-platform kernels starting with Versatile Express, Highbank and mvebu (Armada XP). Perhaps OMAP will happen too. Assuming you wanted to support all these platforms from a single kernel build, how would you manage it?
Perhaps you always keep separate platforms and just symlink things back to a common kernel? You could also argue that the command line options will always be different, so we'll always need to have per platform directories.
We could make the directory list be an environment variable to iterate thru then it's not fixed in the bootloader. We'd still need to come up with defaults though.
Rob
I assumed that the ISA distinctions would be covered with just "arm" and "aarch64", with further distinctions within <subarch> since the SoC decides which ISA features are implemented and each one will need a different kernel anyway. So default.armv7.highbank would be a bit redundant because I would expect the "highbank" part of the name to change for a future v7-pae SoC, and again for a v8 SoC. In other words, "highbank" implies v8 (or does it not?) and thus also encoding this in <arch> would be redundant.
But whichever way we have it, the important thing for me is that I can distinguish between the current existing SoCs right now. I need to know that this is the way forward ASAP so that I can implement this inside MAAS. From my point of view the exact strings aren't hard to change before release provided that the general mechanism is agreed upon (and it sounds like it is).
So with that, may I step out of the loop and let Rob, Stephen and anyone else relevant decide what the exact names should look like? I'll happily accept whatever you come up with, provided it leads to a different default.<...> fallback per required kernel/initrd.
Thanks,
Robie
On Tue, Sep 04, 2012 at 01:10:31PM -0500, Rob Herring wrote:
On 09/03/2012 04:44 AM, Robie Basak wrote:
I got to "highbank" because we need to distinguish the kernels and "highbank", "armadaxp" etc. are the existing Ubuntu kernel flavour names.
That's a very short sighted view.
No, it's not a view at all. I just explained how I reached this suggestion. Also, note that I am NOT assuming a one-to-one mapping between kernel flavours and <subarch> (whatever subarch might mean). I explicitly define is as a separate namespace which will need to have a known mapping in Ubuntu to determine the kernel to use.
In 3.7, we will likely multi-platformkernels starting with Versatile Express, Highbank and mvebu (Armada XP). Perhaps OMAP will happen too. Assuming you wanted to support all these platforms from a single kernel build, how would you manage it?
With multiple default.arm-<whatever> names mapping to single Ubuntu kernel flavours as necessary. What am I missing?
Perhaps you always keep separate platforms and just symlink things back to a common kernel? You could also argue that the command line options will always be different, so we'll always need to have per platform directories.
Sure - we could do that. I don't think it'll matter too much how we do it provided that we can.
We could make the directory list be an environment variable to iterate thru then it's not fixed in the bootloader. We'd still need to come up with defaults though.
This sounds good to me.
One additional point that's just come up - we'd also like to know the MAC address, which at the point of serving a default.<something> file we won't statelessly know if the TFTP server is not in the same ethernet segment. How about also hitting something like pxelinux.cfg/01-<mac>.arm-<somthing> before pxelinux.cfg/01-<mac> first, and then the default fallback behaviour later? This would make my life a bit easier by not having to save state in between calls to know the MAC address at the time of serving the default file.
Robie
On 09/04/2012 01:34 PM, Robie Basak wrote:
On Tue, Sep 04, 2012 at 01:10:31PM -0500, Rob Herring wrote:
On 09/03/2012 04:44 AM, Robie Basak wrote:
I got to "highbank" because we need to distinguish the kernels and "highbank", "armadaxp" etc. are the existing Ubuntu kernel flavour names.
That's a very short sighted view.
No, it's not a view at all. I just explained how I reached this suggestion. Also, note that I am NOT assuming a one-to-one mapping between kernel flavours and <subarch> (whatever subarch might mean). I explicitly define is as a separate namespace which will need to have a known mapping in Ubuntu to determine the kernel to use.
In 3.7, we will likely multi-platformkernels starting with Versatile Express, Highbank and mvebu (Armada XP). Perhaps OMAP will happen too. Assuming you wanted to support all these platforms from a single kernel build, how would you manage it?
With multiple default.arm-<whatever> names mapping to single Ubuntu kernel flavours as necessary. What am I missing?
Just making sure you are fine with that.
Perhaps you always keep separate platforms and just symlink things back to a common kernel? You could also argue that the command line options will always be different, so we'll always need to have per platform directories.
Sure - we could do that. I don't think it'll matter too much how we do it provided that we can.
We could make the directory list be an environment variable to iterate thru then it's not fixed in the bootloader. We'd still need to come up with defaults though.
This sounds good to me.
One additional point that's just come up - we'd also like to know the MAC address, which at the point of serving a default.<something> file we won't statelessly know if the TFTP server is not in the same ethernet segment. How about also hitting something like pxelinux.cfg/01-<mac>.arm-<somthing> before pxelinux.cfg/01-<mac> first, and then the default fallback behaviour later? This would make my life a bit easier by not having to save state in between calls to know the MAC address at the time of serving the default file.
That's really a departure from standard PXELINUX and people have complained already that u-boot PXE is not standard PXELINUX. :) How is this specific to ARM?
Rob
On Tue, Sep 04, 2012 at 02:42:17PM -0500, Rob Herring wrote:
On 09/04/2012 01:34 PM, Robie Basak wrote:
One additional point that's just come up - we'd also like to know the MAC address, which at the point of serving a default.<something> file we won't statelessly know if the TFTP server is not in the same ethernet segment. How about also hitting something like pxelinux.cfg/01-<mac>.arm-<somthing> before pxelinux.cfg/01-<mac> first, and then the default fallback behaviour later? This would make my life a bit easier by not having to save state in between calls to know the MAC address at the time of serving the default file.
That's really a departure from standard PXELINUX and people have complained already that u-boot PXE is not standard PXELINUX. :) How is this specific to ARM?
If U-Boot PXE were identical to pxelinux, it would boot Intel architecture kernels and we wouldn't have a problem :-)
Given that it must boot ARM kernels, I think it is reasonable to differ from pxelinux for architecture detection purposes, but still seek parity with pxelinux in all other respects where architecture differences do not prevent it from doing so.
With pxelinux, a dynamic TFTP server can, on the first request of pxelinux.cfg/01-<mac>, assume that the machine is an Intel architecture and also know its MAC address, even if the booting machine is in a different ethernet segment. This is what MAAS is doing at the moment.
For ARM support with U-Boot, I'd like to determine the architecture and know the machine's MAC address in that same query.
If I do it with default.arm-<details>, then the dynamic TFTP server suddenly has to be stateful, store the MAC in a table against the IP address and returning a "404" first to then get architecture details on the default.arm-<details>, which is more difficult (eg. wrt. state cleanup) - but I admit that it is possible.
So yes - it is a departure from pxelinux, but one that I think makes sense given that this is an alternative architecture. Do you think this is a step too far?
Robie
-
dann frazier -
Rob Herring -
Robie Basak -
Stephen Warren