Rob,
Am 04.12.18 um 19:36 schrieb Rob Herring:
I've put together a script to move the dts files and update the makefiles. It doesn't handle files not following a common prefix which isn't many and some includes within the dts files will need some fixups by hand.
MAINTAINERS will also need updating.
A few questions:
Do we want to move absolutely everything to subdirs?
This refactoring is a terrible idea!
While it would've been nice to have more structure from the start, bootloaders like U-Boot expect a flat structure for arm .dtb files now. If you start installing them into subdirs instead, they won't find the files anymore under the hardcoded name.
Doing this only for new platforms would be much less invasive and allow to prepare bootloaders accordingly. Alternatively, white-list which ones are safe to move around. But don't just script a refactoring because it looks nicer in the source tree, without testing what side effects this can have for board/distro users of the compiled files in practice. We already had that discussion for arm64 because Debian chose to ignore the kernel-installed subdirectories and installed .dtb files into a flat directory, which collided with openSUSE sticking to the kernel choice.
This topic becomes even more important with EBBR: There is neither a mechanism in place to sync .dts files into U-Boot or EDK2 source trees, nor are capsule updates implemented in U-Boot for easily deploying such bootloaders with new .dts sources or paths yet. And I can assure you that just getting users to dd the right bootloader can be difficult... Since DT forward and backward compatibility is often being neglected, for example with optional properties or renamed compatibles that break booting with previous drivers, new kernel versions often need updated Device Trees to make use of new/enhanced drivers. Therefore it is unfortunately often enough a necessity to load newer kernel-based .dtb files matching the kernel (as opposed to the dream of kernel-independent hardware descriptions) when working with the latest -rc or -next kernels at least. For examples of DTs needing updates, look no further than Linaro's 96boards - in case of hikey960/EDK2 GRUB is another layer where .dtb paths may be hardcoded, ditto for arm; and Armada was an example where the upstream bindings for the network IP changed incompatibly.
DT overlays are another topic that is not making any progress upstream according to the ELCE BoF, so beyond the Raspberry Pi the only known working way to apply them is to write a U-Boot boot.scr script, which can either reuse $fdtcontroladdr DT or use the filename $fdtfile or hardcode one, the latter two of which would break with your renaming.
So expect people to be using .dtb files, expect them to be affected by file movements to subdirectories here, and don't expect each user to understand or be able to fix things themselves if they fall apart as result of your changes and they suddenly no longer have Ethernet/Wifi.
Regards, Andreas
On Tue, Dec 4, 2018 at 7:22 PM Andreas Färber afaerber@suse.de wrote:
Rob,
Am 04.12.18 um 19:36 schrieb Rob Herring:
I've put together a script to move the dts files and update the makefiles. It doesn't handle files not following a common prefix which isn't many and some includes within the dts files will need some fixups by hand.
MAINTAINERS will also need updating.
A few questions:
Do we want to move absolutely everything to subdirs?
This refactoring is a terrible idea!
How do you really feel?
While it would've been nice to have more structure from the start, bootloaders like U-Boot expect a flat structure for arm .dtb files now. If you start installing them into subdirs instead, they won't find the files anymore under the hardcoded name.
Doing this only for new platforms would be much less invasive and allow to prepare bootloaders accordingly.
That was my suggestion where this started for the new RDA platform. Olof preferred to move everything and that's my desire too.
Alternatively, white-list which ones are safe to move around.
I'd prefer to know which ones the distros don't want moved. That should be easier to figure out. We also need that anyways in context of what platforms we care about compatibility.
Another option is dtbs_install target could flatten the installed dtbs. That is the only part the distros should depend on.
But don't just script a refactoring because it looks nicer in the source tree, without testing what side effects this can have for board/distro users of the compiled files in practice. We already had that discussion for arm64 because Debian chose to ignore the kernel-installed subdirectories and installed .dtb files into a flat directory, which collided with openSUSE sticking to the kernel choice.
So everyone already deals with subdirs or not with arm and arm64 already, seems like they can deal with this. I will raise the topic on the cross-distro list though.
This topic becomes even more important with EBBR: There is neither a mechanism in place to sync .dts files into U-Boot or EDK2 source trees, nor are capsule updates implemented in U-Boot for easily deploying such bootloaders with new .dts sources or paths yet.
EBBR actually says firmware (including dtbs) goes in directories named by vendor.
And I can assure you that just getting users to dd the right bootloader can be difficult... Since DT forward and backward compatibility is often being neglected, for example with optional properties or renamed compatibles that break booting with previous drivers, new kernel versions often need updated Device Trees to make use of new/enhanced drivers. Therefore it is unfortunately often enough a necessity to load newer kernel-based .dtb files matching the kernel (as opposed to the dream of kernel-independent hardware descriptions) when working with the latest -rc or -next kernels at least. For examples of DTs needing updates, look no further than Linaro's 96boards - in case of hikey960/EDK2 GRUB is another layer where .dtb paths may be hardcoded, ditto for arm; and Armada was an example where the upstream bindings for the network IP changed incompatibly.
Compatibility is an issue, yes, but that really has nothing to do with this.
DT overlays are another topic that is not making any progress upstream according to the ELCE BoF, so beyond the Raspberry Pi the only known working way to apply them is to write a U-Boot boot.scr script, which can either reuse $fdtcontroladdr DT or use the filename $fdtfile or hardcode one, the latter two of which would break with your renaming.
DT overlays also have nothing to do with this as there aren't any in the kernel. I'm not inclined to take any either with a flat tree. We're already at 1800+ files.
Rob
Am 05.12.18 um 05:17 schrieb Rob Herring:
On Tue, Dec 4, 2018 at 7:22 PM Andreas Färber afaerber@suse.de wrote:
Rob,
Am 04.12.18 um 19:36 schrieb Rob Herring:
I've put together a script to move the dts files and update the makefiles. It doesn't handle files not following a common prefix which isn't many and some includes within the dts files will need some fixups by hand.
MAINTAINERS will also need updating.
A few questions:
Do we want to move absolutely everything to subdirs?
This refactoring is a terrible idea!
How do you really feel?
While it would've been nice to have more structure from the start, bootloaders like U-Boot expect a flat structure for arm .dtb files now. If you start installing them into subdirs instead, they won't find the files anymore under the hardcoded name.
Doing this only for new platforms would be much less invasive and allow to prepare bootloaders accordingly.
That was my suggestion where this started for the new RDA platform. Olof preferred to move everything and that's my desire too.
Alternatively, white-list which ones are safe to move around.
I'd prefer to know which ones the distros don't want moved. That should be easier to figure out. We also need that anyways in context of what platforms we care about compatibility.
Another option is dtbs_install target could flatten the installed dtbs. That is the only part the distros should depend on.
I'd be okay with distinguishing source vs. install location. Due to the issue I mention below (and more) we can't use install_dtbs for openSUSE and had to reimplement it, which we'd need to (and can) adjust.
But don't just script a refactoring because it looks nicer in the source tree, without testing what side effects this can have for board/distro users of the compiled files in practice. We already had that discussion for arm64 because Debian chose to ignore the kernel-installed subdirectories and installed .dtb files into a flat directory, which collided with openSUSE sticking to the kernel choice.
So everyone already deals with subdirs or not with arm and arm64 already, seems like they can deal with this. I will raise the topic on the cross-distro list though.
Sounds like you're twisting words... The keyword was "hardcoded" paths - one way or another (not "and") depending on the kernel choices being flat for arm, vendor subdir for arm64.
This topic becomes even more important with EBBR: There is neither a mechanism in place to sync .dts files into U-Boot or EDK2 source trees, nor are capsule updates implemented in U-Boot for easily deploying such bootloaders with new .dts sources or paths yet.
EBBR actually says firmware (including dtbs) goes in directories named by vendor.
Fine, but unrelated.
And I can assure you that just getting users to dd the right bootloader can be difficult... Since DT forward and backward compatibility is often being neglected, for example with optional properties or renamed compatibles that break booting with previous drivers, new kernel versions often need updated Device Trees to make use of new/enhanced drivers. Therefore it is unfortunately often enough a necessity to load newer kernel-based .dtb files matching the kernel (as opposed to the dream of kernel-independent hardware descriptions) when working with the latest -rc or -next kernels at least. For examples of DTs needing updates, look no further than Linaro's 96boards - in case of hikey960/EDK2 GRUB is another layer where .dtb paths may be hardcoded, ditto for arm; and Armada was an example where the upstream bindings for the network IP changed incompatibly.
Compatibility is an issue, yes, but that really has nothing to do with this.
DT overlays are another topic that is not making any progress upstream according to the ELCE BoF, so beyond the Raspberry Pi the only known working way to apply them is to write a U-Boot boot.scr script, which can either reuse $fdtcontroladdr DT or use the filename $fdtfile or hardcode one, the latter two of which would break with your renaming.
DT overlays also have nothing to do with this as there aren't any in the kernel. I'm not inclined to take any either with a flat tree. We're already at 1800+ files.
Read again: a) Breaking DT changes and b) the desire to use Overlays instead of replacing the bootloaders for each change are _reasons_ why people depend on .dtb filenames from the kernel source tree for their boot flow today. Nothing to do with downstream .dtbo files.
For example, remember when I reported that the kernel didn't compile DTs with -@? No reaction except for Frank asking to be CC'ed - was it ever fixed??? Do EDK2's or U-Boot's built-in DTs compile with -@ today? Raspberry Pi overlays in U-Boot work because it switched to passing the DT through from the proprietary firmware. Point being that while it would be nice to get a current, compatible DT via UEFI tables and ignore .dtb filenames outside of a few bootloaders, in reality we're not quite there yet for all platforms.
I see no problem (except for naming choices) moving new targets like RDA to subfolders because we can then hardcode it the new way; I also assume deeply embedded targets like stm32f4 or targets with no mainline bootloaders yet like owl-s500 could be refactored. I do see problems refactoring widely used SBC targets like sunXi though.
Regards, Andreas
On Thu, Dec 6, 2018 at 7:32 AM Andreas Färber afaerber@suse.de wrote:
Am 05.12.18 um 05:17 schrieb Rob Herring:
On Tue, Dec 4, 2018 at 7:22 PM Andreas Färber afaerber@suse.de wrote:
Rob,
Am 04.12.18 um 19:36 schrieb Rob Herring:
I've put together a script to move the dts files and update the makefiles. It doesn't handle files not following a common prefix which isn't many and some includes within the dts files will need some fixups by hand.
MAINTAINERS will also need updating.
A few questions:
Do we want to move absolutely everything to subdirs?
This refactoring is a terrible idea!
How do you really feel?
While it would've been nice to have more structure from the start, bootloaders like U-Boot expect a flat structure for arm .dtb files now. If you start installing them into subdirs instead, they won't find the files anymore under the hardcoded name.
Doing this only for new platforms would be much less invasive and allow to prepare bootloaders accordingly.
That was my suggestion where this started for the new RDA platform. Olof preferred to move everything and that's my desire too.
Alternatively, white-list which ones are safe to move around.
I'd prefer to know which ones the distros don't want moved. That should be easier to figure out. We also need that anyways in context of what platforms we care about compatibility.
Another option is dtbs_install target could flatten the installed dtbs. That is the only part the distros should depend on.
I'd be okay with distinguishing source vs. install location. Due to the issue I mention below (and more) we can't use install_dtbs for openSUSE and had to reimplement it, which we'd need to (and can) adjust.
What would be needed for dtbs_install to work? arm64 needs to support a flat install? If it doesn't work for Debian or openSUSE, I'm not sure why we have it. So I'd like to make it work.
But don't just script a refactoring because it looks nicer in the source tree, without testing what side effects this can have for board/distro users of the compiled files in practice. We already had that discussion for arm64 because Debian chose to ignore the kernel-installed subdirectories and installed .dtb files into a flat directory, which collided with openSUSE sticking to the kernel choice.
So everyone already deals with subdirs or not with arm and arm64 already, seems like they can deal with this. I will raise the topic on the cross-distro list though.
Sounds like you're twisting words... The keyword was "hardcoded" paths - one way or another (not "and") depending on the kernel choices being flat for arm, vendor subdir for arm64.
This topic becomes even more important with EBBR: There is neither a mechanism in place to sync .dts files into U-Boot or EDK2 source trees, nor are capsule updates implemented in U-Boot for easily deploying such bootloaders with new .dts sources or paths yet.
EBBR actually says firmware (including dtbs) goes in directories named by vendor.
Fine, but unrelated.
If the distros want dtbs in a flat dir and EBBR says otherwise, then it is related.
And I can assure you that just getting users to dd the right bootloader can be difficult... Since DT forward and backward compatibility is often being neglected, for example with optional properties or renamed compatibles that break booting with previous drivers, new kernel versions often need updated Device Trees to make use of new/enhanced drivers. Therefore it is unfortunately often enough a necessity to load newer kernel-based .dtb files matching the kernel (as opposed to the dream of kernel-independent hardware descriptions) when working with the latest -rc or -next kernels at least. For examples of DTs needing updates, look no further than Linaro's 96boards - in case of hikey960/EDK2 GRUB is another layer where .dtb paths may be hardcoded, ditto for arm; and Armada was an example where the upstream bindings for the network IP changed incompatibly.
Compatibility is an issue, yes, but that really has nothing to do with this.
DT overlays are another topic that is not making any progress upstream according to the ELCE BoF, so beyond the Raspberry Pi the only known working way to apply them is to write a U-Boot boot.scr script, which can either reuse $fdtcontroladdr DT or use the filename $fdtfile or hardcode one, the latter two of which would break with your renaming.
DT overlays also have nothing to do with this as there aren't any in the kernel. I'm not inclined to take any either with a flat tree. We're already at 1800+ files.
Read again: a) Breaking DT changes and b) the desire to use Overlays instead of replacing the bootloaders for each change are _reasons_ why people depend on .dtb filenames from the kernel source tree for their boot flow today. Nothing to do with downstream .dtbo files.
For example, remember when I reported that the kernel didn't compile DTs with -@? No reaction except for Frank asking to be CC'ed - was it ever fixed??? Do EDK2's or U-Boot's built-in DTs compile with -@ today?
IIRC, Frank objected to changing this globally because it will bloat all dtbs. And then no one did the work to make it a per dtb option. Maybe that was the same issue in another thread.
Raspberry Pi overlays in U-Boot work because it switched to passing the DT through from the proprietary firmware. Point being that while it would be nice to get a current, compatible DT via UEFI tables and ignore .dtb filenames outside of a few bootloaders, in reality we're not quite there yet for all platforms.
I see no problem (except for naming choices) moving new targets like RDA to subfolders because we can then hardcode it the new way; I also assume deeply embedded targets like stm32f4 or targets with no mainline bootloaders yet like owl-s500 could be refactored. I do see problems refactoring widely used SBC targets like sunXi though.
Regards, Andreas
-- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg)
boot-architecture@lists.linaro.org
-
Andreas Färber -
Rob Herring