Someone asked if this worked, and I thought 'that's trivial to test with multiarch' so I did. On Saucy (where there is no multiarch version skew issue between binary versions of packages) the dpkg --add-architecutre armhf apt-get update apt-get install links:armhf part works very nicely. Everything installs as required.
However binaries don't run - they just get killed.
Apparently no-one has tried this for a couple of years since it was last working...(Or is it working on other platforms? - apparently it's OK on android)
Turns out that our arm64 kernel config has: vm.mmap_min_addr=65536 but armhf binaries tend to get mmapped at 0x8000 (32K).
On armhf that value is set to vm.mmap_min_addr=4096
This difference is to protect page0 even if large pages are enabled which seems sensible enough, but has this unfortunate side-effect.
So either we need to stop doing that (What would be the consequences of setting 4096 by default on arm64?) or change something in the loader to stop mapping things below 64K, which I think involves glibc hackery.
Running 32-bit binaries is quite seriously broken until this is fixed. I presume this currently isn't on anyone's list to fix? I'm not sure who's list it should go on.
part2: Once this is fixed with: sudo sysctl -w vm.mmap_min_addr=4096
some binaries work (hello, bzip2) but fancier things still don't (links, wget). They segfault after loading libs. I'm still investigating that, but it looks like we have at least two issues..
So this mail is really to ask what the best fix is and thus who will deal with it? Do I need to file a bug or a card somewhere?
Possibly more to follow when I work out what else is wrong...
Wookey
On Wednesday 29 January 2014 16:36:49 Wookey wrote:
Running 32-bit binaries is quite seriously broken until this is fixed. I presume this currently isn't on anyone's list to fix? I'm not sure who's list it should go on.
Are you running with 4KB or 64KB page size in the kernel? IIRC you cannot really run 32-bit binaries with 64KB page size because of related issues.
Arnd
+++ Arnd Bergmann [2014-01-29 18:14 +0100]:
On Wednesday 29 January 2014 16:36:49 Wookey wrote:
Running 32-bit binaries is quite seriously broken until this is fixed. I presume this currently isn't on anyone's list to fix? I'm not sure who's list it should go on.
Are you running with 4KB or 64KB page size in the kernel? IIRC you cannot really run 32-bit binaries with 64KB page size because of related issues.
I'm not sure. The kernel config has: CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y # CONFIG_ARM64_64K_PAGES is not set CONFIG_PAGEFLAGS_EXTENDED=y # CONFIG_TRANSPARENT_HUGEPAGE is not set
Is that using 64K pages or not?
Is there a dynamic switch I can twiddle or does it need a kernel rebuild?
Wookey
On Wednesday 29 January 2014 17:40:54 Wookey wrote:
+++ Arnd Bergmann [2014-01-29 18:14 +0100]:
On Wednesday 29 January 2014 16:36:49 Wookey wrote:
Running 32-bit binaries is quite seriously broken until this is fixed. I presume this currently isn't on anyone's list to fix? I'm not sure who's list it should go on.
Are you running with 4KB or 64KB page size in the kernel? IIRC you cannot really run 32-bit binaries with 64KB page size because of related issues.
I'm not sure. The kernel config has: CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y # CONFIG_ARM64_64K_PAGES is not set CONFIG_PAGEFLAGS_EXTENDED=y # CONFIG_TRANSPARENT_HUGEPAGE is not set
Is that using 64K pages or not?
This is 4K pages, so I was on the wrong track here.
Is there a dynamic switch I can twiddle or does it need a kernel rebuild?
If you turn on CONFIG_ARM64_64K_PAGES, you probably won't be able to load any 32-bit executables any more, or more will fail.
Also, I guess vm.mmap_min_addr becomes settable only in 64K increments, which would not be helpful for you.
I also found this setting in the kernel:
config DEFAULT_MMAP_MIN_ADDR int "Low address space to protect from user allocation" depends on MMU default 4096 help This is the portion of low virtual memory which should be protected from userspace allocation. Keeping a user from writing to low pages can help reduce the impact of kernel NULL pointer bugs.
For most ia64, ppc64 and x86 users with lots of address space a value of 65536 is reasonable and should cause no problems. On arm and other archs it should not be higher than 32768. Programs which use vm86 functionality or have some need to map this low address space will need CAP_SYS_RAWIO or disable this protection by setting the value to 0.
This value can be changed after boot using the /proc/sys/vm/mmap_min_addr tunable.
Arnd
On Wed, Jan 29, 2014 at 12:34 PM, Arnd Bergmann arnd@arndb.de wrote:
On Wednesday 29 January 2014 17:40:54 Wookey wrote:
+++ Arnd Bergmann [2014-01-29 18:14 +0100]:
On Wednesday 29 January 2014 16:36:49 Wookey wrote:
Running 32-bit binaries is quite seriously broken until this is fixed. I presume this currently isn't on anyone's list to fix? I'm not sure who's list it should go on.
Are you running with 4KB or 64KB page size in the kernel? IIRC you cannot really run 32-bit binaries with 64KB page size because of related issues.
I'm not sure. The kernel config has: CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y # CONFIG_ARM64_64K_PAGES is not set CONFIG_PAGEFLAGS_EXTENDED=y # CONFIG_TRANSPARENT_HUGEPAGE is not set
Is that using 64K pages or not?
This is 4K pages, so I was on the wrong track here.
Is there a dynamic switch I can twiddle or does it need a kernel rebuild?
If you turn on CONFIG_ARM64_64K_PAGES, you probably won't be able to load any 32-bit executables any more, or more will fail.
You definitely need 64K pages disabled and you also need CONFIG_COMPAT=y.
FWIW, I have 64-bit QEMU system level emulation working, but 32-bit userspace is not working for me.
Rob
On 29 January 2014 08:36, Wookey wookey@wookware.org wrote:
Someone asked if this worked, and I thought 'that's trivial to test with multiarch' so I did. On Saucy (where there is no multiarch version skew issue between binary versions of packages) the dpkg --add-architecutre armhf apt-get update apt-get install links:armhf part works very nicely. Everything installs as required.
However binaries don't run - they just get killed.
Apparently no-one has tried this for a couple of years since it was last working...(Or is it working on other platforms? - apparently it's OK on android)
FYI: It seems OK with OE rootfs file systems too. Just booted 64 system with Linaro OE rootfs, nfs mounted similar rootfs but for 32bit, set up symlink for /lib/ld-linux-armhf.so.3, set correct LD_LIBRARY_PATH - don't see any issues with running 32 binaries so far .. tried few executables. Works in BE variant as well.
I was running 3.13-rc3 kernel.
Thanks, Victor
Turns out that our arm64 kernel config has: vm.mmap_min_addr=65536 but armhf binaries tend to get mmapped at 0x8000 (32K).
On armhf that value is set to vm.mmap_min_addr=4096
This difference is to protect page0 even if large pages are enabled which seems sensible enough, but has this unfortunate side-effect.
So either we need to stop doing that (What would be the consequences of setting 4096 by default on arm64?) or change something in the loader to stop mapping things below 64K, which I think involves glibc hackery.
Running 32-bit binaries is quite seriously broken until this is fixed. I presume this currently isn't on anyone's list to fix? I'm not sure who's list it should go on.
part2: Once this is fixed with: sudo sysctl -w vm.mmap_min_addr=4096
some binaries work (hello, bzip2) but fancier things still don't (links, wget). They segfault after loading libs. I'm still investigating that, but it looks like we have at least two issues..
So this mail is really to ask what the best fix is and thus who will deal with it? Do I need to file a bug or a card somewhere?
Possibly more to follow when I work out what else is wrong...
Wookey
Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
+++ Wookey [2014-01-29 16:36 +0000]:
Someone asked if this worked, and I thought 'that's trivial to test with multiarch' so I did. On Saucy (where there is no multiarch version skew issue between binary versions of packages) the dpkg --add-architecutre armhf apt-get update apt-get install links:armhf part works very nicely. Everything installs as required.
However binaries don't run - they just get killed.
Turns out that our arm64 kernel config has: vm.mmap_min_addr=65536 but armhf binaries tend to get mmapped at 0x8000 (32K).
This has been fixed in the default kernel configs now.
part2: Once this is fixed
some binaries work (hello, bzip2) but fancier things still don't (links, wget). They segfault after loading libs. I'm still investigating that, but it looks like we have at least two issues..
So after rather a lot of faffing round related issues I got to the bottom of this problem which turned out to be 'kernel too old'. The Linaro Foundation 3.8 kernels I was using doesn't work in this regard. Changing to last month's release (and munging the kernel command line with the very cool kcmt utility works fine).
So, I've now uploaded A saucy armhf/arm64 configured multiarch image and tarball to: http://people.debian.org/~wookey/bootstrap/rootfs/
2G disk image: (as 92MB file) http://people.debian.org/~wookey/bootstrap/rootfs/saucy-arm64-multiarch.img.... Corresponding kernel: http://people.debian.org/~wookey/bootstrap/rootfs/linux-system-foundation.ax... tarball of same rootfs: http://people.debian.org/~wookey/bootstrap/rootfs/saucy-multiarch.tar.gz
All linked on https://wiki.debian.org/Arm64Port#Pre-built_Rootfses
Wookey
-
Arnd Bergmann -
Rob Herring -
Victor Kamensky -
Wookey