On Tue, Aug 25, 2015 at 9:15 AM, Tyler Baker tyler.baker@linaro.org wrote:

Hi,

On 25 August 2015 at 05:55, kernelci.org bot bot@kernelci.org wrote:

...

*next boot: 179 boots: 37 failed, 136 passed with 4 offline, 2 conflicts (next-20150825)* Full Boot Summary: http://kernelci.org/boot/all/job/next/kernel/next-20150825/ Full Build Summary: http://kernelci.org/build/next/kernel/next-20150825/ Tree: next Branch: local/master Git Describe: next-20150825 Git Commit: 512450361ea46ed5a8c5d81f507fef094181ab91 Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=512450361ea46ed5a8c5d81f507fef094181ab91 Tested: 56 unique boards, 15 SoC families, 22 builds out of 132 *Boot Failures Detected: http://kernelci.org/boot/?next-20150825&fail http://kernelci.org/boot/?next-20150825&fail* arm: multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y: am335x-boneblack http://kernelci.org/boot/am335x-boneblack/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab at91-sama5d3_xplained http://kernelci.org/boot/at91-sama5d3_xplained/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab exynos4412-odroidx2 http://kernelci.org/boot/exynos4412-odroidx2/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab exynos5250-arndale http://kernelci.org/boot/exynos5250-arndale/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab exynos5250-snow http://kernelci.org/boot/exynos5250-snow/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab exynos5420-arndale-octa http://kernelci.org/boot/exynos5420-arndale-octa/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab exynos5422-odroidxu3 http://kernelci.org/boot/exynos5422-odroidxu3/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 2 failed labs exynos5800-peach-pi http://kernelci.org/boot/exynos5800-peach-pi/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab hip04-d01 http://kernelci.org/boot/hip04-d01/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab imx6q-cm-fx6 http://kernelci.org/boot/imx6q-cm-fx6/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab imx6q-sabrelite http://kernelci.org/boot/imx6q-sabrelite/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab imx6q-wandboard http://kernelci.org/boot/imx6q-wandboard/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab omap3-beagle-xm http://kernelci.org/boot/omap3-beagle-xm/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab omap4-panda http://kernelci.org/boot/omap4-panda/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab omap4-panda-es http://kernelci.org/boot/omap4-panda-es/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab qcom-apq8064-ifc6410 http://kernelci.org/boot/qcom-apq8064-ifc6410/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab qcom-apq8084-ifc6540 http://kernelci.org/boot/qcom-apq8084-ifc6540/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab sun7i-a20-bananapi http://kernelci.org/boot/sun7i-a20-bananapi/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab sun7i-a20-cubietruck http://kernelci.org/boot/sun7i-a20-cubietruck/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab sun9i-a80-cubieboard4 http://kernelci.org/boot/sun9i-a80-cubieboard4/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab sun9i-a80-optimus http://kernelci.org/boot/sun9i-a80-optimus/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab tegra124-jetson-tk1 http://kernelci.org/boot/tegra124-jetson-tk1/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 2 failed labs vexpress-v2p-ca15-tc1 http://kernelci.org/boot/vexpress-v2p-ca15-tc1/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab vexpress-v2p-ca15_a7 http://kernelci.org/boot/vexpress-v2p-ca15_a7/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab vexpress-v2p-ca9 http://kernelci.org/boot/vexpress-v2p-ca9/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab zynq-parallella http://kernelci.org/boot/zynq-parallella/job/next/kernel/next-20150825/defconfig/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/: 1 failed lab

I bisected these CONFIG_THUMB2_KERNEL boot failures:

0db805aa8c96f0eac0cb5bc6688999212fb5f5d3 is the first bad commit commit 0db805aa8c96f0eac0cb5bc6688999212fb5f5d3 Author: Russell King rmk+kernel@arm.linux.org.uk Date: Wed Aug 19 20:40:41 2015 +0100

ARM: software-based priviledged-no-access support

Provide a software-based implementation of the priviledged no access
support found in ARMv8.1.

Userspace pages are mapped using a different domain number from the
kernel and IO mappings.  If we switch the user domain to "no access"
when we enter the kernel, we can prevent the kernel from touching
userspace.

However, the kernel needs to be able to access userspace via the
various user accessor functions.  With the wrapping in the previous
patch, we can temporarily enable access when the kernel needs user
access, and re-disable it afterwards.

This allows us to trap non-intended accesses to userspace, eg, caused
by an inadvertent dereference of the LIST_POISON* values, which, with
appropriate user mappings setup, can be made to succeed.  This in turn
can allow use-after-free bugs to be further exploited than would
otherwise be possible.

Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

FYI... The davinci_defconfig fails (on da850-evm and dm365) also bisected to this commit.

Kevin

On Wed, Aug 26, 2015 at 12:30:51AM +0100, Russell King - ARM Linux wrote:

From what I can tell from the oops dumps you have, the failure is in padzero calling clear_user, which then calls __clear_user_std. That path already changes the domain register to make userspace accessible; we can see that in the register values in the oops dump, where r4 is 0x15 and r3 is 0x55. This follows the assembly I have here, where r3 will be written to the DACR, followed by an ISB, then a call to __clear_user_std before restoring it to the state saved in r4.

I wonder whether we need something stronger there, maybe a DSB to prevent the accesses trying to occur before the DACR update. I'll need to read the ARM or talk to Will about that.

I finally found the cause of it - it comes down to the crappy idea that assembler macros which depend on a config option should be done like this:

#ifdef CONFIG_symbol .macro foo ... .endm .macro bar ... .endm #else .macro foo ... .endm .macro bar ... .endm #endif

On the face of it, it looks like a good idea, but it _really_ isn't, especially when we end up with it getting to:

#ifdef CONFIG_symbol .macro foo ... .endm .macro bar ... .endm .macro baz ... .endm #else .macro foo ... .endm .macro bar ... .endm #ifdef CONFIG_symbol2 .macro baz ... .endm #else .macro baz ... .endm #endif #endif

which is exactly what we have for the svc_exit, svc_exit_with_fiq and restore_user macros in entry-header.S.

It's no wonder, therefore, that the Thumb2 SVC exit paths totally missed out on being updated with the DACR restore. So, what was happening in these failure cases was:

- CPU sets DACR to allow userspace access in padzero - CPU drops into arm_clear_user - CPU writes to userspace, but page is not present. - CPU triggers data abort, saves state (including DACR), disables user access, and processes the exception - CPU returns from data abort processing, but _omits_ to restore the DACR. - CPU retries the write to userspace, and hits a domain fault because the DACR is now set to deny access to userspace.

To prevent this kind of oversight in the future, the only sane way to handle this is to ensure locality of definition. The best way to ensure that is not by the above crappy structure, but by:

.macro foo ... common bits ... #ifdef CONFIG_symbol ... #else ... #endif .endm .macro bar #ifdef CONFIG_symbol ... #else ... #endif .endm .macro baz #ifdef CONFIG_symbol ... #elif defined(CONFIG_symbol2) ... #else ... #endif .endm

which (a) has the effect of keeping multiple definitions of the same macro in close proximity, and (b) allows common code (such as in the case of svc_exit) to be kept in exactly one place and not spread across multiple definitions - which is exactly what leads to stuff being missed when it comes to update it.

Now, if I can stop silly git commit from deleting the above preprocessor lines in my commit message which illustrate the problem, I'll push out the updates. (# is such a wonderful comment prefix, except when you want to quote some C preprocessor in the commit message.)