next/master boot: 257 boots: 8 failed, 237 passed with 8 offline, 2 untried/unknown, 2 conflicts (next-20191028)
next/master boot: 257 boots: 8 failed, 237 passed with 8 offline, 2 untried/unknown, 2 conflicts (next-20191028)
Full Boot Summary: https://kernelci.org/boot/all/job/next/branch/master/kernel/next-20191028/ Full Build Summary: https://kernelci.org/build/next/branch/master/kernel/next-20191028/
Tree: next Branch: master Git Describe: next-20191028 Git Commit: 60c1769a45f4b6beddcc48843739d7d41b88dc1c Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git Tested: 93 unique boards, 26 SoC families, 28 builds out of 216
Boot Failures Detected:
arm64: defconfig+CONFIG_RANDOMIZE_BASE=y: gcc-8: meson-gxl-s805x-libretech-ac: 1 failed lab meson-sm1-khadas-vim3l: 1 failed lab meson-sm1-sei610: 1 failed lab
defconfig: gcc-8: apq8096-db820c: 1 failed lab meson-sm1-khadas-vim3l: 1 failed lab meson-sm1-sei610: 1 failed lab
defconfig+CONFIG_CPU_BIG_ENDIAN=y: gcc-8: meson-sm1-khadas-vim3l: 1 failed lab meson-sm1-sei610: 1 failed lab
Offline Platforms:
riscv:
defconfig: gcc-8 sifive_fu540: 1 offline lab
arm:
multi_v7_defconfig: gcc-8 qcom-apq8064-cm-qs600: 1 offline lab sun5i-r8-chip: 1 offline lab sun7i-a20-bananapi: 1 offline lab
sunxi_defconfig: gcc-8 sun5i-r8-chip: 1 offline lab sun7i-a20-bananapi: 1 offline lab
davinci_all_defconfig: gcc-8 dm365evm,legacy: 1 offline lab
qcom_defconfig: gcc-8 qcom-apq8064-cm-qs600: 1 offline lab
Conflicting Boot Failures Detected: (These likely are not failures as other labs are reporting PASS. Needs review.)
arm: exynos_defconfig: exynos5422-odroidxu3: lab-collabora: FAIL (gcc-8) lab-baylibre: PASS (gcc-8)
arm64: defconfig: meson-gxl-s905x-libretech-cc: lab-baylibre-seattle: PASS (gcc-8) lab-clabbe: FAIL (gcc-8) lab-baylibre: PASS (gcc-8)
--- For more info write to info@kernelci.org
On Mon, Oct 28, 2019 at 08:03:08AM -0700, kernelci.org bot wrote:
Today's -next (anf Friday's) fails to boot on db820c:
defconfig: gcc-8: apq8096-db820c: 1 failed lab
It looks like it deadlocks somewhere, the last things in the log are a failure to start ufshcd-qcom and then an RCU stall some time later:
03:03:27.191914 [ 21.156672] ufshcd-qcom 624000.ufshc: ufshcd_populate_vreg: Unable to find vdd-hba-supply regulator, assuming enabled 03:03:27.198061 <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=qup-i2c-driver-present RESULT=pass> 03:03:27.208499 [ 21.175985] ufshcd-qcom 624000.ufshc: ufs_qcom_init: required phy device. hasn't probed yet. err = -517 03:03:27.216720 [ 21.176014] ufshcd-qcom 624000.ufshc: ufshcd_variant_hba_init: variant qcom init failed err -517 03:03:27.226220 <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=qup-i2c-blsp-i2c2-probed RESULT=pass> 03:03:27.239850 [ 21.211424] ufshcd-qcom 624000.ufshc: Initialization failed 03:03:48.157338 [ 42.128777] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: 03:03:48.167648 [ 42.128802] rcu: 3-...!: (0 ticks this GP) idle=dde/1/0x4000000000000000 softirq=1715/1715 fqs=60 03:03:48.171895 [ 42.133839] (detected by 0, t=5252 jiffies, g=2301, q=787)
Full details, including the whole log, at:
On Mon 28 Oct 10:48 PDT 2019, Mark Brown wrote:
Hi Mark
On Mon, Oct 28, 2019 at 08:03:08AM -0700, kernelci.org bot wrote:
Today's -next (anf Friday's) fails to boot on db820c:
defconfig: gcc-8: apq8096-db820c: 1 failed labIt looks like it deadlocks somewhere, the last things in the log are a failure to start ufshcd-qcom and then an RCU stall some time later:
db820c has been failing intermittently for a while now, it seems that booting with kpti enabled causes something to go wrong. There are nothing strange in the kernel logs and ftrace seems to indicate that all the CPUs are idling nicely.
Regards, Bjorn
03:03:27.191914 [ 21.156672] ufshcd-qcom 624000.ufshc: ufshcd_populate_vreg: Unable to find vdd-hba-supply regulator, assuming enabled 03:03:27.198061 <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=qup-i2c-driver-present RESULT=pass> 03:03:27.208499 [ 21.175985] ufshcd-qcom 624000.ufshc: ufs_qcom_init: required phy device. hasn't probed yet. err = -517 03:03:27.216720 [ 21.176014] ufshcd-qcom 624000.ufshc: ufshcd_variant_hba_init: variant qcom init failed err -517 03:03:27.226220 <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=qup-i2c-blsp-i2c2-probed RESULT=pass> 03:03:27.239850 [ 21.211424] ufshcd-qcom 624000.ufshc: Initialization failed 03:03:48.157338 [ 42.128777] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: 03:03:48.167648 [ 42.128802] rcu: 3-...!: (0 ticks this GP) idle=dde/1/0x4000000000000000 softirq=1715/1715 fqs=60 03:03:48.171895 [ 42.133839] (detected by 0, t=5252 jiffies, g=2301, q=787)
Full details, including the whole log, at:
On Mon, Oct 28, 2019 at 11:40:19AM -0700, Bjorn Andersson wrote:
On Mon 28 Oct 10:48 PDT 2019, Mark Brown wrote:
On Mon, Oct 28, 2019 at 08:03:08AM -0700, kernelci.org bot wrote:
Today's -next (anf Friday's) fails to boot on db820c:
defconfig: gcc-8: apq8096-db820c: 1 failed lab
It looks like it deadlocks somewhere, the last things in the log are a failure to start ufshcd-qcom and then an RCU stall some time later:
db820c has been failing intermittently for a while now, it seems that booting with kpti enabled causes something to go wrong. There are nothing strange in the kernel logs and ftrace seems to indicate that all the CPUs are idling nicely.
Oh dear. Adding Catalin and Will. Is it definitely KPTI that's triggering stuff? It did turn up some bugs on other systems, though it's a bit strange it's only manifesting in KernelCI...
On Mon 28 Oct 12:11 PDT 2019, Mark Brown wrote:
On Mon, Oct 28, 2019 at 11:40:19AM -0700, Bjorn Andersson wrote:
On Mon 28 Oct 10:48 PDT 2019, Mark Brown wrote:
On Mon, Oct 28, 2019 at 08:03:08AM -0700, kernelci.org bot wrote:
Today's -next (anf Friday's) fails to boot on db820c:
defconfig: gcc-8: apq8096-db820c: 1 failed labIt looks like it deadlocks somewhere, the last things in the log are a failure to start ufshcd-qcom and then an RCU stall some time later:
db820c has been failing intermittently for a while now, it seems that booting with kpti enabled causes something to go wrong. There are nothing strange in the kernel logs and ftrace seems to indicate that all the CPUs are idling nicely.
Oh dear. Adding Catalin and Will. Is it definitely KPTI that's triggering stuff? It did turn up some bugs on other systems, though it's a bit strange it's only manifesting in KernelCI...
I did a test recently where I booted my db820c 100 times with kpti=yes and 100 times with kpti=no on the kernel command line, and the result was 90% failure to reach console vs 0%. Going back and looking at the logs for the 10% indicated that the boot CPU was fine, but I had stalls reported on other CPUs.
In an effort to rule out driver bugs I reduced the DT to CPUs, the core clocks, gic, timers and serial driver, and I still saw the problem.
I have not looked at this with jtag and hence do not know what secure world is doing.
Regards, Bjorn
On Mon, Oct 28, 2019 at 01:02:19PM -0700, Bjorn Andersson wrote:
On Mon 28 Oct 12:11 PDT 2019, Mark Brown wrote:
On Mon, Oct 28, 2019 at 11:40:19AM -0700, Bjorn Andersson wrote:
On Mon 28 Oct 10:48 PDT 2019, Mark Brown wrote:
On Mon, Oct 28, 2019 at 08:03:08AM -0700, kernelci.org bot wrote:
Today's -next (anf Friday's) fails to boot on db820c:
defconfig: gcc-8: apq8096-db820c: 1 failed labIt looks like it deadlocks somewhere, the last things in the log are a failure to start ufshcd-qcom and then an RCU stall some time later:
db820c has been failing intermittently for a while now, it seems that booting with kpti enabled causes something to go wrong. There are nothing strange in the kernel logs and ftrace seems to indicate that all the CPUs are idling nicely.
Oh dear. Adding Catalin and Will. Is it definitely KPTI that's triggering stuff? It did turn up some bugs on other systems, though it's a bit strange it's only manifesting in KernelCI...
I did a test recently where I booted my db820c 100 times with kpti=yes and 100 times with kpti=no on the kernel command line, and the result was 90% failure to reach console vs 0%. Going back and looking at the logs for the 10% indicated that the boot CPU was fine, but I had stalls reported on other CPUs.
In an effort to rule out driver bugs I reduced the DT to CPUs, the core clocks, gic, timers and serial driver, and I still saw the problem.
I have not looked at this with jtag and hence do not know what secure world is doing.
Hmm. Is this a recent thing? Neither kpti nor the snapdragon 820 are particular new. Might be worth checking that CONFIG_QCOM_FALKOR_ERRATUM_1003 is enabled and getting patched in at runtime -- we had hardware issues during kpti development with this CPU.
Will
On Mon, Oct 28, 2019 at 08:14:18PM +0000, Will Deacon wrote:
Hmm. Is this a recent thing? Neither kpti nor the snapdragon 820 are
It's in mainline, don't seem to have any results from Bjorn's lab for stable.
particular new. Might be worth checking that CONFIG_QCOM_FALKOR_ERRATUM_1003 is enabled and getting patched in at runtime -- we had hardware issues during kpti development with this CPU.
It's enabled in defconfig:
https://storage.kernelci.org/next/master/next-20191028/arm64/defconfig/gcc-8...
but I can't see any sign that it's been announced in the boot log:
https://storage.kernelci.org/next/master/next-20191028/arm64/defconfig/gcc-8...
so that might be it.
kernel-build-reports@lists.linaro.org
-
Bjorn Andersson -
kernelci.org bot -
Mark Brown -
Will Deacon