next/master boot: 275 boots: 23 failed, 233 passed with 7 offline, 12 untried/unknown (next-20200316)
next/master boot: 275 boots: 23 failed, 233 passed with 7 offline, 12 untried/unknown (next-20200316)
Full Boot Summary: https://kernelci.org/boot/all/job/next/branch/master/kernel/next-20200316/ Full Build Summary: https://kernelci.org/build/next/branch/master/kernel/next-20200316/
Tree: next Branch: master Git Describe: next-20200316 Git Commit: 8548fd2f20ed19b0e8c0585b71fdfde1ae00ae3c Git URL: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git Tested: 104 unique boards, 23 SoC families, 30 builds out of 329
Boot Regressions Detected:
arm:
exynos_defconfig: gcc-8: exynos5422-odroidxu3: lab-collabora: new failure (last pass: next-20200313)
imx_v4_v5_defconfig: gcc-8: imx27-phytec-phycard-s-rdk: lab-pengutronix: new failure (last pass: next-20191011)
imx_v6_v7_defconfig: gcc-8: imx6ul-pico-hobbit: lab-pengutronix: new failure (last pass: next-20191011)
multi_v5_defconfig: gcc-8: imx27-phytec-phycard-s-rdk: lab-pengutronix: new failure (last pass: next-20191011)
multi_v7_defconfig: gcc-8: imx6ul-pico-hobbit: lab-pengutronix: new failure (last pass: next-20191011) tegra124-nyan-big: lab-collabora: new failure (last pass: next-20200226)
multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y: gcc-8: exynos5422-odroidxu3: lab-collabora: new failure (last pass: next-20200313) rk3288-veyron-jaq: lab-collabora: failing since 2 days (last pass: next-20200312 - first fail: next-20200313) tegra124-nyan-big: lab-collabora: failing since 2 days (last pass: next-20200312 - first fail: next-20200313)
multi_v7_defconfig+CONFIG_SMP=n: gcc-8: imx6ul-pico-hobbit: lab-pengutronix: new failure (last pass: next-20191011) tegra124-nyan-big: lab-collabora: new failure (last pass: next-20200226)
omap2plus_defconfig: gcc-8: omap3-beagle-xm: lab-baylibre: new failure (last pass: next-20200313)
sama5_defconfig: gcc-8: at91-sama5d4_xplained: lab-baylibre: failing since 27 days (last pass: next-20200214 - first fail: next-20200217)
sunxi_defconfig: gcc-8: sun4i-a10-cubieboard: lab-baylibre-seattle: new failure (last pass: next-20200313) sun8i-h2-plus-libretech-all-h3-cc: lab-baylibre: new failure (last pass: next-20200313)
tegra_defconfig: gcc-8: tegra124-nyan-big: lab-collabora: failing since 2 days (last pass: next-20200312 - first fail: next-20200313)
versatile_defconfig: gcc-8: versatile-pb: lab-collabora: new failure (last pass: next-20200313)
vexpress_defconfig: gcc-8: vexpress-v2p-ca15-tc1: lab-collabora: new failure (last pass: next-20200313) lab-baylibre: new failure (last pass: next-20200313) vexpress-v2p-ca9: lab-collabora: new failure (last pass: next-20200313) lab-baylibre: new failure (last pass: next-20200313)
arm64:
defconfig: gcc-8: apq8096-db820c: lab-bjorn: failing since 9 days (last pass: next-20200304 - first fail: next-20200306) bcm2837-rpi-3-b: lab-baylibre: new failure (last pass: next-20200313) meson-axg-s400: lab-baylibre-seattle: new failure (last pass: next-20200313) meson-g12a-sei510: lab-baylibre: new failure (last pass: next-20200313) meson-gxl-s905d-p230: lab-baylibre: failing since 2 days (last pass: next-20200306 - first fail: next-20200313) meson-sm1-sei610: lab-baylibre: new failure (last pass: next-20200313) sun50i-h5-libretech-all-h3-cc: lab-baylibre: new failure (last pass: next-20200313)
defconfig+CONFIG_CPU_BIG_ENDIAN=y: gcc-8: meson-axg-s400: lab-baylibre-seattle: new failure (last pass: next-20200313) meson-gxl-s905x-libretech-cc: lab-clabbe: new failure (last pass: next-20200313)
defconfig+CONFIG_RANDOMIZE_BASE=y: gcc-8: meson-axg-s400: lab-baylibre-seattle: new failure (last pass: next-20200313) meson-gxm-khadas-vim2: lab-baylibre: new failure (last pass: next-20200313) sun50i-h5-libretech-all-h3-cc: lab-baylibre: new failure (last pass: next-20200313)
Boot Failures Detected:
arm: multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y: gcc-8: exynos5422-odroidxu3: 1 failed lab rk3288-veyron-jaq: 1 failed lab tegra124-nyan-big: 1 failed lab
omap2plus_defconfig: gcc-8: omap3-beagle-xm: 1 failed lab
imx_v4_v5_defconfig: gcc-8: imx27-phytec-phycard-s-rdk: 1 failed lab
sama5_defconfig: gcc-8: at91-sama5d4_xplained: 1 failed lab
multi_v5_defconfig: gcc-8: imx27-phytec-phycard-s-rdk: 1 failed lab
multi_v7_defconfig: gcc-8: bcm2836-rpi-2-b: 1 failed lab imx6ul-pico-hobbit: 1 failed lab rk3288-veyron-jaq: 1 failed lab sun4i-a10-cubieboard: 1 failed lab tegra124-nyan-big: 1 failed lab
tegra_defconfig: gcc-8: tegra124-nyan-big: 1 failed lab
sunxi_defconfig: gcc-8: sun4i-a10-cubieboard: 1 failed lab sun8i-h2-plus-libretech-all-h3-cc: 1 failed lab
imx_v6_v7_defconfig: gcc-8: imx6ul-pico-hobbit: 1 failed lab vf610-colibri-eval-v3: 1 failed lab
exynos_defconfig: gcc-8: exynos5422-odroidxu3: 1 failed lab
multi_v7_defconfig+CONFIG_SMP=n: gcc-8: imx6ul-pico-hobbit: 1 failed lab rk3288-veyron-jaq: 1 failed lab tegra124-nyan-big: 1 failed lab
arm64: defconfig+CONFIG_RANDOMIZE_BASE=y: gcc-8: meson-gxm-khadas-vim2: 1 failed lab
defconfig: gcc-8: apq8096-db820c: 1 failed lab
Offline Platforms:
arm:
multi_v7_defconfig: gcc-8 exynos5800-peach-pi: 1 offline lab qcom-apq8064-cm-qs600: 1 offline lab
exynos_defconfig: gcc-8 exynos5800-peach-pi: 1 offline lab
qcom_defconfig: gcc-8 qcom-apq8064-cm-qs600: 1 offline lab
arm64:
defconfig+CONFIG_RANDOMIZE_BASE=y: gcc-8 meson-axg-s400: 1 offline lab
defconfig: gcc-8 meson-axg-s400: 1 offline lab
defconfig+CONFIG_CPU_BIG_ENDIAN=y: gcc-8 meson-axg-s400: 1 offline lab
--- For more info write to info@kernelci.org
This boot report shows the issue with failing to track some regressions that I mentioned on another thread last week:
On 16/03/2020 13:24, kernelci.org bot wrote:
next/master boot: 275 boots: 23 failed, 233 passed with 7 offline, 12 untried/unknown (next-20200316)
Full Boot Summary: https://kernelci.org/boot/all/job/next/branch/master/kernel/next-20200316/ Full Build Summary: https://kernelci.org/build/next/branch/master/kernel/next-20200316/
Tree: next Branch: master Git Describe: next-20200316 Git Commit: 8548fd2f20ed19b0e8c0585b71fdfde1ae00ae3c Git URL: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git Tested: 104 unique boards, 23 SoC families, 30 builds out of 329
Boot Regressions Detected:
arm:
[...]
2 regressions were detected for multi_v7_defconfig:
multi_v7_defconfig: gcc-8: imx6ul-pico-hobbit: lab-pengutronix: new failure (last pass: next-20191011) tegra124-nyan-big: lab-collabora: new failure (last pass: next-20200226)
[...]
Boot Failures Detected:
arm:
[...]
However, all these platforms failed to boot:
multi_v7_defconfig: gcc-8: bcm2836-rpi-2-b: 1 failed lab imx6ul-pico-hobbit: 1 failed lab rk3288-veyron-jaq: 1 failed lab sun4i-a10-cubieboard: 1 failed lab tegra124-nyan-big: 1 failed lab
There was no mention of conflicting results, in fact the rk3388-veyron-jaq and tegra124-nyan-big are only in Collabora's lab so they can't have conflicts with other labs. And we know these platforms did boot fine at some point in the past, picking a random earlier job:
https://kernelci.org/boot/id/5e2167ead92d42035d74e98a/
As such, they should have been tracked as a regression. It's true that some previous boot test results are missing for multi_v7_defconfig on next/master, probably due to the Jenkins-Docker issue we had in production for a few weeks. Still, that shouldn't be a good enough reason to not detect a regression. I've noticed other similar and more obvious cases in the past, with platforms that started failing but no regression was detected, and bisections not being triggered as a result.
This issue is going away now with the move to have everything as test results and no boot results any more. I'm keeping a close eye on test case regressions and so far they seem to be working in a reliable way.
Guillaume
On Tue, Mar 17, 2020 at 11:50:56AM +0000, Guillaume Tucker wrote:
There was no mention of conflicting results, in fact the rk3388-veyron-jaq and tegra124-nyan-big are only in Collabora's lab so they can't have conflicts with other labs. And we know these platforms did boot fine at some point in the past, picking a random earlier job:
Ah, I think I was only reading that as "new regressions" rather than just plain regressions.
kernel-build-reports@lists.linaro.org
-
Guillaume Tucker -
kernelci.org bot -
Mark Brown