Hi Siddhesh,

This was due to a bug in our bisection scripts. I've disabled notifications to patch authors and linaro-toolchain@ while I'm testing the fix.

Sorry for the noise.

-- Maxim Kuvyrkov https://www.linaro.org

On Jul 11, 2021, at 1:56 PM, ci_notify@linaro.org wrote:

Successfully identified regression in *glibc* in CI configuration tcwg_cross/gnu-master-aarch64-build_cross. So far, this commit has regressed CI configurations:

• tcwg_cross/gnu-master-aarch64-build_cross

Culprit:

<cut> commit fab3a6daf137c734e684471e197148aa058102e3 Author: Siddhesh Poyarekar <siddhesh@sourceware.org> Date: Wed Jul 7 13:25:49 2021 +0530

tests-exclude-mcheck: Fix typo

It's tst-realloc, not tst-posix-realloc. Verified this time to ensure that the total number of tests reduced by 1.

Reported-by: Stefan Liebler stli@linux.ibm.com

</cut>

Results regressed to (for first_bad == fab3a6daf137c734e684471e197148aa058102e3) # reset_artifacts: -10 # true: 0 # build_abe binutils: 1 # build_abe stage1: 2 # build_abe linux: 3 # First few build errors in logs: # 00:00:14 make[2]: *** [/home/tcwg-buildslave/workspace/tcwg_gnu_1/abe/builds/x86_64-pc-linux-gnu/aarch64-linux-gnu/glibc-glibc.git~master/sysd-rules:499: /home/tcwg-buildslave/workspace/tcwg_gnu_1/abe/builds/x86_64-pc-linux-gnu/aarch64-linux-gnu/glibc-glibc.git~master/csu/start.os] Error 1 # 00:00:14 make[2]: *** [/home/tcwg-buildslave/workspace/tcwg_gnu_1/abe/builds/x86_64-pc-linux-gnu/aarch64-linux-gnu/glibc-glibc.git~master/sysd-rules:483: /home/tcwg-buildslave/workspace/tcwg_gnu_1/abe/builds/x86_64-pc-linux-gnu/aarch64-linux-gnu/glibc-glibc.git~master/csu/crtn.o] Error 1 # 00:00:14 make[2]: *** [/home/tcwg-buildslave/workspace/tcwg_gnu_1/abe/builds/x86_64-pc-linux-gnu/aarch64-linux-gnu/glibc-glibc.git~master/sysd-rules:483: /home/tcwg-buildslave/workspace/tcwg_gnu_1/abe/builds/x86_64-pc-linux-gnu/aarch64-linux-gnu/glibc-glibc.git~master/csu/start.o] Error 1 # 00:00:14 make[2]: *** [/home/tcwg-buildslave/workspace/tcwg_gnu_1/abe/builds/x86_64-pc-linux-gnu/aarch64-linux-gnu/glibc-glibc.git~master/sysd-rules:483: /home/tcwg-buildslave/workspace/tcwg_gnu_1/abe/builds/x86_64-pc-linux-gnu/aarch64-linux-gnu/glibc-glibc.git~master/csu/crti.o] Error 1 # 00:00:14 make[2]: *** [../o-iterator.mk:9: /home/tcwg-buildslave/workspace/tcwg_gnu_1/abe/builds/x86_64-pc-linux-gnu/aarch64-linux-gnu/glibc-glibc.git~master/csu/check_fds.os] Error 1 # 00:00:14 make[2]: *** [../o-iterator.mk:9: /home/tcwg-buildslave/workspace/tcwg_gnu_1/abe/builds/x86_64-pc-linux-gnu/aarch64-linux-gnu/glibc-glibc.git~master/csu/check_fds.o] Error 1 # 00:00:14 ../sysdeps/aarch64/nptl/tls.h:91:19: error: ‘__builtin_thread_pointer’ is not supported on this target # 00:00:14 make[2]: *** [/home/tcwg-buildslave/workspace/tcwg_gnu_1/abe/builds/x86_64-pc-linux-gnu/aarch64-linux-gnu/glibc-glibc.git~master/sysd-rules:485: /home/tcwg-buildslave/workspace/tcwg_gnu_1/abe/builds/x86_64-pc-linux-gnu/aarch64-linux-gnu/glibc-glibc.git~master/csu/libc-tls.o] Error 1 # 00:00:14 ../sysdeps/unix/sysv/linux/aarch64/sysdep.h:190:17: error: invalid register name for ‘_x0’ # 00:00:14 ../sysdeps/unix/sysv/linux/aarch64/sysdep.h:174:22: error: invalid register name for ‘_x8’

from (for last_good == 7a5db2e82fbb6c3a6e3fdae02b7166c5d0e8c7a8) # reset_artifacts: -10 # true: 0 # build_abe binutils: 1 # build_abe stage1: 2 # build_abe linux: 3 # build_abe glibc: 4 # build_abe stage2: 5 # build_abe qemu: 6

Artifacts of last_good build: https://ci.linaro.org/job/tcwg_cross-bisect-gnu-master-aarch64-build_cross/3... Artifacts of first_bad build: https://ci.linaro.org/job/tcwg_cross-bisect-gnu-master-aarch64-build_cross/3... Build top page/logs: https://ci.linaro.org/job/tcwg_cross-bisect-gnu-master-aarch64-build_cross/3...

Configuration details:

Reproduce builds:

<cut> mkdir investigate-glibc-fab3a6daf137c734e684471e197148aa058102e3 cd investigate-glibc-fab3a6daf137c734e684471e197148aa058102e3

git clone https://git.linaro.org/toolchain/jenkins-scripts

mkdir -p artifacts/manifests curl -o artifacts/manifests/build-baseline.sh https://ci.linaro.org/job/tcwg_cross-bisect-gnu-master-aarch64-build_cross/3... --fail curl -o artifacts/manifests/build-parameters.sh https://ci.linaro.org/job/tcwg_cross-bisect-gnu-master-aarch64-build_cross/3... --fail curl -o artifacts/test.sh https://ci.linaro.org/job/tcwg_cross-bisect-gnu-master-aarch64-build_cross/3... --fail chmod +x artifacts/test.sh

# Reproduce the baseline build (build all pre-requisites) ./jenkins-scripts/tcwg_gnu-build.sh @@ artifacts/manifests/build-baseline.sh

cd glibc

# Reproduce first_bad build git checkout --detach fab3a6daf137c734e684471e197148aa058102e3 ../artifacts/test.sh

# Reproduce last_good build git checkout --detach 7a5db2e82fbb6c3a6e3fdae02b7166c5d0e8c7a8 ../artifacts/test.sh

cd ..

</cut>

History of pending regressions and results: https://git.linaro.org/toolchain/ci/base-artifacts.git/log/?h=linaro-local/c...

Artifacts: https://ci.linaro.org/job/tcwg_cross-bisect-gnu-master-aarch64-build_cross/3... Build log: https://ci.linaro.org/job/tcwg_cross-bisect-gnu-master-aarch64-build_cross/3...

Full commit (up to 1000 lines):

<cut> commit fab3a6daf137c734e684471e197148aa058102e3 Author: Siddhesh Poyarekar <siddhesh@sourceware.org> Date: Wed Jul 7 13:25:49 2021 +0530

tests-exclude-mcheck: Fix typo

It's tst-realloc, not tst-posix-realloc. Verified this time to ensure that the total number of tests reduced by 1.

Reported-by: Stefan Liebler stli@linux.ibm.com

malloc/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/malloc/Makefile b/malloc/Makefile index 2af1203a0a..37a9a4efab 100644 --- a/malloc/Makefile +++ b/malloc/Makefile @@ -94,7 +94,7 @@ tests-exclude-mcheck = tst-mallocstate \ tst-malloc_info \ tst-memalign \ tst-posix_memalign \

• tst-posix-realloc

• tst-realloc

tests-mcheck = $(filter-out $(tests-exclude-mcheck), $(tests))

</cut>