On 23.02.2024 15:24, ci_notify@linaro.org wrote:
Dear contributor, our automatic CI has detected problems related to your patch(es). Please find some details below. If you have any questions, please follow up on linaro-toolchain@lists.linaro.org mailing list, Libera's #linaro-tcwg channel, or ping your favourite Linaro toolchain developer on the usual project channel.
We appreciate that it might be difficult to find the necessary logs or reproduce the issue locally. If you can't get what you need from our CI within minutes, let us know and we will be happy to help.
So: The specific failure observed makes me guess that in the course of building generated files (opcodes/aarch64-*-2.c) aren't re-generated. If this is indeed the case, sending out such unsolicited emails (I've had a 2nd one in the meantime, for whatever reason) is a waste of everybody's time, even more so when not clearly indicating that fact, such that - quoting above - it is possible "within minutes" to understand what's going on / wrong.
IOW - unless clarified I'm going to assume that the report here is a false negative.
Jan
In binutils_check master-aarch64 after:
| 6 patches in binutils | Patchwork URL: https://patchwork.sourceware.org/patch/86276 | 2d039d3b147 gas/NEWS: drop mention of Arm64's SVE2.1 and SME2.1 | 38eca7294bd Arm64: correct SVE2.1 ld2q (scalar plus scalar) | cb804986590 Arm64: correct SVE2.1 ld{3,4}q / st{3,4}q (scalar plus immediate) | edaf01ca3b4 Arm64: check tied operand specifier in aarch64-gen | 006ce007bc8 Arm64: check matching operands for predicated B16B16 insns | ... and 1 more patches in binutils | ... applied on top of baseline commit: | e346d50a891 x86: rename vec_encoding and vex_encoding_*
FAIL: 1 regressions
regressions.sum: === gas tests ===
Running gas:gas/aarch64/aarch64.exp ... FAIL: Test of SVE2.1 instructions
You can find the failure logs in *.log.1.xz files in
The full lists of regressions and progressions as well as configure and make commands are in
The list of [ignored] baseline and flaky failures are in
The configuration of this build is: CI config tcwg_binutils_check master-aarch64
-----------------8<--------------------------8<--------------------------8<-------------------------- The information below can be used to reproduce a debug environment:
Current build : https://ci.linaro.org/job/tcwg_binutils_check--master-aarch64-precommit/1099... Reference build : https://ci.linaro.org/job/tcwg_binutils_check--master-aarch64-build/797/arti...
Hi Jan,
On Mon, 26 Feb 2024 at 09:05, Jan Beulich jbeulich@suse.com wrote:
On 23.02.2024 15:24, ci_notify@linaro.org wrote:
Dear contributor, our automatic CI has detected problems related to your patch(es). Please find some details below. If you have any questions, please follow up on linaro-toolchain@lists.linaro.org mailing list, Libera's #linaro-tcwg channel, or ping your favourite Linaro toolchain developer on the usual project channel.
We appreciate that it might be difficult to find the necessary logs or reproduce the issue locally. If you can't get what you need from our CI within minutes, let us know and we will be happy to help.
So: The specific failure observed makes me guess that in the course of building generated files (opcodes/aarch64-*-2.c) aren't re-generated. If this is indeed the case, sending out such unsolicited emails (I've had a 2nd one in the meantime, for whatever reason) is a waste of everybody's time, even more so when not clearly indicating that fact, such that
- quoting above - it is possible "within minutes" to understand what's
going on / wrong.
IOW - unless clarified I'm going to assume that the report here is a false negative.
Sorry if these messages weren't helpful.
You are right, the reason for the failures is that these aarch64 files are not regenerated, because our CI does not currently enable maintainer-mode.
We've known that for some time already, and I started working on that recently. Unfortunately, it turns out it's not as simple as adding --enable-maintainer-mode in our build script. I've been chasing random build errors with maintainer-mode enable, at this stage I suspect a race condition in binutils' top-level Makefile leading to regenerating some files several times concurrently, leading to corrupt output. Hopefully I'll manage to track this down and fix it. Otherwise we'd send random regression notifications, which would be even more confusing.
Note that we want to enable maintainer-mode in "precommit" CI only, not in the postcommit one, were we want to build & check patches as they were committed to the repo, not "as they should have been".
Regarding the 2 messages you received, you can notice that they point to 2 different patchwork threads: https://patchwork.sourceware.org/project/binutils/list/?series=31217 https://patchwork.sourceware.org/project/binutils/list/?series=31252
so ISTM it's normal that the CI started 2 independent validations.
We do have several things in place to avoid sending duplicate notifications.
Thanks for your message, it helps us confirm what we have to improve.
Christophe
Jan
In binutils_check master-aarch64 after:
| 6 patches in binutils | Patchwork URL: https://patchwork.sourceware.org/patch/86276 | 2d039d3b147 gas/NEWS: drop mention of Arm64's SVE2.1 and SME2.1 | 38eca7294bd Arm64: correct SVE2.1 ld2q (scalar plus scalar) | cb804986590 Arm64: correct SVE2.1 ld{3,4}q / st{3,4}q (scalar plus immediate) | edaf01ca3b4 Arm64: check tied operand specifier in aarch64-gen | 006ce007bc8 Arm64: check matching operands for predicated B16B16 insns | ... and 1 more patches in binutils | ... applied on top of baseline commit: | e346d50a891 x86: rename vec_encoding and vex_encoding_*
FAIL: 1 regressions
regressions.sum: === gas tests ===
Running gas:gas/aarch64/aarch64.exp ... FAIL: Test of SVE2.1 instructions
You can find the failure logs in *.log.1.xz files in
The full lists of regressions and progressions as well as configure and make commands are in
The list of [ignored] baseline and flaky failures are in
The configuration of this build is: CI config tcwg_binutils_check master-aarch64
-----------------8<--------------------------8<--------------------------8<-------------------------- The information below can be used to reproduce a debug environment:
Current build : https://ci.linaro.org/job/tcwg_binutils_check--master-aarch64-precommit/1099... Reference build : https://ci.linaro.org/job/tcwg_binutils_check--master-aarch64-build/797/arti...
linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-leave@lists.linaro.org
On 26.02.2024 10:08, Christophe Lyon wrote:
On Mon, 26 Feb 2024 at 09:05, Jan Beulich jbeulich@suse.com wrote:
On 23.02.2024 15:24, ci_notify@linaro.org wrote:
Dear contributor, our automatic CI has detected problems related to your patch(es). Please find some details below. If you have any questions, please follow up on linaro-toolchain@lists.linaro.org mailing list, Libera's #linaro-tcwg channel, or ping your favourite Linaro toolchain developer on the usual project channel.
We appreciate that it might be difficult to find the necessary logs or reproduce the issue locally. If you can't get what you need from our CI within minutes, let us know and we will be happy to help.
So: The specific failure observed makes me guess that in the course of building generated files (opcodes/aarch64-*-2.c) aren't re-generated. If this is indeed the case, sending out such unsolicited emails (I've had a 2nd one in the meantime, for whatever reason) is a waste of everybody's time, even more so when not clearly indicating that fact, such that
- quoting above - it is possible "within minutes" to understand what's
going on / wrong.
IOW - unless clarified I'm going to assume that the report here is a false negative.
Sorry if these messages weren't helpful.
You are right, the reason for the failures is that these aarch64 files are not regenerated, because our CI does not currently enable maintainer-mode.
We've known that for some time already, and I started working on that recently. Unfortunately, it turns out it's not as simple as adding --enable-maintainer-mode in our build script. I've been chasing random build errors with maintainer-mode enable, at this stage I suspect a race condition in binutils' top-level Makefile leading to regenerating some files several times concurrently, leading to corrupt output. Hopefully I'll manage to track this down and fix it. Otherwise we'd send random regression notifications, which would be even more confusing.
Right, and I try to stay away from maintainer mode, too, I have to confess. Instead I carry a local patch removing just a few of the @MAINT@ that are getting in the way.
In any event, may I ask that it be clarified in the emails that failure may be due to the lack of re-generation of files, until such time that the issue is under control?
Note that we want to enable maintainer-mode in "precommit" CI only, not in the postcommit one, were we want to build & check patches as they were committed to the repo, not "as they should have been".
Of course.
Regarding the 2 messages you received, you can notice that they point to 2 different patchwork threads: https://patchwork.sourceware.org/project/binutils/list/?series=31217 https://patchwork.sourceware.org/project/binutils/list/?series=31252
so ISTM it's normal that the CI started 2 independent validations.
Hmm, no. The 2nd was about https://patchwork.sourceware.org/patch/86275 which is https://patchwork.sourceware.org/patch/86276 short of one patch. So still odd.
The first of the two links you provided is actually an entirely different series by a different author.
Jan
On Mon, 26 Feb 2024 at 10:41, Jan Beulich jbeulich@suse.com wrote:
On 26.02.2024 10:08, Christophe Lyon wrote:
On Mon, 26 Feb 2024 at 09:05, Jan Beulich jbeulich@suse.com wrote:
On 23.02.2024 15:24, ci_notify@linaro.org wrote:
Dear contributor, our automatic CI has detected problems related to your patch(es). Please find some details below. If you have any questions, please follow up on linaro-toolchain@lists.linaro.org mailing list, Libera's #linaro-tcwg channel, or ping your favourite Linaro toolchain developer on the usual project channel.
We appreciate that it might be difficult to find the necessary logs or reproduce the issue locally. If you can't get what you need from our CI within minutes, let us know and we will be happy to help.
So: The specific failure observed makes me guess that in the course of building generated files (opcodes/aarch64-*-2.c) aren't re-generated. If this is indeed the case, sending out such unsolicited emails (I've had a 2nd one in the meantime, for whatever reason) is a waste of everybody's time, even more so when not clearly indicating that fact, such that
- quoting above - it is possible "within minutes" to understand what's
going on / wrong.
IOW - unless clarified I'm going to assume that the report here is a false negative.
Sorry if these messages weren't helpful.
You are right, the reason for the failures is that these aarch64 files are not regenerated, because our CI does not currently enable maintainer-mode.
We've known that for some time already, and I started working on that recently. Unfortunately, it turns out it's not as simple as adding --enable-maintainer-mode in our build script. I've been chasing random build errors with maintainer-mode enable, at this stage I suspect a race condition in binutils' top-level Makefile leading to regenerating some files several times concurrently, leading to corrupt output. Hopefully I'll manage to track this down and fix it. Otherwise we'd send random regression notifications, which would be even more confusing.
Right, and I try to stay away from maintainer mode, too, I have to confess. Instead I carry a local patch removing just a few of the @MAINT@ that are getting in the way.
In any event, may I ask that it be clarified in the emails that failure may be due to the lack of re-generation of files, until such time that the issue is under control?
Sure, I'll have a look at adding that.
Note that we want to enable maintainer-mode in "precommit" CI only, not in the postcommit one, were we want to build & check patches as they were committed to the repo, not "as they should have been".
Of course.
Regarding the 2 messages you received, you can notice that they point to 2 different patchwork threads: https://patchwork.sourceware.org/project/binutils/list/?series=31217 https://patchwork.sourceware.org/project/binutils/list/?series=31252
so ISTM it's normal that the CI started 2 independent validations.
Hmm, no. The 2nd was about https://patchwork.sourceware.org/patch/86275 which is https://patchwork.sourceware.org/patch/86276 short of one patch. So still odd.
The first of the two links you provided is actually an entirely different series by a different author.
Ha sorry, I was confused because the 2 notifications I mentioned appear in the same thread in my mailer.
So... what our precommit CI does in presence of a patch series is build and check first with the full series, then repeat the process by removing the top-most patches one by one. The reason for this is that we want to detect cases when a patch in the middle of a series would break things (fixed later in the series), because it can misleading when doing bisections.
Thanks,
Christophe
Jan
linaro-toolchain@lists.linaro.org
-
Christophe Lyon -
Jan Beulich