Yeah sure, I am looking at this.

Here the timeout happens after 1200 seconds without any new output on stdout/stderr. I reproduce the issue on tcwg-g3-01 with -j34 (the default on this machine): the build is stuck near the end when building flang objects. The same build succeeds using -j16 on this machine. So I wonder if this could simply be due to insufficient memory.

I will relaunch a few builds to check if something significant changed with last commits.

KO * 0245dc94ebee [test][sanitizers] Disable new test on Android ?? * 795f93007e10 [AArch64] Add test for #62620. ?? * 4fe91e083a25 [llvm-profdata] ProfileReader cleanup - preparation for MD5 refactoring - 3 ?? * be5747e51693 [flang] Fixed global name creation for literal constants. OK * 26a7f4230613 [mlir][tosa] Add accumulator type attribute to TOSA dialect

Antoine

On Mon, 15 May 2023 at 22:37, Maxim Kuvyrkov maxim.kuvyrkov@linaro.org wrote:

Thanks, Andrzej!

Hi Antoine, would you please take a look at this?

Kind regards,

-- Maxim Kuvyrkov https://www.linaro.org

...

On May 15, 2023, at 22:55, Andrzej Warzynski andrzej.warzynski@arm.com

wrote:

...

Hi,

Has anyone noticed that https://lab.llvm.org/buildbot/#/builders/198

(clang-aarch64-sve-vla-2stagehttps://lab.llvm.org/buildbot/#/builders/198) has been timing out for the past few days? "Duration" is often less than 1hr, so that's odd. And all Flang buildbots are green, so it's unlikely caused by changes to that sub-project ( https://lab.llvm.org/buildbot/#/builders/198/builds/1804). Would anyone be able to take a look?

...

Best regards,

Andrzej

IMPORTANT NOTICE: The contents of this email and any attachments are

confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

...

---

linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-leave@lists.linaro.org

+ @Kiran Chandramohanmailto:Kiran.Chandramohan@arm.com

Kiran suggested that this might be this recent change in MLIR that’s increased build times: https://github.com/llvm/llvm-project/issues/62687

-Andrzej

Thanks for looking into this!

Based on the logs and your experiment, looks like it's Flang's fault. 64GB used to be sufficient, but perhaps that's no longer the case (sadly, Flang has insane requirements). I will ask internally.

If this is indeed Flang's fault, I'd be tempted to simply disable it on this bot. We know that that sub-project is tested quite extensively through dedicated Flang bots. What do you think?

-Andrzej On 16/05/2023 07:23, Antoine Moynault wrote: Yeah sure, I am looking at this.

Here the timeout happens after 1200 seconds without any new output on stdout/stderr. I reproduce the issue on tcwg-g3-01 with -j34 (the default on this machine): the build is stuck near the end when building flang objects. The same build succeeds using -j16 on this machine. So I wonder if this could simply be due to insufficient memory.

I will relaunch a few builds to check if something significant changed with last commits. KO * 0245dc94ebee [test][sanitizers] Disable new test on Android ?? * 795f93007e10 [AArch64] Add test for #62620. ?? * 4fe91e083a25 [llvm-profdata] ProfileReader cleanup - preparation for MD5 refactoring - 3 ?? * be5747e51693 [flang] Fixed global name creation for literal constants. OK * 26a7f4230613 [mlir][tosa] Add accumulator type attribute to TOSA dialect

Antoine

On Mon, 15 May 2023 at 22:37, Maxim Kuvyrkov <maxim.kuvyrkov@linaro.orgmailto:maxim.kuvyrkov@linaro.org> wrote: Thanks, Andrzej!

Hi Antoine, would you please take a look at this?

Kind regards,

-- Maxim Kuvyrkov https://www.linaro.org

On May 15, 2023, at 22:55, Andrzej Warzynski <andrzej.warzynski@arm.commailto:andrzej.warzynski@arm.com> wrote:

Hi,

Has anyone noticed that https://lab.llvm.org/buildbot/#/builders/198 (clang-aarch64-sve-vla-2stagehttps://lab.llvm.org/buildbot/#/builders/198) has been timing out for the past few days? "Duration" is often less than 1hr, so that's odd. And all Flang buildbots are green, so it's unlikely caused by changes to that sub-project (https://lab.llvm.org/buildbot/#/builders/198/builds/1804). Would anyone be able to take a look?

Best regards,

Andrzej

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.orgmailto:linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-leave@lists.linaro.orgmailto:linaro-toolchain-leave@lists.linaro.org

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Great detective work, thank you for fixing this :)

-Andrzej

On 16/05/2023 16:55, Antoine Moynault wrote:

Yeah thanks, it could have explained this indeed. But I guess bots would be green again if it was linked to this mlir issue 62687, as the fix for it was merged several hours ago and part of these jobs: https://lab.llvm.org/buildbot/#/builders/198/builds/1879 https://lab.llvm.org/buildbot/#/builders/176/builds/2357

Actually Maxim probably found the issue: The swap was not enabled on these g3 machines, which is a problem as the build of flang (-j34) now needs a bit more than the 58GB of RAM allocated for the buildbot container. I also understand that a custom ninja binary is used on these containers, behaving differently when processes are waiting for something (memory/cpu). Enabling the swap on these linaro-g3 machines will probably fix the bot failures.

And yes indeed, building flang looks to require a lot of memory: ~60Go RAM for a full build (j34) vs <20Go RAM for a build without flang !

Antoine

On Tue, 16 May 2023 at 10:56, Andrzej Warzynski <Andrzej.Warzynski@arm.commailto:Andrzej.Warzynski@arm.com> wrote:

+ @Kiran Chandramohanmailto:Kiran.Chandramohan@arm.com

Kiran suggested that this might be this recent change in MLIR that’s increased build times: https://github.com/llvm/llvm-project/issues/62687

-Andrzej

Thanks for looking into this!

Based on the logs and your experiment, looks like it's Flang's fault. 64GB used to be sufficient, but perhaps that's no longer the case (sadly, Flang has insane requirements). I will ask internally.

If this is indeed Flang's fault, I'd be tempted to simply disable it on this bot. We know that that sub-project is tested quite extensively through dedicated Flang bots. What do you think?

-Andrzej On 16/05/2023 07:23, Antoine Moynault wrote: Yeah sure, I am looking at this.

Here the timeout happens after 1200 seconds without any new output on stdout/stderr. I reproduce the issue on tcwg-g3-01 with -j34 (the default on this machine): the build is stuck near the end when building flang objects. The same build succeeds using -j16 on this machine. So I wonder if this could simply be due to insufficient memory.

I will relaunch a few builds to check if something significant changed with last commits. KO * 0245dc94ebee [test][sanitizers] Disable new test on Android ?? * 795f93007e10 [AArch64] Add test for #62620. ?? * 4fe91e083a25 [llvm-profdata] ProfileReader cleanup - preparation for MD5 refactoring - 3 ?? * be5747e51693 [flang] Fixed global name creation for literal constants. OK * 26a7f4230613 [mlir][tosa] Add accumulator type attribute to TOSA dialect

Antoine

On Mon, 15 May 2023 at 22:37, Maxim Kuvyrkov <maxim.kuvyrkov@linaro.orgmailto:maxim.kuvyrkov@linaro.org> wrote: Thanks, Andrzej!

Hi Antoine, would you please take a look at this?

Kind regards,

-- Maxim Kuvyrkov https://www.linaro.org

On May 15, 2023, at 22:55, Andrzej Warzynski <andrzej.warzynski@arm.commailto:andrzej.warzynski@arm.com> wrote:

Hi,

Has anyone noticed that https://lab.llvm.org/buildbot/#/builders/198 (clang-aarch64-sve-vla-2stagehttps://lab.llvm.org/buildbot/#/builders/198) has been timing out for the past few days? "Duration" is often less than 1hr, so that's odd. And all Flang buildbots are green, so it's unlikely caused by changes to that sub-project (https://lab.llvm.org/buildbot/#/builders/198/builds/1804). Would anyone be able to take a look?

Best regards,

Andrzej

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.orgmailto:linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-leave@lists.linaro.orgmailto:linaro-toolchain-leave@lists.linaro.org

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.