[BUILD] Race condition during build/testing for check-clang/check-clang-tools

[AaronBallman]

[6099/6110] Running clang_tools regression tests
llvm-lit.py: C:\_work\llvm-project\llvm-project\llvm\utils\lit\lit\llvm\config.py:57: note: using lit tools: C:\Program Files\Git\usr\bin
llvm-lit.py: C:\_work\llvm-project\llvm-project\llvm\utils\lit\lit\llvm\config.py:520: note: using clang: c:\_work\llvm-project\llvm-project\build\bin\clang.exe
llvm-lit.py: C:\_work\llvm-project\llvm-project\llvm\utils\lit\lit\llvm\config.py:312: fatal: Couldn't find the include dir for Clang ('c:\_work\llvm-project\llvm-project\build\bin\clang.exe')
c:\_work\llvm-project\llvm-project\build\bin\clang.exe
[6101/6110] Linking CXX executable bin\obj2yaml.exe
FAILED: tools/clang/tools/extra/CMakeFiles/check-clang-tools 
cmd.exe /C "cd /D C:\_work\llvm-project\llvm-project\build\tools\clang\tools\extra && C:\Python39\python.exe C:/_work/llvm-project/llvm-project/build/./bin/llvm-lit.py -v --xunit-xml-output C:/_work/llvm-project/llvm-project/build/test-results.xml --use-unique-output-file-name --timeout=1200 --time-tests C:/_work/llvm-project/llvm-project/build/tools/clang/tools/extra/include-cleaner/test C:/_work/llvm-project/llvm-project/build/tools/clang/tools/extra/clangd/test/../unittests C:/_work/llvm-project/llvm-project/build/tools/clang/tools/extra/clangd/test C:/_work/llvm-project/llvm-project/build/tools/clang/tools/extra/test"
[6103/6110] Linking CXX executable bin\clang.exe

It looks like we try to run the clang-tools-extra regression tests which seem to be looking for clang.exe which isn't yet built.

Some examples of this in the wild include:
https://github.com/llvm/llvm-project/actions/runs/15875229617/job/44761115591?pr=145679
https://github.com/llvm/llvm-project/actions/runs/15860875108/job/44717506023?pr=143424

[llvmbot]

@llvm/issue-subscribers-infrastructure

Author: Aaron Ballman (AaronBallman)

``` [6099/6110] Running clang_tools regression tests llvm-lit.py: C:\_work\llvm-project\llvm-project\llvm\utils\lit\lit\llvm\config.py:57: note: using lit tools: C:\Program Files\Git\usr\bin llvm-lit.py: C:\_work\llvm-project\llvm-project\llvm\utils\lit\lit\llvm\config.py:520: note: using clang: c:\_work\llvm-project\llvm-project\build\bin\clang.exe llvm-lit.py: C:\_work\llvm-project\llvm-project\llvm\utils\lit\lit\llvm\config.py:312: fatal: Couldn't find the include dir for Clang ('c:\_work\llvm-project\llvm-project\build\bin\clang.exe') c:\_work\llvm-project\llvm-project\build\bin\clang.exe [6101/6110] Linking CXX executable bin\obj2yaml.exe FAILED: tools/clang/tools/extra/CMakeFiles/check-clang-tools cmd.exe /C "cd /D C:\_work\llvm-project\llvm-project\build\tools\clang\tools\extra && C:\Python39\python.exe C:/_work/llvm-project/llvm-project/build/./bin/llvm-lit.py -v --xunit-xml-output C:/_work/llvm-project/llvm-project/build/test-results.xml --use-unique-output-file-name --timeout=1200 --time-tests C:/_work/llvm-project/llvm-project/build/tools/clang/tools/extra/include-cleaner/test C:/_work/llvm-project/llvm-project/build/tools/clang/tools/extra/clangd/test/../unittests C:/_work/llvm-project/llvm-project/build/tools/clang/tools/extra/clangd/test C:/_work/llvm-project/llvm-project/build/tools/clang/tools/extra/test" [6103/6110] Linking CXX executable bin\clang.exe ```

It looks like we try to run the clang-tools-extra regression tests which seem to be looking for clang.exe which isn't yet built.

Some examples of this in the wild include:
https://github.com/llvm/llvm-project/actions/runs/15875229617/job/44761115591?pr=145679
https://github.com/llvm/llvm-project/actions/runs/15860875108/job/44717506023?pr=143424

[boomanaiden154]

@cmtice @lnihlen Is the current on call able to investigate this? It could be relatively recent and might just be a CMake missing dependency, but impacts reliability a decent amount.

Not entirely sure it’s in scope for the rotation though given it’s not an infrastructure issue.

[lnihlen]

@Keenuts is primary this week. I'll try to take a look in a bit myself.

[Keenuts]

Looks like the issue was already existing? But since #144033 seems like the CI now runs this, hence the failure showing

[Keenuts]

Not sure why this is happening yet. But what the flakiness I observe, even locally on linux tells me there is some race-condition

[Keenuts]

Hi, I renamed the issuer to reflect what this is about.
I haven't found the culprit, but at least we know this is not an infrastructure issue.

This can be reproduced locally on linux (if lucky since flaky) on a clean build with:

cmake -S ./llvm -B repro -D 'LLVM_ENABLE_PROJECTS=clang;clang-tools-extra;llvm' -G Ninja -D CMAKE_BUILD_TYPE=Release -D LLVM_ENABLE_ASSERTIONS=ON -D LLVM_BUILD_EXAMPLES=ON -D COMPILER_RT_BUILD_LIBFUZZER=OFF -D 'LLVM_LIT_ARGS=-v --use-unique-output-file-name --timeout=1200 --time-tests' -D COMPILER_RT_BUILD_ORC=OFF -D MLIR_ENABLE_BINDINGS_PYTHON=ON -D LLVM_PARALLEL_COMPILE_JOBS=64 -D LLVM_PARALLEL_LINK_JOBS=64 && ninja -C repro check-clang check-clang-tools