[CI] CoordinatorTests testAllSearchesExecuted failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #7455 / zulu8_java-matrix
• elasticsearch-periodic-platform-support #7344 / ubuntu-2404_platform-support-unix

Reproduction Line:

./gradlew ':x-pack:plugin:enrich:test' --tests "org.elasticsearch.xpack.enrich.action.CoordinatorTests.testAllSearchesExecuted" -Dtests.seed=383D3FDDCAFA585A -Dtests.locale=et -Dtests.timezone=America/Costa_Rica -Druntime.java=8

Applicable branches:
7.17

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: null

Issue Reasons:

• [7.17] 2 failures in test testAllSearchesExecuted (0.3% fail rate in 730 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[cbuescher]

I cannot reproduce locally so far. Will try a bit more on a linux box with the exact JVM since the failures are all in the matrix tests.

Test fails here waiting for a task completion countdown latch to reach 0:
https://github.com/elastic/elasticsearch/blob/7.17/x-pack/plugin/enrich/src/test/java/org/elasticsearch/xpack/enrich/action/CoordinatorTests.java#L382

The timeout has been increased to 20sec already at some point in the past but that seems ample time to finish.

Failure history shows that this test only failed on 7.17 in the last three months, only twice in total, so its not muted. Marking as low risk for that reason as I don't believe there have been any significant code changes recently in that area. I will dig a bit into whether there is anything else in the build scans that would explain machine slowness and take a look at the commit history around the time of the first failures that happened here on March 30th.