[CI] ManyShardsIT testCancelUnnecessaryRequests failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #7349 / openjdk23_entitlements_false_checkpart3_java-matrix
• elasticsearch-pull-request #65500 / part-3
• elasticsearch-periodic-platform-support #7252 / almalinux-9_platform-support-unix

Reproduction Line:

./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.action.ManyShardsIT.testCancelUnnecessaryRequests" -Dtests.seed=2B36B5E6A61236E2 -Dtests.jvm.argline="-Des.entitlements.enabled=false" -Dtests.locale=tt-RU -Dtests.timezone=America/Belize -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: safeGet: listener was completed exceptionally

Issue Reasons:

• [main] 3 failures in test testCancelUnnecessaryRequests (0.3% fail rate in 871 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testCancelUnnecessaryRequests (0.3% fail rate in 871 executions)

Build Scans:

• elasticsearch-periodic #7349 / openjdk23_entitlements_false_checkpart3_java-matrix
• elasticsearch-pull-request #65500 / part-3
• elasticsearch-periodic-platform-support #7252 / almalinux-9_platform-support-unix

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[alex-spies]

I can see

 Caused by: java.util.concurrent.ExecutionException: org.elasticsearch.action.NoShardAvailableActionException: no such shard
 at org.elasticsearch.action.support.PlainActionFuture$Sync.getValue(PlainActionFuture.java:279)
 at org.elasticsearch.action.support.PlainActionFuture$Sync.get(PlainActionFuture.java:253)
 at org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:74)
 at org.elasticsearch.test.ESTestCase.safeGet(ESTestCase.java:2533)
 ... 44 more

I don't see any WARN logs for this test that would immediately explain what happened.

@dnhatn , git blame has your name (and only your name) plastered over this test suite :D So I hope it's okay to throw this your way - do you have an idea of this failure's risk level?

[idegtiarenko]

I looked on this test and noticed that we execute the query immediately after the new data node has started. Please note, we do not wait for all shard migrations to complete.

It would really help if we have stack trace from NoShardAvailableActionException: no such shard, but I suspect we do not handle well the case when shard moves to another node between can match and sending data request call.

If such cases are expected to be retried on the client, I could add a wait for no shard movements.
Otherwise, it is a bit trickier case to fix.

[idegtiarenko]

I can also add logging to confirm this theory.