[CI] SearchWithRandomDisconnectsIT testSearchWithRandomDisconnects failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #11566 / graalvm-ce17_checkpart1_java-matrix
• elasticsearch-periodic-platform-support #11539 / oraclelinux-9_platform-support-unix
• elasticsearch-periodic-platform-support #11526 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-periodic-platform-support #11521 / almalinux-8_platform-support-unix

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.search.basic.SearchWithRandomDisconnectsIT.testSearchWithRandomDisconnects" -Dtests.seed=8BB8480BCB83781E -Dtests.locale=doi -Dtests.timezone=America/Boa_Vista -Druntime.java=17

Applicable branches:
8.19

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: timed out waiting for green state

Issue Reasons:

• [8.19] 4 failures in test testSearchWithRandomDisconnects (2.4% fail rate in 167 executions)
• [8.19] 3 failures in pipeline elasticsearch-periodic-platform-support (60.0% fail rate in 5 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testSearchWithRandomDisconnects (0.3% fail rate in 948 executions)
• [main] 2 failures in step part-1 (0.3% fail rate in 595 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (0.3% fail rate in 592 executions)

Build Scans:

• elasticsearch-periodic-platform-support #11421 / amazonlinux-2023_platform-support-aws
• elasticsearch-pull-request #103810 / part-1
• elasticsearch-pull-request #103072 / part-1

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[elasticsearchmachine]

This has been muted on branch 9.2

Mute Reasons:

• [9.2] 2 failures in test testSearchWithRandomDisconnects (1.6% fail rate in 122 executions)
• [9.2] 2 failures in step part1 (9.1% fail rate in 22 executions)
• [9.2] 2 failures in pipeline elasticsearch-intake (9.1% fail rate in 22 executions)

Build Scans:

• elasticsearch-intake #31379 / part1
• elasticsearch-intake #31241 / part1

[benchaplin]

I unmuted this recently after fixing the bug causing it to fail (#122707).

There are several failures here. Most look to be related to cluster management, e.g.

java.lang.AssertionError: java.lang.RuntimeException: Cluster is already closed	
	at __randomizedtesting.SeedInfo.seed([98A9691449CF97F5]:0)	
	at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:427)	
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:330)	
	at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:355)	
	at org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:267)	
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)	
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)	
	at org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:423)	
	at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:234)	
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)	
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)	
	at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)	
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:360)	
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)	
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)	
	at org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:423)	
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)	
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)	
	at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)	
	at org.elasticsearch.action.search.TransportSearchAction$SearchTelemetryListener.onFailure(TransportSearchAction.java:2123)	
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)	
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)	
	at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)	
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:360)	
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)	
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)	
	at org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:423)	
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncAction.java:647)	
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:624)	
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:326)	
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:669)	
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.finishOneShard(AbstractSearchAsyncAction.java:430)	
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:422)	
	at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:279)	
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:54)	
	at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:654)	
	at org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1735)	
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1515)	
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1649)	
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1624)	
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44)	
	at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:45)	
	at org.elasticsearch.search.SearchService.lambda$wrapListenerForErrorHandling$4(SearchService.java:636)	
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:202)	
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)	
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:207)	
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:152)	
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:29)	
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)	
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1088)	
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)	
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)	
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)	
	at java.base/java.lang.Thread.run(Thread.java:1474)	
Caused by: java.lang.RuntimeException: Cluster is already closed	
	at org.elasticsearch.test.InternalTestCluster.ensureOpen(InternalTestCluster.java:666)	
	at org.elasticsearch.test.InternalTestCluster.getRandomNodeAndClient(InternalTestCluster.java:693)	
	at org.elasticsearch.test.InternalTestCluster.client(InternalTestCluster.java:869)	
	at org.elasticsearch.test.ESIntegTestCase.client(ESIntegTestCase.java:670)	
	at org.elasticsearch.test.ESIntegTestCase.client(ESIntegTestCase.java:663)	
	at org.elasticsearch.test.ESIntegTestCase.prepareSearch(ESIntegTestCase.java:1294)	
	at org.elasticsearch.search.basic.SearchWithRandomDisconnectsIT.prepareRandomSearch(SearchWithRandomDisconnectsIT.java:105)	
	at org.elasticsearch.search.basic.SearchWithRandomDisconnectsIT$1.runMoreSearches(SearchWithRandomDisconnectsIT.java:75)	
	at org.elasticsearch.search.basic.SearchWithRandomDisconnectsIT$1.onFailure(SearchWithRandomDisconnectsIT.java:69)	
	at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:418)	
	... 52 more	
	Suppressed: Failed to execute phase [dfs], all shards failed; shardFailures {[_na_][test-0][0]: org.elasticsearch.action.NoShardAvailableActionException	
}{[_na_][test-1][0]: org.elasticsearch.action.NoShardAvailableActionException	
}{[_na_][test-10][0]: org.elasticsearch.action.NoShardAvailableActionException	
}{[8xDN0JrORE-CPDK72B09kw][test-11][0]: org.elasticsearch.action.NoShardAvailableActionException: [node_s1][127.0.0.1:27432][indices:data/read/search[phase/dfs]]	
}{[8xDN0JrORE-CPDK72B09kw][test-2][0]: org.elasticsearch.action.NoShardAvailableActionException: [node_s1][127.0.0.1:27432][indices:data/read/search[phase/dfs]]	
}{[_na_][test-3][0]: org.elasticsearch.action.NoShardAvailableActionException	
}{[_na_][test-4][0]: org.elasticsearch.action.NoShardAvailableActionException	
}{[8xDN0JrORE-CPDK72B09kw][test-5][0]: org.elasticsearch.action.NoShardAvailableActionException: [node_s1][127.0.0.1:27432][indices:data/read/search[phase/dfs]]	
}{[_na_][test-6][0]: org.elasticsearch.action.NoShardAvailableActionException	
}{[_na_][test-7][0]: org.elasticsearch.action.NoShardAvailableActionException	
}{[8xDN0JrORE-CPDK72B09kw][test-8][0]: org.elasticsearch.action.NoShardAvailableActionException: [node_s1][127.0.0.1:27432][indices:data/read/search[phase/dfs]]	
}{[_na_][test-9][0]: org.elasticsearch.action.NoShardAvailableActionException	
}	
		... 26 more	
	Caused by: org.elasticsearch.action.NoShardAvailableActionException

This leads me to believe these current issues are unrelated to batched query execution, and simply weren't caught as the test was muted for ~9 months. It will take some time to investigate, but I'll mark as low-risk for now as this appears to stem from test code.

[elasticsearchmachine]

This has been muted on branch 9.1

Mute Reasons:

• [9.1] 2 failures in test testSearchWithRandomDisconnects (1.9% fail rate in 108 executions)

Build Scans:

• elasticsearch-intake #31740 / part1
• elasticsearch-periodic #11493 / concurrent-search-tests