[CI] FieldSortIT testSortMixedFieldTypes failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #9199 / rhel-8_platform-support-unix
• elasticsearch-pull-request #78601 / part-1

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.search.sort.FieldSortIT.testSortMixedFieldTypes" -Dtests.seed=26C0313418883E0F -Dtests.locale=cv-RU -Dtests.timezone=Australia/Tasmania -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: an empty collection
     but: <[LEAK: resource was not cleaned up before it was garbage-collected.
Recent access records: 
#1:
	in [elasticsearch[node_s2][search][T#2]][testSortMixedFieldTypes]
	org.elasticsearch.action.ActionListener.respondAndRelease(ActionListener.java:388)
	org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
	org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
	org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
	org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
	org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
[truncated]

Issue Reasons:

• [main] 2 failures in test testSortMixedFieldTypes (0.3% fail rate in 654 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[cbuescher]

Test history suggests this test suite has been flaky on 8.19 (8.x) and main since the end of march

All test methods are, symptoms seem to be timeouts, java.lang.AssertionError: java.lang.IllegalStateException: Attempted to close with partial reduce in-flight and "java.lang.AssertionError: invalid decRef call: already closed". Need to dig into those failures a bit more to see if they look related.

[cbuescher]

One of the very early ones from March 31st already show Leak issues related to "testSortMixedFieldTypes"

https://gradle-enterprise.elastic.co/s/aqlr4gxvmas4o

Will spot-check some others to be sure this is a main cause

[cbuescher]

All failures I spot checked had either a Leak message with access in "testSortMixedFieldTypes" or a java.lang.IllegalStateException: Attempted to close with partial reduce in-flight in "testSortMixedFieldTypes" in the logs somewhere, e.g. here

[benwtrent]

This failed in my PR: https://buildkite.com/elastic/elasticsearch-pull-request/builds/78601#0197cb1c-8cc8-48f0-8bca-30fdeeebd656

 ./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.search.sort.FieldSortIT.testSortMixedFieldTypes" -Dtests.seed=43DA5706726F932E -Dtests.locale=ff-SN -Dtests.timezone=Asia/Manila -Druntime.java=24

Trace:

FieldSortIT > testSortMixedFieldTypes FAILED
  | java.lang.AssertionError:
  | Expected: an empty collection
  | but: <[LEAK: resource was not cleaned up before it was garbage-collected.
  | Recent access records:
  | #1:
  | in [elasticsearch[node_s2][search][T#3]][testSortMixedFieldTypes]
  | org.elasticsearch.action.ActionListener.respondAndRelease(ActionListener.java:388)
  | org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
  | org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
  | org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
  | org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  | org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
  | org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
  | org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
  | java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
  | java.base/java.lang.Thread.run(Thread.java:1447)
  | #2:
  | in [elasticsearch[node_s2][search][T#3]][testSortMixedFieldTypes]
  | org.elasticsearch.action.search.ArraySearchPhaseResults.consumeResult(ArraySearchPhaseResults.java:47)
  | org.elasticsearch.action.search.QueryPhaseResultConsumer.consumeResult(QueryPhaseResultConsumer.java:160)
  | org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardResult(AbstractSearchAsyncAction.java:507)
  | org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.onShardResult(SearchQueryThenFetchAsyncAction.java:182)
  | org.elasticsearch.action.search.AbstractSearchAsyncAction$1.innerOnResponse(AbstractSearchAsyncAction.java:270)
  | org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:34)
  | org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:19)
  | org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:63)
  | org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:27)
  | org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:49)
  | org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleResponse(SearchTransportService.java:592)
  | org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleResponse(TransportService.java:1718)
  | org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1496)
  | org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1594)
  | org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1568)
  | org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:35)
  | org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:33)
  | org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:20)
  | org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:198)
  | org.elasticsearch.action.ActionListener.respondAndRelease(ActionListener.java:386)
  | org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
  | org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
  | org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
  | org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  | org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
  | org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
  | org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
  | java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
  | java.base/java.lang.Thread.run(Thread.java:1447)
  | #3:
  | in [elasticsearch[node_s2][search][T#3]][testSortMixedFieldTypes]
  | org.elasticsearch.core.Releasables.close(Releasables.java:39)
  | org.elasticsearch.core.Releasables.closeExpectNoException(Releasables.java:72)
  | org.elasticsearch.action.ActionListenerImplementations$2.run(ActionListenerImplementations.java:51)
  | org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:218)
  | org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:413)
  | org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:403)
  | org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:323)
  | org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:368)
  | org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:259)
  | org.elasticsearch.search.internal.SearchContext.close(SearchContext.java:115)
  | org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:904)
  | org.elasticsearch.search.SearchService.lambda$executeQueryPhase$7(SearchService.java:710)
  | org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
  | org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
  | org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
  | org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  | org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
  | org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
  | org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
  | java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
  | java.base/java.lang.Thread.run(Thread.java:1447)
  | #4:
  | in [elasticsearch[node_s2][search][T#3]][testSortMixedFieldTypes]
  | org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:901)
  | org.elasticsearch.search.SearchService.lambda$executeQueryPhase$7(SearchService.java:710)
  | org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
  | org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
  | org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
  | org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  | org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
  | org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
  | org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
  | java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
  | java.base/java.lang.Thread.run(Thread.java:1447)
  | Created at:
  | in [elasticsearch[node_s2][search][T#3]][testSortMixedFieldTypes]
  | org.elasticsearch.search.query.QuerySearchResult.<init>(QuerySearchResult.java:110)
  | org.elasticsearch.search.DefaultSearchContext.addQueryResult(DefaultSearchContext.java:342)
  | org.elasticsearch.search.SearchService$ResultsType$2.addResultsObject(SearchService.java:1820)
  | org.elasticsearch.search.SearchService.createContext(SearchService.java:1330)
  | org.elasticsearch.search.MockSearchService.createContext(MockSearchService.java:142)
  | org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:869)
  | org.elasticsearch.search.SearchService.lambda$executeQueryPhase$7(SearchService.java:710)
  | org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
  | org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
  | org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
  | org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  | org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
  | org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
  | org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
  | java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
  | java.base/java.lang.Thread.run(Thread.java:1447)]>
  | at __randomizedtesting.SeedInfo.seed([43DA5706726F932E:19AFEDF12748A0C6]:0)
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
  | at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2666)
  | at org.elasticsearch.test.ESTestCase.checkStaticState(ESTestCase.java:816)
  | at org.elasticsearch.test.ESTestCase.after(ESTestCase.java:610)
  | at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  | at java.base/java.lang.reflect.Method.invoke(Method.java:565)
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004)
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  | at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  | at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  | at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  | at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  | at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  | at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  | at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  | at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  | at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  | at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  | at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  | at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  | at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  | at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  | at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  | at java.base/java.lang.Thread.run(Thread.java:1447)
  |  

<br class="Apple-interchange-newline">

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testSortMixedFieldTypes (0.3% fail rate in 654 executions)

Build Scans:

• elasticsearch-periodic-platform-support #9199 / rhel-8_platform-support-unix
• elasticsearch-pull-request #78601 / part-1

[cbuescher]

I spent some more time with this on Friday but unfortunately haven't gotten closer to a reproduction or solution. Some findings and a few theories I haven't been able to prove or disprove though:

• I increased logging SearchQueryResult reference counting in “normal” non failing cases but the locations where we inc/decRef look the same as in the failure cases (i.e. )

Leaks

java.lang.AssertionError: 	
Expected: an empty collection	
   but: <[LEAK: resource was not cleaned up before it was garbage-collected.	
Recent access records: 	
#1:	
  in [elasticsearch[node_s2][search][T#2]][testSortMixedFieldTypes]	
  org.elasticsearch.action.ActionListener.respondAndRelease(ActionListener.java:388)	
  org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)	
  org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)	
  org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)	
  org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)	
  org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)	
  org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)	
  org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)	
  java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)	
  java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)	
  java.base/java.lang.Thread.run(Thread.java:1447)	
#2:	
  in [elasticsearch[node_s2][search][T#2]][testSortMixedFieldTypes]	
  org.elasticsearch.action.search.ArraySearchPhaseResults.consumeResult(ArraySearchPhaseResults.java:47)	
  org.elasticsearch.action.search.QueryPhaseResultConsumer.consumeResult(QueryPhaseResultConsumer.java:160)	
  org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardResult(AbstractSearchAsyncAction.java:507)	
  org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.onShardResult(SearchQueryThenFetchAsyncAction.java:182)	
  org.elasticsearch.action.search.AbstractSearchAsyncAction$1.innerOnResponse(AbstractSearchAsyncAction.java:270)	
  org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:34)	
  org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:19)	
  org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:63)	
  org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:27)	
  org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:49)	
  org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleResponse(SearchTransportService.java:592)	
  org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleResponse(TransportService.java:1718)	
  org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1496)	
  org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1594)	
  org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1568)	
  org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:35)	
  org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:33)	
  org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:20)	
  org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:198)	
  org.elasticsearch.action.ActionListener.respondAndRelease(ActionListener.java:386)	
  org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)	
  org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)	
  org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)	
  org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)	
  org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)	
  org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)	
  org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)	
  java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)	
  java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)	
  java.base/java.lang.Thread.run(Thread.java:1447)	
#3:	
  in [elasticsearch[node_s2][search][T#2]][testSortMixedFieldTypes]	
  org.elasticsearch.core.Releasables.close(Releasables.java:39)	
  org.elasticsearch.core.Releasables.closeExpectNoException(Releasables.java:72)	
  org.elasticsearch.action.ActionListenerImplementations$2.run(ActionListenerImplementations.java:51)	
  org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:218)	
  org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:413)	
  org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:403)	
  org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:323)	
  org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:368)	
  org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:259)	
  org.elasticsearch.search.internal.SearchContext.close(SearchContext.java:115)	
  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:904)	
  org.elasticsearch.search.SearchService.lambda$executeQueryPhase$7(SearchService.java:710)	
  org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)	
  org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)	
  org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)	
  org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)	
  org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)	
  org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)	
  org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)	
  java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)	
  java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)	
  java.base/java.lang.Thread.run(Thread.java:1447)	
#4:	
  in [elasticsearch[node_s2][search][T#2]][testSortMixedFieldTypes]	
  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:901)	
  org.elasticsearch.search.SearchService.lambda$executeQueryPhase$7(SearchService.java:710)	
  org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)	
  org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)	
  org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)	
  org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)	
  org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)	
  org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)	
  org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)	
  java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)	
  java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)	
  java.base/java.lang.Thread.run(Thread.java:1447)	
Created at:	
  in [elasticsearch[node_s2][search][T#2]][testSortMixedFieldTypes]	
  org.elasticsearch.search.query.QuerySearchResult.<init>(QuerySearchResult.java:110)	
  org.elasticsearch.search.DefaultSearchContext.addQueryResult(DefaultSearchContext.java:342)	
  org.elasticsearch.search.SearchService$ResultsType$2.addResultsObject(SearchService.java:1820)	
  org.elasticsearch.search.SearchService.createContext(SearchService.java:1330)	
  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:869)	
  org.elasticsearch.search.SearchService.lambda$executeQueryPhase$7(SearchService.java:710)	
  org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)	
  org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)	
  org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)	
  org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)	
  org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)	
  org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)	
  org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)	
  java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)	
  java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)	
  java.base/java.lang.Thread.run(Thread.java:1447)]>	
  

at __randomizedtesting.SeedInfo.seed([26C0313418883E0F:7CB58BC34DAF0DE7]:0)	
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)	
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)	
  at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2710)	
  at org.elasticsearch.test.ESTestCase.checkStaticState(ESTestCase.java:860)	
  at org.elasticsearch.test.ESTestCase.after(ESTestCase.java:654)	
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)	
  at java.lang.reflect.Method.invoke(Method.java:565)	
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)	
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004)	
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)	
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)	
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)	
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)	
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)	
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)	
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)	
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)	
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)	
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)	
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)	
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)	
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)	
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)	
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)	
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)	
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)	
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)	
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)	
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)	
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)	
  at java.lang.Thread.run(Thread.java:1447)

• tried to (unsuccessfully) reproduce locally
• investigated role of expected “IllegalArgumentExceptions” on invalid sort
  • the failing test expects searches to fail because of incompatible sort types (i.e. long vs. double). The IEA is thrown in SearchPhaseController#validateSameSortTypeAndMaybeRewrite I think should be caught and handled in the SearchQueryThenFetchAsyncAction#onShardDone. However in local test I ran I occasionally saw these messsages logged in the Error logger (most likely this handler with an additional "failed to merge on data node"). This doesn't show up in the CI logs to my knowledge so might be a red hering
• investigated role of MockSearchService showing up in some failure cases, doesn’t seem to be the issue though because there are failures with it in the object creation path as well and forcing all tests to use MocksearchService doesn't increase likelihood of failure

Its important to note that the begining of this failure correlates with the introduction of "batched query execution", so my current working theory is that we run into a specific edge case with these search failures and maybe have a race somewhere between error handling and closing and cleaning up resources.

One other thing thats maybe worth noting:

I tried increasing the number of documents used in the test (currently the test only indexes one doc per sort type) to e.g. 20 which largely increases the likelihood of this test failing. However, this is with a different error reason than the Leak/"Attempted to close with partial reduce in-flight" messages:

java.lang.AssertionError: [org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.search.TransportSearchAction$SearchResponseActionListener/
[...]
Caused by: java.lang.AssertionError: [org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.search.TransportSearchAction$SearchResponseActionListener/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.tasks.TaskManager$1{org.elasticsearch.client.internal.support.AbstractClient$RefCountedFuture@21383b4e}{CancellableTask{Task{id=329, type='transport', action='indices:data/read/search', description='null', parentTask=unset, startTime=1751890516860, headers={}, startTimeNanos=17299138524750}, reason='Fatal failure during search: failed to merge result [Can't sort on field [foo]; the field has incompatible sort types: [DOUBLE] and [LONG] across shards!]', isCancelled=true}}/org.elasticsearch.action.support.TransportAction$$Lambda/0x00001c00018e55a0@42058342/org.elasticsearch.action.search.AbstractSearchAsyncAction$$Lambda/0x00001c0001b16c28@6be94945]
	... 42 more
Caused by: org.elasticsearch.ElasticsearchException: executed already
	at org.elasticsearch.action.ActionListener$3.assertFirstRun(ActionListener.java:403)
	... 41 more

This is coming from an assertion in ActionListener

Here's a local run of 10 with 4 of these failure: https://gradle-enterprise.elastic.co/s/oiufje2arvhla

[cbuescher]

This has been muted on branch main

Unmuted in #130755 to collect more information on failure circumstances