IllegalStateException during search: "Error retrieving path ..."

[alex-spies]

Got a stack trace from a suppressed REST error on Serverless (anonymized). Looks like it could be originating from here in SearchHit.java, although the stack trace doesn't really show this.

Failed to execute phase [query], all shards failed; shardFailures {[pZ7he6e-S6adoEWgKemdyA][some_index][0]: org.elasticsearch.transport.RemoteTransportException: [es-es-search-...][...][indices:data/read/search[phase/query]]
  Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
  Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [3197]]
  Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [3194]]
  Caused by: java.lang.IllegalStateException: Error retrieving path some_path
  }{[pZ7he6e-S6adoEWgKemdyA][some_index][1]: org.elasticsearch.transport.RemoteTransportException: [es-es-search-...][...][indices:data/read/search[phas
  e/query]]
  Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
  Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [6231]]
  Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [6228]]
  Caused by: java.lang.IllegalStateException: Error retrieving path some_path
  }{[pZ7he6e-S6adoEWgKemdyA][some_index][2]: org.elasticsearch.transport.RemoteTransportException: [es-es-search-...][...][indices:data/read/search[phase/query]]
  Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
  Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [6615]]
  Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [6612]]    
  Caused by: java.lang.IllegalStateException: Error retrieving path some_path
  }

 │ at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:693)
  │ at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:360)
  │ at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:738)
  │ at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:454)
  │ at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:312)
  │ at [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)
  │ at [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)
  │ at [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
  │ at [email protected]/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:54)
  │ at [email protected]/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:593)
  │ at [email protected]/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1776)
  │ at [email protected]/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1509)
  │ at [email protected]/org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1643)
  │ at [email protected]/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1618)
  │ at [email protected]/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44)
  │ at [email protected]/org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:45)
  │ at [email protected]/org.elasticsearch.search.SearchService.lambda$maybeWrapListenerForStackTrace$1(SearchService.java:540)
  │ at [email protected]/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:188)
  │ at [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)
  │ at [email protected]/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:193)
  │ at [email protected]/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:152)
  │ at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:29)
  │ at [email protected]/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:34)
  │ at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
  │ at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  │ at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  │ at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  │ at java.base/java.lang.Thread.run(Thread.java:1575)
  Caused by: org.elasticsearch.ElasticsearchException$1: Error retrieving path  some_path
  │ at [email protected]/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:725)
  │ at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:358)
  │ ... 26 more
  Caused by: java.lang.IllegalStateException: Error retrieving path some_path

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[javanna]

This has to do with some unexpected scenario encountered when retrieving nested hits as part of the fetch phase.

[original-brownbear]

This has come up on discuss once before https://discuss.elastic.co/t/error-retrieving-path-of-nested-field/362359 where an example query can be found.

[original-brownbear]

@benchaplin actually found a reproducer for this. This is definitely caused by concurrency as a result of running the search context across multiple threads. There's just some debate on the correct fix :)