Better handling of node shutdown in SearchScrollAsyncAction

[piergm]

SearchScrollAsyncAction seems to handle poorly node shutdown.
During a node shutdown we performed multiple scroll search path: /_search/scroll and got back 500 with the following exception:

Failed to execute phase [query], all shards failed; shardFailures {java.lang.IllegalStateException: node [XYZ] is not available

Node XYZ was SIGTERMd 30 seconds prior to the first 500 request and completed the shutdown process 6 seconds before the first 500 request.
I suspect that we could improve on how we handle node shutdown in SearchScrollAsyncAction.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[jimczi]

Note that PIT (Point in Time) would behave similarly, so it would be beneficial to extend the scope to include all persistent readers.

[smalyshev]

I just got a bunch of those as suppressed REST - did we change something to make this happen more frequently or is it just a coincidence?

[craigtaverner]

Saw more suppressed REST errors today.