[CI] FrozenSearchableSnapshotsIntegTests testCreateAndRestorePartialSearchableSnapshot failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #19146 / part2
• elasticsearch-intake #18891 / part2

Reproduction Line:

./gradlew ":x-pack:plugin:searchable-snapshots:internalClusterTest" --tests "org.elasticsearch.xpack.searchablesnapshots.FrozenSearchableSnapshotsIntegTests.testCreateAndRestorePartialSearchableSnapshot" -Dtests.seed=2CDF29528A255FC4 -Dtests.locale=hu-Latn-HU -Dtests.timezone=ROK -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Searchable snapshot directory does not support the operation [createOutput(corrupted_gTNu35_ERpCZ9TsjuuE44w, IOContext[context=DEFAULT, mergeInfo=null, flushInfo=null, readAdvice=RANDOM])], current directory files: _c.cfe,_c.cfs,_c.si,_c_h.fnm,_c_h_Asserting_0.dvd,_c_h_Asserting_0.dvm,_m.cfe,_m.cfs,_m.si,_m_8.fnm,_m_8_Asserting_0.dvd,_m_8_Asserting_0.dvm,_t.cfe,_t.cfs,_t.si,segments_4

Issue Reasons:

• [main] 2 failures in test testCreateAndRestorePartialSearchableSnapshot (0.4% fail rate in 464 executions)
• [main] 2 failures in step part2 (3.8% fail rate in 53 executions)
• [main] 2 failures in pipeline elasticsearch-intake (3.8% fail rate in 53 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testCreateAndRestorePartialSearchableSnapshot (0.4% fail rate in 464 executions)
• [main] 2 failures in step part2 (3.8% fail rate in 53 executions)
• [main] 2 failures in pipeline elasticsearch-intake (3.8% fail rate in 53 executions)

Build Scans:

• elasticsearch-intake #19146 / part2
• elasticsearch-intake #18891 / part2

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[bcully]

This does reproduce locally, just took 1000 iters.

[bcully]

Looks like another shutdown race most likely.

[2025-03-01T07:51:08,554][INFO ][o.e.n.Node               ] [testCreateAndRestorePartialSearchableSnapshot] closed
...
márc. 01, 2025 7:51:08 DE. com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException

The stack trace looks like we're doing a shard recovery (probably automatically) that is prepared to handle an AlreadyClosedException but we're getting a different kind of IOException which is causing us to try to mark the store as corrupted.

We are in the middle of a full cluster restart triggered here. In this run, we're also doing expensive checksum validation at shard startup.

[2025-03-01T07:51:08,468][INFO ][o.e.t.InternalTestCluster] [testCreateAndRestorePartialSearchableSnapshot] Stopping and resetting node [node_s1] 
...
[2025-03-01T07:51:08,554][INFO ][o.e.n.Node               ] [testCreateAndRestorePartialSearchableSnapshot] closed	
[2025-03-01T07:51:08,554][INFO ][o.e.i.s.IndexShard       ] [node_s1] [tvttlliquc][0] check index [ok]: checksum check passed on [_c_h.fnm]	
[2025-03-01T07:51:08,555][INFO ][o.e.i.s.IndexShard       ] [node_s1] [tvttlliquc][0] check index [ok]: checksum check passed on [_m_8.fnm]	
[2025-03-01T07:51:08,555][INFO ][o.e.i.s.IndexShard       ] [node_s1] [tvttlliquc][0] check index [ok]: checksum check passed on [_m.cfs]	
[2025-03-01T07:51:08,555][INFO ][o.e.i.s.IndexShard       ] [node_s1] [tvttlliquc][0] check index [ok]: checksum check passed on [_m_8_Asserting_0.dvd]	
[2025-03-01T07:51:08,555][INFO ][o.e.i.s.IndexShard       ] [node_s1] [tvttlliquc][0] check index [ok]: checksum check passed on [_t.cfs]	
[2025-03-01T07:51:08,556][WARN ][o.e.i.s.IndexShard       ] [node_s1] [tvttlliquc][0] check index [failure]: checksum failed on [_c.cfs]	
java.io.IOException: failed to read data from cache for [CacheFileReference{cacheKey='CacheKey[snapshotUUID=Ex9qOi2mQkGxeAS37EhaaQ, snapshotIndexName=bpynpmvnoc, shardId=[tvttlliquc][0], fileName=_c.cfs]', fileLength=481290, acquired=false}]

looks like we're doing validation on this shard (during opening?) after the node has been stopped.