[CI] SearchableSnapshotsIntegTests testCreateAndRestoreSearchableSnapshot failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #21307 / part2
• elasticsearch-pull-request #65935 / part-2

Reproduction Line:

./gradlew ":x-pack:plugin:searchable-snapshots:internalClusterTest" --tests "org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsIntegTests.testCreateAndRestoreSearchableSnapshot" -Dtests.seed=CA65EAE19FFC2A5C -Dtests.locale=fur-IT -Dtests.timezone=SystemV/CST6CDT -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Index{id='tlSFDTRjRM/C0kqyDz8R0iy18ccAYF4Rw/6IdUt1iTR2mem7xfzwzZRA/5/_c_Lucene101_1.doc/@0', seqNo=841, primaryTerm=1, version=1, autoGeneratedIdTimestamp=-1}
Expected: <2L>
     but: was <1L>

Issue Reasons:

• [main] 2 failures in test testCreateAndRestoreSearchableSnapshot (0.7% fail rate in 273 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[piergm]

The test did not get muted and did not fail in the last 17 days, plus it's not reproducible locally. I am going to close this for now.

[thecoop]

A similar failure happened - https://gradle-enterprise.elastic.co/s/jlasxj7sgru44/console-log?anchor=20165&page=21. Doesn't reproduce locally, but looks like it may be an intermittent issue?

[javanna]

Seen another one today in 8.18:

REPRODUCE WITH: ./gradlew ":x-pack:plugin:searchable-snapshots:internalClusterTest" --tests "org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsIntegTests.testCreateAndRestoreSearchableSnapshot" -Dtests.seed=4A4B064A5713CE1C -Dtests.locale=ar-LB -Dtests.timezone=America/Belem -Druntime.java=24
  2> java.lang.AssertionError: Index{id='Ljkcprlviy/0TLjdQ4GSnSUnpZROdaN4g/aPHX5oqsTPSmMh42ODqE-A/7/_0.cfs/@2232', seqNo=193, primaryTerm=1, version=1, autoGeneratedIdTimestamp=-1}
    Expected: <2L>
         but: was <1L>
        at __randomizedtesting.SeedInfo.seed([4A4B064A5713CE1C:C0287131C4849FAF]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2617)
        at org.elasticsearch.index.engine.EngineTestCase.assertConsistentHistoryBetweenTranslogAndLuceneIndex(EngineTestCase.java:1378)
        at org.elasticsearch.index.shard.IndexShardTestCase.assertConsistentHistoryBetweenTranslogAndLucene(IndexShardTestCase.java:987)
        at org.elasticsearch.test.InternalTestCluster.assertConsistentHistoryBetweenTranslogAndLuceneIndex(InternalTestCluster.java:1409)
        at org.elasticsearch.snapshots.AbstractSnapshotIntegTestCase.assertConsistentHistoryInLuceneIndex(AbstractSnapshotIntegTestCase.java:131)

Relabelling this as well because from the assertion that fails, this does not seem to be a search problem.

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testCreateAndRestoreSearchableSnapshot (0.7% fail rate in 273 executions)

Build Scans:

• elasticsearch-intake #21307 / part2
• elasticsearch-pull-request #65935 / part-2

[arteam]

I'm going to reassign this issue to `:StorageEngine/Logs, since the assertion started failing after the translog operation assertions changes in #119330. This test seems to failing on checking the primary terms of the lucene ops and translog ops, and is not fail on synthetic source assertions, but I wonder if #119330 had some side-effects.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[arteam]

Maybe we need to supply TranslogOperationAsserter.withEngineConfig(config) in ReadOnlyEngine instead of TranslogOperationAsserter.DEFAULT

[lkts]

It looks like it fails here meaning primary term somehow does not match?

https://github.com/elastic/elasticsearch/blob/main/test/framework/src/main/java/org/elasticsearch/index/engine/EngineTestCase.java#L1400

Note that in order to get to this line the engine must be an InternalEngine.

[lkts]

I don't believe this is related to #119330 since it was failing before as well https://es-delivery-stats.elastic.dev/app/r/s/sD7Lr.

[lkts]

The only somewhat notable thing i can see is that the document from the assert seems to be from the .snapshot-blob-cache index.

[lkts]

I have opened #127131 which may or may not be helpful. Given that it does not look like it's related to #119330 i'll assign it back.