[CI] FileSettingsServiceTests testInvalidJSON failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #16057 / part1
• elasticsearch-pull-request #51498 / part-1-fips
• elasticsearch-periodic-platform-support #5927 / sles-15_platform-support-unix
• elasticsearch-periodic-platform-support #5927 / rocky-9_platform-support-unix

Reproduction Line:

./gradlew ":server:test" --tests "org.elasticsearch.reservedstate.service.FileSettingsServiceTests.testInvalidJSON" -Dtests.seed=1305FC3681068953 -Dtests.locale=ar-IL -Dtests.timezone=Etc/GMT-1 -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
No

Failure History:
See dashboard

Failure Message:

org.mockito.exceptions.misusing.UnfinishedStubbingException: 
Unfinished stubbing detected here:
-> at org.elasticsearch.reservedstate.service.FileSettingsServiceTests.testInvalidJSON(FileSettingsServiceTests.java:312)

E.g. thenReturn() may be missing.
Examples of correct stubbing:
    when(mock.isOk()).thenReturn(true);
    when(mock.isOk()).thenThrow(exception);
    doThrow(exception).when(mock).someVoidMethod();
Hints:
 1. missing thenReturn()
 2. you are trying to stub a final method, which is not supported
 3. you are stubbing the behaviour of another mock inside before 'thenReturn' instruction is completed

Issue Reasons:

• [main] 4 failures in test testInvalidJSON (23.5% fail rate in 17 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 4 failures in test testInvalidJSON (23.5% fail rate in 17 executions)

Build Scans:

• elasticsearch-intake #16057 / part1
• elasticsearch-pull-request #51498 / part-1-fips
• elasticsearch-periodic-platform-support #5927 / sles-15_platform-support-unix
• elasticsearch-periodic-platform-support #5927 / rocky-9_platform-support-unix

[prdoyle]

Works for me. 🤔

[prdoyle]

The stubbing code from FileSettingsServiceTests.java:312:

        doAnswer((Answer<?>) invocation -> {
            try {
                return invocation.callRealMethod();
            } finally {
                awaitOrBust(fileChangeBarrier);
            }
        }).when(fileSettingsService).onProcessFileChangesException(any());

I'm not seeing a problem here.

[prdoyle]

Here's a link to this code in the SHA reported in the build scans. It's definitely the code I pointed out above.

[prdoyle]

For the "hints":

Hints:
 1. missing thenReturn()
 2. you are trying to stub a final method, which is not supported
 3. you are stubbing the behaviour of another mock inside before 'thenReturn' instruction is completed

1. I can't use thenReturn because it's a void method.
2. The onProcessFileChangesException is not final.
3. I am stubbing just this one method.

[prdoyle]

According to this guy, Mockito uses some static global state, and can't tolerate any kind of concurrent access.

If that's the case, perhaps I can reorder the mocking/stubbing in such a way as to avoid races with the ES logic that uses the mocks.