Skip to content

Removing assertion in asynchronous operation to avoid flaky test #1193

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 4, 2021

Conversation

denini08
Copy link
Contributor

@denini08 denini08 commented Nov 4, 2021

What is the purpose of this PR

  • This PR fixes a flaky test to avoid an assertion on an asynchronous operation org.java_websocket.issues.Issue677Test#testIssue

Reproduce the test failure

  • The test fails non-deterministically when execution delays occur. To observe a test failure, the unmodified test can be re-executed many times. the likelihood of observing a test failure increases when the test executes on a slow machine. To deterministically reproduce the test failure, introduce an execution delay:
@Test
public void testIssue() throws Exception {
    ...
    server.start();
    countServerDownLatch.await();
    webSocket0.connectBlocking();
    assertTrue( "webSocket.isOpen()", webSocket0.isOpen() );
    webSocket0.close();
    Thread.sleep(20);
    assertTrue( "webSocket.isClosing()", webSocket0.isClosing() ); //fail
    ...
}

Expected result:

  • The test should run successfully in all runs.

Actual result:

  • In 52 out of 100 repetitions of the test suite, the test fails with:
java.lang.AssertionError: webSocket.isClosing()

Why the test fails

  • In this test, a WebSocket server and 2 clients are created. First, the test starts the server and then connects client1 to the server. After the connection, webSocket0.close() is called (this operation is done in WebSocketClient's thread to close client1's connection to the server). Next, the server is checked to see if it is closing. Sometimes when the assertion runs, the connection with the server has already been closed (and is not in the process of closing) or is still open, causing the assertion to fail. We suggest that the assertion should be removed, i.e., the test should not check if the connection is closing, instead it should check only if the connection is opened or closed (giving some time between the call and the assertion).

Fix

  • Remove the assertion on an asynchronous operation.

@marci4 marci4 added the Test label Nov 4, 2021
@marci4 marci4 added this to the Release 1.5.3 milestone Nov 4, 2021
@marci4 marci4 merged commit bcfc967 into TooTallNate:master Nov 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants