Skip to content

fix the paramterized flaky tests in the file ToOpenApiActionImplTest.java #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ThugJudy
Copy link
Owner

@ThugJudy ThugJudy commented Nov 7, 2023

This PR aims to fix the following 16 flaky test cases:
link to the test

io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[1]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[2]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[3]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[4]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[5]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[6]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[7]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[8]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[9]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[10]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[11]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[12]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[13]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[14]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[15]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[16]

I found and confirmed the flaky behavior using an open-source research tool NonDex, which shuffles implementations of nondeterminism operations.

Problem:
The writeValueAsString() function is nondeterministic and when it tries to convert a YAML object to a string, it does not account for the ordering of elements of the YAML object.


Hence, all the tests mentioned above when they try to call the SchemaUtils.writeValueAsString() function, are prone to flakiness.

Solution:

Since we do not have access to the internal library, I have added a check, that converts the YAML string to YAML objects and then compares the objects.

Steps to reproduce

Integrate NonDex:

Add the following snippet to the top of the build.gradle in $PROJ_DIR:

plugins {
    id 'edu.illinois.nondex' version '2.1.1-1'
}

Append to build.gradle in $PROJ_DIR:

apply plugin: 'edu.illinois.nondex'

Execute Test with Gradle:

./gradlew --info test --tests=io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test

Run NonDex:

./gradlew --info nondexTest --tests=io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test --nondexRuns=50

Test Environment:

Java version "1.8.0_381"
macOS Venture Version 13.4.1 (22F82)

Please let me know if you have any concerns or questions.

@zzjas
Copy link

zzjas commented Nov 14, 2023

Suggestion on text is the same as #1

String expected = Resources.toString(expectedUrl, StandardCharsets.UTF_8).replace("\r", "");
String actual = new String(Files.readAllBytes(actualFile.toPath()), StandardCharsets.UTF_8).replace("\r", "");
Assertions.assertEquals(expected, actual);
Assertions.assertEquals(Optional.ofNullable(yaml.load(expected)), Optional.ofNullable(yaml.load(actual)));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is Optional.ofNullable needed in this test but not in other tests? You probably want to justify its usage.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to explicit type casting.

Copy link

@zzjas zzjas Nov 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My question is actually:

In #1, you used assertEquals with yaml.load without the need to cast. Why do you need to cast here? What's the difference between these two PRs that makes the cast necessary?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason for this is that, we are typecasting it into map object in the previous PR. The return data type is ambiguous hence we have to explicitly specify it. An alternate approach to maintain uniformity would be to replace line 39 and 40 with the following lines:

Map<String, Object> expected = yaml.load(Resources.toString(expectedUrl, StandardCharsets.UTF_8).replace("\r", ""));
Map<String, Object> actual = yaml.load(new String(Files.readAllBytes(actualFile.toPath()), StandardCharsets.UTF_8).replace("\r", ""));

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Keeping the cast inline looks better.

@ThugJudy
Copy link
Owner Author

Suggestion on text is the same as #1

done

@ThugJudy ThugJudy force-pushed the fix-flaky-test-to-open-api-action-impl branch from 04fcbf4 to 463d1f4 Compare November 15, 2023 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants