-
Notifications
You must be signed in to change notification settings - Fork 0
fix the paramterized flaky tests in the file ToOpenApiActionImplTest.java #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Suggestion on text is the same as #1 |
String expected = Resources.toString(expectedUrl, StandardCharsets.UTF_8).replace("\r", ""); | ||
String actual = new String(Files.readAllBytes(actualFile.toPath()), StandardCharsets.UTF_8).replace("\r", ""); | ||
Assertions.assertEquals(expected, actual); | ||
Assertions.assertEquals(Optional.ofNullable(yaml.load(expected)), Optional.ofNullable(yaml.load(actual))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is Optional.ofNullable
needed in this test but not in other tests? You probably want to justify its usage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed to explicit type casting.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My question is actually:
In #1, you used assertEquals
with yaml.load
without the need to cast. Why do you need to cast here? What's the difference between these two PRs that makes the cast necessary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The reason for this is that, we are typecasting it into map object in the previous PR. The return data type is ambiguous hence we have to explicitly specify it. An alternate approach to maintain uniformity would be to replace line 39 and 40 with the following lines:
Map<String, Object> expected = yaml.load(Resources.toString(expectedUrl, StandardCharsets.UTF_8).replace("\r", ""));
Map<String, Object> actual = yaml.load(new String(Files.readAllBytes(actualFile.toPath()), StandardCharsets.UTF_8).replace("\r", ""));
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense. Keeping the cast inline looks better.
done |
04fcbf4
to
463d1f4
Compare
This PR aims to fix the following 16 flaky test cases:
link to the test
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[1]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[2]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[3]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[4]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[5]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[6]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[7]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[8]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[9]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[10]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[11]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[12]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[13]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[14]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[15]
io.github.vlsergey.springdatarestutils.ToOpenApiActionImplTest.test(String)[16]
I found and confirmed the flaky behavior using an open-source research tool NonDex, which shuffles implementations of nondeterminism operations.
Problem:
The writeValueAsString() function is nondeterministic and when it tries to convert a YAML object to a string, it does not account for the ordering of elements of the YAML object.
spring-data-rest-utils/src/main/java/io/github/vlsergey/springdatarestutils/SchemaUtils.java
Line 33 in 8cd30df
Hence, all the tests mentioned above when they try to call the SchemaUtils.writeValueAsString() function, are prone to flakiness.
Solution:
Since we do not have access to the internal library, I have added a check, that converts the YAML string to YAML objects and then compares the objects.
Steps to reproduce
Integrate NonDex:
Add the following snippet to the top of the build.gradle in $PROJ_DIR:
Append to build.gradle in $PROJ_DIR:
Execute Test with Gradle:
Run NonDex:
Test Environment:
Please let me know if you have any concerns or questions.