-
Notifications
You must be signed in to change notification settings - Fork 907
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
improve: allow flaky tests run with specify parameters #4543
base: master
Are you sure you want to change the base?
Conversation
Signed-off-by: Zhangjian He <[email protected]>
0369cce
to
75c317b
Compare
@lhotari PTAL, after half year, I finally closed this thing. :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 thanks
<configuration> | ||
<excludedGroups>flaky</excludedGroups> | ||
</configuration> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change isn't consistent with the PR description "This adjustment ensures that all tests, including those marked as flaky, are executed as intended unless explicitly excluded".
Wouldn't this exclude the flaky group by default?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lhotari I would like to excluded flaky tests in maven command line, but can be easily run in IDE.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this the original meaning of the "flaky" test group in the project? If so, please document this in the FlakyTest annotation. The PR description should also clearly express this that in CI, the tests aren't run at all.
stream/distributedlog/core/src/test/java/org/apache/distributedlog/bk/TestLedgerAllocator.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please check the review comments
Signed-off-by: Zhangjian He <[email protected]>
fc989cc
to
02d6e0f
Compare
Signed-off-by: Zhangjian He <[email protected]>
Signed-off-by: Zhangjian He <[email protected]>
@lhotari I updated the annotation's annotation, PTAL again, thanks
|
Fix #3249
Motivation
This PR addresses the issue where tests annotated with @FlakyTest were not being executed by JUnit unless they were also annotated with @test.
To resolve this, we have updated our Maven Surefire plugin configuration to exclude tests tagged with flaky during regular CI runs, allowing these tests to be run separately.
This adjustment ensures that all tests, including those marked as flaky, are executed as intended unless explicitly excluded.(In CI still don't run it)
Moreover, I've planned the addition of a daily CI job and non-required CI to specifically run tests tagged as flaky, ensuring continuous monitoring and quicker identification of intermittent issues without affecting the main test pipeline. This setup also enhances the ability to run tests directly from the IDE, making it more convenient for developers to execute and debug individual tests as needed.