Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve: allow flaky tests run with specify parameters #4543

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

hezhangjian
Copy link
Member

@hezhangjian hezhangjian commented Jan 8, 2025

Fix #3249

Motivation

This PR addresses the issue where tests annotated with @FlakyTest were not being executed by JUnit unless they were also annotated with @test.

To resolve this, we have updated our Maven Surefire plugin configuration to exclude tests tagged with flaky during regular CI runs, allowing these tests to be run separately.

This adjustment ensures that all tests, including those marked as flaky, are executed as intended unless explicitly excluded.(In CI still don't run it)

Moreover, I've planned the addition of a daily CI job and non-required CI to specifically run tests tagged as flaky, ensuring continuous monitoring and quicker identification of intermittent issues without affecting the main test pipeline. This setup also enhances the ability to run tests directly from the IDE, making it more convenient for developers to execute and debug individual tests as needed.

@hezhangjian
Copy link
Member Author

@lhotari PTAL, after half year, I finally closed this thing. :)

Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 thanks

Comment on lines +921 to +923
<configuration>
<excludedGroups>flaky</excludedGroups>
</configuration>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change isn't consistent with the PR description "This adjustment ensures that all tests, including those marked as flaky, are executed as intended unless explicitly excluded".
Wouldn't this exclude the flaky group by default?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lhotari I would like to excluded flaky tests in maven command line, but can be easily run in IDE.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the original meaning of the "flaky" test group in the project? If so, please document this in the FlakyTest annotation. The PR description should also clearly express this that in CI, the tests aren't run at all.

Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the review comments

Signed-off-by: Zhangjian He <[email protected]>
@hezhangjian
Copy link
Member Author

@lhotari I updated the annotation's annotation, PTAL again, thanks

 * <p>Flaky tests are tests that produce inconsistent results,
 * occasionally failing or passing without changes to the code under test.
 * This could be due to external factors such as timing issues, resource contention,
 * dependency on non-deterministic data, or integration with external systems.
 *
 * <p>Tests marked with this annotation are excluded from execution
 * in CI pipelines and Maven commands by default, to ensure a reliable and
 * deterministic build process. However, they can still be executed manually
 * or in specific environments for debugging and resolution purposes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Multiple tests marked as "@FlakyTest" and de-facto not running
3 participants