Threshold report: add new YOASTCS_THRESHOLD_EXACT_MATCH constant
#375
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The threshold report is used in the YoastCS repos in combination with a (Composer) script which overrules the exit code from PHPCS to only exit with a non-zero exit code when the _actual_ total number of errors/warnings exceeds the _expected_ number of errors/warnings.
This could lead to situations where issues are being fixed in PR A, but the threshold has not been updated (i.e. errors/warnings _below_ threshold). This would not fail a build, which means this could go unnoticed and get merged. This, in turn, then allows for a subsequent PR B to introduce _new issues_ into a codebase without those new issues failing the PHPCS build, as long as the number of new issues being introduced in PR B did not exceed the number of previously _fixed_ issues from PR A.
This is undesired.
This commit adds a new global `YOASTCS_THRESHOLD_EXACT_MATCH` constant which can be used by the calling script to fail a build with a non-zero exit code when the _actual_ total number of errors/warnings doesn't **_exactly_** match the _expected_ number of errors/warnings.
Notes:
* The old constant is not being removed as that would be a breaking change and would require a major release.
* I considered deprecating the old `YOASTCS_ABOVE_THRESHOLD` constant, but having both available provides the calling scripts flexibility to choose which logic to apply.
If so desired, the old constant can still be deprecated in a later minor.
Includes a minor tweak to make the values of the constants testable.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The threshold report is used in the YoastCS repos in combination with a (Composer) script which overrules the exit code from PHPCS to only exit with a non-zero exit code when the actual total number of errors/warnings exceeds the expected number of errors/warnings.
This could lead to situations where issues are being fixed in PR A, but the threshold has not been updated (i.e. errors/warnings below threshold). This would not fail a build, which means this could go unnoticed and get merged. This, in turn, then allows for a subsequent PR B to introduce new issues into a codebase without those new issues failing the PHPCS build, as long as the number of new issues being introduced in PR B did not exceed the number of previously fixed issues from PR A.
This is undesired.
This commit adds a new global
YOASTCS_THRESHOLD_EXACT_MATCHconstant which can be used by the calling script to fail a build with a non-zero exit code when the actual total number of errors/warnings doesn't exactly match the expected number of errors/warnings.Notes:
YOASTCS_ABOVE_THRESHOLDconstant, but having both available provides the calling scripts flexibility to choose which logic to apply. If so desired, the old constant can still be deprecated in a later minor.Includes a minor tweak to make the values of the constants testable.