Collect test types in CI in parallel #30450
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
One of the steps in our CI is to collect tests, which is to prevent parallel tests from even starting when we know test collection will fail (it is terrible waste of resources to start 20 test jobs and initialize databases etc. when we know it is not needed. This however introduced a single point of delay in the CI process, which with the recent collection protection implemented in apache#30315 piled up to more than 5 minutes occassionally on the CI machines of ours, especially on public runners. This PR utilises our existing test framework to be able to parallelise test collection (Pytest does not have paralllel collection mechanism) - also for localised PRs it will only run test collection for the test types that are going to be executed, which will speed it up quite a lot. This might get further sped up if we split Provider tests into smaller groups to parallelise them even more.
eladkal
approved these changes
Apr 4, 2023
|
We have now |
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Apr 4, 2023
We've added a new reference to test-types in apache#30450 and it clashed with parameter rename in apache#30424. This resulted in bad merge (not too dangerous, just causing missing optimisation in collection elapsed time in case only a subset of test types were to be executed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One of the steps in our CI is to collect tests, which is to prevent parallel tests from even starting when we know test collection will fail (it is terrible waste of resources to start 20 test jobs and initialize databases etc. when we know it is not needed.
This however introduced a single point of delay in the CI process, which with the recent collection protection implemented in #30315 piled up to more than 5 minutes occassionally on the CI machines of ours, especially on public runners.
This PR utilises our existing test framework to be able to parallelise test collection (Pytest does not have paralllel collection mechanism) - also for localised PRs it will only run test collection for the test types that are going to be executed, which will speed it up quite a lot.
This might get further sped up if we split Provider tests into smaller groups to parallelise them even more.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.