[bazel] Improve build determinism #25842
Merged
+63
−8
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Make OTP image stamping (including a comment "Generated on [DATE] by gen_otp_img.py ...") conditional on the presence of Bazel's standard stamp argument, such that by default generated files are not stamped, but the original behaviour can be retained when running Bazel with the `--stamp` flag. This will make OTP image building deterministic, and as such should help improve cache hit rates and thus reduce build times (if Bazel thinks it needs to re-build the OTP but nothing has changed, making the OTP image deterministic will allow it to re-use results that use the identical OTP images). Signed-off-by: Alex Jones <[email protected]>
Sorts the set of word annotations for a MEM line in the OTP image, such that the MEM line being read by Vivado is the same, but the word annotations always appear in the same order, improving the determinism of the builds. Previously, usage of Python's sets meant that for MEM lines with multiple word attributes, they could appear in different orders between builds, causing slight differences in the output images (and thus potentially requiring later actions to be re-run instead of cache hits due to differences in the generated OTP image). Signed-off-by: Alex Jones <[email protected]>
AlexJones0
force-pushed
the
build_determinism
branch
from
582fae0 to
f4ee6d7
Compare
jwnrt
approved these changes
Jan 10, 2025
nbdd0121
reviewed
Jan 10, 2025
Move rules_foreign_cc from the root MODULE.bazel to its own `third_party/foreign_cc` directory so that we can patch it to fix an issue that has not yet been supported upstream yet, in the same way as we do for e.g. `rules_rust`. When `rules_foreign_cc` builds using tools such as e.g. Configure+Make, output files Configure.log and BootstrapGNUMake.log are generated and emitted which contain references to the absolute path. When building inside Bazel's sandbox, this produces a non-determinstic output which changes between each run, as a result, rebuilding OpenOCD will then change the inputs of any Bazel actions that use OpenOCD, resulting in a reduced number of cache hits and increasing build times. By making this determinstic, we improve reproducibility and potentially reduce build times. To fix this, this PR introduces a simple patch to completely remove all the content of this logging output, as it is not needed by any later steps as part of the build. Signed-off-by: Alex Jones <[email protected]>
AlexJones0
force-pushed
the
build_determinism
branch
from
f4ee6d7 to
945b309
Compare
nbdd0121
approved these changes
Jan 10, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes three commits, which, in addition to the crate updates from e7868c7 in #25795, should help improve the determinism of our builds and as such should improve the cache hit rate for Bazel builds, reducing build times. See more details in the commit messages.