Sanitize BEP contents before emitting them #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…d#19042) * Support multiple remote execution digest functions The following PR on the remote-apis side adds support for remote execution services to announce support for multiple digest functions: bazelbuild/remote-apis#236 This makes migrating from one digest function to the other more graceful. This change extends Bazel's server capabilities checking code to take the new field in the execution capabilities into account. Partial commit for third_party/*, see bazelbuild#16791. Signed-off-by: Sunil Gowroji <[email protected]> (cherry picked from commit d0cba55) * Support multiple remote execution digest functions The following PR on the remote-apis side adds support for remote execution services to announce support for multiple digest functions: bazelbuild/remote-apis#236 This makes migrating from one digest function to the other more graceful. This change extends Bazel's server capabilities checking code to take the new field in the execution capabilities into account. Closes bazelbuild#16791. PiperOrigin-RevId: 517084447 Change-Id: I72afce6c1fae9e624f9e7ed1936c744ae9c81280 (cherry picked from commit f8c8875) --------- Co-authored-by: Ed Schouten <[email protected]>
* Include stack trace in all gRPC errors when --verbose_failures is set. Also refactor a couple places where the stack trace was included in an ad-hoc manner, and force Utils.grpcAwareErrorMessage callers to be explicit to avoid future instances. Closes bazelbuild#16086. PiperOrigin-RevId: 502854490 Change-Id: Id2d6a1728630fffea9399b406378c7f173b247bd * Avoid discarding SRE state for IO cause Unwrapping all StatusRuntimeExceptions in in ReferenceCountedChannel when caused by IOException will discard critical tracing and retriability. The Retrier evaluations may not see an SRE in the causal chain, and presume it is invariably an unretriable exception. In general, IOExceptions as SRE wrappers are unsuitable containers and are routinely misued either for identification (grpc aware status), or capture (handleInitError). Partially addresses bazelbuild#18764 (retries will occur with SSL handshake timeout, but the actual connection will not be retried) Closes bazelbuild#18836. PiperOrigin-RevId: 546037698 Change-Id: I7f6efcb857c557aa97ad3df085fc032c8538eb9a * Operation stream termination is not an error According to the GrpcRemoteExecutor when it occurs after a !done operation response. Remove the error from the ExperimentalRemoteGrpcExecutor and reinforce both with tests. Update the FakeExecutionService to generate compatible error responses that appear in the ExecuteResponse, rather than the operation error field, per the REAPI spec. Made required adjustments to ExGRE Test invocations to avoid the ExecutionStatusException interpretation of DEADLINE_EXCEEDED -> FAILED_PRECONDITION in ExecuteResponse. Closes bazelbuild#18785. PiperOrigin-RevId: 546925894 Change-Id: I7a489c8bc936a83cfd94e0138437f3fe6d152da8 * Done operations must be reexecuted Any operation with done == true as reported by the server is not expected to change its result on subsequent waitExecution calls. To properly retry, this action must be reexecuted, if it was truly transient, to achieve a definitive result. Submit a transient status for retry, disallow special behaviors for NOT_FOUND as covered by done observation, and consider method type when handling operation streams. Closes bazelbuild#18943. PiperOrigin-RevId: 548680656 Change-Id: Ib2c9887ead1fbd3de97761db6e8b4077783ad03c --------- Co-authored-by: Tiago Quelhas <[email protected]>
Currently, it is included in runtime library with glob which is shared between bazel and blaze. Moving it to bazel package to prevent from packaging it when building blaze. An upcoming change (in bazelbuild#18784) will package BLAKE3 to this module and we don't want it in blaze. PiperOrigin-RevId: 547516042 Change-Id: I673fa3d6824d56c85e85b02d13b4eb56571edda9 (cherry picked from commit 6d6bbdc) Co-authored-by: Googler <[email protected]>
…ttribute (bazelbuild#19053) for cc_test. Add a test covering the behavior. PiperOrigin-RevId: 504294442 Change-Id: If9f96b631ca958e746613563c56103044c973277 (cherry picked from commit a63d8eb) Co-authored-by: Googler <[email protected]>
The `isolate` parameter of `use_extension` and the `is_isolated` field on `module_ctx` are only available with the `--experimental_isolated_extension_usages` flag. Closes bazelbuild#19021. PiperOrigin-RevId: 550582124 Change-Id: I7486a3b344ca9d29614c26044f1943bfedfcc654 Co-authored-by: Fabian Meumertzheim <[email protected]>
When set to true, this flag causes Bazel to abort the build whenever it encounters an action that is not cached. This is very useful when trying to troubleshoot action caching issues across machines because it allows running a build on one and having it fail on another as soon as there is a problem without tainting what already exists in the cache. My workflow is to essentially do: 1. Machine 1: bazel clean 2. Machine 1: bazel build ... 3. Machine 2: bazel clean 4. Machine 2: bazel build --remote_require_cached ... which makes step 4 fail on the first action that wasn't cached as expected. Then I can address that problem and re-run step 4 to encounter the next issue. Closes bazelbuild#18942. PiperOrigin-RevId: 549242966 Change-Id: Ib46a2eb8cce6f4444968882e99c21284fc6bc4f8 Co-authored-by: Julio Merino <[email protected]>
…d#19088) * Advertise CcInfo from cc_import Fixes bazelbuild#19056 * Update BUILD.builtin_test Co-authored-by: oquenchil <[email protected]>
…azelbuild#19049) In the following PR we extended most of the REv2 *Request messages to take an explicit digest function: bazelbuild/remote-apis#235 This eliminates the ambiguity between digest functions that have the same hash length (e.g., MD5 and MURMUR3, SHA256 and SHA256TREE). This change extends the REv2 client in Bazel to set the digest_function field explicitly, so that the server does not need to use any heuristics to pick a digest function when processing requests. As we are now going to call DigestUtil.getDigestFunction() far more frequently, alter DigestUtil to precompute the value upon construction. This change was tested by letting Bazel forward all traffic through an instance of bb_clientd that was patched up to require the use of `digest_function`. PiperOrigin-RevId: 520074622 Change-Id: Ie92dc368c502b9bc3fddef56a5eb0a238760f673 (cherry picked from commit 0a8380b) Signed-off-by: Brentley Jones <[email protected]> Co-authored-by: Ed Schouten <[email protected]>
…19134) Users only have to copy&paste a single command this way. Closes bazelbuild#19119. PiperOrigin-RevId: 551887901 Change-Id: Id285563e83a18ac9a4db3b2d4e8d731d5b3f5813 Co-authored-by: Fabian Meumertzheim <[email protected]>
…azelbuild#19164) Fixes errors such as the following when a module provides two extensions that share a name: ``` ERROR: /my/workspace/library_path/BUILD:4:13: no such package '@_main~my_ext~2~xyz//': The repository '@_main~my_ext~2~xyz' could not be resolved: Repository '@_main~my_ext~2~xyz' is not defined and referenced by '//library_path:library_name' ERROR: Analysis of target '//my_path:my_target' failed; build aborted: ``` This was a consequence of the extension unique names followed by `~` being ``` _main~my_ext~2~ _main~my_ext~ ``` since 19a9710. Before, they were of the following form, which was prefix-free: ``` _main~my_ext2~ _main~my_ext~ ``` Fixes bazelbuild#19155 Closes bazelbuild#19156. PiperOrigin-RevId: 553567725 Change-Id: I98650663fea3bfee77752a06a99132e507d91aef Co-authored-by: Fabian Meumertzheim <[email protected]>
* Add 'environ' attribute to module extensions and update lockfile - Add 'environ' to module extensions - Store the variables and their values in the lockfile - Compare the variables and values with the lockfile and re-evaluate or error if they differ PiperOrigin-RevId: 548964193 Change-Id: Ic2d52fe3332e93095c414d8bca4c9b4312bca8c2 # Conflicts: # src/main/java/com/google/devtools/build/lib/bazel/bzlmod/ModuleExtension.java # src/main/java/com/google/devtools/build/lib/bazel/bzlmod/SingleExtensionEvalFunction.java # src/main/java/com/google/devtools/build/lib/bazel/repository/starlark/StarlarkRepositoryModule.java # src/main/java/com/google/devtools/build/lib/skyframe/ActionEnvironmentFunction.java # src/main/java/com/google/devtools/build/lib/skyframe/ClientEnvironmentFunction.java # src/main/java/com/google/devtools/build/lib/starlarkbuildapi/repository/RepositoryModuleApi.java # src/main/java/com/google/devtools/build/skydoc/fakebuildapi/repository/FakeRepositoryModule.java # src/test/java/com/google/devtools/build/lib/bazel/bzlmod/StarlarkBazelModuleTest.java # src/test/py/bazel/bzlmod/bazel_lockfile_test.py * Remove extra changes * Re-run extension if its files change fixes bazelbuild#19068 PiperOrigin-RevId: 552823534 Change-Id: I87256b2cf954b932e24c70e22386020599f21a6f * Do not fail on new fields added to lockfile data Fixes bazelbuild#19105 PiperOrigin-RevId: 553068023 Change-Id: I877bc8ece0641c01119a9295e09175a2d0a3a0c1 --------- Co-authored-by: Ian (Hee) Cha <[email protected]>
Using 'local = True' seems to run this too frequently. Realistically the user's installed Xcode versions don't change very often so similar to cc_autoconf this should be ok. Closes bazelbuild#19121. PiperOrigin-RevId: 553557177 Change-Id: I66368ec31d97badd24814aa13177261099e56dc7 Co-authored-by: Keith Smiley <[email protected]>
…d. This will be useful for testing later. (bazelbuild#19179) * Format android_sdk_repository_template.bzl. RELNOTES: None PiperOrigin-RevId: 527336290 Change-Id: I4212e7c6f8ffd912bcf89b47f5a0ec182866badb * Add a Starlark flag that allows disabling proguard. This will be useful for testing later. PiperOrigin-RevId: 527367618 Change-Id: Ia2054aa4b85b52e054791ddb0cac3b72452e51ca
…ld#19196) Ideally we would also include the location of the offending bazel_dep, but that requires a (slightly?) bigger change to record the locations of all bazel_deps. Fixes bazelbuild#17126 PiperOrigin-RevId: 554406088 Change-Id: I46865c31dc983de18ac64a37e8ee15ebec624937 Co-authored-by: Googler <[email protected]>
All dependencies for `select` expressions are collected in an implicit `$config_dependencies` attribute. Even with `--incompatible_visibility_private_attributes_at_definition`, visibility for this attribute is now checked relative to the use rather than the definition of the rule. Fixes bazelbuild#19126 Closes bazelbuild#19139. PiperOrigin-RevId: 553830083 Change-Id: Ic4af0f5ad9a0c627c973cd5ce88dd1e1bf51474e Co-authored-by: Fabian Meumertzheim <[email protected]>
* Add BLAKE3 digest function to remote_execution.proto (cherry picked from commit e382abf) * Add BLAKE3 source code to third_party (cherry picked from commit 0ff4f6d) * Add BLAKE3 source code to third_party This PR adds the BLAKE3 C and asm sources to third_party, and includes a BUILD file to build them. PiperOrigin-RevId: 541539341 Change-Id: I49b1edce20a7d0f986e29712e6050e4e0b9c1d44 (cherry picked from commit a3a569e) * Support new-style digest functions Support new-style digest functions. This PR adds support for new-style digest functions to the remote execution library code. The remote-apis spec says: ``` // * `digest_function` is a lowercase string form of a `DigestFunction.Value` // enum, indicating which digest function was used to compute `hash`. If the // digest function used is one of MD5, MURMUR3, SHA1, SHA256, SHA384, SHA512, // or VSO, this component MUST be omitted. In that case the server SHOULD // infer the digest function using the length of the `hash` and the digest // functions announced in the server's capabilities. ``` PiperOrigin-RevId: 543691155 Change-Id: If8c386d923db1b24dff6054c8ab3f783409b7f13 (cherry picked from commit 88412ce) * Add BLAKE3 hasher to vfs This PR adds the Blake3Hasher and Blake3HashFunction classes to vfs and makes them available under the flag --digest_function=BLAKE3. PiperOrigin-RevId: 550525978 Change-Id: Iedc0886c51755585d56b4d8f47676d3be5bbedba (cherry picked from commit cc49d68) * Simplify blake3 hasher and improve performance Improve BLAKE3 performance by reducing copies. Rather than buffering bytes, it's faster to just update the blake3 hasher as bytes are hashed. Performance with buffering: ``` Benchmark (inputSize) Mode Cnt Score Error Units Blake3Benchmark.blake3Hash 1 avgt 10 1766.697 ± 709.515 ns/op Blake3Benchmark.blake3Hash 10 avgt 10 1466.253 ± 19.474 ns/op Blake3Benchmark.blake3Hash 100 avgt 10 1522.845 ± 15.480 ns/op Blake3Benchmark.blake3Hash 1000 avgt 10 2254.156 ± 8.588 ns/op Blake3Benchmark.blake3Hash 10000 avgt 10 4660.881 ± 28.637 ns/op Blake3Benchmark.blake3Hash 100000 avgt 10 24283.191 ± 32.754 ns/op Blake3Benchmark.sha256Hash 1 avgt 10 742.091 ± 6.307 ns/op Blake3Benchmark.sha256Hash 10 avgt 10 757.844 ± 12.042 ns/op Blake3Benchmark.sha256Hash 100 avgt 10 942.902 ± 555.874 ns/op Blake3Benchmark.sha256Hash 1000 avgt 10 1208.336 ± 508.392 ns/op Blake3Benchmark.sha256Hash 10000 avgt 10 4871.231 ± 494.507 ns/op Blake3Benchmark.sha256Hash 100000 avgt 10 40686.231 ± 63.814 ns/op ``` Performance without buffering (after this CL): ``` Benchmark (inputSize) Mode Cnt Score Error Units Blake3Benchmark.blake3Hash 1 avgt 10 1021.075 ± 11.640 ns/op Blake3Benchmark.blake3Hash 10 avgt 10 1029.561 ± 19.850 ns/op Blake3Benchmark.blake3Hash 100 avgt 10 1070.509 ± 12.140 ns/op Blake3Benchmark.blake3Hash 1000 avgt 10 1685.043 ± 13.963 ns/op Blake3Benchmark.blake3Hash 10000 avgt 10 3939.516 ± 13.212 ns/op Blake3Benchmark.blake3Hash 100000 avgt 10 21730.550 ± 22.976 ns/op Blake3Benchmark.sha256Hash 1 avgt 10 745.943 ± 9.853 ns/op Blake3Benchmark.sha256Hash 10 avgt 10 747.649 ± 17.381 ns/op Blake3Benchmark.sha256Hash 100 avgt 10 962.802 ± 590.879 ns/op Blake3Benchmark.sha256Hash 1000 avgt 10 1189.069 ± 412.327 ns/op Blake3Benchmark.sha256Hash 10000 avgt 10 4594.978 ± 21.833 ns/op Blake3Benchmark.sha256Hash 100000 avgt 10 39224.248 ± 229.265 ns/op ``` PiperOrigin-RevId: 551225043 Change-Id: I57dc0700b2f44d6faf75ffbd29f1607544e54f29 (cherry picked from commit d922ab3) * Ensure namedSetOfFiles URIs specify blob type correctly I noticed when testing the BLAKE3 digest function that uploaded files were being referenced incorrectly in the BES because they were missing the digest type. This CL fixes that. Before: https://github.com/bazelbuild/bazel/assets/141737/52781f1b-b897-48f0-8956-f63c57b59436 After: https://github.com/bazelbuild/bazel/assets/141737/01ebc61b-3512-4ca5-8e2d-f47ad5f086b7 PiperOrigin-RevId: 553405394 Change-Id: Ic976e5a58f80ab8b5270b4aedc6204c22562533a (cherry picked from commit 1929367) Signed-off-by: Brentley Jones <[email protected]> * Separate new-style-hash content in DiskCache DiskCache always stores files in /root/{cas/ac}/digestHash. This change keeps the current behavior, but for new style digest functions inserts a directory between /root/ and {cas/ac} to disambiguate the digest function type. This prevents issues that could theoretically happen if there were hash collisions between two digest functions sharing the same cache directory. PiperOrigin-RevId: 554477775 Change-Id: Ibef994e432764c260a3cab821ca6583c231c5b50 (cherry picked from commit f17b280) Signed-off-by: Brentley Jones <[email protected]> --------- Signed-off-by: Brentley Jones <[email protected]> Co-authored-by: Tyler Williams <[email protected]>
…bazelbuild#19198) Otherwise, the config settings will not be correctly resolved resulting in build error when compiling blake3. PiperOrigin-RevId: 550568252 Change-Id: I8323f7f89aae6ca47db20fa8d177c027c754e1fb (cherry picked from commit 12ac28c) Co-authored-by: Googler <[email protected]>
The disk cache root directory was previously being manually created in two places before creating the disk cache client. This CL instead ensures that the disk cache root directory is created when createDiskCache() is called, and simplifies the other callsites. This fixes an issue where the disk cache directory might not exist if using --digest_function=BLAKE3. PiperOrigin-RevId: 555429326 Change-Id: Ifcfa8c686df30c03c9fca24be860b2db780a6374 (cherry picked from commit 00dfa2e) Co-authored-by: Tyler Williams <[email protected]>
…w `--ouput=streamed_jsonproto` implementation. (bazelbuild#19226) Fix valid json when using jsonproto output in queries with new `--ouput=streamed_jsonproto` implementation. Closes bazelbuild#18701. Commit bazelbuild@2cd583a PiperOrigin-RevId: 555417403 Change-Id: I30eb06f734188f8511884954f43c5f5b3c0091a3 Co-authored-by: Chirag Ramani <[email protected]>
The `bootclasspath` rule in `rules_java` will soon use this toolchain type instead of the regular Java runtime toolchain type, which naturally carries a constraint on the target platform. Also adds more detailed explanations of the now three Java toolchain types. Work towards bazelbuild#17085 Work towards bazelbuild#18265 Work towards bazelbuild/rules_java#64 Split off from bazelbuild#18262 Closes bazelbuild#18841. PiperOrigin-RevId: 545756139 Change-Id: Ib9dd7a1c20c32315375722d6a023a89859daea9c Fixes bazelbuild#19201
…#19076) Also deletes some of the implemented methods (the ones not used natively) and methods which are obsolete. PiperOrigin-RevId: 483673510 Change-Id: Ifb0179f7ba6de2be19af6563f7e94ddca42b844f Co-authored-by: Googler <[email protected]>
Previously, `str(native.cc_test)` included `cc_test_wrapper`, which is an implementation detail of `cc_test`. Closes bazelbuild#18871. Commit 70e31da PiperOrigin-RevId: 546836622 Change-Id: I80e02a2be80ab01610c5848ab1cd506df11c9557
Cherry-picked three commits (bazelbuild@138f06f , bazelbuild@d0de5e0 and bazelbuild@5647b26) to fix a Windows issue. 1.Disable using NEON instructions on Windows arm64 Commit bazelbuild@d0de5e0 PiperOrigin-RevId: 555467146 Change-Id: Ib34811467d7f0f6980bfa4976b8d1ba2fb041b16 2.Select the correct toolchain for Windows arm64 after enabling cc tool… Commit bazelbuild@138f06f PiperOrigin-RevId: 555413912 Change-Id: I3ccc0d51973849aaa891dd0944a00432ce3d2cfb 3.Enable cc toolchain resolution for building Bazel Commit bazelbuild@5647b26 PiperOrigin-RevId: 551218176 Change-Id: Ia01c3b83b071b50b353f116dbd360bd3fe04f0c3 --------- Co-authored-by: Googler <[email protected]> Co-authored-by: Chi Wang <[email protected]>
…ion (bazelbuild#19281) Bash completion now also recognized `WORKSPACE.bazel`, `MODULE.bazel` and `REPO.bazel`. Also fixes a shellcheck finding. Closes bazelbuild#19268. Commit bazelbuild@a0f1976 PiperOrigin-RevId: 557931944 Change-Id: Ia2b6e463f4643d9c0d829845fee4228103cf90a0 Co-authored-by: Fabian Meumertzheim <[email protected]>
…build#19283) This allows non-hermetic information (such as Starlark `Location`s) to be included in `fail` messages without allowing access to it from Starlark. It also aligns the content of `fail` output with that of `print`. This change has very mild implications for backwards compatibility: it only affects the contents of `fail` messages, which aren't accessible from Starlark; `debugPrint`'s default implementation is `str` and no built-in Starlark type overrides it; if a type overrides `debugPrint`, it would usually contain at least as detailed information as `str`; `debugPrint` is a "Bazelism" that isn't governed by the Starlark spec. Work towards bazelbuild#17375 Closes bazelbuild#18818. Co-authored-by: Alexandre Rostovtsev <[email protected]> Commit bazelbuild@3abbf20 PiperOrigin-RevId: 557916312 Change-Id: I8f202cd8530bcebb2d99f57745289b3992d03cac Co-authored-by: Fabian Meumertzheim <[email protected]> Co-authored-by: Alexandre Rostovtsev <[email protected]>
For an exported rule `foo_library`, `repr(foo_library)` now evaluates to `<rule foo_library>`, not `<rule>`, matching the behavior of native rules more closely. Fixes bazelbuild#17483 Closes bazelbuild#18392. Commit bazelbuild@002490b PiperOrigin-RevId: 538458291 Change-Id: I331955655756a3c235be0a93f1394716ddf9849d Co-authored-by: Fabian Meumertzheim <[email protected]>
…ld#19300) By caching `BazelRegistryJson` in `IndexRegistry` and caching `IndexRegistry` instances per registry URL, `bazel_registry.json` is only downloaded once per registry instead of once for each module in the final dependency graph in `computeFinalDepGraph`. On my local machine, this shaves 4s off of the time spent on module resolution for Bazel itself. Closes bazelbuild#19292. Commit bazelbuild@8337dd7 PiperOrigin-RevId: 558940780 Change-Id: I89b03a4c246b10f39b89a79852c922a6504f00bf Co-authored-by: Fabian Meumertzheim <[email protected]>
…sages (bazelbuild#19285) When passed to `print` or `fail`, a module extension tag now results in a string such as `'foo' tag at /ws/MODULE.bazel:3:4`, which can be used to form error messages referencing tags without leaking non-hermetic information to the extension implementation function. Closes bazelbuild#19274. Commit bazelbuild@cc5889c PiperOrigin-RevId: 558163928 Change-Id: If20086c62356bb0635d0a4bdf91029267122f62b Co-authored-by: Fabian Meumertzheim <[email protected]> Co-authored-by: Xùdōng Yáng <[email protected]>
Fixes: bazelbuild#18181. Closes: bazelbuild#18280. Closes bazelbuild#18280. PiperOrigin-RevId: 530246609 Change-Id: I6b49d1bd827e98ca65db87c34c1bb13106ffc232 (cherry picked from commit 978cd23) Co-authored-by: David Ostrovsky <[email protected]>
…ld#19836) Fixes bazelbuild#19146 Closes bazelbuild#19725. Commit bazelbuild@e55fee2 PiperOrigin-RevId: 573751305 Change-Id: I9b5df85dc5e52822b3a0b44fc42d90b727a5abf0 Co-authored-by: Thi Don <[email protected]>
bazelbuild#19848) …update Ensure that `BazelLockFileModule` only updates `MODULE.bazel.lock` if the content of the file needs to change. Every such update changes the file's metadata, which results in Skyframe invalidation of, in particular, all configurations. This broke `bazel config`, which uses `MemoizingEvaluator#getDoneValues()` to directly observe Skyframe state. Compared to the original commit bazelbuild@78db9ae, this cherry-pick does not include the change to `bazel config` as it may not be backwards compatible (changes the exit code in certain situations). Fixes bazelbuild#19823 Closes bazelbuild#19842. PiperOrigin-RevId: 574133346 Change-Id: I5886c91fc6b7b938a7dee59ea75aa7b8afb5b161 Fixes bazelbuild#19843
Baseline: 0f231ac Release Notes: + Support multiple remote execution digest functions (bazelbuild#19042) + Release 6.4.0 remote (bazelbuild#18959) + Move BazelFileSystemModule into bazel package (bazelbuild#19043) + Fix a bug where frozen targets list was mutated while expanding env attribute (bazelbuild#19053) + Mark isolated extension usages as experimental (bazelbuild#19065) + Add the remote_require_cached flag (bazelbuild#19075) + Advertise CcInfo from cc_import (bazelbuild#19086) (bazelbuild#19088) + Update java_tools version to 12.6 (bazelbuild#19092) + Set the digest_function field as part of all relevant gRPC requests (bazelbuild#19049) + Merge `use_repo` buildifier fixups into a single command (bazelbuild#19134) + Ensure that extension unique names followed by `~` are prefix-free (bazelbuild#19164) + Lockfile updates & fixes (bazelbuild#19153) + Switch xcode_autoconf to use 'configure = True' (bazelbuild#19174) + Cherry pick Add a Starlark flag that allows disabling proguard. This will be useful for testing later. (bazelbuild#19179) + Update CODEOWNERS for 6.4.0 (bazelbuild#19194) + Friendlier error message for `bazel_dep`s without `version` (bazelbuild#19196) + Always check `$config_dependencies` visibility at use (bazelbuild#19197) + Add support for the BLAKE3 digest function (bazelbuild#19191) + Enable cc toolchain resolution when cross compiling to windows arm64. (bazelbuild#19198) + Ensure disk cache root exists (bazelbuild#19225) + Fix valid json when using jsonproto output in queries with new `--ouput=streamed_jsonproto` implementation. (bazelbuild#19226) + Add toolchain type for Java bootstrap runtime (bazelbuild#19220) + Add Starlark implementation for several CcCommon methods. (bazelbuild#19076) + Rename `cc_test_wrapper` to `cc_test` (bazelbuild#19231) + Cherry-pick commits to fix a Windows issue (bazelbuild#19232) + Add support for more workspace boundary files to bash completion (bazelbuild#19281) + Use `debugPrint` instead of `str` for `fail` arguments (bazelbuild#19283) + Include name in `repr` of exported `rule`s (bazelbuild#19229) + Download `BazelRegistryJson` only once per registry (bazelbuild#19300) + Make module extension tag's `debugPrint` useful for error messages (bazelbuild#19285) + Intern repository mapping entries (bazelbuild#19293) + Add `additional_linker_inputs` option to `cc_library` rule (bazelbuild#19264) + Do not rerun module extensions when only imports or locations change (bazelbuild#19284) + Add profiling for Bzlmod operations (bazelbuild#19313) + Retry on javax.net.ssl.SSLException ... BAD_DECRYPT (bazelbuild#19346) + Fetch `RepoSpecs` in parallel (bazelbuild#19354) + Make `MODULE.bazel.lock` deterministic (bazelbuild#19370) + Ensure lockfile is updated after reset to pre-build state (bazelbuild#19371) + build-runfiles: remove temporary file prior to creating it (bazelbuild#19386) + Always fail on unknown attributes (bazelbuild#19404) + Ignore Starlark options on commands with `allowResidue = False` (bazelbuild#19417) + Separate PackageSpecificationProvider from its target (PackageGroupConfiguredTarget) (bazelbuild#19420) + Expose PackageSpecificationInfo provider as a top level symbol (bazelbuild#19422) + Revert "Report remote execution messages as events" (bazelbuild#19415) + [6.4] Add --incompatible_disable_objc_library_transition (bazelbuild#19393) + Create .bazelversion to address postsubmit timeout issues (bazelbuild#19435) + Add `contains` method inside `PackageSpecificationProvider` (bazelbuild#19425) + Wrong include path to Clang 16 on Windows (bazelbuild#19430) + Simplify release notes by just printing the first line of the commit … (bazelbuild#19448) + Remove PackageGroupConfiguredTarget.isAvailableFor function (bazelbuild#19444) + Remove default -s flag from macOS libtool invocation (bazelbuild#19454) + Turn off lockfile feature by default (bazelbuild#19462) + Take the no-remote-exec tag into account when computing the action salt (bazelbuild#19457) + Add `--incompatible_merge_fixed_and_default_shell_env` (bazelbuild#19319) + Improve error when a label is provided in `config_setting`'s `values` (bazelbuild#19484) + Mark tool inputs in the execution log. (bazelbuild#19483) + Add visionOS support (bazelbuild#19436) + Intern empty `Depset`s (bazelbuild#19443) + Do not allow applicable_licenses on platform. (bazelbuild#19426) + Cherry pick Bzlmod fixes (bazelbuild#19494) + Optimize classpath pre-processing in java_stub_template.txt (bazelbuild#19491) + Add output name to CacheNotFoundException (bazelbuild#19452) + feat: add option to exit early if analysis cache is discarded (bazelbuild#19503) + Cherry pick platform dependent lockfile (bazelbuild#19498) + Print Passed and Failed methods in detailed test summary (bazelbuild#19505) + Add formatted timestamp entries to volatile workspace status file. (bazelbuild#19499) + Write an explicit line ending to the lockfile (bazelbuild#19519) + Only use `/showIncludes` if supported (bazelbuild#19521) + Also apply `NestedSet` optimizations to `Depset` (bazelbuild#19492) + Add diff_against_dynamic_baseline option to experimental_output_direc… (bazelbuild#19514) + Update java_tools to v12.7 (bazelbuild#19522) + Advertise CcInfo from cc_proto_library (bazelbuild#19534) + Update unknown Xcode version error message and provide an environment variable to force re-evaluation (bazelbuild#19540) + Print dep chain leading to a module that is in error (bazelbuild#19543) + Show fetch progress for the `mod` command (bazelbuild#19542) + Revert "Switch xcode_autoconf to use 'configure = True' (bazelbuild#19174)" (bazelbuild#19550) + Explain the use of `str(Label(...))` in the docs (bazelbuild#19554) + Add `--consistent_labels` flag to all query commands (bazelbuild#19567) + Inject builtin modules at the end of the MODULE.bazel file (bazelbuild#19573) + Disable bzlmod_query_test for RBE build (bazelbuild#19585) + Clear runfiles environment variables for `bazel run` (bazelbuild#19606) + cc_library: propagate data dependencies via implementation_deps. (bazelbuild#19590) + Error on potential unsupported `/showIncludes` lines (bazelbuild#19611) + Use case-insensitive comparison for Windows paths in `runfiles.bash` (bazelbuild#19626) + Show test labels in summaries in display form (bazelbuild#19625) + Remove stale extension entries from lockfile (bazelbuild#19683) + Keep leading zero in formatted date (bazelbuild#19694) + Bzlmod lockfile: fix pretty printing for attributes (bazelbuild#19691) + MODULE.bazel.lock file contains user specific paths (bazelbuild#19698) + Consider RCs equivalent to release for `bazel_compatibility` (bazelbuild#19689) + Use `Label` in `@bazel_tools//tools/jdk` macros (bazelbuild#19675) + Remove stale extension entries from lockfile if module order changes (bazelbuild#19730) + Update rules_java 5.5.1 (bazelbuild#19701) + Fix Java compilation for jdk21 (bazelbuild#19735) + Fix output materialized as symlink when building without the bytes. (bazelbuild#19739) + Make lockfile's `RepoSpec` attributes more readable (bazelbuild#19748) + Merge rule and aspect validation output groups (bazelbuild#19745) + Fix handling of non-ASCII characters in archive entry file names (bazelbuild#19765) + Raise an early error on invalid labels in transitions inputs/outputs (bazelbuild#19764) + Handle synthetic method parameters entries that don't have names (bazelbuild#19758) + Bazel release process: Fix push justification. (bazelbuild#19768) + Flip --experimental_cc_implementation_deps (bazelbuild#19751) + Add blake3 NEON instructions on linux arm64 (bazelbuild#19804) + Fix crash when `environ` contains duplicate entries (bazelbuild#19827) + Bump c++ standard to c++14 per default (bazelbuild#19794) + Collect debug info context from implementation deps (bazelbuild#19836) + Fix unconditional Skyframe invalidation with --lockfile_mode=… (bazelbuild#19848) Acknowledgements: This release contains contributions from many people at Google, as well as Andreas Herrmann, bazel.build machine account, Brentley Jones, buildbreaker2021, Chirag Ramani, David Ostrovsky, Ed Schouten, Fabian Meumertzheim, George Gensure, Greg, John Laxson, Julio Merino, Keith Smiley, Matt Mackay, Mauricio G, NelsonLi0701, nglevin, Nicholas Junge, oquenchil, Orion Hodson, Roman Salvador, Ted Kaplan, Thi Doan, Thi Don, Tyler Williams, Xùdōng Yáng.
jmmv
force-pushed
the
redact-bep
branch
2 times, most recently
from
19e0f23 to
89c4b00
Compare
…ilding without the bytes. (bazelbuild#20409) This is the same bug as bazelbuild#19143, except that the fix in 3a48457 missed the case where the symlink occurs inside an output directory. Fixes bazelbuild#20408.
Backported from bazelbuild@367994a Fixes bazelbuild#20468
…on (bazelbuild#20520) Cherry-picks: bazelbuild@7a537ca, bazelbuild@0656103, bazelbuild@bc1df91, bazelbuild@7b97147 Fixes bazelbuild#20480 --------- Co-authored-by: Fabian Meumertzheim <[email protected]>
…lbuild#20531) By upgrading the Apache Commons Compress library to 1.20. I'm deliberately not upgrading to the most recent one (1.24.0) because it would require an additional JDK module (java.desktop) and significantly regress the binary size. Fixes bazelbuild#20090. Closes bazelbuild#20110. PiperOrigin-RevId: 580935354 Change-Id: I6c9728ac3fd925432f44a55efaef8f5b52d428c0 Co-authored-by: Tiago Quelhas <[email protected]>
…azelbuild#20550) After an action was executed remotely, RemoteSpawnRunner would use the timestamps in the execution metadata to record appropriate timing phases into the JSON profile. However, there are durations in-between the existing phases that are unaccounted for. Depending on the RBE server implemenation, these phases could mean different things: - Sandbox preparation - Cleaning up sandbox environments post-execution - Others Missing these durations inside the timing profile would cause confusion to end users as it would be interpreted as nothing happened in between the existing phases. Add these durations into the profile as "pre-X" phases so that user is aware of activities could still be happening during that time. RBE server implementation should be able to alter these label programmatically if necessary. Closes bazelbuild#20387. Commit bazelbuild@fe9e9e0 PiperOrigin-RevId: 590816782 Change-Id: I2bee36be928db24a14fab18bc519c3893723b7d6 Co-authored-by: Son Luong Ngoc <[email protected]>
…azelbuild#20549) Without this there can be large gaps in the profile when using remote execution that are hard to reason about. Closes bazelbuild#20474. Commit bazelbuild@75fffbf PiperOrigin-RevId: 590475989 Change-Id: Ic6e042c36f85e8098a468c73c62bd45cc367423e Co-authored-by: Brentley Jones <[email protected]>
Due to bazelbuild#20660 RELNOTES: None Commit bazelbuild@d9dc4fd PiperOrigin-RevId: 593115090 Change-Id: Ifc2a282dbd6dd8a3abfec987398388e5844af91c --------- Co-authored-by: Googler <[email protected]> Co-authored-by: Ian (Hee) Cha <[email protected]>
) This should fix https://buildkite.com/bazel-trusted/publish-bazel-binaries/builds/19494, which was caused by recent OS update.
…tall_base` (bazelbuild#20568) Currently if the `--install_base` path passed is not writable by the user invoking Bazel, the Bazel client crashes: ```console ❯ bazel --install_base=/some/read/only/path version FATAL: failed to set timestamp on '/some/read/only/path': (error: 30): Read-only file system ``` This happens because the Bazel client (unconditionally) attempts to update the `mtime` of this path: https://github.com/bazelbuild/bazel/blob/a3c677dfea2de636a719d50345a5a97af96fae60/src/main/cpp/blaze.cc#L1010-L1021 This commit updates the client to ignore such errors. See bazelbuild#20373 for context. Closes bazelbuild#20442. Commit bazelbuild@7f782e3 PiperOrigin-RevId: 591266054 Change-Id: If53e7cad48cb62406f7883f72b413e4b5a0bb8e2 Co-authored-by: Rahul Butani <[email protected]> Co-authored-by: Ian (Hee) Cha <[email protected]>
…onment (bazelbuild#20667) When a repository rule is fetch attributes are iterated over in `enforceLabelAttributes` to prepopulate the environment, restarting the fetch each time a new dependency is discovered. This is faster than calling `repository_ctx.path(...)` early in the repository rule implementation function but still has considerable overhead. This PR defers throwing `NeedsSkyframeRestartException` till after every attribute has been processed, greatly reducing the number of restarts and iterations. In an internal project these optimisations are particularly noticeable. Before: 2min 8s, 2min 7s After: 1min 35s, 1min 27s Closes bazelbuild#20434. Commit bazelbuild@e8ac96a PiperOrigin-RevId: 588090528 Change-Id: I7917b137d6e60b6d6a73189cf396418a85b3ec28 Co-authored-by: Jordan Mele <[email protected]> Co-authored-by: Xùdōng Yáng <[email protected]>
For the bootstrap VanillaJavaBuilder, it is important that the AutoValue plugin classes do not end up in the deploy jar. If they do, the processor class is loaded from the app classloader (instead of the processor path classloader). When this happens, AutoValueProcessor uses the app classloader to load extensions (such as auto-value-gson needed by bzlmod), instead of the processor path classloader. Unless all extensions are also correctly present in the app classloder (i.e. in the VanillaJavaBuilder deploy jar), they won't be loaded. Unfortunately, simply adding the jars to the deploy jar is insufficient, we need to also correctly merge the `META-INF/service/...` files as well, which does not happen in our bootstrap java_binary/java_library rules. So, to fix, we remove the auto value plugins from the deploy jar, and use them only to compile our sources that require them. Added a regression test for the crash. We should probably improve it so that we can actually build with the bootstapped Bazel in tests without network access. But I don't think we need block this patch release on that. Fixes bazelbuild#20501 PiperOrigin-RevId: 592266992 Change-Id: I08ac91ee74140df0c4f22ad2553a8c017b497181 Co-authored-by: Googler <[email protected]>
…lbuild#20861) to allow users temporarily disable remote action building throttle. Workaround for bazelbuild#20478. Closes bazelbuild#20558. Commit bazelbuild@294c904 PiperOrigin-RevId: 597445193 Change-Id: Ib2c7133adf86139b35156d94e39cbf9e17906439 Co-authored-by: Chi Wang <[email protected]>
The toolchain was passing -l:name. This mechanism doesn' t exist on macOS, instead the full path to the shared library should be passed. Mainline commit: bazelbuild@f0ade80 As it says on that mainline commit, it cannot be checked in with a test unlike this PR, because the macOS toolchain was moved outside of Bazel. Therefore, the change must go in, the macOS toolchain needs to be updated, then the test can be added. This change does go in with a test because the toolchain is still embedded in Bazel for 6.5. Fixes bazelbuild#20487
Fixes: bazelbuild#20921 List of cherrypicks: 5a24c8a [email protected] Mon Oct 9 02:27:10 2023 -0700 Support automatic exec groups in proto_common.compile 3e1e061 [email protected] Tue Oct 3 03:32:01 2023 -0700 Use proto toolchains in cc_proto_library 8c38be3 [email protected] Mon Oct 2 07:08:55 2023 -0700 Use proto toolchains in java_lite_proto_library 51970d2 [email protected] Mon Oct 2 07:05:16 2023 -0700 Use proto toolchains in java_proto_library 20bc11f [email protected] Mon Oct 2 06:56:47 2023 -0700 Decouple java_lite_proto_library from java_proto_library 3b18d3f [email protected] Mon Oct 2 04:11:24 2023 -0700 Refactor proto toolchainsation support utilities 42800a8 [email protected] Fri Sep 22 03:04:18 2023 -0700 Use proto compiler from proto_toolchain rule d435c6d [email protected] Thu Sep 21 07:05:40 2023 -0700 Use proto_toolchain in proto_library f5fb2f6 [email protected] Wed Sep 20 05:50:59 2023 -0700 Remove protoc from proto_lang_toolchain rule 108ef55 [email protected] Tue Sep 19 08:40:00 2023 -0700 Use MockProtoSupport.setup where protos are used 11cf1b7 [email protected] Sun Sep 17 21:57:54 2023 -0700 Implement incompatible_enable_proto_toolchain_resolution
jmmv
pushed a commit
that referenced
this pull request
Feb 2, 2024
This is to facilitate moving the rule implementation out of Bazel itself and into rules_python. This also makes the tests pass when an allowlist is used to enforce that the Python rules are being loaded through rules_python instead of used from builtins. This the same change as before, but with some internal fixes applied. PiperOrigin-RevId: 565113515 Change-Id: I999dbfaace8cce2168e718093fe418d548ab1b73
jmmv
pushed a commit
that referenced
this pull request
Feb 2, 2024
This is to facilitate moving the rule implementation out of Bazel itself and into rules_python. This also makes the tests pass when an allowlist is used to enforce that the Python rules are being loaded through rules_python instead of used from builtins. This the same change as before, but with some internal fixes applied. PiperOrigin-RevId: 565113515 Change-Id: I999dbfaace8cce2168e718093fe418d548ab1b73 Partial commit for third_party/*, see bazelbuild#19472. Signed-off-by: Hee Cha <[email protected]>
jmmv
pushed a commit
that referenced
this pull request
Feb 2, 2024
… order.
1. `TestAttempt` events would wait for the `TargetCompleteEvent` to be posted before being posted.
2. There was an implicit requirement for the `TestAttempt` events to be posted in a specific order.
3. This didn't break in the noskymeld case because we fulfilled this ordering by using the order of performing the attempts themselves. The sequence would look like:
+ post `TargetCompleteEvent`
-> perform attempt #1
-> post `TestAttempt` #1
-> perform attempt #2
-> post `TestAttempt` #2
4. With skymeld, however, it could happen like this:
+ defer `TargetCompleteEvent` to wait for `CoverageActionFinishedEvent`
+ perform attempt #1 -> defer posting `TestAttempt` #1 & wait for `TargetCompleteEvent`
+ perform attempt #2 -> defer posting `TestAttempt` #2 & wait for `TargetCompleteEvent`
+ `CoverageActionFinishedEvent` -> release & post `TargetCompleteEvent`
+ `TargetCompleteEvent` -> release & post `TestAttempt` #2
+ `TargetCompleteEvent` -> release & post `TestAttempt` #1
Due to (2), the undefined ordering in (4) would cause an issue.
This CL fixes that by ensuring a FIFO ordering of the deferred events.
PiperOrigin-RevId: 572165337
Change-Id: Iac4d023d946865b8b81f15b119417192dc4b5c53
jmmv
pushed a commit
that referenced
this pull request
Feb 28, 2024
*** Reason for rollback *** In the past few days, I have done more research to compare between (1) adding a new parallelism (which is the follow-up work of this cl) and (2) using the existing skyframe FJP. And I am now more inclined to adopt #2. PiperOrigin-RevId: 610840138 Change-Id: I8caae6641103e5b712bf34f32d02f88415ee23fa
This change adds logic to clean up the contents of the BEP before it
leaves Bazel in an attempt to prevent propagating secrets to a remote
service.
In particular, this implements protections for the following:
* Secrets stored in the environment which are unconditionally leaked
via the hidden --client_env flag.
* Secrets passed via --bes_header, --test_env, --test_arg, and similar
options. This is not completely safe to do because a determined
user could inject secrets into the BEP in other ways. So... treat
this as just a safety net for common innocent mistakes.
* Secrets passed to tools run via "bazel run". Think, for example,
doing "bazel run //some/docker:command -- --password 1234".
For the purposes of this change, a "secret" is whatever string we did
not explicitly allow. This code does not implement secret detection
and it does not indend to, because such code would be based on
heuristics as well.
Note that services like BuildBuddy implement their own redaction of
secrets when they receive the BEP... but it's better if we can strip
them out at the source to minimize exposure. Defense in depth.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change adds logic to clean up the contents of the BEP before it leaves Bazel in an attempt to prevent propagating secrets to a remote service.
In particular, this implements protections for the following:
Secrets stored in the environment which are unconditionally leaked via the hidden --client_env flag.
Secrets passed via --test_env, --test_arg, and similar options. This is not completely safe to do because a determined user could inject secrets into the BEP in other ways. So... treat this as just a safety net for common innocent mistakes.
Secrets passed to tools run via "bazel run". Think, for example, doing "bazel run //some/docker:command -- --password 1234".
For the purposes of this change, a "secret" is whatever string we did not explicitly allow. This code does not implement secret detection and it does not indend t, because such code would be based on heuristics as well.
Note that services like BuildBuddy implement their own redaction of secrets when they receive the BEP... but it's better if we can strip them out at the source to minimize exposure. Defense in depth.