Skip to content

Commit

Permalink
chore: optimize out retriggering 2nd stage of build + of wasmopt stag…
Browse files Browse the repository at this point in the history
…e in tests context (#292)

reason for this pr is that [LazyLock initialization
block](https://github.com/NEAR-DevHub/neardevhub-contract/blob/083fdde69d147f43bd320094db1b3a8acf64c841/tests/test_env/mod.rs#L20-L27)
doesn't work to ensure tested contracts are built only once for `cargo
nextest run` as **cargo-nextest** uses a process-per-test runner,
and for usual `cargo test` workflows it may not be convenient/possible
to unite all tests into a single test module, thus contracts' build will
run at least once per each test module.

This pr minimizes time spent on contracts' rebuilding after initial
build, so that LazyLock init block isn't that much needed any more, but
it depends on separating target folders of different contracts being
tested similar to
dj8yfo/neardevhub-contract@cbbf0ee
to avoid partially rebuilding contracts starting with `near-sdk-macros`
crate, if it's shared in common release profile when different contracts
are in a workspace.

---

this results in 

```bash
• Building contract
 │     Finished `release` profile [optimized] target(s) in 0.08s

• Running an optimize for size post-step with wasm-opt...
/home/user/Documents/code/near/neardevhub-contract/target/wasm32-unknown-unknown/release/devhub.wasm -> /tmp/optimized-s2bmSK.wasm
• done

✓ Contract successfully built! (in CARGO_NEAR_BUILD_ENVIRONMENT=host)
     -                Binary: /home/user/Documents/code/near/neardevhub-contract/target/near/devhub.wasm
     - SHA-256 checksum hex : 1f568ea4c5eabf9c947a201f3adf38a4e47056658aa512fdfc722f17de9d3aa3
     - SHA-256 checksum bs58: 37LC32jmZFmZN8KesPipwUn4ZyhW4994coXqSWMdnDQv
     -                   ABI: /home/user/Documents/code/near/neardevhub-contract/target/near/devhub_abi.json
     -          Embedded ABI: /home/user/Documents/code/near/neardevhub-contract/target/near/devhub_abi.zst
    Finished cargo near build in 5s
Here is the console command if you ever need to re-run it again:
cargo near build non-reproducible-wasm
```

instead of

```bash
• Building contract
 │    Compiling near-sdk-macros v5.5.0
 │    Compiling near-sdk v5.5.0
 │    Compiling devhub_common v0.1.0 (/home/user/Documents/code/near/neardevhub-contract/d
evhub_common)
 │    Compiling near-contract-standards v5.5.0
 │    Compiling devhub v0.2.0 (/home/user/Documents/code/near/neardevhub-contract)
 │     Finished `release` profile [optimized] target(s) in 15.52s

• Running an optimize for size post-step with wasm-opt...
/home/user/Documents/code/near/neardevhub-contract/target/wasm32-unknown-unknown/release/d
evhub.wasm -> /tmp/optimized-FbzCfP.wasm
• done

✓ Contract successfully built! (in CARGO_NEAR_BUILD_ENVIRONMENT=host)
     -                Binary: /home/user/Documents/code/near/neardevhub-contract/target/ne
ar/devhub.wasm
     - SHA-256 checksum hex : 1f568ea4c5eabf9c947a201f3adf38a4e47056658aa512fdfc722f17de9d3aa3
     - SHA-256 checksum bs58: 37LC32jmZFmZN8KesPipwUn4ZyhW4994coXqSWMdnDQv
     -                   ABI: /home/user/Documents/code/near/neardevhub-contract/target/ne
ar/devhub_abi.json
     -          Embedded ABI: /home/user/Documents/code/near/neardevhub-contract/target/ne
ar/devhub_abi.zst
    Finished cargo near build in 20s
```
on rebuilds. 

and wasm-opt stage skipping was handled in
e746699
. No file locks were added, it would still waste some cpu, if multiple
builders enter the wasm-opt stage around the same time, but it does save
on time and locks are perceived as unneeded complexity at this time.

---

vaguely related to Near-One/omni-bridge#184

---------

Co-authored-by: dj8yf0μl <[email protected]>
Co-authored-by: Vlad Frolov <[email protected]>
  • Loading branch information
3 people authored Jan 22, 2025
1 parent 4a5d1cc commit 2851dc6
Show file tree
Hide file tree
Showing 4 changed files with 75 additions and 10 deletions.
6 changes: 3 additions & 3 deletions cargo-near-build/src/fs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ pub fn copy(from: &Utf8Path, out_dir: &Utf8Path) -> eyre::Result<Utf8PathBuf> {
/// Copy a file to a file destination.
///
/// Does nothing if the destination is the same as the source to avoid truncating the file.
pub fn copy_to_file(from: &Utf8Path, to: &Utf8Path) -> eyre::Result<Utf8PathBuf> {
pub fn copy_to_file(from: &Utf8Path, to: &Utf8Path) -> eyre::Result<()> {
tracing::debug!("Copying file `{}` -> `{}`", from, to,);

if !from.is_file() {
Expand All @@ -40,14 +40,14 @@ pub fn copy_to_file(from: &Utf8Path, to: &Utf8Path) -> eyre::Result<Utf8PathBuf>
from,
to,
);
return Ok(to.to_path_buf());
return Ok(());
}
}
if from != to {
std::fs::copy(from, to)
.wrap_err_with(|| format!("failed to copy `{}` to `{}`", from, to))?;
}
Ok(to.to_path_buf())
Ok(())
}

/// Create the directory if it doesn't exist, and return the absolute path to it.
Expand Down
15 changes: 15 additions & 0 deletions cargo-near-build/src/near/abi/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,21 @@ pub fn write_to_file(
crate_metadata.formatted_package_name(),
abi_types::file_extension(format, compression)
));

// this prevents doing `touch target/near/{contract_crate_name}_abi.zst` and similar
// and doing a partial project's rebuild during 2nd phase of build (into wasm)
if out_path_abi.is_file() {
let existing_content = std::fs::read(&out_path_abi)?;

if existing_content == near_abi_compressed {
tracing::debug!(
"skipped wrting file `{}` on identical contents",
out_path_abi,
);
return Ok(abi_types::Result { path: out_path_abi });
}
}

std::fs::write(&out_path_abi, near_abi_compressed)?;

Ok(abi_types::Result { path: out_path_abi })
Expand Down
54 changes: 47 additions & 7 deletions cargo-near-build/src/near/build/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -164,12 +164,23 @@ pub fn run(args: Opts) -> eyre::Result<CompilationArtifact> {
)?;

wasm_artifact.path = {
let (from_path, _maybe_tmpfile) =
maybe_wasm_opt_step(&wasm_artifact.path, args.no_wasmopt)?;
crate::fs::copy_to_file(
&from_path,
&out_dir.join(wasm_artifact.path.file_name().expect("has filename")),
)?
let prev_artifact_path = wasm_artifact.path;
let target_path = out_dir.join(prev_artifact_path.file_name().expect("has filename"));

// target file does not yet exist `!target_path.is_file()` condition is implied by
// `is_newer_than(...)` predicate, but it's redundantly added here for readability 🙏
if !target_path.is_file() || is_newer_than(&prev_artifact_path, &target_path) {
let (from_path, _maybe_tmpfile) =
maybe_wasm_opt_step(&prev_artifact_path, args.no_wasmopt)?;
crate::fs::copy_to_file(&from_path, &target_path)?;
} else {
println!();
pretty_print::step(
"Skipped running wasm-opt as final target exists and is newer than wasm produced by cargo",
);
println!();
}
target_path
};

wasm_artifact.builder_version_info = Some(builder_version_info);
Expand Down Expand Up @@ -203,6 +214,33 @@ pub fn run(args: Opts) -> eyre::Result<CompilationArtifact> {
Ok(wasm_artifact)
}

fn is_newer_than(prev: &Utf8PathBuf, next: &Utf8PathBuf) -> bool {
// (1) if `next` does not yet exist, `metadata_of_prev.modified()` will be greater than
// `std::time::SystemTime::UNIX_EPOCH`;
// (2) if `m.modified()` isn't available on current platform, the predicate will always
// return true
// (3) non-monotonic nature of `std::time::SystemTime` won't be a problem:
// if the next_time and prev_time are too close in time so that next_time registers
// before prev_time, it will only affect that skipping build won't occur, but doesn't
// affect correctnes, as the build will run next time due to prev_time > next_time
let prev_time = std::fs::metadata(prev)
.and_then(|m| m.modified())
.unwrap_or_else(|_| std::time::SystemTime::now());
let next_time = std::fs::metadata(next)
.and_then(|m| m.modified())
.unwrap_or(std::time::SystemTime::UNIX_EPOCH);
let debug_msg = format!(
"{prev:?} = {prev_time:?}\n\
{next:?} = {next_time:?}"
);
println!();
println!(
"Modification timestamps of:\n{}",
pretty_print::indent_payload(&debug_msg)
);
prev_time > next_time
}

fn maybe_wasm_opt_step(
input_path: &Utf8PathBuf,
no_wasmopt: bool,
Expand All @@ -216,13 +254,15 @@ fn maybe_wasm_opt_step(
pretty_print::handle_step(
"Running an optimize for size post-step with wasm-opt...",
|| {
println!(
let start = std::time::Instant::now();
tracing::debug!(
"{} -> {}",
format!("{}", input_path).cyan(),
format!("{}", opt_destination.path().to_string_lossy()).cyan()
);
wasm_opt::OptimizationOptions::new_optimize_for_size()
.run(input_path, opt_destination.path())?;
pretty_print::duration_millis(start, "wasm-opt -O");
Ok(())
},
)?;
Expand Down
10 changes: 10 additions & 0 deletions cargo-near-build/src/pretty_print.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,16 @@ pub fn duration(start: Instant, activity: &str) {
);
}

pub fn duration_millis(start: Instant, activity: &str) {
let duration = std::time::Duration::from_millis(start.elapsed().as_millis() as u64);
println!(
" {} {} in {}",
"Finished".bold().truecolor(90, 90, 90),
activity,
humantime::format_duration(duration)
);
}

pub fn indent_payload(s: &str) -> String {
use std::fmt::Write;

Expand Down

0 comments on commit 2851dc6

Please sign in to comment.