Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark reports including comparisons from CI #97

Open
Ma27 opened this issue Jul 11, 2022 · 3 comments
Open

Benchmark reports including comparisons from CI #97

Ma27 opened this issue Jul 11, 2022 · 3 comments
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@Ma27
Copy link
Member

Ma27 commented Jul 11, 2022

Description

Changes like #94 (and the previous ones) have shown that it's still a thing to underestimate the performance impact of a patch.

We should thus generate benchmark reports that are posted to each PR (including comparison to e.g. the previous commit or master).

@Ma27 Ma27 added the enhancement New feature or request label Jul 11, 2022
@Ma27 Ma27 added this to the 0.11.0 milestone Jul 11, 2022
@Ma27 Ma27 self-assigned this Jul 11, 2022
@oberblastmeister
Copy link
Contributor

I feel like the benchmarks could take way too long and I am not sure we can trust github actions to provide a consistent benchmarking environment. I think it would be better to run benchmarks on your own machine.

@Ma27
Copy link
Member Author

Ma27 commented Jul 11, 2022

Well, t his isn't supposed to be a 100% perfect environment for that, but I still think it's reasonable to to get some automated feedback if the situation is generally improving or getting worse. Also, an automated check won't decide whether something is mergeable or not, a human should always investigate anyways ;-)

@fogti
Copy link
Contributor

fogti commented Jul 11, 2022

[zseri@orca:~/devel/rnix-parser]$ git lfs checkout
Skipped checkout for "benches/all-packages.nix", content not local. Use fetch to download.                                                                                                                                                     
Checking out LFS objects: 100% (1/1), 803 KB | 0 B/s, done.

[nix-shell:~/devel/rnix-parser/src]$ cargo bench
[...]
all-packages            time:   [9.3041 us 9.4207 us 9.5911 us]
                        thrpt:  [13.026 MiB/s 13.261 MiB/s 13.428 MiB/s]
[...]
[nix-shell:~/devel/rnix-parser/src]$ git switch master
Previous HEAD position was ec908d6 Merge pull request #91 from darichey/rust-2021
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
branchless: processing checkout

[nix-shell:~/devel/rnix-parser/src]$ cargo bench
[...]
all-packages/all-packages
                        time:   [6.9461 us 6.9626 us 6.9775 us]
                        thrpt:  [17.905 MiB/s 17.943 MiB/s 17.986 MiB/s]
[...]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants