Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: improve --total-size performance #1291

Open
TENX-S opened this issue Jan 3, 2025 · 1 comment
Open

feat: improve --total-size performance #1291

TENX-S opened this issue Jan 3, 2025 · 1 comment

Comments

@TENX-S
Copy link

TENX-S commented Jan 3, 2025

# macOS 15.2, M3 Pro
# under $HOME
$ time eza -Tlahmo -L1 --no-permissions --total-size --no-user
eza -Tlahmo -L1 --no-permissions --total-size --no-user   1.17s user 14.00s system 44% cpu 33.772 total

Without --total-size will be much faster.
Is there a way we can parallel this size computing?

@PThorpe92
Copy link
Member

I have thought about this as well, for this and the --git flags, which can both be quite slow in some circumstances.

With async generally being the solution to IO bound tasks, eza is perhaps not an ideal candidate for something like tokio. Directory/path traversal and the sheer amount of syscalls is the bottleneck, so it is likely going to be using something like rayon's parallel bridge.

Another option would be using io_uring, to submit batches of statx with callbacks in place to do the calculations on the returned metatdata, although that would obviously only help linux users. Not to mention would require an immense refactor, and to ensure we are doing any additional work needed asyncronously.

Definitely open to ideas here. A quick test reveals we are actually already faster than du at calculating the total size of my Projects directory, but still about half as fast as a modern rewrite.

eza --total-size -l:
image

du --total --human-readable:
image

dust a du rewrite using rayon
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants