Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of filesystem queries by avoiding them #19

Closed
SamStudio8 opened this issue Jan 27, 2021 · 2 comments
Closed

Improve performance of filesystem queries by avoiding them #19

SamStudio8 opened this issue Jan 27, 2021 · 2 comments
Assignees
Labels

Comments

@SamStudio8
Copy link
Member

Before Elan begins, user uploads are mapped to Majora metadata to determine what work is to be done for the day. This mapping process involves running find over the upload directories and piping the file list to ocarina_resolve.py. Additionally, the resolver hits cephfs again to query the mtime to determine whether the file is old or new.

During periods of heavy I/O, this find step, or the following mtime check can add delay. This delay is not so significant that this needs to be fixed urgently, but the delay will increase linearly as we continue to accrue uploads.

This problem also extends to other areas (see #17), so it would be good to consider alternatives, so far we've thought about:

  • Using cephfs-shell to query (however it seems to be unpackaged for our target dist)
  • inotify or similar to monitor user upload directories and generate manifests on changes
  • Using Majora information where possible (e.g. Improve performance of Asklepian export #17)
@SamStudio8 SamStudio8 added enhancement New feature or request elan labels Jan 27, 2021
@SamStudio8 SamStudio8 self-assigned this Jan 27, 2021
@SamStudio8
Copy link
Member Author

SamStudio8 commented Jan 27, 2021

Another such example of a find to remove: https://github.com/SamStudio8/elan-nextflow/blob/master/bin/control/cog-publish.sh#L116

@SamStudio8
Copy link
Member Author

Added to backlog #62

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant