You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Before Elan begins, user uploads are mapped to Majora metadata to determine what work is to be done for the day. This mapping process involves running find over the upload directories and piping the file list to ocarina_resolve.py. Additionally, the resolver hits cephfs again to query the mtime to determine whether the file is old or new.
During periods of heavy I/O, this find step, or the following mtime check can add delay. This delay is not so significant that this needs to be fixed urgently, but the delay will increase linearly as we continue to accrue uploads.
This problem also extends to other areas (see #17), so it would be good to consider alternatives, so far we've thought about:
Using cephfs-shell to query (however it seems to be unpackaged for our target dist)
inotify or similar to monitor user upload directories and generate manifests on changes
Before
Elan
begins, user uploads are mapped to Majora metadata to determine what work is to be done for the day. This mapping process involves runningfind
over the upload directories and piping the file list toocarina_resolve.py
. Additionally, the resolver hits cephfs again to query themtime
to determine whether the file is old or new.During periods of heavy I/O, this find step, or the following
mtime
check can add delay. This delay is not so significant that this needs to be fixed urgently, but the delay will increase linearly as we continue to accrue uploads.This problem also extends to other areas (see #17), so it would be good to consider alternatives, so far we've thought about:
cephfs-shell
to query (however it seems to be unpackaged for our target dist)inotify
or similar to monitor user upload directories and generate manifests on changesThe text was updated successfully, but these errors were encountered: