Enhance overall VFolder removal experience

[kyujin-cho]

Motivation

File operations take a long time and are very fragile.Backend.AI should assume that vfolder deletion process can fail at any time.

Currently, there are two design issues:

• Purging a user account depends on the deletion of their owned vfolders.* If vfolder deletion takes a long time or fails, it may prevent the admins from purging user accounts.

• VFolders in the trash bin should always be restorable.* We should hide the vfolders "being deleted" in the filesystem from the users.

• However, we should show them to the admin so that they can inspect and delete interrupted ones manually.

This is a follow-up to:

• Introduce vfolder status #575 (feat: Introduce vfolder status #713)

• refactor: re-define vfolder delete status #1892

• Clean up vfolder statuses #1915
  and the completion of:

• Advanced vfolder deletion features (aka trash bin) #767

  Proposed implementation details

Decoupling user account purging and vfolder deletion

• We need to make the user account purging done instantly.* [ ] fix: Immediately move vfolders to trash bin when purging user #2405* replacing/rewriting fix: Purge user after deleting records that reference the user #2374

• Either:* [ ] Introduce a placeholder user (like @ghost) who owns any vfolder when its owner user is purged.

• Or, refactor: Remove the foreign key constraint of vfolders.{user,group} #2404.

Moving to trash bin

• When a user moves a vfolder to trash bin by calling DELETE /vfolders or DELETE /vfolders/{name} API* The vfolder state transits to DELETE_PENDING.
• The filesystem remains unaffected.

Deleting forever

• Upon one of the following events:* [x] When the user deletes the vfolder from the trash bin via DELETE /vfolders/delete-from-trash-bin API* Including when the admin triggers manual deletion of a vfolder in the trash bin via the control panel* [ ] Update the control-panel UI to show the vfolders in the DELETE_{ONGOING|ERROR} state.

• Update the control-panel UI to show the orphan vfolders in the DELETE_PENDING state with their owner users/projects gone.

• When the user "flushes" the entire trash bin via POST /vfolders/flush,

• When the user or an external service triggers immediate deletion of a vfolder via ... API,

• When the system triggers periodic cleanup of old vfolders in the trash bin,

• Or, when the system triggers periodic cleanup of interrupted vfolder deletions,

• Do:* [x] The vfolder state transits to DELETE_ONGOING.

• Hide it from the user-observable trash bin as its content is no longer fully recoverable.

• The storage-proxy initiates a background filesystem-level deletion task.* When this is successfully done, the state now moves to DELETE_COMPLETED.

• Whne this fails, move the state to DELETE_ERROR (and store the error log somewhere?)

• Since a vfolder may enter DELETE_{ONGOING|ERROR} states multiple times, we need refactor: Replace vfolder's status_history's type map with list #2116 as well.

• Reliable VFolder deletion(purge) #1797 (feat: improve reliability & usability of VFolder removal API #1884)

History cleanup

• Never delete the vfolders records during the deletion process. Only delete vfolders in the DELETE_COMPLETE state after the data rentention period.
• _{User removes database record of VFolder in DELETE_COMPLETED status by calling DELETE /vfolders/purge API}* [ ] Replace this with backend.ai mgr clear-history.

[fregataa]

We made a glossary related to VFolder deletion.

• move to trash: transit VFolder status to DELETE_PENDING and actual contents remains
• delete forever: delete the actual contents of the VFolder
• restore: transit VFolder status from DELETE_PENDING to READY

Does this issue suggest to replace delete forever with flush or add support for flush?

[achimnol]

Since the filesystem-level deletion process can take a long time, or gets failed and retried afterwards, we should show a small notice to the users like: "The deletion process and restoring the available storage space may take up to 12 hours."