Workspace: Add homefs

[ConnorNelson]

This PR introduces homefs, our solution to improve the performance of persistent home directories, favoring btrfs (that is capable of migrating around nodes) over nfs.

---

Some TODOs (probably to be addressed largely by other PRs):

• Migrate production homes from ext4 to btrfs
• Remove nfs logic
• Restore as_user workspace support
• Support periodically synchronizing incremental snapshots back to the main node
• Support home migration

[ConnorNelson]

Closes #572.

[ConnorNelson]

This is how the migration was performed:

#!/bin/bash

set -e

log_file="/data/log"
exec > >(tee -a "$log_file") 2>&1

parallel_jobs=20

users=$(ls /tank/homes | sort -n)

log_error() {
    user=$1
    touch "/data/homes/$user.error"
}

process_user() {
    user=$1
    date
    echo "Processing user: $user"

    tmp_mount="/data/homes/tmp_$user"

    mkdir -p "$tmp_mount"

    mount /tank/homes/$user "$tmp_mount" || { echo "Failed to mount for $user"; log_error "$user"; return 1; }

    if [[ ! -d "/data/homes/$user" ]]; then
        btrfs subvolume create /data/homes/$user || { echo "Failed to create subvolume for $user"; log_error "$user"; return 1; }
    fi

    if [[ ! -d "/data/homes/$user/snapshots" ]]; then
        btrfs subvolume create /data/homes/$user/snapshots || { echo "Failed to create snapshots subvolume for $user"; log_error "$user"; return 1; }
    fi

    if [[ ! -d "/data/homes/$user/active" ]]; then
        btrfs subvolume create /data/homes/$user/active || { echo "Failed to create active subvolume for $user"; log_error "$user"; return 1; }
    fi

    btrfs qgroup limit 1G /data/homes/$user/active || { echo "Failed to set qgroup limit for $user"; log_error "$user"; return 1; }

    rsync -aH --sparse --delete --ignore-errors --info=progress2 "$tmp_mount/" /data/homes/$user/active/ || { echo "Rsync failed for $user"; log_error "$user"; return 1; }

    umount "$tmp_mount" || { echo "Failed to unmount for $user"; log_error "$user"; return 1; }

    rmdir "$tmp_mount" || { echo "Failed to remove temp mount for $user"; log_error "$user"; return 1; }
}

export -f process_user
export -f log_error

echo "$users" | xargs -n 1 -P $parallel_jobs -I {} bash -c 'process_user "$@"' _ {}

echo -e "\nAll users processed successfully."

We --ignore-errors because some paths are too long (file system mazes for the race condition module). We accept that some of these files are not migrated. Additionally --sparse is critical, because many files (like core dumps) are sparse, and not transferring them sparsely would consume more data than necessary (and in some cases go above the 1GB quota).