Fix directory rename hiding files from lower branches

[Tuupertunut]

Fixes #91

This PR fixes the old issue #91 with directory rename. Previously after renaming a directory, only the contents of the highest branch were cow-copied up to the read-write branch, hiding all content the directory had in lower branches. In this PR a new function is introduced that can do a cow-copy recursively for a directory tree in every branch. Now if a file is renamed and it's a non-directory, we do a normal cow-copy, but for directories we do this recursive cow-copy instead. Everything should now work as expected from a proper directory rename.

Doing a cow-copy recursively for a whole directory is significantly more complex that for a single file:

• We have to copy the directory from every branch because each of them might contain a different part of the directory's contents.
• We have to recursively copy every file in the directory in every branch.
• We have to check for each file if it already exists in the destination so we don't overwrite files from higher branches.
• We have to check for each file that it doesn't have a whiteout in any of the higher branches so we don't accidentally reintroduce it by copying.

Caveat: directory renaming is now much slower than before. That's the price of it working correctly.

[rpodgorny]

hi @Tuupertunut ! thank you very much for your contribution! i will take a look at this some time next week. anyway, some tests seem to be failing (see below) - can you look into it meanwhile? thanks!

[Tuupertunut]

Yep, that's the case where cow is not enabled. Directory renaming heavily uses cow-copying so I didn't consider that it sometimes might need to work without cow enabled and I'm still not exactly sure how it should behave. I'll figure something out.

[Tuupertunut]

Fixed the case with cow disabled by introducing a similar shortcut in the recursive cow-copy function as in the non-recursive one. Now all the tests pass.

However I'm not sure the original shortcut code was working as intended and there was probably a bug already in the non-recursive function too: Rename fails when we have cow enabled, two read-write branches and we rename a file from the lower branch to a name already existing in upper branch. I'm not sure how unionfs is supposed to behave when cow is enabled but there are multiple read-write branches. It's not necessary to fix this right now but just something I discovered.

[rpodgorny]

thanks for the updates! i've quick-reviewed the code and it looks reasonable. i especially like the explanatory comments - great work!
...as for the tests, i'm not really sure why you changed the os.path.isdir to os.path.exists in some cases but i've changed it back (and more) in my "pre-merge" branch: https://github.com/rpodgorny/unionfs-fuse/tree/Tuupertunut-master
...so, i'll let this sink in for few more days and will merge it then. thanks again!

[Tuupertunut]

Those assertions are all assertFalse, so asserting that a directory does not exist. In that case exists is a stronger assertion than isdir, which might allow there to be a file.

[rpodgorny]

Those assertions are all assertFalse, so asserting that a directory does not exist. In that case exists is a stronger assertion than isdir, which might allow there to be a file.

yes, you are correct! i knew i must have overlooked something. ;-) ...so i've "fixed" the other checks and added a comment. other than that, i've merged your branch without changes. thanks again for your contribution!