Catch PermissionError in fsspec.core when attempting to auto_mkdir

[DanielTsiang]

Catch PermissionError in fsspec.core when attempting to auto_mkdir for parent dir, as the user may not have permission to create the parent dir.

This should hopefully allow the user to carry on without error if the parent dir already exists. If not, a later exception should be thrown when trying to write to a file when the parent dir doesn't exist.

This attempts to address the following issue:

• Writing to a new file in an existing bucket throws "unspecified location constraint" error as it incorrectly tries to create new bucket #1404

Comments on how to improve this PR are welcome, as this is my first time contributing to the repo.

[martindurant]

Does PermissionError actually catch the S3 case?

We could probably test this with a mock on makedirs for some filesystem like memoryFS.

I feel like it would be nice if we kept the exception around in the OpenFile, so that if writing fails later, we can refer to it - else when the directory doens't exist and the user couldn't create it, they can know why their write fails.

[DanielTsiang]

@martindurant I'm not sure why the tests have started failing because of pytest import error:

 Skipped: could not import 'moto': No module named 'moto'

Do you have any ideas?

[martindurant]

This is an upstream issue in pytest-asyncio. Waiting on pytest-dev/pytest-asyncio#656

[DanielTsiang]

This is an upstream issue in pytest-asyncio. Waiting on pytest-dev/pytest-asyncio#656

I see, thanks. Since it's broken in 0.22.0, worth pinning to 0.21.1 in the meantime?

[martindurant]

I'll give them a couple of days, see if it gets resolved.

[DanielTsiang]

I'll give them a couple of days, see if it gets resolved.

Sounds reasonable.

In terms of this PR, once the pytest-asyncio issue is resolved, would you be comfortable merging this PR in?

[martindurant]

Yes, this PR is good to go.

[seifertm]

I saw this PR linked in the pytest-asyncio repository.

pytest-asyncio v0.22 was so bad that it was yanked from PyPI. There are currently no plans to release a bugfix version. It would be best if you pinned the version pytest-asyncio!=0.22.0 and went ahead with this PR.

[martindurant]

Thanks @seifertm . Will it also be removed from conda-forge?

[martindurant]

(please merge from master - I don't have permissions on your repo to do so)

[DanielTsiang]

(please merge from master - I don't have permissions on your repo to do so)

I've now merged latest from master branch into this PR branch.

[seifertm]

Thanks @seifertm . Will it also be removed from conda-forge?

@martindurant Unlikely. I couldn't find any information on how to do that. I'd appreciate if you had any pointers on this.

[martindurant]

@seifertm , the README at https://github.com/conda-forge/admin-requests gives instructions. We are excluding the build from our test environment for now.

[martindurant]

@DanielTsiang , thanks for cleaning up the history - not sure what happened there!

[seifertm]

@martindurant Thanks for the pointer. I filed a PR to mark pytest-asyncio-0.22.0 broken in conda-forge:
conda-forge/admin-requests#857