Installing s3fs in SageMaker Studio removes fsspec

[trevorpfiz]

When I install s3fs in the SageMaker Distribution 2.0 image, it breaks fsspec for other libraries that depend on it.

%pip install --disable-pip-version-check torch sagemaker transformers huggingface-hub esm s3fs
%pip uninstall tensorflow -y

Then I get:

Using cached fsspec-2024.9.0-py3-none-any.whl (179 kB)
Installing collected packages: fsspec
  Attempting uninstall: fsspec
    Found existing installation: fsspec None
error: uninstall-no-record-file

× Cannot uninstall fsspec None
╰─> The package's contents are unknown: no RECORD file was found for fsspec.

hint: You might be able to recover from this via: pip install --force-reinstall --no-deps fsspec==2023.6.0

What it looks like without s3fs:

[martindurant]

Sorry, I have no idea how you end up with "version: None", and why pip refuses to remove it. Can you force install fsspec 2024.9.0 ?

[trevorpfiz]

Sorry, I have no idea how you end up with "version: None", and why pip refuses to remove it. Can you force install fsspec 2024.9.0 ?

Right now I am just using s3fs=='0.4.2' with no issues.

[martindurant]

You could plausibly find the old fsspec install in your distribution's site-packages location (check sys.path inside a python session, for example; or import fsspec and check the __file__ attribute of the module). There may be a code folder and a separate .egg metadata folder.

I am just using s3fs=='0.4.2'

I can't stop you, but you should know that this is very old, and I expect you will face some problems.