The requests distribution was not found and is required by PyGithub

[chrisdane]

Describe the bug
On albedo suddenly (most recent esm-tools branch release, a1d545b) this error occurs:

esm_tools --version
Traceback (most recent call last):
  File "/albedo/home/cdanek/.local/bin/esm_tools", line 33, in <module>
    sys.exit(load_entry_point('esm-tools', 'console_scripts', 'esm_tools')())
  File "/albedo/home/cdanek/.local/bin/esm_tools", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/albedo/soft/sw/spack-sw/python/3.9.12-aohn73j/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/albedo/soft/sw/spack-sw/python/3.9.12-aohn73j/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/albedo/home/cdanek/esm/esm_tools/src/esm_tools/__init__.py", line 37, in <module>
    import pkg_resources
  File "/albedo/soft/sw/spack-sw/python/3.9.12-aohn73j/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3243, in <module>
    def _initialize_master_working_set():
  File "/albedo/soft/sw/spack-sw/python/3.9.12-aohn73j/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3226, in _call_aside
    f(*args, **kwargs)
  File "/albedo/soft/sw/spack-sw/python/3.9.12-aohn73j/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3255, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/albedo/soft/sw/spack-sw/python/3.9.12-aohn73j/lib/python3.9/site-packages/pkg_resources/__init__.py", line 568, in _build_master
    ws.require(__requires__)
  File "/albedo/soft/sw/spack-sw/python/3.9.12-aohn73j/lib/python3.9/site-packages/pkg_resources/__init__.py", line 886, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/albedo/soft/sw/spack-sw/python/3.9.12-aohn73j/lib/python3.9/site-packages/pkg_resources/__init__.py", line 772, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'requests>=2.14.0' distribution was not found and is required by PyGithub

System (please complete the following information):

• Supercomputer: albedo

Cheers,
Chris

[nwieters]

Hi @chrisdane,

I can't reproduce this error.
For esm_tools --version
I get
esm_tools, version 6.25.1
on Albedo.

[nwieters]

Hi @chrisdane ,
since when do you get this error? Maybe it is related to some other issue on Albedo?

[chrisdane]

I can't tell for sure but a few days ago it worked. The problem remains if I switch to other branches, so yes, seems rather related to the machine than the esm_tools? Any solution suggestion to the machine admins?

[nwieters]

Hi @chrisdane,
have you tried to reinstall the esm_tools? Maybe then all missing requirements are installed again.

[chrisdane]

The docu does not tell how to reinstall the esm_tools (related to #1054). I tried ./install.sh and it solved the issue above.

./install.sh, however, complained that

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
dask 2023.9.2 requires pyyaml>=5.3.1, but you have pyyaml 5.1 which is incompatible.
jupyter-events 0.6.3 requires pyyaml>=5.3, but you have pyyaml 5.1 which is incompatible.

How to solve that?

[nwieters]

Hi @chrisdane,
yes, sorry. Issue #1054 is on my todo list. And I will also include a section of reinstalling esm-tools. But I meant
./install.sh
Good that it solved your issue.

Regarding the latter error. I also have pyyaml 5.1 installed. But I don't have dask 2023.9.2 or jupyter-events. I understand the error that dask and jupyter-events are complaining about pyyaml version.

I found a solution to deinstall the conflicting python package (in your case this seems to be pyyaml 5.1) and install the requested one (pyyaml>=5.3). But I havn't tried it myself.

Maybe you can try

pip uninstall pyyaml==5.1
pip install pyyaml==5.3

I found it here
https://stackoverflow.com/questions/72672196/error-pips-dependency-resolver-does-not-currently-take-into-account-all-the-pa

Hope this may help. Please let me know.

[chrisdane]

That changed the error to

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
dask 2023.9.2 requires pyyaml>=5.3.1, but you have pyyaml 5.3 which is incompatible.
esm-tools 6.24.2 requires pyyaml==5.1, but you have pyyaml 5.3 which is incompatible.

[nwieters]

So for esm-tools you would have to install pyyaml version 5.1 again.
Removing dask would propably help. But I guess you need that for other things?

[nwieters]

Maybe another solution is to downgrade/uninstall dask in the local python installation to a version that has no conflicts with pyyaml 5.1, and run your other application that needs this particular dask version in a conda environment.