esm_runscripts py3.9 vs py3.10 different softlink handling

[chrisdane]

Describe the bug
A check run results in different files in <expid>/.venv_esmtools depending on the python version that was used to install the esm_tools. The affected files are softlinks:

cd /home/a/a270073/esm/esm_tools/namelists/jsbach/3.20
drwxr-xr-x 4 a270073 ab0246 4096 Dec 18 11:06 ssp/
lrwxrwxrwx 1 a270073 ab0246    3 Dec 18 11:06 ssp119 -> ssp/
lrwxrwxrwx 1 a270073 ab0246    3 Dec 18 11:06 ssp126 -> ssp/
lrwxrwxrwx 1 a270073 ab0246    3 Dec 18 11:06 ssp245 -> ssp/
lrwxrwxrwx 1 a270073 ab0246    3 Dec 18 11:06 ssp370 -> ssp/
lrwxrwxrwx 1 a270073 ab0246    3 Dec 18 11:06 ssp534os -> ssp/
lrwxrwxrwx 1 a270073 ab0246    3 Dec 18 11:06 ssp585 -> ssp/

(or here: https://github.com/esm-tools/esm_tools/tree/fix_echam_hist_ssp_forcing/namelists/jsbach/3.20)

To Reproduce

er39=/home/a/a270073/.local/bin/esm_runscripts
er310=/home/a/a270211/.local/bin/esm_runscripts
runscript=/home/a/a270073/esm/runscripts/awicm_recom_levante_ssp.run.yaml
$er39 $runscript -e test39 -c
$er310 $runscript -e test310 -c

leads to different venv files:

# py3.9 case
cd /work/ab1095/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/test39/.venv_esmtools/lib/python3.9/site-packages/esm_tools/namelists/jsbach/3.20
ls -dl ssp*
drwxr-sr-x 4 a270073 ab1095 4096 Mar 11 15:13 ssp/
drwxr-sr-x 4 a270073 ab1095 4096 Mar 11 15:13 ssp119/
drwxr-sr-x 4 a270073 ab1095 4096 Mar 11 15:13 ssp126/
drwxr-sr-x 4 a270073 ab1095 4096 Mar 11 15:13 ssp245/
drwxr-sr-x 4 a270073 ab1095 4096 Mar 11 15:13 ssp370/
drwxr-sr-x 4 a270073 ab1095 4096 Mar 11 15:13 ssp534os/
drwxr-sr-x 4 a270073 ab1095 4096 Mar 11 15:13 ssp585/

i.e. with python3.9 all softlinks were dereferenced successfully (recursively) but with python3.10, only one dir exists (and I don't understand why exactly this one?):

# py3.10 case
cd /work/ba1103/a270211/out/awicm-1.0-recom/sofia/ssp585/.venv_esmtools/lib/python3.10/site-packages/esm_tools/namelists/jsbach/3.20
ls -dl ssp*
drwxr-sr-x 4 a270211 ba1103 4096 Mar 11 13:15 ssp534os/

EDIT: that directory is from a colleague and was already overwritten by a new esm_runscripts call. But this is how the directory looked like. The same

general:
    use_venv: true
    install_esm_tools_branch: fix_echam_hist_ssp_forcing

was used of course.

Expected behavior
Same esm_tools repo files should exist in <expid>/.venv_esmtools no matter which python version is used.

System (please complete the following information):

• Supercomputer: levante
• Version: 6.23.4

[mandresm]

That is indeed scary... @pgierz, can you look at this next week?

[pgierz]

.....yes, this is absolutely frightening. I am very confused by this. The actual employed Python version should have absolutely no affect on the files used, as far as I am aware nothing in our code checks the Python version. @chrisdane I will try to reproduce this and look why it is happening.

[pgierz]

EDIT: that directory is from a colleague and was already overwritten by a new esm_runscripts call.

This makes debugging a bit more difficult :-(

[pgierz]

@chrisdane: Can you please ensure that no one touches the fix_echam_hist_ssp_forcing branch for a few days?

[chrisdane]

Yes sure however I think the problem does not depend on that specific esm_tools branch.

You could just make a new branch in which you add a softlink to the repo and make the venv check run with two different python versions, i.e. installing the esm_tools from scratch with different python versions.

[pgierz]

It does not, but I want to reproduce your problem as closely as possible 😉 Otherwise there will always be "something else" that might be causing issues.