Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSP/SSPMATRIX test problems with new case github workflow #2913

Open
ekluzek opened this issue Dec 18, 2024 · 4 comments · May be fixed by #2921
Open

SSP/SSPMATRIX test problems with new case github workflow #2913

ekluzek opened this issue Dec 18, 2024 · 4 comments · May be fixed by #2921
Assignees
Labels
bug something is working incorrectly

Comments

@ekluzek
Copy link
Collaborator

ekluzek commented Dec 18, 2024

This is in ctsm5.3.015

The SSPMATRIX test on Izumi is failing with this problem. I think this is a problem with the new workflow of saving the case details to a local git repository, and likely will be fixed with the next updates to cime.

2024-12-18 13:10:33: ERROR: Command: 'git -C /scratch/cluster/erik/tests_ctsm5315acl/SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-ciso_monthly.GC.ctsm5315acl_int commit -m "
ERROR: RUN FAIL: Command '  /scratch/cluster/erik/tests_ctsm5315acl/SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-ciso_monthly.GC.ctsm5315acl_int/bld/cesm.exe   >> cesm.log.$LID 2>&1 ' failed
See log file for details: /scratch/cluster/erik/tests_ctsm5315acl/SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-ciso_monthly.GC.ctsm5315acl_int.step0-AD/run/cesm.log.707138.izumi.cgd.ucar.edu.241218-131031"' failed with error 'On branch SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-ciso_monthly.GC.ctsm5315acl_int
nothing to commit, working tree clean' from dir '/scratch/cluster/erik/tests_ctsm5315acl/SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-ciso_monthly.GC.ctsm5315acl_int.step0-AD'

Originally posted by @ekluzek in #2900 (comment)

@ekluzek ekluzek self-assigned this Dec 18, 2024
@ekluzek ekluzek added this to the cesm3_0_beta05 milestone Dec 18, 2024
@ekluzek ekluzek added bug something is working incorrectly next this should get some attention in the next week or two. Normally each Thursday SE meeting. labels Dec 18, 2024
@ekluzek
Copy link
Collaborator Author

ekluzek commented Dec 19, 2024

I'm still seeing this in ctsm5.3.016 and think it will take some more work to get working. The SSP and SSPMATRIX tests were based on older testing code and could be refactored to use system_tests_compare_two or system_tests_compare_n.py. The update would probably increase robustness and future protect these tests as well.

The PVT.py test and some others may need similar treatment.

@ekluzek ekluzek removed the next this should get some attention in the next week or two. Normally each Thursday SE meeting. label Dec 19, 2024
@ekluzek
Copy link
Collaborator Author

ekluzek commented Dec 19, 2024

One problem I did see is that the final test run is one of the clones made rather than the original test. I think rearranging the order so that the final test run is the original test would help this. That was something arbitrarily done when these tests were made that now with changes has caused a problem. I also realized that the ERI test still does things similar to the SSP tests and modeling what it does might help here. So it might not need a refactoring to use the system_tests_compare*.py infrastructure, and likely that might be easier.

@ekluzek ekluzek changed the title SSPMATRIX test problem on Izumi with new case github workflow SSP/SSPMATRIX test problems with new case github workflow Dec 19, 2024
@ekluzek
Copy link
Collaborator Author

ekluzek commented Dec 19, 2024

I played around with this a bit, but didn't get it working.

I think some code similar to this needs to be added to handle the drv rpointer file.

+        drvrest = "rpointer.cpl"
+        drvrest += self._rest_time
+        self._set_drv_restart_pointer(drvrest)

One catch is that _rest_time doesn't work in above, without using self._set_restart_interval. So that needs to be added in.

@ekluzek
Copy link
Collaborator Author

ekluzek commented Dec 20, 2024

Running ctsm_sci testlist on ctsm5.3.016 the only unexpected fail was this one:

SSP_Ld4.f09_g17.I1850Clm50BgcCrop.derecho_intel.clm-ciso_rtmColdSSP

@jedwards4b jedwards4b linked a pull request Jan 2, 2025 that will close this issue
@ekluzek ekluzek modified the milestones: cesm3_0_beta05, cesm3_0_beta06 Jan 5, 2025
@ekluzek ekluzek moved this from Todo to In Progress in LMWG: Near Term Priorities Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug something is working incorrectly
Projects
Status: In Progress
Development

Successfully merging a pull request may close this issue.

1 participant