PI-CTRL scenario dependent on year

[PengyangSong]

Dear developers,
I had some simulations and I found that the PI-CTRL run is dependent on year.
The spinup is 500 years with PI-CTRL scenario. After that, one branch is the restart from 2501 to 2550; the other branch is restart from 1851 to 1900.

2001--------2500---2501--------2550
2001--------2500---1851--------1900

But the result shows differences (I plotted the AMOC). So I am wondering what causes the difference in time. I didn't change the yaml script except for the year time.

Results:
/work/ollie/psong/paleo_work/AWIESM-2.1-wiso/PI_CTRL_prod.yaml
/work/ollie/psong/paleo_work/AWIESM-2.1-wiso/PI_CTRL_prod/ [2501-2586]
/work/ollie/psong/paleo_work/AWIESM-2.1-wiso/PI_CTRL_equil.yaml
/work/ollie/psong/paleo_work/AWIESM-2.1-wiso/PI_CTRL_equil/ [1851-1936]

[mandresm]

Hi @PengyangSong, I compared your 2 experiments input file by input file, namelist by namelist, binaries, finsihed_configs, restart by restart and I can confirm that they are identical (or equivalent where dates need to differ), except for the namelist.config where in the clockinit section yearnew is 2001 for your the prod experiment, and 1850 for the equil, but I don't think this could be causing the problem. As far as I understand the only requirement for yearnew is that for a restart it needs to be a smaller number than the initial year of your current run.

So my answer is, I don't understand what's going on. Any ideas @pgierz and @christian-stepanek ?

[PengyangSong]

I am quite sure yearnew is not the problem from FESOM2 side. Thanks for your confirmation.

[christian-stepanek]

I did not understand it either when I looked at it with Pengyang yesterday, hence I suggested him to open an issue.
Generally we are sure that on the same machine a simulation should be binary reproducible with ESM-tools, am I right? The only condition is that the forcing is time-independent (or, if the forcing is time-dependent, the forcing and climate state must fit to each other, i.e. same year used), all submodels are restarted correctly using the same settings and parameters as the other simulations, and the restart files are identical.

In addition I would make sure that all other boundary condition files are the same as well. If I remember well then ECHAM/JSBACH has some files that are also read in restart runs, although I am not sure which information is actually used and could overwrite restart information.

There is one thing that could of course always happen, although I am not sure how large the probability is for this to happen: If there was a bad node involved that creates computational errors in only one of the simulations, then reproducibility would be impeded. I guess this could only be checked by running the "faulty" simulation once more and see if the problem persists.

Sorry, but right now I do not have more ideas.

[PengyangSong]

There is no problem in reproducibility, don't panic.
The only thing is that there must be something in the PI-CTRL setting that is dependent on year time, which should be found out.

[mandresm]

I did not understand it either when I looked at it with Pengyang yesterday, hence I suggested him to open an issue. Generally we are sure that on the same machine a simulation should be binary reproducible with ESM-tools, am I right?

Yes, you are right, simulations are bit reproducible, I've tried that multiple times with different models in different computers.

The only condition is that the forcing is time-independent (or, if the forcing is time-dependent, the forcing and climate state must fit to each other, i.e. same year used), all submodels are restarted correctly using the same settings and parameters as the other simulations, and the restart files are identical.

I was wondering, if could be that some forcing file was time dependent... so in the same lines of @PengyangSong 's comment...

[mandresm]

somthing I am finding strange... I have compared with cdo diff this files among the prod and equil:

• PI_CTRL_prod/outdata/echam/PI_CTRL_prod_250112.01_echam with PI_CTRL_equil/outdata/echam/PI_CTRL_equil_185101.01_echam
• PI_CTRL_prod/outdata/fesom/sst.fesom.250101.01.nc with PI_CTRL_equil/outdata/fesom/sst.fesom.185101.01.nc

They are identical. I would expect these files not to be identical if this triggers somehow different AMOCs in the future, wouldn't you expect the same?

[christian-stepanek]

1852 was a leap year. :o/

[mandresm]

That would make a lot of sense, one extra day will certainly change things... then we should see no differences if we run a simulation starting on 1853, correct?

[PengyangSong]

Yes, either you or I can make that test.

[mandresm]

If you are not too busy, can you try that out? I'm trying to code today some needed features of ESM-Tools. Otherwise, I will run those tomorrow (tomorrow is my support day)

[PengyangSong]

leave it to me then

[PengyangSong]

Problem solved. It is caused by the leap year. Thank you! @mandresm @christian-stepanek
But it is really surprising that one more day simulation would cause such a big difference in AMOC.

Blue, green and red lines are identical. And at year 70, the orange line show different "phase" of oscillation compared with the blue line.

[PengyangSong]

The label of green line should be 2501 restart.

[christian-stepanek]

Modes of internal variability are very sensitive to even small changes in a simulation's initial state. In the case that is present here, addition of an additional day of simulation at the end of February will lead to climate being different on the first of March. As a result, the AMOC will develop completely independently in both simulations from first of March on.

That the differences in amplitude is so large is actually not surprising to me. We see here chaos in action. What should still be fairly similar, though, is the mean state of the AMOC, i.e. the average over, lets say, 100 model years.

In essence, adding a day of simulations should not impede the scientific outcome of your simulation (as long as you look at climatological averages beyond time scales of internal variability).