b4b-dev: Plumber2 Implementation

[TeaganKing]

This PR will use the TowerSite class (parent to both NEON and PLUBMER sites) and create a Plumber2Site class as well as implement capabilities for running single point simulations at PLUMBER sites.

Contributors other than yourself, if any:
@ekluzek
@adrifoster

CTSM Issues Fixed (include github issue #):
Addresses part of #1487

Are answers expected to change (and if so in what way)?
No, this PR should be BFB. However, it should expand existing capabilities to allow users to run at PLUMBER sites.

Any User Interface Changes (namelist or namelist defaults changes)?
Additional flags will be implemented for run_neon

[TeaganKing]

This PR will also address #2186

[TeaganKing]

We'll also need to add the new CDEPS tag to this PR.

[TeaganKing]

Per conversation with Erik, we'll move this to bfb once the other Plumber-related PRs are in bfb next week.

[TeaganKing]

To Do for this PR:

• Update branch / resolve merge conflicts
• Add new CDEPS tag to this PR once the CDEPS PR is merged.
• Bring in CTSM PLUMBER5.2 PR
• Check for any sort of equivalent of user_version for plumber
• Include any plumber-specific user namelist lines
• Update check_plumber_data
• Determine valid sites based on tower_type?
• Define available_plumber_list
• Be able to run NEON & PLUMBER simultaneously (if both are same run type)
• Provide option for None for both NEON and PLUMBER cases so they can be run indepdently
• Ensure that we don't use a NEON base case for PLUMBER or vice versa
• Update run_type defaults in arg parse
• Resolve unknown case.run error for PLUMBER given need for setup/build
• Test out various tower options/arguments
• Update tests
• Update documentation

[TeaganKing]

We don't currently have restart files for PLUMBER2, which has brought up the question: to what extent do we want to provide support for PLUMBER tower sites? Two options that @wwieder and I discussed this morning are described below:

1. Users spin up their own AD & postAD cases and then can run a transient case (no restart files needed). This is somewhat contrary to the purpose of run_neon because it requires users to run case setup, build, submit, and archive.
2. My vote would be that we provide (and update upon changes to model versions) restart files and allow users to run a transient case out of the box (this is how NEON tower sites work, and consistency with run_tower would be nice as well as ease of use, but there is more overhead to maintain the feature). This would be a slightly easier implementation given that run_tower currently expects a transient case (and automatically submits case.run, which is not automatically created for the ad run but is automatically created for transient cases).

@danicalombardozzi @slevis-lmwg @olyson let me know if you have thoughts on this. We were planning to chat on Tuesday at 9am but it looks like there is a NCAR-NEON-Community partnership meeting that will be taking that time slot and I'd ideally like to make a decision before August 20th; also happy to tag up an additional time if that's easier than a GitHub discussion.

[olyson]

My guess is that most users will want to change the model and will likely have to do their own spinups anyway. So I might lean toward not providing initial files if it requires significant work to generate, keep track of, and update those files.

[dlawrenncar]

I would agree with this. These runs are cheap and the point is to have a set of tower sites that can be run quickly and easily with different model configurations and with new model versions. The ICs quickly become irrelevant as the model changes.

…

On Tue, Aug 6, 2024 at 7:02 PM Keith Oleson ***@***.***> wrote: My guess is that most users will want to change the model and will likely have to do their own spinups anyway. So I might lean toward not providing initial files if it requires significant work to generate, keep track of, and update those files. — Reply to this email directly, view it on GitHub <#2406 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFABYVB663VOB5R7GAWPZJDZQFIYNAVCNFSM6AAAAABELMMUA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZSGMYDEMBTHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[TeaganKing]

Ok, thank you both for this feedback!

[wwieder]

Thanks for weighing in here. Teagan since Dave and Keith came to the same conclusion we did yesterday lets move forward with the plan to support 'spinup' and 'transient' PLUMBER2 cases, without providing initial conditions.

It might be nice to differentiate how we do this for SP (fast spinup, ~20 or 40 years) vs. BGC (more involved, like NEON) cases. Would that be difficult to integrate into the run_towers workflow?

[TeaganKing]

That sounds good. I have successfully run the AD & postAD case (with a bit of hard-coding), so this will require the following updates, which should be feasible:

• finidat file name updates for each site in user_nl_clm (expecting <SITE>.ad.clm2.r.YYYY-MM-DD-00000.nc files but we have for instance *50400.nc files)
• Update run_type defaults in run_case in both tower_site.py and plumber_site.py such that the AD case is the default (instead of the transient default that's set in neon_site.py).

@wwieder To clarify, regarding the option for SP vs BGC, are you envisioning a flag that allows users to pick between these options?

[wwieder]

A few comments:

• I don't know that we need to "automatically" run postAD and transient runs, as I think it's a good idea for users to have to check the status of their AD and postAD cases to check that they meet steady state criteria (e.g., Alaska NEON sites take significantly longer to spin up than CONUS NEON sites).
• What I like about the current run_neon implementation is that running a postAD cases automatically creates a branch that looks for an similarly names AD case and then using the --run from postAD flag also creates a transient case uses the off the postAD ref case and initial conditions. Both of these options just take the last restart files that are available (regardless of simulation date).
• Regarding SP vs. BGC cases, yes a flag that allows users to chose between these two options would be excellent. I'm not sure about the best way to implement this and don't want to introduce scope creep / complexity to this work but having the option to set up SP vs. BGC cases would be very useful for PLUMBER2 (and even NEON) cases.

Happy to have a quick that if it's helpful.

[TeaganKing]

Ok, in that case, I'll just plan to change the default for PLUMBER to AD and add some documentation on this anticipated workflow for users. Thanks for clarifying!

Why don't I plan to get that first part implemented and then we can talk more about the BGC vs. SP options.

[danicalombardozzi]

It seems like perhaps the SP vs BGC option should come later. It will be useful for both PLUMBER and NEON and therefore might require more in-depth work to do it well. We should also create a separate tutorial for people to check the stability of their site spin up. We apply a standard number of years for the AD and post-AD NEON site simulations and it would be useful to help users evaluate the stability after making changes. I don't recall that we evaluated all sites for stability thoroughly, either, so perhaps it would also help us to know if the number of years we use is effective everywhere.

…

On Wed, Aug 7, 2024 at 11:41 AM Teagan King ***@***.***> wrote: Ok, in that case, I'll just plan to change the default for PLUMBER to AD and add some documentation on this anticipated workflow for users. Thanks for clarifying! Why don't I plan to get that first part implemented and then we can talk more about the BGC vs. SP options. — Reply to this email directly, view it on GitHub <#2406 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGHW2QOYGE75H6BHKVQVDGLZQJL4RAVCNFSM6AAAAABELMMUA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZTHE4TINRTGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Dr. Danica Lombardozzi she/her/hers Terrestrial Sciences Section Climate and Global Dynamics NSF National Center for Atmospheric Research Boulder, CO 80305 email: ***@***.*** office: (303) 497-1777

[TeaganKing]

This is about ready to go, but once #2485 is in, I'll run the unit and system tests once more and perform a few final tests to make sure the various run_tower flags work as expected.

[TeaganKing]

Hi @ekluzek ,

I have now addressed the comments from @wwieder 's review. Two outstanding issues that Will and I discussed are the following (more details in comments above):

• The Plumber2Site class is called in a somewhat clunky way with dummy years that get overwritten. I'm not sure of a better way to do this, and it works, but let me know if you have alternative suggestions?
• Running system tests takes a while. There are of course benefits to doing more testing (as I have done here), but the time the tests take may be on the excessive side. Your feedback on whether these tests are worth it would be helpful!

And a software-focused review of the code would be helpful now that Will reviewed some of the functionality pieces.

Before merging (and after @ekluzek 's review), I will be sure to run the following tests one last time (I ran them before Will's review but there are changes that we should test again before merging):

• run ./run_tower --plumber-sites with various options as a final check
• make all
• run ctsm system tests with ./python/run_ctsm_py_tests -s

Lastly, note that there are some related outstanding issues that have been documented that are outside the scope of this PR.

[wwieder]

It seems like this PR also fixes #2739

[ekluzek]

Hi @TeaganKing thanks for all your work on this. So sorry it's been so long. I know your head isn't into this anymore. This is something that @wwieder wants me to work on, and I will.

I mostly suggest making some changes to make this Object Oriented. Those changes will be worth getting in as it will make it more flexible, extensible and maintainable. With the things needed for NEON isolated in NEON modules and things needed for PLUMBER2 isolated in it's own module as well. It'll also make it easier for these things to evolve separately as well as bring in some new class for something like AmeriFlux or a class to users handle their own list of generic tower sites. Or whatever the future needs are dreamed up...

I know you don't have time to do coding, but if possible and you are up for it, I would like to talk design with you. Would you be up for that?

Also @adrifoster this seems like a good thing to bring you in on as well. Could we maybe meet Thursday when I come in?

[ekluzek]

I think we remove the prism option for PLUMBER2 right? You might have to include it in the interface, but we then should check to make sure it's False.

[ekluzek]

Another way to handle this would be to have PRISM only within the NEON class and only able to be set there.

[TeaganKing]

Right, PRISM is not available for PLUMBER-- having this in the NEON class could work.

[ekluzek]

Rather than returning a list for both valid plumber sites and neon sites maybe it should be more generic of valid sites?

[ekluzek]

I think we should either rename this to

run_plumber

OR remove run_neon and only have run_tower which can run both NEON or PLUMBER sites. And then can be extended to work with generic tower sites, or another set of data similar to NEON or PLUMBER.

[ekluzek]

Again, if we have a generic list we don't have to know if it's plumber or neon (or something else).

[ekluzek]

Again remove returning both site lists and just have a general list of sites.

[ekluzek]

This would be lifted to the neon specific implementation.

[ekluzek]

Likewise this would go to the PLUMBER2 specific implementation.

[ekluzek]

Rather than having this do both NEON and PLUMBER2 I suggest there should be a NEON and PLUMBER2 specific versions that just do their own things rather than both.

Doing it this way is a more OO programming pattern and will help extensibility and keeping NEON and PLUMBER2 logic separated. As such will be easier to maintain.

[TeaganKing]

Hi @ekluzek , happy to tag up to chat about this (and share some challenges that resulted in me implementing this a certain way). My calendar is up to date and lists the days that I'm on site-- feel free to just schedule something on there with me at some point when we're both in person! Or we can meet remotely.

[TeaganKing]

Per discussion with @adrifoster and @ekluzek , we decided that it would be best to focus on updating the user interface. We will implement updates in a way that is similar to how making surface datasets uses point and global parameters and then has additional optional parameters based on whether it is a point or global case.

In this case, we will provide a user interface that looks like the following:
./run_tower neon|plumber2 --<additional parameters that apply to either neon OR plumber2>

The additional parameters that should apply to NEON are:

• base_case_root: str, opt; file path of base case*
• run_type: str, opt; transient, post_ad, or ad case, default transient**
• prism: bool, opt; if True, use PRISM precipitation, default False
• user_version: str, opt; default 'latest' (could be useful later for PLUMBER, but not currently used)
• overwrite: bool, opt; default False*
• setup_only: bool, opt; default False; if True, set up but do not run case*
• no_batch: bool, opt; default False*
• rerun: bool, opt; default False*
• experiment: str, opt; name of experiment, default False*

The additional parameters that should apply to PLUMBER2 are:

• base_case_root: str, opt; file path of base case*
• run_type: str, opt; transient, post_ad, or ad case, default ad**
  • (ad case is default because PLUMBER requires spinup)
• overwrite: bool, opt; default False*
• setup_only: bool, opt; default False; if True, set up but do not run case*
• no_batch: bool, opt; default False*
• rerun: bool, opt; default False*
• experiment: str, opt; name of experiment, default False*

* parameter applies exactly the same way to NEON and PLUMBER2
** parameter applies to both but with different defaults

[wwieder]

Thanks for this work, Tegan.

Minor comment, default cases for plumber should be SP, which doesn't require an ad spin up. This may be most relevant with the user interface topic you all discussed today.