CESM notes from pull 233

[hkershaw-brown]

From pull/233

Based on the discussion here and in the software standup of 2021-8-19
I think that we need to make a plan with managably sized pieces.
The decisions were to:

1. Discuss later which released CESMs to use; functional or software supported or scientifically supported.
2. Change the setup philosophy from single script/job to "resources" file + setup scripts?
3. Put CESM SourceMods into DART.
4. Unanswered question; should I update external documentation simultaneously with the code?

Here's my rough description of the effort that will be required to implement those decisions.
I think that in the name of incremental development and committing useful code that works, this PR and issue should be completed using CESM version and DART code that already exists, and further discussion and development should be done in new issues.
One we've decided on the overall strategy, I'll work on answering whichever questions remain in this PR.

1. Deciding which CESM releases are worthy of a DART interface probably won't have a clear answer. It depends on user needs and the state of evolution of CESM, which is unpredictable. After a choice is made it will be an unknown amount of work to figure out how CESM might have broken the interface and fix it. This will be even harder than in the past because of CESM's migration to NUOPC.

2. The set of script variables a user needs to change will vary from a few to many, depending on their experiment and machine. We'll need to decide which should be in a resources file and which will be left in the setup scripts. The cutoff will almost certainly be arbitrary, so we'll need to document which are set where. Speaking of documentation, roughly a third of the lines in the setup scripts are documentation of the variables. Will that documentation migrate with the variables into the resource file, making it potentially hundreds of lines? Or will it be left in the setup scripts where the variables are used, which would make the resource file small but only understandable by looking at some documentation at the same time? In the end we'll be asking all users to interact with the resources file, and many to interact with the setup script too, and possibly not being able to provide thorough guidance about which is necessary. The style of resources file we're discussing here differs from what I used in the Reanalysis. There the resources file was created by the setup script, using values defined in the setup script.

3. This one is relatively easy, but there are questions about which parts of CESM (and CIME) can be fixed for DART using SourceMods, and how to apply all the kinds of fixes.

4. Documentation of the code outside of the code as it evolves through issues and pull requests will result in needing to rewrite recently written documentation and possibly slowing the code development. External documention after the code is set would be less (re)writing, but doesn't provide the reviewers with as much information, and brings in the risk of delaying the documentation. Will there be a general rule about this, or decided on a case-by-case basis?

Originally posted by @kdraeder in #233 (comment)