Quick start

Building access-om3 executable (optional)

It's easiest to use avoid building access-om3 by using a precompiled executable from /g/data/ik11/inputs/access-om3/bin/. This requires membership of the ik11 project - apply here if needed.

However, if you want to make code changes you'll need to build access-om3 yourself. See here to find out how.

Setting up a configuration

There are several configurations available, listed here. You should use the accessom3_exe branch of these.

For example, to run the MOM6-CICE6 configuration, clone and switch to the accessom3_exe branch:

git clone https://github.com/COSIMA/MOM6-CICE6.git
cd COSIMA/MOM6-CICE6
git checkout accessom3_exe

It's best to do this somewhere in your /home directory, as this is the only filesystem that's backed up.

If you've compiled your own executable you'll need to edit the exe entry in config.yaml to point to it.

Running

Running ACCESS-OM3 requires an updated payu, available from analysis3-23.04 (currently analysis3-unstable) or later, and the executable requires some custom modules, so you need to do this before you can run (only needs to be done once per log in, or you can put this in your ~/.bash_profile to do it automatically each login):

module use /g/data/hh5/public/modules
module use /g/data/ik11/spack/0.20.1/share/modules/linux-rocky8-cascadelake
module load conda/analysis3-unstable

Now you're ready to run:

payu run

This uses the payu workflow management tool to prepare the run and submit it as a job to the PBS job queue. See the Gadi User Guide to learn more about PBS job management.

Check the status of the job (state 'Q'=waiting in queue, 'R'=running, 'E'=exiting, 'H'=held) with

uqstat -c

While it's running, you can check the date it's up to with

grep currtime work/log/med.log

To kill the run early, do qdel N, where N is the job number (first column given by uqstat). If you kill the job (or it crashes), a work directory will be left behind after the job has disappeared from uqstat and you'll need to do payu sweep before you can run again.

When your run has finished successfully, payu puts its output in archive/output000 and removes the work directory. payu also records a log of your experiment in the git history, including the identity of the inputs and executables used (see the files in manifests).

To do another run, just type payu run again. Or to do (say) 10 runs, type payu run -n 10 and they'll automatically be submitted one after the other.

The outputs from each run will be in numbered subdirectories in archive.

Each run creates a restart directory in archive which is used as the initial condition for the next run. These restarts can accumulate and consume disk space, but only the most recent one is needed (unless you plan to restart a new experiment from an intermediate state). See the payu documentation for more information.

If the run fails, the work directory will be retained. You can find diagnostic messages in access-om3.*, MOM6-CICE6.[oe]*, work/log/*, work/logfile* and other files in work. You will need to do payu sweep to delete the work directory before you will be able to do another payu run. This will also save copies of the PBS logs into archive/pbs_logs.

WARNING restarts and outputs are stored on /scratch and will therefore be deleted if unread for 100 days, so if you value them you should move them somewhere safer, e.g. /g/data. Note that /home is the only filesystem that is backed up, so your configuration should live there, but you probably won't have room for outputs and restarts.

Customising your experiment

See the Configurations section to find out how to set various parameters.

Before running, commit your changes with an informative message, e.g. git commit -am "initial setup for experiment to test... bla bla"