suite-writing.tex

\section{In-Depth Suite Writing Tutorial}
\label{In-Depth Suite Writing Tutorial}

\begin{figure}[htp]
\includegraphics[width=0.3\columnwidth]{resources/suite-writing.png}
\end{figure}

\subsection{Introduction}

This tutorial walks you through creating a non-trivial suite based around the requirements of a piece of code that needs to be compiled, run, and supplied with inputs. Often, people won't create suites from scratch like this, but will base new suites on copies of existing ones.

\subsection{Starting Out}

This example supposes:
\begin{itemize}
    \item We're on a sailing ship, making a passage
    \item We're navigating using a Fortran program
\end{itemize}

Our Fortran program reads in some position data and then pretends to calculate a new one based on a compass direction and a 5 knot speed for a given period of time.

If you are reading this tutorial via the cylc VM then the Fortran code is located at {\em \$HOME/tutorial/suites/navigation/src/dead\_reckoning.f90}. Otherwise, the code is given in Appendix~\ref{Appendix In-Depth Suite Writing Tutorial}.

We need to analyse our program to figure out what the dependencies are.

Have a quick look through the code and look for:
\begin{itemize}
    \item what files or environment it needs to run
    \item what it produces
    \item when it might need to run
\end{itemize}

\subsection{Inputs}

The inputs to this Fortran code are:
\begin{itemize}
    \item Two environment variables, {\em TIME\_INTERVAL\_HRS} and {\em POSITION\_FILEPATH}
    \item An input (and output) file located at {\em \$POSITION\_FILEPATH} that stores the latitude and longitude.
\end{itemize}

When we run the compiled program, we'll need all these inputs to be present.

We want to run this program every 3 hours, on the hour - so there is a repeated dependency on the time or cycle.

We also want to build the program to begin with, so we need a compilation task that runs at the start.

\subsection{Suite Creation}

If you are running this tutorial on the cylc VM \footnote{
    If and only if you are {\em not} running on the VM:
\begin{itemize}
    \item create a new directory called {\em navigation}) somewhere in your homespace, which will be the top-level suite directory.
    \item inside that directory, create a subdirectory called {\em src}
    \item copy the Fortran code into a file called {\em src/dead\_reckoning.f90}
    \item create a suite.rc file under {\em navigation} with the contents below in Section~\ref{Suite Writing suite.rc file}.
\end{itemize}
} , the raw directory structure of the suite is set up for you. Change directory to {\em \$HOME/tutorial/suites/navigation/}. This is our suite top-level directory.

\subsubsection{suite.rc file}
\label{Suite Writing suite.rc file}

An initial suite.rc file is already present in the {\em \$HOME/tutorial/suites/navigation/} directory.
It looks like this:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
[cylc]
    UTC mode = True # Ignore DST
[scheduling]
    initial cycle point = 20160601T00Z  # 1 June 2016
    final cycle point = 20160603T00Z  # 3 June 2016
    [[dependencies]]
        [[[R1]]]  # Run once at the start of the suite.
            graph = compile_navigate
[runtime]
    [[root]]
        pre-script = sleep 5  # Slow down tasks for visualization.
    [[compile_navigate]]
        script = """
            gfortran $CYLC_SUITE_DEF_PATH/src/dead_reckoning.f90 \
                -o $CYLC_SUITE_SHARE_PATH/dead_reckoning.exe
        """
\end{lstlisting}

    We've set the suite to run from midnight, 1 June 2016 to midnight, 3 June 2016. This is just a model time or label. It isn't synchronised with real clock time.

    We've set the \lstinline{compile_navigate} task to run once at the initial cycle point (i.e. midnight, 1 June 2016). This will compile the source code and produce an executable in a directory {\em \$HOME/cylc-run/navigation/share/}. This directory, {\em \$CYLC\_SUITE\_SHARE\_PATH} is provided by cylc as a place for inter-task 'sharing' of files.

    All our tasks inherit from \lstinline{root} and will run \lstinline{sleep 5} from the \lstinline{root} {\em pre-script} setting.

\subsubsection{Initial Run}

    We can test that this works as it is.
    
    Run \lstinline{rose suite-run}\footnote {Section~\ref{Hello World in cylc} explained why we're running this instead of cylc commands.}.
    
    The \lstinline{compile_navigate} task should succeed, and the suite will shut down automatically. After that, there should be a newly created {\em dead\_reckoning.exe} file under {\em \$HOME/cylc-run/navigation/share/}. Have a look in that directory.

\subsubsection{Adding a task}

Let's now add our navigation task to run our executable. There are two steps involved in adding a task in the {\em suite.rc}:
\begin{itemize}
    \item Add the task to the dependencies so cylc knows when to run it.
    \item Configure what the task actually does, in the runtime section!
\end{itemize}

Modify the {\em suite.rc} to have a dependencies section that looks like this:
\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
    [[dependencies]]
        [[[R1]]]  # Run once at the start of the suite.
            graph = compile_navigate => navigate
        [[[PT3H]]]  # Run every 3 hours (ISO 8601 date-time syntax).
            graph = navigate[-PT3H] => navigate
\end{lstlisting}

Here, we've made \lstinline{navigate} run every 3 hours, with each instance waiting for the previous one to finish.

We've configured the dependency between \lstinline{compile_navigate} and \lstinline{navigate}.

We also need to tell the \lstinline{navigate} task what to run. Add a new entry under the runtime section:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
    [[navigate]]
        script = $CYLC_SUITE_SHARE_PATH/dead_reckoning.exe
        [[[environment]]]
            POSITION_FILEPATH = $CYLC_SUITE_SHARE_PATH/position
            TIME_INTERVAL_HRS = 3
\end{lstlisting}

This instructs cylc to run the executable made by \lstinline{compile_navigate} and sets up the necessary environment variables.

Our navigation executable will read and write the {\em \$POSITION\_FILEPATH} position file. Each run of the executable will read the previous run's output. However, at the first cycle point, the file will not currently exist. We'll need to make it and include start coordinates.

\subsubsection{Adding another initial-only task}

We'll add a task called \lstinline{write_start_position} to create this initial file. Add it as a run-once ({\em R1}) task by replacing the graph for the \lstinline{[[[R1]]]} section with:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
        [[[R1]]]  # Run once at the start of the suite.
            graph = """
                compile_navigate => navigate
                write_start_position => navigate
            """
\end{lstlisting}

Finally, add these lines under the \lstinline{runtime} section, at the end of the suite.rc file:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
    [[write_start_position]]
        script = echo 50.0 -3.0 >$CYLC_SUITE_SHARE_PATH/position
\end{lstlisting}

This initialises our location for the \lstinline{navigate} task via a file (pointed to via {\em \$POSITION\_FILEPATH}). Our start coordinates are 50.0 north, 3.0 west, which is in the English Channel (a.k.a. La Manche, Canal da Mancha, etc). You can change these to another location if you like.

\subsection{Checking the suite}

Your suite should now look something like this:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
[cylc]
    UTC mode = True # Ignore DST
[scheduling]
    initial cycle point = 20160601T00Z  # 1 June 2016
    final cycle point = 20160603T00Z  # 3 June 2016
    [[dependencies]]
        [[[R1]]]  # Run once at the start of the suite.
            graph = """
                compile_navigate => navigate
                write_start_position => navigate
            """
        [[[PT3H]]]  # Run every 3 hours (ISO 8601 date-time syntax).
            graph = navigate[-PT3H] => navigate
[runtime]
    [[root]]
        pre-script = sleep 5  # Slow down tasks for visualization.
    [[compile_navigate]]
        script = """
            gfortran $CYLC_SUITE_DEF_PATH/src/dead_reckoning.f90 \
                -o $CYLC_SUITE_SHARE_PATH/dead_reckoning.exe
        """
    [[navigate]]
        script = $CYLC_SUITE_SHARE_PATH/dead_reckoning.exe
        [[[environment]]]
            POSITION_FILEPATH = $CYLC_SUITE_SHARE_PATH/position
            TIME_INTERVAL_HRS = 3
    [[write_start_position]]
        script = echo 50.0 -3.0 >$CYLC_SUITE_SHARE_PATH/position
\end{lstlisting}

\subsection{Running our Suite}

Run:

\begin{lstlisting}[mathescape, language=bash]
$\$$ rose suite-run
\end{lstlisting}

If everything has been set up successfully, after running that command, \lstinline{cylc gui} will launch with your running suite.

\subsection{Finished Output}

You can look at the finished output by running:

\begin{lstlisting}[mathescape, language=bash]
$\$$ rose suite-log
\end{lstlisting}

The position will be written in the out file for each navigate task.

\subsubsection{Nicer Output}

If you want a quick and easy way of visualising the output, try replacing the line:

\lstset{language=Fortran}
\begin{lstlisting}[columns=fullflexible]
PRINT*, "New position, me hearties:",new_lat," ",new_long
\end{lstlisting}

in {\em dead\_reckoning.f90} with

\lstset{language=Fortran}
\begin{lstlisting}[columns=fullflexible]
lat = (180.0/pi) * lat
long = (180.0/pi) * long
CALL get_environment_variable("CYLC_TASK_LOG_ROOT",value=task_log_root,status=code)
OPEN(1,file=TRIM(task_log_root) // '-map.html',action='write')
WRITE(1,'(A, F7.4, A, F7.4, A, F7.4, A, F7.4, A, F7.4, A, F7.4, A)') &
      "<img src='https://maps.googleapis.com/maps/api/staticmap?center=",&
      lat,",",long,"&zoom=7&size=600x300&markers=color:blue|label:A|",&
      lat,",",long,"&markers=color:red|label:B|",new_lat,",",new_long,&
      "'/>"
CLOSE(1)
\end{lstlisting}

\note{Beware of pdf copy-and-paste introducing line breaks before and after the asterisks (*) in the code above.}

This will produce a {\em job-map.html} file per job which you can click through to in \lstinline{Rose Bush} (via running \lstinline{rose suite-log}).

\subsection{More Cycling}

Our suite has a start cycle point and a single cycling period of 3 hours, but we could have other cycle definitions in the same suite. Let's add a task called \lstinline{take_sun_sight} that runs at 12:00 each day. This will correct our latitude.

Add these lines below \lstinline{[[dependencies]]} in your suite.rc file:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
        [[[T12]]]  # Run at 12:00Z each day.
            graph = navigate => take_sun_sight
\end{lstlisting}

This will run after the \lstinline{navigate} task from that cycle point finishes.

In order to make the \lstinline{navigate} task wait for our new \lstinline{take_sun_sight} task, we'll need to add some extra configuration for the 15 hour cycle - add the following lines in the same way as you did for \lstinline{[[[T12]]]}:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
        [[[T15]]]  # Run at 15:00Z each day.
            graph = take_sun_sight[-PT3H] => navigate
\end{lstlisting}


\subsection{Adding a Script to the Suite}

You can put scripts in the {\em bin/} directory of a suite, and they will be added to cylc's path.

Change directory to the root of your suite, and create a {\em bin/} directory.

Create an empty file in the {\em bin/} directory called {\em bin/sun\_sight}.

Open this file with a text editor. Paste the following text into the {\em sun\_sight} file:

\lstset{language=Python}
\begin{lstlisting}[columns=fullflexible]
#!/usr/bin/env python

import random
import sys


if __name__ == "__main__":
    random.seed()
    with open(sys.argv[1], "r") as f:
        (lat, long) = f.read().split()
    lat = float(lat) + random.uniform(-0.05, 0.05)
    print "Yarr! Our corrected position be {0}, {1}".format(lat, long)
    with open(sys.argv[1], "w") as f:
        f.write("{0} {1}\n".format(lat, long))
\end{lstlisting}

Save the file and make sure the indentation is correct. Make it executable by running:

\begin{lstlisting}[mathescape, language=bash]
$\$$ chmod +x bin/sun_sight
\end{lstlisting}

We need to reference this script explicitly in the {\em suite.rc} for our \lstinline{take_sun_sight} task - append these lines to the {\em suite.rc} file:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
    [[take_sun_sight]]
        script = sun_sight $CYLC_SUITE_SHARE_PATH/position
\end{lstlisting}

\subsection{Results}

Run the suite by invoking:
\begin{lstlisting}[mathescape, language=bash]
$\$$ rose suite-run
\end{lstlisting}

Our extra task should run at 20160601T1200Z and 20160602T1200Z. If it does, congratulations!

If you run into any problems, you can refer to a completed suite at {\em \$HOME/tutorial/suites/navigation-complete}.

You can check the graphing by running:

\begin{lstlisting}[mathescape, language=bash]
$\$$ cylc graph navigation 20160601T0000Z 20160603T0000Z
\end{lstlisting}

\subsection{Rose (optional - feel free to skip)}
\label{Rose}

\subsubsection{Introduction}

\begin{figure}[h]
\includegraphics[width=1.0\columnwidth]{resources/rose-edit.png}
\caption{rose edit, a generic configuration GUI showing a dynamically generated panel for a section in the inputs for the Met Office Unified Model. The contents of the panel are generated from the relevant part of a Rose app file plus some simple metadata, both specified in flat INI-based text files. \label{rose edit}}
\end{figure}

\begin{lstlisting}[mathescape, language=bash, title={rose-app.conf snippet used to drive Figure~\ref{rose edit}}]
[namelist:run_rivers]
i_river_vn=1
!!l_inland=.false.
l_rivers=.false.
!!river_mcoef=3.1
!!river_step=10800
!!river_vel=0.5
\end{lstlisting}

Rose has a powerful set of utilities for the configuration and running of executables, suite
storage and version control, and also includes some top-level suite running functionality that will be migrated to cylc.

In this part of the tutorial, we'll demonstrate a couple of pieces of Rose functionality that can enhance suites and
tasks:
\begin{itemize}
    \item{Top-level suite.rc configuration - {\em rose-suite.conf}}
    \item{Metadata for the top-level inputs in {\em rose-suite.conf}, which is used for checking and presentation via \lstinline{rose edit}}
    \item{{\em app} configuration to configure inputs for particular executables in tasks}
    \item{Metadata for the {\em app}, as for the suite, and \lstinline{rose edit} presentation of it}
\end{itemize}

\subsubsection{rose-suite.conf}

Let's pull out our initial starting longitude and latitude into top-level {\em rose-suite.conf} Jinja2 template inputs. Jinja2 allows us to effectively script the {\em suite.rc} file when we need to.

Add this line to the top of the suite.rc file:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
#!jinja2
\end{lstlisting}

and replace:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
    [[write_start_position]]
        script = echo 50.0 -3.0 > $CYLC_SUITE_SHARE_PATH/position
\end{lstlisting}

with:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
    [[write_start_position]]
        script = """
            echo {{START_LATITUDE}} {{START_LONGITUDE}} > \
                $CYLC_SUITE_SHARE_PATH/position
        """
\end{lstlisting}

We can now pull out \lstinline{START_LATITUDE} and \lstinline{START_LONGITUDE} into top-level {\em rose-suite.conf} inputs. Open the (currently empty) {\em rose-suite.conf} file and change it to read:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
[jinja2:suite.rc]
START_LATITUDE=50.0
START_LONGITUDE=-3.0
\end{lstlisting}

When the suite is run with \lstinline{rose suite-run}, these variables will be inserted into the \lstinline{write_start_position} task's script and it will run as before.

\subsubsection{rose-suite.conf Metadata}

Rose has a concept of metadata for inputs like these, which help check their validity and improve the presentation in the \lstinline{rose edit} configuration editor GUI. We can take a shortcut to fill out some of that metadata.

In the top-level directory of your suite, run:

\begin{lstlisting}[mathescape, language=bash]
$\$$ rose metadata-gen --auto-type
\end{lstlisting}

It will create a subdirectory called {\em meta} with a file called {\em rose-meta.conf} inside. Open that file and edit it to add some description and allowed value ranges:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
[jinja2:suite.rc]
title=Starting Position

[jinja2:suite.rc=START_LATITUDE]
compulsory=true
description=Starting latitude in degrees
range=-90:90
type=real

[jinja2:suite.rc=START_LONGITUDE]
compulsory=true
description=Starting longitude in degrees
range=-180:180
type=real
\end{lstlisting}
 
Now run the command \lstinline{rose edit} - this launches a nice GUI interface to our inputs which will check for errors. Try e.g altering the longitude to 300 degrees. An error should display.

You can run the suite directly through \lstinline{rose edit} via the 'Play' button in the toolbar (although for safety, this button will be disabled if there are unsaved changes).

\subsubsection{Rose Apps (rose-app.conf)}

We can also add configuration for our Fortran program via a Rose app configuration. This allows us to pull out internal task-specific configuration into a separate location and add metadata and GUI support for it. Inter-task configuration, such as the \lstinline{$POSITION_FILEPATH} variable, would normally belong in the suite.rc file where it helps to define workflow.

Alter the \lstinline{navigate} task \lstinline{script} in the \lstinline{suite.rc} file to read:
\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
    [[navigate]]
        script = rose task-run -v
\end{lstlisting}

and delete the \lstinline{[[[environment]]]} settings for the task there too.

Create an {\em app/navigate/} subdirectory in your suite:

\begin{lstlisting}[mathescape, language=bash]
$\$$ mkdir -p app/navigate/
\end{lstlisting}

Create a file underneath that directory called {\em rose-app.conf} ({\em app/navigate/rose-app.conf}). Alter that file to read:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
[command]
default=$CYLC_SUITE_SHARE_PATH/dead_reckoning.exe

[env]
POSITION_FILEPATH=$CYLC_SUITE_SHARE_PATH/position
TIME_INTERVAL_HRS=3
\end{lstlisting}

The new \lstinline{[env]} section in the app replaces the \lstinline{[[[environment]]]} settings for \lstinline{[[navigate]]} in the {\em suite.rc} file.

\subsubsection{rose-app.conf Metadata}

Create a further {\em meta} subdirectory for some app metadata:

\begin{lstlisting}[mathescape, language=bash]
$\$$ mkdir -p app/navigate/meta
\end{lstlisting}

Inside that new directory, create a {\em rose-meta.conf} file ({\em app/navigate/meta/rose-meta.conf}) that looks like this:

\lstset{language=suiterc}
\begin{lstlisting}[columns=fullflexible]
[env=TIME_INTERVAL_HRS]
range=1:24
type=integer
\end{lstlisting}

If you save these files and then launch \lstinline{rose edit}, you should see configuration appear for the \lstinline{navigate} app. As we have declared \lstinline{TIME_INTERVAL_HRS} to be an integer, it should have a numeric widget. It should display an error if the value is less than 1 or greater than 24.

\note{This tutorial covered only a limited subset of the functionality that Rose apps and metadata can provide - there is also inter-variable triggering, a Pythonic mini-language, dynamic page assignment, deep Fortran namelist integration, and much more. See the Rose documentation for more details.}