diff --git a/.github/actions/parse-ci-config/action.yml b/.github/actions/parse-ci-config/action.yml index 0375e50..6948320 100644 --- a/.github/actions/parse-ci-config/action.yml +++ b/.github/actions/parse-ci-config/action.yml @@ -3,22 +3,22 @@ description: Action to parse model-config-tests configurations for CI tests inputs: check: required: true - description: Type of check (e.g. reproducibility, qa) + description: Type of check/test to run (e.g. "reproducibility", "qa" or "scheduled") branch-or-tag: required: true description: Name of Git branch or tag to run CI testing on config-filepath: required: true - description: Path to configuration file + description: Path to CI configuration file outputs: model-config-tests-version: value: ${{ steps.read-config.outputs.model-config-tests-version }} description: A version of the model-config-tests package python-version: - value: ${{ steps.read-config.python-version }} + value: ${{ steps.read-config.outputs.python-version }} description: The python version used to create test virtual environment markers: - value: ${{ steps.read-config.markers }} + value: ${{ steps.read-config.outputs.markers }} description: A python expression of markers to pass to model-config-tests pytests runs: using: "composite" @@ -50,4 +50,4 @@ runs: echo "markers=$(echo "$output" | jq -r '.["markers"]')" >> $GITHUB_OUTPUT echo "python-version=$(echo "$output" | jq -r '.["python-version"]')" >> $GITHUB_OUTPUT - echo "model-config-tests-=$(echo "$output" | jq -r '.["model-config-tests-version"]')" >> $GITHUB_OUTPUT + echo "model-config-tests-version=$(echo "$output" | jq -r '.["model-config-tests-version"]')" >> $GITHUB_OUTPUT diff --git a/README-DEV.md b/README-DEV.md index 10cb9d6..43f658a 100644 --- a/README-DEV.md +++ b/README-DEV.md @@ -33,7 +33,7 @@ The first job, `commit-check`, is used to short-circuit execution of this workfl This job is used as a check before running [`repro-ci`](#repro-ci) checks, which are only run on PRs `dev-*` -> `release-*`. It also makes sure that the branches are formatted correctly. -#### `config` +##### `config` This job reads configuration file `config/ci.json` to obtain pytest markers, `model-config-tests` version and python version for running QA and reproducibility tests. @@ -126,3 +126,32 @@ This workflow is used to check that modifications to `*.json` files are in line #### Initial Checksum Creation: `generate-initial-checksums.yml` This `workflow_dispatch`-triggered workflow generates checksums of a given config branch, and optionally commits them. This is useful for generating checksums for an entirely new config branch, so the workflows above have something to compare against. + +### CI Configuration File + +This is the `config/ci.json` configuration file for specifying different test markers, or test versions based on type of the test to run, and the name of the git branch or tag. The different types of test are defined as: +- `scheduled`: These are scheduled monthly reproducibility tests that are run as part of [`schedule-2-start.yml`](#repro-ci-1). The keys under these tests represent released config tags to run scheduled checks on. +- `reproducibility`: These are reproducibility tests are run as part of [`pr-1-ci.yml`](#repro-ci). The keys under these tests represent the target branches into which pull requests are being merged. +- `qa` - These are quick QA tests are run as part of [`pr-1-ci.yml`](#qa-ci). The keys under these tests represent the target branches into which pull requests are being merged. + +As most of the tests use the same test and python versions, and similar markers, there are two levels of defaults. There's a default at test type level which is useful for defining test markers - this selects certain pytests to run in `model-config-tests`. There is an outer global default, which is used if a property is not defined for a given branch/tag and it is not defined for the test default. + +#### parse-ci-config + +This action parses the CI testing configuration file. The caller of the action needs to checkout the branch where the config file is defined before running this action. + +#### Inputs + +| Name | Type | Description | Required | Example | +| ---- | ---- | ----------- | -------- | ------- | +| check | `string` | The type of check/test to run | true | `scheduled` | +| branch-or-tag | `string` | The name of git branch or tag | true | 'release-1deg_jra55_ryf-2.0' | +| config-filepath | `string` | Path to configuration file | true | `config/ci.json` | + +#### Outputs + +| Name | Type | Description | Example | +| ---- | ---- | ----------- | -------- | +| markers | `string` | Markers used for the pytest checks, in the python format | `checksum` | +| model-config-tests-version | `string` | The version of the model-config-tests | `0.0.1` | +| python-version | `string` | The python version used to create test virtual environment | `3.11.0` | diff --git a/config/ci.json b/config/ci.json index 3c885b1..171133d 100644 --- a/config/ci.json +++ b/config/ci.json @@ -14,7 +14,7 @@ }, "qa": { "default": { - "markers": "access-om2 or config" + "markers": "access_om2 or config" } }, "default": { diff --git a/config/ci.schema.json b/config/ci.schema.json index a111ce6..6151f47 100644 --- a/config/ci.schema.json +++ b/config/ci.schema.json @@ -17,7 +17,7 @@ }, "python-version": { "type": "string", - "description": "The python version used to create test virtual environment." + "description": "The python version used to create test virtual environment" } }, "additionalProperties": false @@ -33,7 +33,8 @@ "$ref": "#/$defs/config", "required": [ "markers" - ] + ], + "description": "The default configuration for this check" } }, "required": [ @@ -48,22 +49,23 @@ }, "scheduled": { "$ref": "#/$defs/check", - "description": "Scheduled reproducibility checks" + "description": "Scheduled reproducibility checks. The keys are config tags to run scheduled checks on" }, "reproducibility": { "$ref": "#/$defs/check", - "description": "Reproducibility checks" + "description": "Reproducibility checks. The keys are the target branch names for pull requests" }, "qa": { "$ref": "#/$defs/check", - "description": "Quick quality assurance checks" + "description": "Quick quality assurance checks. The keys are the target branch names for pull requests" }, "default": { "$ref": "#/$defs/config", "required": [ "model-config-tests-version", "python-version" - ] + ], + "description": "Global default configuration" } }, "required": [