Data Hub: K8s: Reduce config duplication using yaml anchors

[de-code]

part of https://github.com/elifesciences/data-hub-issues/issues/1003

Reduce duplication in the config using yaml anchors.

[de-code]

This can be tested using the CLI in the Core DAGs PR:

python -m data_pipeline.kubernetes.cli.dump_config \
    --config-file=/path/to/elife-flux-cluster/kustomizations/apps/data-hub-configs/kubernetes-pipeline--stg.config.yaml \
    --deployment-env=staging \
    --data-pipeline-id=Web_API_elife_observer_api_publication_dates_Kubernetes

airflow:
  dagParameters:
    max_active_runs: 1
    schedule: 25 2 * * *
    tags:
    - Kubernetes
    - Web API
arguments:
- python
- -m
- data_pipeline.generic_web_api.cli
- --data-pipeline-id=elife_observer_api_publication_dates
dataPipelineId: Web_API_elife_observer_api_publication_dates_Kubernetes
env:
- name: DEPLOYMENT_ENV
  value: staging
- name: GOOGLE_APPLICATION_CREDENTIALS
  value: /dag_secret_files/gcloud/credentials.json
- name: AWS_CONFIG_FILE
  value: /dag_secret_files/aws/credentials
- name: WEB_API_CONFIG_FILE_PATH
  value: /dag_config_files/web-api-data-pipeline.config.yaml
image: docker.io/elifesciences/data-hub-core-dags_unstable:latest
imagePullPolicy: Always
resources:
  limits:
    cpu: 1000m
    memory: 600Mi
  requests:
    cpu: 100m
    memory: 500Mi
volumeMounts:
- mountPath: /dag_secret_files/gcloud/
  name: gcloud-secret-volume
  readOnly: true
- mountPath: /dag_secret_files/aws
  name: aws-secret-volume
  readOnly: true
- mountPath: /dag_config_files/
  name: data-hub-config-volume
  readOnly: true
volumes:
- name: aws-secret-volume
  secret:
    secretName: credentials
- name: gcloud-secret-volume
  secret:
    secretName: gcloud
- configMap:
    name: data-hub-configs
  name: data-hub-config-volume

[de-code]

One thing I didn't realise is that merge keys were removed in YAML 1.2, but are still supported by many libraries.

[de-code]

Closing this, as we seem to be in favour in implementing default config at the application level rather than using yaml anchors