Merge pull request #396 from NationalGenomicsInfrastructure/monthly

Monthy merge February 2020

README.md

@@ -67,14 +67,20 @@ Start by forking the repository https://github.com/NationalGenomicsInfrastructur
 
 If you want to test your roles/playbook run `ansible-playbook install.yml`. This will install your development run in `/lupus/ngi/irma3/devel-root/<username>_<branch_name>`.
 
-When you are satisfied with your changes you need to test it in staging. First create a pull request from your feature branch into upstream irma-provisioning's master branch.
+When you are satisfied with your changes you need to test it in staging. To do this, you must create a pull request to one of the two staging branches of irma-provision.
 
-Do the following once the feature has been approved:
+There are two staging branches in the repository, one called monthly and the other bimonthly. The changes that do not need extensive testing and validation on stage should be pushed to the monthly branch, which is used to make stage and production deployments on a monthly cycle. Changes that need time for validation on stage (like pipeline version updates) should be pushed to the bimonthly branch.
+
+Production deployments on the monthly release cycle are done on the last Monday of each month. A stage deployment from the monthly branch will be made two weeks before this date and the introduction of new changes to the monthly branch will be frozen. Only fixes to changes already merged will be approved to the monthly branch during this time.
+
+Stage deployments from the bimonthly branch can be made outside of the monthly release cycle. The changes on the bimonthly branch would generally be pulled to the monthly branch and deployed to production in the monthly release cycle once all validations are complete.
+
+Do the following once the feature has been approved. The deployment version should be `$(date +%y%m%d).$(git rev-parse --short HEAD)-bimonthly` in case of the bimonthly branch.
 
 ```
     cd /lupus/ngi/irma3/deploy
-    git checkout master
-    git pull
+    git checkout [monthly/bimonthly]
+    git pull origin [monthly/bimonthly]
     ansible-playbook install.yml -e deployment_environment=staging -e deployment_version=$(date +%y%m%d).$(git rev-parse --short HEAD)
     python sync.py -e staging -d deployment_version
 ```
@@ -85,7 +91,7 @@ If you want to stage test many feature branches at the same time then an alterna
 
 When everything is synced over to Irma properly then login to e.g. `irma1` as your personal user and run `source /lupus/ngi/staging/latest/conf/sourceme_<site>.sh && source activate NGI` (where `site` is `upps` or `sthlm` depending on location). For convenience add this to your personal file bash init file `~/.bashrc`. This will load the staging environment for your user with the appropriate staging variables set.
 
-When the staged environment has been verified to work OK (TODO: add test protocol, manual or automated sanity checks) proceed with making a production release at https://github.com/NationalGenomicsInfrastructure/irma-provision/releases/new. Make sure to write a good release note that summarizes all the significant changes that are included since the last production release.
+When the staged environment has been verified to work OK (TODO: add test protocol, manual or automated sanity checks) proceed with making a pull request from the staging branch to the master branch of the repository. Once your pull request is approved and merged, create a production release at https://github.com/NationalGenomicsInfrastructure/irma-provision/releases/new. Make sure to write a good release note that summarizes all the significant changes that are included since the last production release.
 
 ##### Arteria staging
 
@@ -95,7 +101,7 @@ So in essence it should work almost as usual. You run something similar to:
 
 ```
 ansible-playbook install.yml -e deployment_environment=staging -e deployment_version=arteria-staging-FOO -e arteria_checksum_version=660a8ff
-python sync.py -e staging 
+python sync.py -e staging
 ```
 
 if you want to stage test a specific commit hash of `arteria-checksum` and the default versions of the other arteria services.
@@ -114,6 +120,7 @@ To install software and sync to the cluster, run the following commands:
 
 ```
    cd /lupus/ngi/irma3/deploy
+   git checkout master
    git fetch --tags
    git checkout tags/vX.Y
    ansible-playbook install.yml -e deployment_environment=production

host_vars/127.0.0.1/main.yml

@@ -19,7 +19,7 @@ bash_env_sthlm_script: sourceme_sthlm.sh
 
 ngi_pipeline_repo: https://github.com/NationalGenomicsInfrastructure/ngi_pipeline.git
 ngi_pipeline_dest: "{{ sw_path }}/ngi_pipeline"
-ngi_pipeline_version: 1e2e11955c07f58512f93c6dfb5bf18912324f1c
+ngi_pipeline_version: e4d2d19942c67b213f25dca5bb3331f30b4a6b1f
 NGI_venv_name: "NGI"
 NGI_venv_py3_name: "NGI_py3"
 ngi_pipeline_venv: "{{ sw_path }}/anaconda/envs/{{ NGI_venv_name }}"

roles/arteria-delivery-ws/defaults/main.yml

@@ -7,7 +7,7 @@
 #
 # This will set corresponding paths and use the appropriate port. 
 arteria_delivery_repo: https://github.com/arteria-project/arteria-delivery.git
-arteria_delivery_version: v2.4.0
+arteria_delivery_version: v2.4.1
 
 arteria_install_path: "{{ sw_path }}/arteria"
 conda_bin: "{{ sw_path }}/anaconda/bin/conda"

roles/nextflow/templates/nextflow_irma_site.config.j2

@@ -1,6 +1,8 @@
 params {
     project = "{{ item.project_id }}"
     multiqc_config = "{{ ngi_pipeline_conf }}/multiqc_{{ item.site }}_config.yml"
+    sequencing_center = "SciLifeLab National Genomics Infrastructure {% if item.site=='sthlm' %}Stockholm{% else %}Uppsala{% endif %}"
+    seq_center = "SciLifeLab National Genomics Infrastructure {% if item.site=='sthlm' %}Stockholm{% else %}Uppsala{% endif %}"
 }
 
 {% if item.site == "sthlm" %}

roles/ngi_pipeline/templates/create_ngi_pipeline_dirs.sh.j2

@@ -36,3 +36,8 @@ ln -s /lupus/ngi/{{ deployment_environment }}/latest/sw/ngi_pipeline/DELIVERY.RE
 ln -s /lupus/ngi/{{ deployment_environment }}/latest/sw/ngi_pipeline/scripts/applyRecalibration.sh {{ proj_root }}/$1/nobackup/NGI/softlinks/applyRecalibration.sh
 ln -s /lupus/ngi/{{ deployment_environment }}/latest/sw/ngi_pipeline/scripts/bam2fastq.sh {{ proj_root }}/$1/nobackup/NGI/softlinks/bam2fastq.sh
 ln -s /lupus/ngi/{{ deployment_environment }}/latest/sw/ngi_pipeline/$ACKNOWLEDGE_SITE {{ proj_root }}/$1/nobackup/NGI/softlinks/ACKNOWLEDGEMENTS.txt
+
+#if sthlm, source delivery readme script
+if [ $1 == "ngi2016003" ]; then
+. create_delivery_readme_softlinks.sh
+fi

roles/ngi_reports/defaults/main.yml

@@ -1,7 +1,7 @@
 ---
 ngi_reports_repo: https://github.com/NationalGenomicsInfrastructure/ngi_reports.git
 ngi_reports_dest: "{{ sw_path }}/ngi_reports"
-ngi_reports_version: f68f5b6522754243055e0ee4cf6d01da20a3eaa2
+ngi_reports_version: eb4185756a22223fc3c746ee9fb969f75055e576
 ngi_reports_log: "/log/ngi_reports.log"
 ngi_reports_log_sthlm: "{{ ngi_pipeline_sthlm_path }}/{{ ngi_reports_log }}"
 

roles/taca/defaults/main.yml

@@ -1,7 +1,7 @@
 ---
 taca_ngi_repo: https://github.com/SciLifeLab/taca-ngi-pipeline.git
 taca_ngi_dest: "{{ sw_path }}/taca-ngi-pipeline"
-taca_ngi_version: e502a67ce2297787b467c9d52e07055da7cfe795
+taca_ngi_version: edbe7504b1ac67dbbf06e6c46f10ce0e6ca9291f
 
 statusdb_repo: "https://github.com/SciLifeLab/statusdb.git"
 statusdb_dest: "{{ sw_path }}/statusdb"

roles/taca/tasks/main.yml

@@ -18,7 +18,7 @@
   file: path="{{ ngi_pipeline_conf }}/TACA" state=directory mode=g+s
 
 - name: Create TACA resources folders for each site
-  file: 
+  file:
     path: "{{ ngi_resources }}/TACA/{{ item }}"
     state: directory
     mode: g+s
@@ -74,6 +74,9 @@
 - name: Deploy runfolder delivery stockholm config
   template: src="site_taca_runfolder_delivery.yml.j2" dest="{{ ngi_pipeline_conf }}/TACA/{{ site }}_taca_runfolder_delivery.yml"
 
+- name: Deploy script for softlinking sthlm delivery Readmes
+  template: src="create_delivery_readme_softlinks.sh.j2" dest="{{ ngi_resources }}/create_delivery_readme_softlinks.sh"
+
 - name: Store taca tools version on deployment
   lineinfile:
     dest: "{{ deployed_tool_versions }}"

roles/taca/templates/create_delivery_readme_softlinks.sh.j2

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+# Minimalistic script that initiates project/site specific
+# directories for delivery readmes under respective
+# site's project dirs. Currently only used by sthlm
+
+# To be run as func account:
+#
+# funk_006 -> ngi2016003
+
+if [ $# -ne 1 ]; then
+        echo "$0: usage: create_delivery_readme_softlinks.sh <project id>"
+        exit 1
+fi
+
+ln -s /lupus/ngi/{{ deployment_environment }}/latest/sw/taca-ngi-pipeline/delivery_readmes/DELIVERY.README.RAW_DATA.txt {{ proj_root }}/$1/nobackup/NGI/softlinks/DELIVERY.README.RAW_DATA.txt
+ln -s /lupus/ngi/{{ deployment_environment }}/latest/sw/taca-ngi-pipeline/delivery_readmes/DELIVERY.README.MethylSeq.txt {{ proj_root }}/$1/nobackup/NGI/softlinks/DELIVERY.README.MethylSeq.txt

roles/taca/templates/site_app_specific_delivery.yml.j2

@@ -40,5 +40,4 @@ deliver:
             - <STAGINGPATH>/00-Reports
 {% if "sthlm" == site %}
     save_meta_info: True
-    xmlgen_ignore_lib_prep: False
 {% endif %}

roles/taca/templates/site_fastq_delivery.yml.j2

@@ -29,6 +29,8 @@ deliver:
             - <ANALYSISPATH>/*multiqc_report.html
             - <STAGINGPATH>/00-Reports
 {% if site == "sthlm" %}
+        -
+            - {{ ngi_site_softlinks }}/DELIVERY.README.RAW_DATA.txt 
+            - <STAGINGPATH>
     save_meta_info: True
-    xmlgen_ignore_lib_prep: False
 {% endif %}

roles/taca/templates/site_taca_delivery.yml.j2

@@ -130,5 +130,4 @@ deliver:
             - <ANALYSISPATH>/*multiqc_report.html
             - <STAGINGPATH>/00-Reports
     save_meta_info: True
-    xmlgen_ignore_lib_prep: False
 {% endif %}

roles/taca/templates/site_taca_runfolder_delivery.yml.j2

@@ -8,5 +8,4 @@ deliver:
     operator: "{{ recipient_mail }}"
 {% if "sthlm" == site %}
     save_meta_info: True
-    xmlgen_ignore_lib_prep: False
 {% endif %}