[validator] Add additional countries to adm1

[tomrconnor]

Brief description

COG-UK may begin to receive samples from British Overseas Territories or Crown Dependencies. This will require an update to adm1 in order to provide the option for sequences from these locations to be correctly linked. These may require some form of suppression post-upload, for example in the generation of data for the public MicroReact.

Detail

Validator	Field	Proposed option	Justification
adm1	adm1	FK	Falkland Islands are not covered by current adm1 options and have their own ISO 3166-2 identifier
adm1	adm1	GI	Gibraltar is not covered by current adm1 options and have their own ISO 3166-2 identifier
adm1	adm1	JE	Jersey is not covered by current adm1 options and have their own ISO 3166-2 identifier
adm1	adm1	IM	The Isle of Man is not covered by current adm1 options and have their own ISO 3166-2 identifier
adm1	adm1	GG	Guernsey is not covered by current adm1 options and have their own ISO 3166-2 identifier

---

Proposed by: Connor, T. R. PHWC

• I have read the documentation for the existing validators and there is no suitable entry

---

• Add field validation to Majora forms
• Add field to API docs
• Push to Majora prod
• Update validator spreadsheet (if applicable)

[SamStudio8]

Discussed this with TRC on Slack but adding the salient points here:

• Additions to adm1 validator is a straightforward request but there'll be a few tasks to tick off before we can set this up;

• Deductive disclosure concerns are currently handled post-hoc before individual sequence-level data is made available to public sources (e.g. via Microreact). Standard approaches should apply here.

@tomrconnor can you confirm that the sequences from these new country codes will not need any other special handling? ie. Is there a problem with automatically uploading genomes from these countries with the basic public metadata using the same system currently provided for the home nations? Or providing the consortium level metadata to COG through the usual means?

---

• Flag this to DIPI
• Audit all code and pipelines that use adm1 and ensure they'll be compatible with upcoming changes e.g.:
  • Automated GISAID and ENA submissions will need to have their country code to full name mappings expanded
  • Datapipe geography munging code will need to be aware of the changes
  • Microreact team will need to be aware of the changes

[tomrconnor]

As far as I am aware, deductive disclosure issues aside, there shouldn't be any need for special handling. I think the advice to the locations concerned will be to not upload other details, so we may need to think about what they put into adm2. I would expect that the upload of metadata would use the same system as the rest of COG-UK.

The locations concerned will be signing the data access agreement and will basically be treated like other sites.

[SamStudio8]

Perfect. adm2 is only made public through sources like Microreact where deductive disclosure concerns are already handled routinely. DIPI meets tomorrow so I'll raise it with the various pipeline teams.

…

On Tue, Apr 13, 2021 at 3:16 PM tomrconnor ***@***.***> wrote: As far as I am aware, deductive disclosure issues aside, there shouldn't be any need for special handling. I think the advice to the locations concerned will be to not upload other details, so we may need to think about what they put into adm2. I would expect that the upload of metadata would use the same system as the rest of COG-UK. The locations concerned will be signing the data access agreement and will basically be treated like other sites. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#50 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIN6OU6SFMKFEIHEVUB433TIRG23ANCNFSM423MWGYQ> .

[SamStudio8]

Flagged this to DIPI. RC will follow up with VH and AT on geography cleaning steps. AU confirms this should fit in to standard practice with Microreact. Test data might be useful.

[SamStudio8]

@tomrconnor Although adm2 is not controlled by Majora it would be useful to get an idea of what the adm2 data for each of these countries may look like, do you have any example sample-level metadata or aggregate adm2 counts that we might be able to feed back to geography teams so they can update their scripts?

[SamStudio8]

I have conducted a small scale audit of my software directory and spot the following:

• Majora API (validator)
• Majora API response (adm1_trans)
• ocarina_resolve / Elan (used to convert country for inclusion in FASTA header via Elan)
• ocarina (no direct handling)
• asklepian (no direct handling)
• ENA outbound (Elan/bam) (uses Majora adm1_trans, but fixed adm0)
• ENA assembly outbound (no direct handling)
• GISAID outbound (uses adm1_trans, but has fixed UK adm0)

I've realised, there is a slightly wider problem here, in that we've never asked for adm0, because it has always been assumed to be United Kingdom. All samples have their adm0 set to UK in Majora, but we will need to automatically fill in and handle adm0 with something appropriate going forward. Note that adm0 cannot be blank as it is a proxy for whether the biosample has been filled in (which is a different future issue).

Will need to compile the accepted formats for these new countries at GISAID and ENA.

Additionally, if PHE needs to receive the local lab ID field then we will need to revisit the configuration of the agreement within Majora to expand the country list from UK-ENG.

• Ask GB GISAID for country list
• Ask NR ENA about country validation

[SamStudio8]

Outbound lookups as follows:

adm1	Country	Genbank[1]	GISAID
FK	Falkland Islands	Falkland Islands (Islas Malvinas)	South America / Falkland Islands
GI	Gibraltar	Gibraltar	Europe / Gibraltar
JE	Jersey	Jersey	Europe / Jersey
IM	Isle of Man	Isle of Man	Europe / Isle of Man
GG	Guernsey	Guernsey	Europe / Guernsey

[1] https://www.ncbi.nlm.nih.gov/genbank/collab/country/

[rmcolq]

Just to check, the proposed adm1s would mean that instead of e.g. UK-ENG you might see FK (not UK-FK)?

[rambaut]

I thought it was GB-ENG?

[SamStudio8]

@rmcolq Yes; the proposed adm1 will be integrated as-is rather than prefixed (ie. using the proper ISO 3166-2 codes). The home nations will continue to use the existing adm1 (which are modified ISO 3166-2 using UK over GB).

[rambaut]

Complication is none of those new places are technically part of the UK (but then Northern Ireland is not part of Great Britain but still has GB-NIR under ISO 3166-2).

[SamStudio8]

@rambaut Indeed - there is a bunch of hard coding assuming United Kingdom dotted around a few things on this end of Elan. I'm not familiar with the NUTS (hah) and bolts of the geography cleaning but presumably this is a bit of work to integrate. There won't be any changes in Majora until we're confident that everything downstream will accept the new codes without undefined behaviour.

[rmcolq]

Verity tells me that geography_cleaning.py now handles these 5 additional adm1 (and those that were added before with postcodes from these regions and an adm1 of England, although I assume these will be overwritten in majora once the new ones are allowed?). I've a few small changes to datapipe and publishing steps which filter on 'is_uk' but these would not break the pipeline, only cause sequences to be included/excluded where we don't want.

[SamStudio8]

Awesome. I see the changes in COG-UK/geography_cleaning@54f5c76 tracked by COG-UK/geography_cleaning#2. @ViralVerity from the look of the patch this will work even if users don't change to the "correct" adm1 too? I don't plan to automatically update the existing data immediately so wouldn't want to break things downstream.

[rmcolq]

Yes it will work even if you don't update the existing ones. Datapipe/Phylopipe2 changes made (largely just the publishing step), just testing now.

[ViralVerity]

Yes - it should propagate up to country level! There's been quite a lot of sequences from the new adm1s so far that just had postcodes, so it will pick those up. I'm also happy to take advisement on what the cleaned up country looks like in terms of underscores vs spaces and capitalisation.

[SamStudio8]

@rmcolq @ViralVerity Thanks both! Looks like it's mostly with me to sort out the outbound pipes and Majora itself now... The main thing I still need to sort out is whether these new country codes need to come under the same treatment as the UK-ENG code in terms of specimen ID data sharing with PHE.

@ViralVerity I'll be using the countries as in the table above but whatever is more consistent with the cleaning you do already should be fine I think. We could set up a page on the docs site about geography cleaning if it would be helpful for people?

[ViralVerity]

Great ok, yeah I've got those as inputs!
I think adding the geography cleaning readme onto the docs site would be a good idea, I suspect some of the columns as not quite self explanatory

[SamStudio8]

Health Informatics group has asked for an update on this. It looks like we're ready downstream. Before we carry on I just want to chase up whether each of these adm1 are signed on to the agreement such that we can handle them as standard.

[ViralVerity]

I can confirm geography cleaning takes the countries exactly as above (ie no "UK" and in capitals) and will return prettier, human readable versions, in line with how the current adm1s are treated.

[SamStudio8]

@tomrconnor Do you know if all these new adm1 have signed up to COG now, don't want to inadvertently trigger an ethics meeting!

[tomrconnor]

They were in the process of it, I think. PHE are sequencing for some of these locations; think that the other sites won't be told they can have access to CLIMB until the HI group knows that this work has been done. So the downstream work being done is great, and I think the next thing is to pass this back to the HI group for them to manage the next actions.

[SamStudio8]

@tomrconnor Any update from HI?

[SamStudio8]

Bumping to backlog #62