Release from staging to master - 2021-03-17 #1371

ESapenaVentura · 2021-03-17T12:59:23Z

Release notes

Versions:

system/links.json - v3.0.0
module/protocol/matrix.json - v1.0.0
type/protocol/analysis/analysis_protocol.json - v9.2.0
type/file/analysis_file.json - v6.3.0
core/file/file_core.json - v6.2.0
type/file/sequence_file.json - v9.3.0
type/file/supplementary_file.json - v2.3.0
type/file/reference_file.json - v3.3.0
type/file/image_file.json - v2.3.0
module/ontology/data_use_ontology.json - v1.0.0
type/project/project.json - v15.0.0

Test metadata:
HumanCellAtlas/schema-test-data#8

Functionality:

Fixed javascript validator (Points to correct URL)
Added fail_test for links.json
Fixed schema linter (Points to correct URL)
Fixed pullapprove and updated lists
Added test data spreadsheet

Additional notes:

type/project/project.json has advanced 1 major version because of a rollback. The major change should have no impact as it involves the deletion of a field that was never used.

[HOTFIX] Pullapprove config changes

…schema into ms-fix-tests

Added pending status while travis test is happening

…schema into ms-fix-tests

fixed check_runs object in conditions

…schema into ms-fix-tests

Tried with success endpoint instead of successful

…schema into ms-fix-tests

Fix failing travis tests

Bumps [ini](https://github.com/isaacs/ini) from 1.3.5 to 1.3.8. - [Release notes](https://github.com/isaacs/ini/releases) - [Commits](npm/ini@v1.3.5...v1.3.8) Signed-off-by: dependabot[bot] <[email protected]>

…tests/ini-1.3.8 Bump ini from 1.3.5 to 1.3.8 in /tests

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.21. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.15...4.17.21) Signed-off-by: dependabot[bot] <[email protected]>

Bumps [bl](https://github.com/rvagg/bl) from 4.0.2 to 4.1.0. - [Release notes](https://github.com/rvagg/bl/releases) - [Commits](rvagg/bl@v4.0.2...v4.1.0) Signed-off-by: dependabot[bot] <[email protected]>

…ub.com/HumanCellAtlas/metadata-schema into dependabot/npm_and_yarn/tests/bl-4.1.0

…tests/bl-4.1.0 Bump bl from 4.0.2 to 4.1.0 in /tests

Improved description of file_source thanks to Hannes-ucsc. Co-authored-by: Hannes Schmidt <[email protected]>

mshadbolt · 2021-03-30T10:29:07Z

Just so the reviewers who have reviewed so far are aware (@hannes-ucsc @ruchim). We have taken your feedback on board and we are now preparing a new feature branch to merge into staging with your suggested fixes.

We are also removing all traces of the data_use_ontology_module and manually rolling back the project versioning and re-releasing the schemas to staging so that it was like the module, and the version of the project schema that referred to it, never existed.

We will also re-generate the test data. Once this is all completed we will re-request review of the release of the modified staging release into master.

Thanks for all your feedback.

json_schema/type/protocol/analysis/analysis_protocol.json

kbergin

Had a question about the analysis_protocol json changes. Outside of that question, everything else shouldn't impact analysis as far as I can tell.

hannes-ucsc

A few more comments.

json_schema/type/file/analysis_file.json

json_schema/type/protocol/analysis/analysis_protocol.json

hannes-ucsc · 2021-03-31T00:53:51Z

json_schema/core/file/file_core.json

+            "enum":[
+                "DCP",
+                "Contributor"
+            ],


Suggested change

"enum":[

"DCP",

"Contributor"

],

"enum":[

"DCP/2 Analysis",

"Contributor",

"Array Express",

"HCA Release",

"GEO",

"SCEA",

"SCP",

"DCP/1 Matrix Service",

"LungMAP",

"Zenodo",

"Publication",

],

to be consistent with the set of sources that we currently track (https://github.com/DataBiosphere/azul/blob/6b45e0e30fbe90c7b3944ee691c95328c7883ab2/src/azul/plugins/metadata/hca/transform.py#L271).

It would be undesirable to be less informative than the current "hacky" mechanism.

@hannes-ucsc it might be nice to also represent if a file source type is "internal" or "external". We could do this by making this an enum of dictionaries instead of strings, with an added property called "type". This would be of use in simplifying to users the distinction between DCP and CGM matrices.

@NoopDog I did some searching but I can't find anything about having an enum of objects in json schema. Or are you suggesting an additional field in the schema? If it is a 'nice to have' would it be possible to make a ticket on the repo with your proposal and we will prioritise it. Thanks!

@hannes-ucsc I have incorporated the enums as requested in the PR I am making with all feedback from the review process (#1381). We'll wait until test data has been reviewed and all expected changes are made in this branch before internal review, then squash and merge to minimise commits, then re-publish the schemas on staging.

NoopDog

Hi there, I would like to propose an alternative to making an optional field for matrix_cell_count on analysis_file.json.

Another way to do this is to make a matrix_file.json that includes analysis_file.json, basically extending it and making a new type, and then adding matrix_cell_count as a required field to matrix_file.json.

We already have image_file, reference_file, sequence_file and it seems to me that since matrix files are one of our primary outputs, that we would do well to model them explicitly.

This would provide a location to attach other matrix-related properties in the future and allow the schema to specify how to validate them, and also enable filtering by the file type.

* manually removed evidence of duo module and field * removed data_use_ontology module * manually edit project version * wording changes for matrix module * Modified descriptions based on feedback * slight wording change * add suggested enums * removed space from array express * edited examples, guidelines * Fixed array of enums inclusion. Co-authored-by: ESapenaVentura <[email protected]>

tests/schema_test_files/system/test_fail_links.json

json_schema/system/links.json

json_schema/module/protocol/matrix.json

json_schema/type/file/analysis_file.json

json_schema/module/protocol/matrix.json

json_schema/type/protocol/analysis/analysis_protocol.json

tests/schema_test_files/system/test_fail_links.json

json_schema/core/file/file_core.json

hannes-ucsc · 2021-04-29T14:05:41Z

json_schema/module/protocol/matrix.json

+            "description": "Method(s) used to normalize data in the matrix",
+            "type": "array",
+            "user_friendly": "Data normalization method(s)",
+            "items":{


Suggested change

"items":{

"items": {

Not a deal-breaker. @clairerye: "Will be fixed before merged". I will post formatting configuration for Amnon.

tests/schema_test_files/system/test_fail_links.json

hannes-ucsc · 2021-04-29T14:09:31Z

tests/schema_test_files/system/test_fail_links.json

+      ]
+    }
+  ]
+


Suggested change

See formatting fixes above.

ruchim

all good

Linted matrix schema.

Fixed tabulation issue

Added updated HCA template spreadsheet with the following schema changes Added EGA and dbGAP accession to Project Tab Added Analysis File and Analysis Protocol Tabs Added File_source to Analysis File tab Added Matrix_Cell_Count to Analysis File Tab Added Derivation_process and data_normalization_method to Analysis Protocol tab See schema change PR here: HumanCellAtlas/metadata-schema#1371

ESapenaVentura and others added 30 commits February 22, 2021 16:39

Changed protocol_type in links schema to enum. Fixes #1333

7897b4d

Changed protocol_type in links schema to enum. Fixes #1333

33108de

updated ontology address

104db86

added longer timeout

115254f

Fixed schema linter ontology URL

b5d6ffd

Triggering checks

2935e91

Test 2 with another empty commit

3fafc33

Merge pull request #1354 from HumanCellAtlas/esv-pullapprove-hotfix2

8e472a1

[HOTFIX] Pullapprove config changes

Merge branch 'develop' of https://github.com/HumanCellAtlas/metadata-…

5be9572

…schema into ms-fix-tests

Added a pending status in checks

2af655c

Rolled back change

e1231a2

Added pending status while travis test is happening

b3ccbb1

Merge pull request #1355 from HumanCellAtlas/esv-pullapprove-hotfix2

8b5c3ad

Added pending status while travis test is happening

Merge branch 'develop' of https://github.com/HumanCellAtlas/metadata-…

d4ffd5f

…schema into ms-fix-tests

fixed check_runs object in conditions

dabdb45

Merge pull request #1356 from HumanCellAtlas/esv-pullapprove-hotfix2

79ecdc6

fixed check_runs object in conditions

Merge branch 'develop' of https://github.com/HumanCellAtlas/metadata-…

253579a

…schema into ms-fix-tests

Tried with success endpoint instead of successful

922d19d

Merge pull request #1357 from HumanCellAtlas/esv-pullapprove-hotfix2

d27930f

Tried with success endpoint instead of successful

Merge branch 'develop' of https://github.com/HumanCellAtlas/metadata-…

f3f3f71

…schema into ms-fix-tests

update devs and wranglers

598d993

add Rachel to reviewer list

488b5d2

Merge pull request #1341 from HumanCellAtlas/ms-fix-tests

896822b

Fix failing travis tests

Bump ini from 1.3.5 to 1.3.8 in /tests

4dc2330

Bumps [ini](https://github.com/isaacs/ini) from 1.3.5 to 1.3.8. - [Release notes](https://github.com/isaacs/ini/releases) - [Commits](npm/ini@v1.3.5...v1.3.8) Signed-off-by: dependabot[bot] <[email protected]>

Merge pull request #1358 from HumanCellAtlas/dependabot/npm_and_yarn/…

c90fbae

…tests/ini-1.3.8 Bump ini from 1.3.5 to 1.3.8 in /tests

Bump lodash from 4.17.15 to 4.17.21 in /tests

db8ffd2

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.21. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.15...4.17.21) Signed-off-by: dependabot[bot] <[email protected]>

Bump bl from 4.0.2 to 4.1.0 in /tests

6ab3db0

Bumps [bl](https://github.com/rvagg/bl) from 4.0.2 to 4.1.0. - [Release notes](https://github.com/rvagg/bl/releases) - [Commits](rvagg/bl@v4.0.2...v4.1.0) Signed-off-by: dependabot[bot] <[email protected]>

Bump bl from 4.0.2 to 4.1.0 in /tests

fcbecbb

Bumps [bl](https://github.com/rvagg/bl) from 4.0.2 to 4.1.0. - [Release notes](https://github.com/rvagg/bl/releases) - [Commits](rvagg/bl@v4.0.2...v4.1.0) Signed-off-by: dependabot[bot] <[email protected]>

Merge branch 'dependabot/npm_and_yarn/tests/bl-4.1.0' of https://gith…

ee84e7c

…ub.com/HumanCellAtlas/metadata-schema into dependabot/npm_and_yarn/tests/bl-4.1.0

Merge pull request #1351 from HumanCellAtlas/dependabot/npm_and_yarn/…

c009a35

…tests/bl-4.1.0 Bump bl from 4.0.2 to 4.1.0 in /tests

Update json_schema/core/file/file_core.json

6443d3a

Improved description of file_source thanks to Hannes-ucsc. Co-authored-by: Hannes Schmidt <[email protected]>

kbergin reviewed Mar 30, 2021

View reviewed changes

json_schema/type/protocol/analysis/analysis_protocol.json Show resolved Hide resolved

kbergin reviewed Mar 30, 2021

View reviewed changes

hannes-ucsc requested changes Mar 31, 2021

View reviewed changes

mshadbolt mentioned this pull request Mar 31, 2021

Adjust schemas based on feedback, manually roll back project #1381

Merged

NoopDog reviewed Apr 5, 2021

View reviewed changes

melainalegaspi mentioned this pull request Apr 7, 2021

Index organically described CGMs DataBiosphere/azul#2958

Closed

hannes-ucsc requested changes Apr 13, 2021

View reviewed changes

dsotirho-ucsc reviewed Apr 22, 2021

View reviewed changes

json_schema/core/file/file_core.json Show resolved Hide resolved

Fixed regex in file_descriptor describedBy field.

5c09fed

ESapenaVentura force-pushed the staging branch from 017a8c9 to 5c09fed Compare April 28, 2021 15:37

hannes-ucsc requested changes Apr 29, 2021

View reviewed changes

kbergin approved these changes Apr 29, 2021

View reviewed changes

hannes-ucsc approved these changes Apr 29, 2021

View reviewed changes

ruchim approved these changes Apr 29, 2021

View reviewed changes

clairerye requested a review from aherbst-broad April 29, 2021 19:47

hannes-ucsc requested a review from NoopDog April 30, 2021 00:54

aherbst-broad approved these changes Apr 30, 2021

View reviewed changes

NoopDog approved these changes Apr 30, 2021

View reviewed changes

NoopDog mentioned this pull request Apr 30, 2021

Review Metadata PR 1371 DataBiosphere/data-browser#1719

Closed

ESapenaVentura and others added 5 commits May 5, 2021 09:27

Linted schema.

3642884

Linted schema

f789373

Merge pull request #1386 from HumanCellAtlas/esv-lint-matrix-schema

8fe3cf3

Linted matrix schema.

Fixed tabulation issue

1aa9e03

Merge pull request #1387 from HumanCellAtlas/esv-lint-matrix-schema

99c993d

Fixed tabulation issue

ESapenaVentura merged commit c1e477d into master May 5, 2021

hannes-ucsc mentioned this pull request May 21, 2021

Evaluate and respond to proposed metadata schema changes DataBiosphere/azul#2821

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release from staging to master - 2021-03-17 #1371

Release from staging to master - 2021-03-17 #1371

ESapenaVentura commented Mar 17, 2021 •

edited

Loading

mshadbolt commented Mar 30, 2021 •

edited

Loading

kbergin left a comment

hannes-ucsc left a comment

hannes-ucsc Mar 31, 2021 •

edited

Loading

NoopDog Apr 5, 2021

mshadbolt Apr 7, 2021 •

edited

Loading

mshadbolt Apr 7, 2021

NoopDog left a comment

hannes-ucsc Apr 29, 2021

hannes-ucsc Apr 29, 2021

hannes-ucsc Apr 29, 2021

hannes-ucsc Apr 29, 2021

ruchim left a comment

-            "enum":[
-                "DCP",
-                "Contributor"
-            ],
+            "enum":[
+                "DCP/2 Analysis",
+                "Contributor",
+                "Array Express",
+                "HCA Release",
+                "GEO",
+                "SCEA",
+                "SCP",
+                "DCP/1 Matrix Service",
+                "LungMAP",
+                "Zenodo",
+                "Publication",
+            ],

Release from staging to master - 2021-03-17 #1371

Release from staging to master - 2021-03-17 #1371

Conversation

ESapenaVentura commented Mar 17, 2021 • edited Loading

Release notes

mshadbolt commented Mar 30, 2021 • edited Loading

kbergin left a comment

Choose a reason for hiding this comment

hannes-ucsc left a comment

Choose a reason for hiding this comment

hannes-ucsc Mar 31, 2021 • edited Loading

Choose a reason for hiding this comment

NoopDog Apr 5, 2021

Choose a reason for hiding this comment

mshadbolt Apr 7, 2021 • edited Loading

Choose a reason for hiding this comment

mshadbolt Apr 7, 2021

Choose a reason for hiding this comment

NoopDog left a comment

Choose a reason for hiding this comment

hannes-ucsc Apr 29, 2021

Choose a reason for hiding this comment

hannes-ucsc Apr 29, 2021

Choose a reason for hiding this comment

hannes-ucsc Apr 29, 2021

Choose a reason for hiding this comment

hannes-ucsc Apr 29, 2021

Choose a reason for hiding this comment

ruchim left a comment

Choose a reason for hiding this comment

ESapenaVentura commented Mar 17, 2021 •

edited

Loading

mshadbolt commented Mar 30, 2021 •

edited

Loading

hannes-ucsc Mar 31, 2021 •

edited

Loading

mshadbolt Apr 7, 2021 •

edited

Loading