Scoring datasets: Measure of the quality of data submitted

[CS76]

In nmrXiv, a dataset comprises spectra files (raw instrument data or processed files with measurements), chemical structure identifications and associated metadata (sample information, assay parameters - instrument provided data and reported by the chemist/researcher). We want to devise a scoring algorithm to qualify the datasets in nmrXiv and give our users a score representing the data/reporting quality/submitter interaction etc.

We would like to hear out your ideas/comments on what to consider, how best we can represent this etc.
@steinbeck @jliermann @sneumann @febach @stefhk3

[jliermann]

Talking of assigned datasets the following things should be considered:

• Is a structure attached? Does the dataset contain assignment information?
• Is the assignment plausible compared to an automatic shift prediction?
• Is the assignment plausible with respect to 2D correlations?

The latter point would be most interesting as I don't know any product that has actually tried to tackle this. Ideally, there would be an automatic 2D peak picking and an algorithm would rate the assignment based on all existing correlations as a whole. That means that the compound would get two ratings: a quality rating based on the agreement of structure and correlation, and a reliability rating based on the number of correlations available compared to the complexity of the molecule.

[stefhk3]

We have done that in nmrshiftdb2. I upload a sample report here. It does 1D and 2D spectra. We can discuss this in more detail if useful.
10016310.pdf

[stefhk3]

Also, I would like to see something like the QuickCheck in nmrshiftdb2. Basically an option to get the score without submitting data to the database.

[febach]

I like the idea of transparent scores a lot. However, be careful to label it with a data quality score. A dataset may be surprising and therefore labeled bad quality by automatic shift prediction etc. but still be valid (ok, not very often). So my suggestion is to make the labels very transparent, e.g. "passed 2D shift prediction autotest" or so, not a number representing data quality. Then a dataset can get many labels, for example also other users could check it...