development
This section contains various technical information on how to develop and run the code.
- 2023-10-09
- 2023-10-09
- Contributors
-
This section contains various technical information on how to develop and run the code.
This section contains various technical information on how to develop and run the code.
This section contains various technical information on how to develop and run the code.
Airflow code is located in src/airflow. Make sure to execute all of the instructions from that directory, unless stated otherwise.
We will be running a local Airflow setup using Docker Compose. First, make sure it is installed (this and subsequent commands are tested on Ubuntu):
sudo apt install docker-compose
+ Running Airflow workflows - Open Targets Genetics Running Airflow workflows¶
Airflow code is located in src/airflow. Make sure to execute all of the instructions from that directory, unless stated otherwise.
Set up Docker¶
We will be running a local Airflow setup using Docker Compose. First, make sure it is installed (this and subsequent commands are tested on Ubuntu):
sudo apt install docker-compose
Next, verify that you can run Docker. This should say "Hello from Docker":
docker run hello-world
If the command above raises a permission error, fix it and reboot:
sudo usermod -a -G docker $USER
newgrp docker
diff --git a/development/contributing/index.html b/development/contributing/index.html
index 2f29aeae3..c92987e33 100644
--- a/development/contributing/index.html
+++ b/development/contributing/index.html
@@ -1 +1 @@
- Contributing guidelines - Open Targets Genetics Contributing guidelines¶
One-time configuration¶
The steps in this section only ever need to be done once on any particular system.
Google Cloud configuration: 1. Install Google Cloud SDK: https://cloud.google.com/sdk/docs/install. 1. Log in to your work Google Account: run gcloud auth login and follow instructions. 1. Obtain Google application credentials: run gcloud auth application-default login and follow instructions.
Check that you have the make utility installed, and if not (which is unlikely), install it using your system package manager.
Check that you have java installed.
Environment configuration¶
Run make setup-dev to install/update the necessary packages and activate the development environment. You need to do this every time you open a new shell.
It is recommended to use VS Code as an IDE for development.
How to run the code¶
All pipelines in this repository are intended to be run in Google Dataproc. Running them locally is not currently supported.
In order to run the code:
-
Manually edit your local workflow/dag.yaml file and comment out the steps you do not want to run.
-
Manually edit your local pyproject.toml file and modify the version of the code.
- This must be different from the version used by any other people working on the repository to avoid any deployment conflicts, so it's a good idea to use your name, for example:
1.2.3+jdoe. - You can also add a brief branch description, for example:
1.2.3+jdoe.myfeature. - Note that the version must comply with PEP440 conventions, otherwise Poetry will not allow it to be deployed.
- Do not use underscores or hyphens in your version name. When building the WHL file, they will be automatically converted to dots, which means the file name will no longer match the version and the build will fail. Use dots instead.
-
Run make build.
- This will create a bundle containing the neccessary code, configuration and dependencies to run the ETL pipeline, and then upload this bundle to Google Cloud.
- A version specific subpath is used, so uploading the code will not affect any branches but your own.
- If there was already a code bundle uploaded with the same version number, it will be replaced.
-
Submit the Dataproc job with poetry run python workflow/workflow_template.py
- You will need to specify additional parameters, some are mandatory and some are optional. Run with
--help to see usage. - The script will provision the cluster and submit the job.
- The cluster will take a few minutes to get provisioned and running, during which the script will not output anything, this is normal.
- Once submitted, you can monitor the progress of your job on this page: https://console.cloud.google.com/dataproc/jobs?project=open-targets-genetics-dev.
- On completion (whether successful or a failure), the cluster will be automatically removed, so you don't have to worry about shutting it down to avoid incurring charges.
Contributing checklist¶
When making changes, and especially when implementing a new module or feature, it's essential to ensure that all relevant sections of the code base are modified. - [ ] Run make check. This will run the linter and formatter to ensure that the code is compliant with the project conventions. - [ ] Develop unit tests for your code and run make test. This will run all unit tests in the repository, including the examples appended in the docstrings of some methods. - [ ] Update the configuration if necessary. - [ ] Update the documentation and check it with make build-documentation. This will start a local server to browse it (URL will be printed, usually http://127.0.0.1:8000/)
For more details on each of these steps, see the sections below.
Documentation¶
- If during development you had a question which wasn't covered in the documentation, and someone explained it to you, add it to the documentation. The same applies if you encountered any instructions in the documentation which were obsolete or incorrect.
- Documentation autogeneration expressions start with
:::. They will automatically generate sections of the documentation based on class and method docstrings. Be sure to update them for: - Dataset definitions in
docs/reference/dataset (example: docs/reference/dataset/study_index/study_index_finngen.md) - Step definition in
docs/reference/step (example: docs/reference/step/finngen.md)
Configuration¶
- Input and output paths in
config/datasets/gcp.yaml - Step configuration in
config/step/STEP.yaml (example: config/step/finngen.yaml)
Classes¶
- Dataset class in
src/org/dataset/ (example: src/otg/dataset/study_index.py → StudyIndexFinnGen) - Step main running class in
src/org/STEP.py (example: src/org/finngen.py)
Tests¶
- Test study fixture in
tests/conftest.py (example: mock_study_index_finngen in that module) - Test sample data in
tests/data_samples (example: tests/data_samples/finngen_studies_sample.json) - Test definition in
tests/ (example: tests/dataset/test_study_index.py → test_study_index_finngen_creation)
- 2023-05-30
- 2023-10-26
- Contributors
-
\ No newline at end of file
+ Contributing guidelines - Open Targets Genetics Contributing guidelines¶
One-time configuration¶
The steps in this section only ever need to be done once on any particular system.
Google Cloud configuration: 1. Install Google Cloud SDK: https://cloud.google.com/sdk/docs/install. 1. Log in to your work Google Account: run gcloud auth login and follow instructions. 1. Obtain Google application credentials: run gcloud auth application-default login and follow instructions.
Check that you have the make utility installed, and if not (which is unlikely), install it using your system package manager.
Check that you have java installed.
Environment configuration¶
Run make setup-dev to install/update the necessary packages and activate the development environment. You need to do this every time you open a new shell.
It is recommended to use VS Code as an IDE for development.
How to run the code¶
All pipelines in this repository are intended to be run in Google Dataproc. Running them locally is not currently supported.
In order to run the code:
-
Manually edit your local workflow/dag.yaml file and comment out the steps you do not want to run.
-
Manually edit your local pyproject.toml file and modify the version of the code.
- This must be different from the version used by any other people working on the repository to avoid any deployment conflicts, so it's a good idea to use your name, for example:
1.2.3+jdoe. - You can also add a brief branch description, for example:
1.2.3+jdoe.myfeature. - Note that the version must comply with PEP440 conventions, otherwise Poetry will not allow it to be deployed.
- Do not use underscores or hyphens in your version name. When building the WHL file, they will be automatically converted to dots, which means the file name will no longer match the version and the build will fail. Use dots instead.
-
Run make build.
- This will create a bundle containing the neccessary code, configuration and dependencies to run the ETL pipeline, and then upload this bundle to Google Cloud.
- A version specific subpath is used, so uploading the code will not affect any branches but your own.
- If there was already a code bundle uploaded with the same version number, it will be replaced.
-
Submit the Dataproc job with poetry run python workflow/workflow_template.py
- You will need to specify additional parameters, some are mandatory and some are optional. Run with
--help to see usage. - The script will provision the cluster and submit the job.
- The cluster will take a few minutes to get provisioned and running, during which the script will not output anything, this is normal.
- Once submitted, you can monitor the progress of your job on this page: https://console.cloud.google.com/dataproc/jobs?project=open-targets-genetics-dev.
- On completion (whether successful or a failure), the cluster will be automatically removed, so you don't have to worry about shutting it down to avoid incurring charges.
Contributing checklist¶
When making changes, and especially when implementing a new module or feature, it's essential to ensure that all relevant sections of the code base are modified. - [ ] Run make check. This will run the linter and formatter to ensure that the code is compliant with the project conventions. - [ ] Develop unit tests for your code and run make test. This will run all unit tests in the repository, including the examples appended in the docstrings of some methods. - [ ] Update the configuration if necessary. - [ ] Update the documentation and check it with make build-documentation. This will start a local server to browse it (URL will be printed, usually http://127.0.0.1:8000/)
For more details on each of these steps, see the sections below.
Documentation¶
- If during development you had a question which wasn't covered in the documentation, and someone explained it to you, add it to the documentation. The same applies if you encountered any instructions in the documentation which were obsolete or incorrect.
- Documentation autogeneration expressions start with
:::. They will automatically generate sections of the documentation based on class and method docstrings. Be sure to update them for: - Dataset definitions in
docs/reference/dataset (example: docs/reference/dataset/study_index/study_index_finngen.md) - Step definition in
docs/reference/step (example: docs/reference/step/finngen.md)
Configuration¶
- Input and output paths in
config/datasets/gcp.yaml - Step configuration in
config/step/STEP.yaml (example: config/step/finngen.yaml)
Classes¶
- Dataset class in
src/org/dataset/ (example: src/otg/dataset/study_index.py → StudyIndexFinnGen) - Step main running class in
src/org/STEP.py (example: src/org/finngen.py)
Tests¶
- Test study fixture in
tests/conftest.py (example: mock_study_index_finngen in that module) - Test sample data in
tests/data_samples (example: tests/data_samples/finngen_studies_sample.json) - Test definition in
tests/ (example: tests/dataset/test_study_index.py → test_study_index_finngen_creation)
- 2023-05-30
- 2023-10-26
- Contributors
-
\ No newline at end of file
diff --git a/development/troubleshooting/index.html b/development/troubleshooting/index.html
index d685c9f7f..dc96298ea 100644
--- a/development/troubleshooting/index.html
+++ b/development/troubleshooting/index.html
@@ -1 +1 @@
- Troubleshooting - Open Targets Genetics Troubleshooting¶
BLAS/LAPACK¶
If you see errors related to BLAS/LAPACK libraries, see this StackOverflow post for guidance.
Pyenv and Poetry¶
If you see various errors thrown by Pyenv or Poetry, they can be hard to specifically diagnose and resolve. In this case, it often helps to remove those tools from the system completely. Follow these steps:
- Close your currently activated environment, if any:
exit - Uninstall Poetry:
curl -sSL https://install.python-poetry.org | python3 - --uninstall - Clear Poetry cache:
rm -rf ~/.cache/pypoetry - Clear pre-commit cache:
rm -rf ~/.cache/pre-commit - Switch to system Python shell:
pyenv shell system - Edit
~/.bashrc to remove the lines related to Pyenv configuration - Remove Pyenv configuration and cache:
rm -rf ~/.pyenv
After that, open a fresh shell session and run make setup-dev again.
Java¶
Officially, PySpark requires Java version 8 (a.k.a. 1.8) or above to work. However, if you have a very recent version of Java, you may experience issues, as it may introduce breaking changes that PySpark hasn't had time to integrate. For example, as of May 2023, PySpark did not work with Java 20.
If you are encountering problems with initialising a Spark session, try using Java 11.
Pre-commit¶
If you see an error message thrown by pre-commit, which looks like this (SyntaxError: Unexpected token '?'), followed by a JavaScript traceback, the issue is likely with your system NodeJS version.
One solution which can help in this case is to upgrade your system NodeJS version. However, this may not always be possible. For example, Ubuntu repository is several major versions behind the latest version as of July 2023.
Another solution which helps is to remove Node, NodeJS, and npm from your system entirely. In this case, pre-commit will not try to rely on a system version of NodeJS and will install its own, suitable one.
On Ubuntu, this can be done using sudo apt remove node nodejs npm, followed by sudo apt autoremove. But in some cases, depending on your existing installation, you may need to also manually remove some files. See this StackOverflow answer for guidance.
After running these commands, you are advised to open a fresh shell, and then also reinstall Pyenv and Poetry to make sure they pick up the changes (see relevant section above).
- 2023-07-04
- 2023-10-17
- Contributors
-
\ No newline at end of file
+ Troubleshooting - Open Targets Genetics Troubleshooting¶
BLAS/LAPACK¶
If you see errors related to BLAS/LAPACK libraries, see this StackOverflow post for guidance.
Pyenv and Poetry¶
If you see various errors thrown by Pyenv or Poetry, they can be hard to specifically diagnose and resolve. In this case, it often helps to remove those tools from the system completely. Follow these steps:
- Close your currently activated environment, if any:
exit - Uninstall Poetry:
curl -sSL https://install.python-poetry.org | python3 - --uninstall - Clear Poetry cache:
rm -rf ~/.cache/pypoetry - Clear pre-commit cache:
rm -rf ~/.cache/pre-commit - Switch to system Python shell:
pyenv shell system - Edit
~/.bashrc to remove the lines related to Pyenv configuration - Remove Pyenv configuration and cache:
rm -rf ~/.pyenv
After that, open a fresh shell session and run make setup-dev again.
Java¶
Officially, PySpark requires Java version 8 (a.k.a. 1.8) or above to work. However, if you have a very recent version of Java, you may experience issues, as it may introduce breaking changes that PySpark hasn't had time to integrate. For example, as of May 2023, PySpark did not work with Java 20.
If you are encountering problems with initialising a Spark session, try using Java 11.
Pre-commit¶
If you see an error message thrown by pre-commit, which looks like this (SyntaxError: Unexpected token '?'), followed by a JavaScript traceback, the issue is likely with your system NodeJS version.
One solution which can help in this case is to upgrade your system NodeJS version. However, this may not always be possible. For example, Ubuntu repository is several major versions behind the latest version as of July 2023.
Another solution which helps is to remove Node, NodeJS, and npm from your system entirely. In this case, pre-commit will not try to rely on a system version of NodeJS and will install its own, suitable one.
On Ubuntu, this can be done using sudo apt remove node nodejs npm, followed by sudo apt autoremove. But in some cases, depending on your existing installation, you may need to also manually remove some files. See this StackOverflow answer for guidance.
After running these commands, you are advised to open a fresh shell, and then also reinstall Pyenv and Poetry to make sure they pick up the changes (see relevant section above).
- 2023-07-04
- 2023-10-17
- Contributors
-
\ No newline at end of file
diff --git a/index.html b/index.html
index 1e329946d..2110f6f99 100644
--- a/index.html
+++ b/index.html
@@ -1,4 +1,4 @@
- Open Targets Genetics - Open Targets Genetics Open Targets Genetics
Open Targets Genetics
Installation¶
TBC
- 2022-10-01
- 2023-10-16
- Contributors
-
\ No newline at end of file
+ Installation - Open Targets Genetics Installation¶
TBC
- 2022-10-01
- 2023-10-31
- Contributors
-
\ No newline at end of file
diff --git a/python_api/_python_api/index.html b/python_api/_python_api/index.html
index 74651bc3c..5375ccc98 100644
--- a/python_api/_python_api/index.html
+++ b/python_api/_python_api/index.html
@@ -1 +1 @@
- Python API - Open Targets Genetics Python API
- 2023-01-15
- 2023-10-16
- Contributors
-
\ No newline at end of file
+ Python API - Open Targets Genetics Python API
- 2023-01-15
- 2023-10-16
- Contributors
-
\ No newline at end of file
diff --git a/python_api/dataset/_dataset/index.html b/python_api/dataset/_dataset/index.html
index 895e1fe55..e0a2997da 100644
--- a/python_api/dataset/_dataset/index.html
+++ b/python_api/dataset/_dataset/index.html
@@ -1,4 +1,4 @@
- Dataset - Open Targets Genetics Dataset¶
Bases: ABC
Open Targets Genetics Dataset.
Dataset is a wrapper around a Spark DataFrame with a predefined schema. Schemas for each child dataset are described in the json.schemas module.
Source code in src/otg/dataset/dataset.py
17
+ Dataset - Open Targets Genetics Dataset
otg.dataset.dataset.Dataset dataclass ¶
Bases: ABC
Open Targets Genetics Dataset.
Dataset is a wrapper around a Spark DataFrame with a predefined schema. Schemas for each child dataset are described in the json.schemas module.
Source code in src/otg/dataset/dataset.py
17
18
19
20
@@ -285,7 +285,7 @@
"""
self.df = self._df.unpersist()
return self
-
df: DataFrame property writable ¶
Dataframe included in the Dataset.
Returns:
Name Type Description DataFrame DataFrame Dataframe included in the Dataset
schema: StructType property ¶
Dataframe expected schema.
Returns:
Name Type Description StructType StructType Dataframe expected schema
from_parquet(session: Session, path: str, **kwargs: dict[str, Any]) -> Dataset classmethod ¶
Reads a parquet file into a Dataset with a given schema.
Parameters:
Name Type Description Default session Session Spark session
required path str Path to the parquet file
required **kwargs dict[str, Any] Additional arguments to pass to spark.read.parquet
{}
Returns:
Name Type Description Dataset Dataset Dataset with the parquet file contents
Source code in src/otg/dataset/dataset.py
69
+
df: DataFrame property writable ¶
Dataframe included in the Dataset.
Returns:
Name Type Description DataFrame DataFrame Dataframe included in the Dataset
schema: StructType property ¶
Dataframe expected schema.
Returns:
Name Type Description StructType StructType Dataframe expected schema
from_parquet(session: Session, path: str, **kwargs: dict[str, Any]) -> Dataset classmethod ¶
Reads a parquet file into a Dataset with a given schema.
Parameters:
Name Type Description Default session Session Spark session
required path str Path to the parquet file
required **kwargs dict[str, Any] Additional arguments to pass to spark.read.parquet
{}
Returns:
Name Type Description Dataset Dataset Dataset with the parquet file contents
Source code in src/otg/dataset/dataset.py
69
70
71
72
@@ -318,7 +318,7 @@
schema = cls.get_schema()
df = session.read_parquet(path=path, schema=schema, **kwargs)
return cls(_df=df, _schema=schema)
-
get_schema() -> StructType abstractmethod classmethod ¶
Abstract method to get the schema. Must be implemented by child classes.
Returns:
Name Type Description StructType StructType Schema for the Dataset
Source code in src/otg/dataset/dataset.py
59
+
get_schema() -> StructType abstractmethod classmethod ¶
Abstract method to get the schema. Must be implemented by child classes.
Returns:
Name Type Description StructType StructType Schema for the Dataset
Source code in src/otg/dataset/dataset.py
59
60
61
62
@@ -335,7 +335,7 @@
StructType: Schema for the Dataset
"""
pass
-
persist() -> Dataset ¶
Persist in memory the DataFrame included in the Dataset.
Returns:
Name Type Description Dataset Dataset Persisted Dataset
Source code in src/otg/dataset/dataset.py
144
+
persist() -> Dataset ¶
Persist in memory the DataFrame included in the Dataset.
Returns:
Name Type Description Dataset Dataset Persisted Dataset
Source code in src/otg/dataset/dataset.py
144
145
146
147
@@ -350,7 +350,7 @@
"""
self.df = self._df.persist()
return self
-
unpersist() -> Dataset ¶
Remove the persisted DataFrame from memory.
Returns:
Name Type Description Dataset Dataset Unpersisted Dataset
Source code in src/otg/dataset/dataset.py
153
+
unpersist() -> Dataset ¶
Remove the persisted DataFrame from memory.
Returns:
Name Type Description Dataset Dataset Unpersisted Dataset
Source code in src/otg/dataset/dataset.py
153
154
155
156
@@ -365,7 +365,7 @@
"""
self.df = self._df.unpersist()
return self
-
validate_schema() -> None ¶
Validate DataFrame schema against expected class schema.
Raises:
Type Description ValueError DataFrame schema is not valid
Source code in src/otg/dataset/dataset.py
87
+
validate_schema() -> None ¶
Validate DataFrame schema against expected class schema.
Raises:
Type Description ValueError DataFrame schema is not valid
Source code in src/otg/dataset/dataset.py
87
88
89
90
@@ -476,4 +476,4 @@
raise ValueError(
f"The following fields present differences in their datatypes: {fields_with_different_observed_datatype}."
)
-
- 2023-01-15
- 2023-10-16
- Contributors
-
\ No newline at end of file
+
- 2023-01-15
- 2023-10-31
- Contributors
-
\ No newline at end of file
diff --git a/python_api/dataset/colocalisation/index.html b/python_api/dataset/colocalisation/index.html
index 571a39c8f..b88d04c05 100644
--- a/python_api/dataset/colocalisation/index.html
+++ b/python_api/dataset/colocalisation/index.html
@@ -1,17 +1,4 @@
- Colocalisation - Open Targets Genetics Colocalisation¶
Schema¶
root
- |-- leftStudyLocusId: long (nullable = false)
- |-- rightStudyLocusId: long (nullable = false)
- |-- chromosome: string (nullable = false)
- |-- colocalisationMethod: string (nullable = false)
- |-- numberColocalisingVariants: long (nullable = false)
- |-- h0: double (nullable = true)
- |-- h1: double (nullable = true)
- |-- h2: double (nullable = true)
- |-- h3: double (nullable = true)
- |-- h4: double (nullable = true)
- |-- log2h4h3: double (nullable = true)
- |-- clpp: double (nullable = true)
-
Class¶
Bases: Dataset
Colocalisation results for pairs of overlapping study-locus.
Source code in src/otg/dataset/colocalisation.py
14
+ Colocalisation - Open Targets Genetics Colocalisation
otg.dataset.colocalisation.Colocalisation dataclass ¶
Bases: Dataset
Colocalisation results for pairs of overlapping study-locus.
Source code in src/otg/dataset/colocalisation.py
14
15
16
17
@@ -34,7 +21,7 @@
StructType: Schema for the Colocalisation dataset
"""
return parse_spark_schema("colocalisation.json")
-
get_schema() -> StructType classmethod ¶
Provides the schema for the Colocalisation dataset.
Returns:
Name Type Description StructType StructType Schema for the Colocalisation dataset
Source code in src/otg/dataset/colocalisation.py
18
+
get_schema() -> StructType classmethod ¶
Provides the schema for the Colocalisation dataset.
Returns:
Name Type Description StructType StructType Schema for the Colocalisation dataset
Source code in src/otg/dataset/colocalisation.py
18
19
20
21
@@ -49,4 +36,17 @@
StructType: Schema for the Colocalisation dataset
"""
return parse_spark_schema("colocalisation.json")
-
- 2023-01-20
- 2023-10-16
- Contributors
-
\ No newline at end of file
+
Schema¶
root
+ |-- leftStudyLocusId: long (nullable = false)
+ |-- rightStudyLocusId: long (nullable = false)
+ |-- chromosome: string (nullable = false)
+ |-- colocalisationMethod: string (nullable = false)
+ |-- numberColocalisingVariants: long (nullable = false)
+ |-- h0: double (nullable = true)
+ |-- h1: double (nullable = true)
+ |-- h2: double (nullable = true)
+ |-- h3: double (nullable = true)
+ |-- h4: double (nullable = true)
+ |-- log2h4h3: double (nullable = true)
+ |-- clpp: double (nullable = true)
+
- 2023-01-20
- 2023-10-31
- Contributors
-
\ No newline at end of file
diff --git a/python_api/dataset/gene_index/index.html b/python_api/dataset/gene_index/index.html
index cbaa4dc89..ae9657802 100644
--- a/python_api/dataset/gene_index/index.html
+++ b/python_api/dataset/gene_index/index.html
@@ -1,16 +1,4 @@
- GeneIndex - Open Targets Genetics GeneIndex¶
Schema¶
root
- |-- geneId: string (nullable = false)
- |-- chromosome: string (nullable = false)
- |-- approvedSymbol: string (nullable = true)
- |-- biotype: string (nullable = true)
- |-- approvedName: string (nullable = true)
- |-- obsoleteSymbols: array (nullable = true)
- | |-- element: string (containsNull = true)
- |-- tss: long (nullable = true)
- |-- start: long (nullable = true)
- |-- end: long (nullable = true)
- |-- strand: integer (nullable = true)
-
Class¶
Bases: Dataset
Gene index dataset.
Gene-based annotation.
Source code in src/otg/dataset/gene_index.py
17
+ Gene Index - Open Targets Genetics Gene Index
otg.dataset.gene_index.GeneIndex dataclass ¶
Bases: Dataset
Gene index dataset.
Gene-based annotation.
Source code in src/otg/dataset/gene_index.py
17
18
19
20
@@ -125,7 +113,7 @@
).alias("geneSymbol"),
"*",
)
-
filter_by_biotypes(biotypes: list) -> GeneIndex ¶
Filter by approved biotypes.
Parameters:
Name Type Description Default biotypes list List of Ensembl biotypes to keep.
required
Returns:
Name Type Description GeneIndex GeneIndex Gene index dataset filtered by biotypes.
Source code in src/otg/dataset/gene_index.py
33
+
filter_by_biotypes(biotypes: list) -> GeneIndex ¶
Filter by approved biotypes.
Parameters:
Name Type Description Default biotypes list List of Ensembl biotypes to keep.
required
Returns:
Name Type Description GeneIndex GeneIndex Gene index dataset filtered by biotypes.
Source code in src/otg/dataset/gene_index.py
33
34
35
36
@@ -146,7 +134,7 @@
"""
self.df = self._df.filter(f.col("biotype").isin(biotypes))
return self
-
get_schema() -> StructType classmethod ¶
Provides the schema for the GeneIndex dataset.
Returns:
Name Type Description StructType StructType Schema for the GeneIndex dataset
Source code in src/otg/dataset/gene_index.py
24
+
get_schema() -> StructType classmethod ¶
Provides the schema for the GeneIndex dataset.
Returns:
Name Type Description StructType StructType Schema for the GeneIndex dataset
Source code in src/otg/dataset/gene_index.py
24
25
26
27
@@ -161,7 +149,7 @@
StructType: Schema for the GeneIndex dataset
"""
return parse_spark_schema("gene_index.json")
-
locations_lut() -> DataFrame ¶
Gene location information.
Returns:
Name Type Description DataFrame DataFrame Gene LUT including genomic location information.
Source code in src/otg/dataset/gene_index.py
45
+
locations_lut() -> DataFrame ¶
Gene location information.
Returns:
Name Type Description DataFrame DataFrame Gene LUT including genomic location information.
Source code in src/otg/dataset/gene_index.py
45
46
47
48
@@ -188,7 +176,7 @@
"strand",
"tss",
)
-
symbols_lut() -> DataFrame ¶
Gene symbol lookup table.
Pre-processess gene/target dataset to create lookup table of gene symbols, including obsoleted gene symbols.
Returns:
Name Type Description DataFrame DataFrame Gene LUT for symbol mapping containing geneId and geneSymbol columns.
Source code in src/otg/dataset/gene_index.py
60
+
symbols_lut() -> DataFrame ¶
Gene symbol lookup table.
Pre-processess gene/target dataset to create lookup table of gene symbols, including obsoleted gene symbols.
Returns:
Name Type Description DataFrame DataFrame Gene LUT for symbol mapping containing geneId and geneSymbol columns.
Source code in src/otg/dataset/gene_index.py
60
61
62
63
@@ -217,4 +205,16 @@
).alias("geneSymbol"),
"*",
)
-
- 2023-01-15
- 2023-10-20
- Contributors
-
\ No newline at end of file
+
Schema¶
root
+ |-- geneId: string (nullable = false)
+ |-- chromosome: string (nullable = false)
+ |-- approvedSymbol: string (nullable = true)
+ |-- biotype: string (nullable = true)
+ |-- approvedName: string (nullable = true)
+ |-- obsoleteSymbols: array (nullable = true)
+ | |-- element: string (containsNull = true)
+ |-- tss: long (nullable = true)
+ |-- start: long (nullable = true)
+ |-- end: long (nullable = true)
+ |-- strand: integer (nullable = true)
+
- 2023-01-15
- 2023-10-31
- Contributors
-
\ No newline at end of file
diff --git a/python_api/dataset/intervals/index.html b/python_api/dataset/intervals/index.html
index 1febf5321..9284ee362 100644
--- a/python_api/dataset/intervals/index.html
+++ b/python_api/dataset/intervals/index.html
@@ -1,15 +1,4 @@
- Intervals - Open Targets Genetics Intervals¶
Schema¶
root
- |-- chromosome: string (nullable = false)
- |-- start: string (nullable = false)
- |-- end: string (nullable = false)
- |-- geneId: string (nullable = false)
- |-- resourceScore: double (nullable = true)
- |-- score: double (nullable = true)
- |-- datasourceId: string (nullable = false)
- |-- datatypeId: string (nullable = false)
- |-- pmid: string (nullable = true)
- |-- biofeature: string (nullable = true)
-
Class¶
Bases: Dataset
Intervals dataset links genes to genomic regions based on genome interaction studies.
Source code in src/otg/dataset/intervals.py
22
+ Intervals - Open Targets Genetics Intervals
otg.dataset.intervals.Intervals dataclass ¶
Bases: Dataset
Intervals dataset links genes to genomic regions based on genome interaction studies.
Source code in src/otg/dataset/intervals.py
22
23
24
25
@@ -176,7 +165,7 @@
),
_schema=V2G.get_schema(),
)
-
from_source(spark: SparkSession, source_name: str, source_path: str, gene_index: GeneIndex, lift: LiftOverSpark) -> Intervals classmethod ¶
Collect interval data for a particular source.
Parameters:
Name Type Description Default spark SparkSession Spark session
required source_name str Name of the interval source
required source_path str Path to the interval source file
required gene_index GeneIndex Gene index
required lift LiftOverSpark LiftOverSpark instance to convert coordinats from hg37 to hg38
required
Returns:
Name Type Description Intervals Intervals Intervals dataset
Raises:
Type Description ValueError If the source name is not recognised
Source code in src/otg/dataset/intervals.py
35
+
from_source(spark: SparkSession, source_name: str, source_path: str, gene_index: GeneIndex, lift: LiftOverSpark) -> Intervals classmethod ¶
Collect interval data for a particular source.
Parameters:
Name Type Description Default spark SparkSession Spark session
required source_name str Name of the interval source
required source_path str Path to the interval source file
required gene_index GeneIndex Gene index
required lift LiftOverSpark LiftOverSpark instance to convert coordinats from hg37 to hg38
required
Returns:
Name Type Description Intervals Intervals Intervals dataset
Raises:
Type Description ValueError If the source name is not recognised
Source code in src/otg/dataset/intervals.py
35
36
37
38
@@ -259,7 +248,7 @@
source_class = source_to_class[source_name]
data = source_class.read(spark, source_path)
return source_class.parse(data, gene_index, lift)
-
get_schema() -> StructType classmethod ¶
Provides the schema for the Intervals dataset.
Returns:
Name Type Description StructType StructType Schema for the Intervals dataset
Source code in src/otg/dataset/intervals.py
26
+
get_schema() -> StructType classmethod ¶
Provides the schema for the Intervals dataset.
Returns:
Name Type Description StructType StructType Schema for the Intervals dataset
Source code in src/otg/dataset/intervals.py
26
27
28
29
@@ -274,7 +263,7 @@
StructType: Schema for the Intervals dataset
"""
return parse_spark_schema("intervals.json")
-
v2g(variant_index: VariantIndex) -> V2G ¶
Convert intervals into V2G by intersecting with a variant index.
Parameters:
Name Type Description Default variant_index VariantIndex Variant index dataset
required
Returns:
Name Type Description V2G V2G Variant-to-gene evidence dataset
Source code in src/otg/dataset/intervals.py
78
+
v2g(variant_index: VariantIndex) -> V2G ¶
Convert intervals into V2G by intersecting with a variant index.
Parameters:
Name Type Description Default variant_index VariantIndex Variant index dataset
required
Returns:
Name Type Description V2G V2G Variant-to-gene evidence dataset
Source code in src/otg/dataset/intervals.py
78
79
80
81
@@ -329,4 +318,15 @@
),
_schema=V2G.get_schema(),
)
-
- 2023-01-15
- 2023-10-16
- Contributors
-
\ No newline at end of file
+
Schema¶
root
+ |-- chromosome: string (nullable = false)
+ |-- start: string (nullable = false)
+ |-- end: string (nullable = false)
+ |-- geneId: string (nullable = false)
+ |-- resourceScore: double (nullable = true)
+ |-- score: double (nullable = true)
+ |-- datasourceId: string (nullable = false)
+ |-- datatypeId: string (nullable = false)
+ |-- pmid: string (nullable = true)
+ |-- biofeature: string (nullable = true)
+
- 2023-01-15
- 2023-10-31
- Contributors
-
root
- |-- variantId: string (nullable = false)
- |-- chromosome: string (nullable = false)
- |-- ldSet: array (nullable = false)
- | |-- element: struct (containsNull = false)
- | | |-- tagVariantId: string (nullable = false)
- | | |-- rValues: array (nullable = false)
- | | | |-- element: struct (containsNull = false)
- | | | | |-- population: string (nullable = false)
- | | | | |-- r: double (nullable = false)
- Bases: Dataset
Dataset containing linkage desequilibrium information between variants.
src/otg/dataset/ld_index.py14 + |
root
+ |-- variantId: string (nullable = false)
+ |-- chromosome: string (nullable = false)
+ |-- ldSet: array (nullable = false)
+ | |-- element: struct (containsNull = false)
+ | | |-- tagVariantId: string (nullable = false)
+ | | |-- rValues: array (nullable = false)
+ | | | |-- element: struct (containsNull = false)
+ | | | | |-- population: string (nullable = false)
+ | | | | |-- r: double (nullable = false)
+root
- |-- studyId: string (nullable = false)
- |-- projectId: string (nullable = false)
- |-- studyType: string (nullable = false)
- |-- traitFromSource: string (nullable = false)
- |-- traitFromSourceMappedIds: array (nullable = true)
- | |-- element: string (containsNull = true)
- |-- pubmedId: string (nullable = true)
- |-- publicationTitle: string (nullable = true)
- |-- publicationFirstAuthor: string (nullable = true)
- |-- publicationDate: string (nullable = true)
- |-- publicationJournal: string (nullable = true)
- |-- backgroundTraitFromSourceMappedIds: array (nullable = true)
- | |-- element: string (containsNull = true)
- |-- initialSampleSize: string (nullable = true)
- |-- nCases: long (nullable = true)
- |-- nControls: long (nullable = true)
- |-- nSamples: long (nullable = true)
- |-- ldPopulationStructure: array (nullable = true)
- | |-- element: struct (containsNull = false)
- | | |-- ldPopulation: string (nullable = true)
- | | |-- relativeSampleSize: double (nullable = true)
- |-- discoverySamples: array (nullable = true)
- | |-- element: struct (containsNull = false)
- | | |-- sampleSize: long (nullable = true)
- | | |-- ancestry: string (nullable = true)
- |-- replicationSamples: array (nullable = true)
- | |-- element: struct (containsNull = false)
- | | |-- sampleSize: long (nullable = true)
- | | |-- ancestry: string (nullable = true)
- |-- summarystatsLocation: string (nullable = true)
- |-- hasSumstats: boolean (nullable = true)
- Bases: Dataset
Study index dataset.
A study index dataset captures all the metadata for all studies including GWAS and Molecular QTL.
src/otg/dataset/study_index.py21 + |
root
+ |-- studyId: string (nullable = false)
+ |-- projectId: string (nullable = false)
+ |-- studyType: string (nullable = false)
+ |-- traitFromSource: string (nullable = false)
+ |-- traitFromSourceMappedIds: array (nullable = true)
+ | |-- element: string (containsNull = true)
+ |-- pubmedId: string (nullable = true)
+ |-- publicationTitle: string (nullable = true)
+ |-- publicationFirstAuthor: string (nullable = true)
+ |-- publicationDate: string (nullable = true)
+ |-- publicationJournal: string (nullable = true)
+ |-- backgroundTraitFromSourceMappedIds: array (nullable = true)
+ | |-- element: string (containsNull = true)
+ |-- initialSampleSize: string (nullable = true)
+ |-- nCases: long (nullable = true)
+ |-- nControls: long (nullable = true)
+ |-- nSamples: long (nullable = true)
+ |-- ldPopulationStructure: array (nullable = true)
+ | |-- element: struct (containsNull = false)
+ | | |-- ldPopulation: string (nullable = true)
+ | | |-- relativeSampleSize: double (nullable = true)
+ |-- discoverySamples: array (nullable = true)
+ | |-- element: struct (containsNull = false)
+ | | |-- sampleSize: long (nullable = true)
+ | | |-- ancestry: string (nullable = true)
+ |-- replicationSamples: array (nullable = true)
+ | |-- element: struct (containsNull = false)
+ | | |-- sampleSize: long (nullable = true)
+ | | |-- ancestry: string (nullable = true)
+ |-- summarystatsLocation: string (nullable = true)
+ |-- hasSumstats: boolean (nullable = true)
+root
- |-- studyLocusId: long (nullable = false)
- |-- variantId: string (nullable = false)
- |-- chromosome: string (nullable = true)
- |-- position: integer (nullable = true)
- |-- studyId: string (nullable = false)
- |-- beta: double (nullable = true)
- |-- oddsRatio: double (nullable = true)
- |-- oddsRatioConfidenceIntervalLower: double (nullable = true)
- |-- oddsRatioConfidenceIntervalUpper: double (nullable = true)
- |-- betaConfidenceIntervalLower: double (nullable = true)
- |-- betaConfidenceIntervalUpper: double (nullable = true)
- |-- pValueMantissa: float (nullable = true)
- |-- pValueExponent: integer (nullable = true)
- |-- effectAlleleFrequencyFromSource: float (nullable = true)
- |-- standardError: double (nullable = true)
- |-- subStudyDescription: string (nullable = true)
- |-- qualityControls: array (nullable = true)
- | |-- element: string (containsNull = false)
- |-- finemappingMethod: string (nullable = true)
- |-- ldSet: array (nullable = true)
- | |-- element: struct (containsNull = true)
- | | |-- tagVariantId: string (nullable = true)
- | | |-- r2Overall: double (nullable = true)
- |-- locus: array (nullable = true)
- | |-- element: struct (containsNull = true)
- | | |-- is95CredibleSet: boolean (nullable = true)
- | | |-- is99CredibleSet: boolean (nullable = true)
- | | |-- logABF: double (nullable = true)
- | | |-- posteriorProbability: double (nullable = true)
- | | |-- variantId: string (nullable = true)
- | | |-- pValueMantissa: float (nullable = true)
- | | |-- pValueExponent: integer (nullable = true)
- | | |-- pValueMantissaConditioned: float (nullable = true)
- | | |-- pValueExponentConditioned: integer (nullable = true)
- | | |-- beta: double (nullable = true)
- | | |-- standardError: double (nullable = true)
- | | |-- betaConditioned: double (nullable = true)
- | | |-- standardErrorConditioned: double (nullable = true)
- | | |-- r2Overall: double (nullable = true)
- Bases: Dataset
Study-Locus dataset.
This dataset captures associations between study/traits and a genetic loci as provided by finemapping methods.
src/otg/dataset/study_locus.py67 + |
assign_study_locus_id(study_id_col: Column, variant_id_col: Column) -> Column staticmethod ¶Hashes a column with a variant ID and a study ID to extract a consistent studyLocusId.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
study_id_col | Column | column name with a study ID | required |
variant_id_col | Column | column name with a variant ID | required |
Returns:
| Name | Type | Description |
|---|---|---|
Column | Column | column with a study locus ID |
Examples:
>>> df = spark.createDataFrame([("GCST000001", "1_1000_A_C"), ("GCST000002", "1_1000_A_C")]).toDF("studyId", "variantId")
>>> df.withColumn("study_locus_id", StudyLocus.assign_study_locus_id(*[f.col("variantId"), f.col("studyId")])).show()
+----------+----------+--------------------+
| studyId| variantId| study_locus_id|
@@ -941,7 +901,7 @@
<BLANKLINE>
"""
return f.xxhash64(*[study_id_col, variant_id_col]).alias("studyLocusId")
-clump() -> StudyLocus ¶Perform LD clumping of the studyLocus.
Evaluates whether a lead variant is linked to a tag (with lowest p-value) in the same studyLocus dataset.
Returns:
| Name | Type | Description |
|---|---|---|
StudyLocus | StudyLocus | with empty credible sets for linked variants and QC flag. |
src/otg/dataset/study_locus.py367
+ |
clump() -> StudyLocus ¶Perform LD clumping of the studyLocus.
Evaluates whether a lead variant is linked to a tag (with lowest p-value) in the same studyLocus dataset.
Returns:
| Name | Type | Description |
|---|---|---|
StudyLocus | StudyLocus | with empty credible sets for linked variants and QC flag. |
src/otg/dataset/study_locus.py367 368 369 370 @@ -1008,7 +968,7 @@ .drop("is_lead_linked") ) return self - |
filter_credible_set(credible_interval: CredibleInterval) -> StudyLocus ¶Filter study-locus tag variants based on given credible interval.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
credible_interval | CredibleInterval | Credible interval to filter for. | required |
Returns:
| Name | Type | Description |
|---|---|---|
StudyLocus | StudyLocus | Filtered study-locus dataset. |
src/otg/dataset/study_locus.py225
+ |
filter_credible_set(credible_interval: CredibleInterval) -> StudyLocus ¶Filter study-locus tag variants based on given credible interval.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
credible_interval | CredibleInterval | Credible interval to filter for. | required |
Returns:
| Name | Type | Description |
|---|---|---|
StudyLocus | StudyLocus | Filtered study-locus dataset. |
src/otg/dataset/study_locus.py225 226 227 228 @@ -1041,7 +1001,7 @@ f.expr(f"filter(locus, tag -> (tag.{credible_interval.value}))"), ) return self - |
find_overlaps(study_index: StudyIndex) -> StudyLocusOverlap ¶Calculate overlapping study-locus.
Find overlapping study-locus that share at least one tagging variant. All GWAS-GWAS and all GWAS-Molecular traits are computed with the Molecular traits always appearing on the right side.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
study_index | StudyIndex | Study index to resolve study types. | required |
Returns:
| Name | Type | Description |
|---|---|---|
StudyLocusOverlap | StudyLocusOverlap | Pairs of overlapping study-locus with aligned tags. |
src/otg/dataset/study_locus.py243
+ |
find_overlaps(study_index: StudyIndex) -> StudyLocusOverlap ¶Calculate overlapping study-locus.
Find overlapping study-locus that share at least one tagging variant. All GWAS-GWAS and all GWAS-Molecular traits are computed with the Molecular traits always appearing on the right side.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
study_index | StudyIndex | Study index to resolve study types. | required |
Returns:
| Name | Type | Description |
|---|---|---|
StudyLocusOverlap | StudyLocusOverlap | Pairs of overlapping study-locus with aligned tags. |
src/otg/dataset/study_locus.py243 244 245 246 @@ -1108,7 +1068,7 @@ # study-locus overlap by aligning overlapping variants return self._align_overlapping_tags(loci_to_overlap, peak_overlaps) - |
get_schema() -> StructType classmethod ¶Provides the schema for the StudyLocus dataset.
Returns:
| Name | Type | Description |
|---|---|---|
StructType | StructType | schema for the StudyLocus dataset. |
src/otg/dataset/study_locus.py216
+ |
get_schema() -> StructType classmethod ¶Provides the schema for the StudyLocus dataset.
Returns:
| Name | Type | Description |
|---|---|---|
StructType | StructType | schema for the StudyLocus dataset. |
src/otg/dataset/study_locus.py216 217 218 219 @@ -1123,7 +1083,7 @@ StructType: schema for the StudyLocus dataset. """ return parse_spark_schema("study_locus.json") - |
neglog_pvalue() -> Column ¶Returns the negative log p-value.
Returns:
| Name | Type | Description |
|---|---|---|
Column | Column | Negative log p-value |
src/otg/dataset/study_locus.py301
+ |
neglog_pvalue() -> Column ¶Returns the negative log p-value.
Returns:
| Name | Type | Description |
|---|---|---|
Column | Column | Negative log p-value |
src/otg/dataset/study_locus.py301 302 303 304 @@ -1142,7 +1102,7 @@ self.df.pValueMantissa, self.df.pValueExponent, ) - |
unique_variants_in_locus() -> DataFrame ¶All unique variants collected in a StudyLocus dataframe.
Returns:
| Name | Type | Description |
|---|---|---|
DataFrame | DataFrame | A dataframe containing |
src/otg/dataset/study_locus.py278
+ |
unique_variants_in_locus() -> DataFrame ¶All unique variants collected in a StudyLocus dataframe.
Returns:
| Name | Type | Description |
|---|---|---|
DataFrame | DataFrame | A dataframe containing |
src/otg/dataset/study_locus.py278 279 280 281 @@ -1185,7 +1145,7 @@ ) .distinct() ) - |
Bases: Enum
Study-Locus quality control options listing concerns on the quality of the association.
Attributes:
| Name | Type | Description |
|---|---|---|
SUBSIGNIFICANT_FLAG | str | p-value below significance threshold |
NO_GENOMIC_LOCATION_FLAG | str | Incomplete genomic mapping |
COMPOSITE_FLAG | str | Composite association due to variant x variant interactions |
VARIANT_INCONSISTENCY_FLAG | str | Inconsistencies in the reported variants |
NON_MAPPED_VARIANT_FLAG | str | Variant not mapped to GnomAd |
PALINDROMIC_ALLELE_FLAG | str | Alleles are palindromic - cannot harmonize |
AMBIGUOUS_STUDY | str | Association with ambiguous study |
UNRESOLVED_LD | str | Variant not found in LD reference |
LD_CLUMPED | str | Explained by a more significant variant in high LD (clumped) |
src/otg/dataset/study_locus.py26
+ |
otg.dataset.study_locus.StudyLocusQualityCheck ¶ Bases: Enum
Study-Locus quality control options listing concerns on the quality of the association.
Attributes:
| Name | Type | Description |
|---|---|---|
SUBSIGNIFICANT_FLAG | str | p-value below significance threshold |
NO_GENOMIC_LOCATION_FLAG | str | Incomplete genomic mapping |
COMPOSITE_FLAG | str | Composite association due to variant x variant interactions |
VARIANT_INCONSISTENCY_FLAG | str | Inconsistencies in the reported variants |
NON_MAPPED_VARIANT_FLAG | str | Variant not mapped to GnomAd |
PALINDROMIC_ALLELE_FLAG | str | Alleles are palindromic - cannot harmonize |
AMBIGUOUS_STUDY | str | Association with ambiguous study |
UNRESOLVED_LD | str | Variant not found in LD reference |
LD_CLUMPED | str | Explained by a more significant variant in high LD (clumped) |
src/otg/dataset/study_locus.py26 27 28 29 @@ -1234,7 +1194,7 @@ UNRESOLVED_LD = "Variant not found in LD reference" LD_CLUMPED = "Explained by a more significant variant in high LD (clumped)" NO_POPULATION = "Study does not have population annotation to resolve LD" - |
Bases: Enum
Credible interval enum.
Interval within which an unobserved parameter value falls with a particular probability.
Attributes:
| Name | Type | Description |
|---|---|---|
IS95 | str | 95% credible interval |
IS99 | str | 99% credible interval |
src/otg/dataset/study_locus.py53
+ |
otg.dataset.study_locus.CredibleInterval ¶ Bases: Enum
Credible interval enum.
Interval within which an unobserved parameter value falls with a particular probability.
Attributes:
| Name | Type | Description |
|---|---|---|
IS95 | str | 95% credible interval |
IS99 | str | 99% credible interval |
src/otg/dataset/study_locus.py53 54 55 56 @@ -1257,4 +1217,44 @@ IS95 = "is95CredibleSet" IS99 = "is99CredibleSet" - |
root
+ |-- studyLocusId: long (nullable = false)
+ |-- variantId: string (nullable = false)
+ |-- chromosome: string (nullable = true)
+ |-- position: integer (nullable = true)
+ |-- studyId: string (nullable = false)
+ |-- beta: double (nullable = true)
+ |-- oddsRatio: double (nullable = true)
+ |-- oddsRatioConfidenceIntervalLower: double (nullable = true)
+ |-- oddsRatioConfidenceIntervalUpper: double (nullable = true)
+ |-- betaConfidenceIntervalLower: double (nullable = true)
+ |-- betaConfidenceIntervalUpper: double (nullable = true)
+ |-- pValueMantissa: float (nullable = true)
+ |-- pValueExponent: integer (nullable = true)
+ |-- effectAlleleFrequencyFromSource: float (nullable = true)
+ |-- standardError: double (nullable = true)
+ |-- subStudyDescription: string (nullable = true)
+ |-- qualityControls: array (nullable = true)
+ | |-- element: string (containsNull = false)
+ |-- finemappingMethod: string (nullable = true)
+ |-- ldSet: array (nullable = true)
+ | |-- element: struct (containsNull = true)
+ | | |-- tagVariantId: string (nullable = true)
+ | | |-- r2Overall: double (nullable = true)
+ |-- locus: array (nullable = true)
+ | |-- element: struct (containsNull = true)
+ | | |-- is95CredibleSet: boolean (nullable = true)
+ | | |-- is99CredibleSet: boolean (nullable = true)
+ | | |-- logABF: double (nullable = true)
+ | | |-- posteriorProbability: double (nullable = true)
+ | | |-- variantId: string (nullable = true)
+ | | |-- pValueMantissa: float (nullable = true)
+ | | |-- pValueExponent: integer (nullable = true)
+ | | |-- pValueMantissaConditioned: float (nullable = true)
+ | | |-- pValueExponentConditioned: integer (nullable = true)
+ | | |-- beta: double (nullable = true)
+ | | |-- standardError: double (nullable = true)
+ | | |-- betaConditioned: double (nullable = true)
+ | | |-- standardErrorConditioned: double (nullable = true)
+ | | |-- r2Overall: double (nullable = true)
+root
- |-- leftStudyLocusId: long (nullable = false)
- |-- rightStudyLocusId: long (nullable = false)
- |-- chromosome: string (nullable = false)
- |-- tagVariantId: string (nullable = false)
- |-- statistics: struct (nullable = false)
- | |-- left_pValueMantissa: float (nullable = true)
- | |-- left_pValueExponent: integer (nullable = true)
- | |-- right_pValueMantissa: float (nullable = true)
- | |-- right_pValueExponent: integer (nullable = true)
- | |-- left_beta: double (nullable = true)
- | |-- right_beta: double (nullable = true)
- | |-- left_logABF: double (nullable = true)
- | |-- right_logABF: double (nullable = true)
- | |-- left_posteriorProbability: double (nullable = true)
- | |-- right_posteriorProbability: double (nullable = true)
- Bases: Dataset
Study-Locus overlap.
This dataset captures pairs of overlapping StudyLocus: that is associations whose credible sets share at least one tagging variant.
Note
This is a helpful dataset for other downstream analyses, such as colocalisation. This dataset will contain the overlapping signals between studyLocus associations once they have been clumped and fine-mapped.
src/otg/dataset/study_locus_overlap.py17 + |
root
+ |-- leftStudyLocusId: long (nullable = false)
+ |-- rightStudyLocusId: long (nullable = false)
+ |-- chromosome: string (nullable = false)
+ |-- tagVariantId: string (nullable = false)
+ |-- statistics: struct (nullable = false)
+ | |-- left_pValueMantissa: float (nullable = true)
+ | |-- left_pValueExponent: integer (nullable = true)
+ | |-- right_pValueMantissa: float (nullable = true)
+ | |-- right_pValueExponent: integer (nullable = true)
+ | |-- left_beta: double (nullable = true)
+ | |-- right_beta: double (nullable = true)
+ | |-- left_logABF: double (nullable = true)
+ | |-- right_logABF: double (nullable = true)
+ | |-- left_posteriorProbability: double (nullable = true)
+ | |-- right_posteriorProbability: double (nullable = true)
+root
- |-- studyId: string (nullable = false)
- |-- variantId: string (nullable = false)
- |-- chromosome: string (nullable = false)
- |-- position: integer (nullable = false)
- |-- beta: double (nullable = false)
- |-- betaConfidenceIntervalLower: double (nullable = true)
- |-- betaConfidenceIntervalUpper: double (nullable = true)
- |-- pValueMantissa: float (nullable = false)
- |-- pValueExponent: integer (nullable = false)
- |-- effectAlleleFrequencyFromSource: float (nullable = true)
- |-- standardError: double (nullable = true)
- Bases: Dataset
Summary Statistics dataset.
A summary statistics dataset contains all single point statistics resulting from a GWAS.
src/otg/dataset/summary_statistics.py20 + |
root
+ |-- studyId: string (nullable = false)
+ |-- variantId: string (nullable = false)
+ |-- chromosome: string (nullable = false)
+ |-- position: integer (nullable = false)
+ |-- beta: double (nullable = false)
+ |-- betaConfidenceIntervalLower: double (nullable = true)
+ |-- betaConfidenceIntervalUpper: double (nullable = true)
+ |-- pValueMantissa: float (nullable = false)
+ |-- pValueExponent: integer (nullable = false)
+ |-- effectAlleleFrequencyFromSource: float (nullable = true)
+ |-- standardError: double (nullable = true)
+root
- |-- variantId: string (nullable = false)
- |-- chromosome: string (nullable = false)
- |-- position: integer (nullable = false)
- |-- gnomad3VariantId: string (nullable = false)
- |-- referenceAllele: string (nullable = false)
- |-- alternateAllele: string (nullable = false)
- |-- chromosomeB37: string (nullable = true)
- |-- positionB37: integer (nullable = true)
- |-- alleleType: string (nullable = true)
- |-- rsIds: array (nullable = true)
- | |-- element: string (containsNull = true)
- |-- alleleFrequencies: array (nullable = false)
- | |-- element: struct (containsNull = true)
- | | |-- populationName: string (nullable = true)
- | | |-- alleleFrequency: double (nullable = true)
- |-- cadd: struct (nullable = true)
- | |-- phred: float (nullable = true)
- | |-- raw: float (nullable = true)
- |-- vep: struct (nullable = false)
- | |-- mostSevereConsequence: string (nullable = true)
- | |-- transcriptConsequences: array (nullable = true)
- | | |-- element: struct (containsNull = true)
- | | | |-- aminoAcids: string (nullable = true)
- | | | |-- consequenceTerms: array (nullable = true)
- | | | | |-- element: string (containsNull = true)
- | | | |-- geneId: string (nullable = true)
- | | | |-- lof: string (nullable = true)
- | | | |-- polyphenScore: double (nullable = true)
- | | | |-- polyphenPrediction: string (nullable = true)
- | | | |-- siftScore: double (nullable = true)
- | | | |-- siftPrediction: string (nullable = true)
- Bases: Dataset
Dataset with variant-level annotations.
src/otg/dataset/variant_annotation.py21 + |
root
+ |-- variantId: string (nullable = false)
+ |-- chromosome: string (nullable = false)
+ |-- position: integer (nullable = false)
+ |-- gnomad3VariantId: string (nullable = false)
+ |-- referenceAllele: string (nullable = false)
+ |-- alternateAllele: string (nullable = false)
+ |-- chromosomeB37: string (nullable = true)
+ |-- positionB37: integer (nullable = true)
+ |-- alleleType: string (nullable = true)
+ |-- rsIds: array (nullable = true)
+ | |-- element: string (containsNull = true)
+ |-- alleleFrequencies: array (nullable = false)
+ | |-- element: struct (containsNull = true)
+ | | |-- populationName: string (nullable = true)
+ | | |-- alleleFrequency: double (nullable = true)
+ |-- cadd: struct (nullable = true)
+ | |-- phred: float (nullable = true)
+ | |-- raw: float (nullable = true)
+ |-- vep: struct (nullable = false)
+ | |-- mostSevereConsequence: string (nullable = true)
+ | |-- transcriptConsequences: array (nullable = true)
+ | | |-- element: struct (containsNull = true)
+ | | | |-- aminoAcids: string (nullable = true)
+ | | | |-- consequenceTerms: array (nullable = true)
+ | | | | |-- element: string (containsNull = true)
+ | | | |-- geneId: string (nullable = true)
+ | | | |-- lof: string (nullable = true)
+ | | | |-- polyphenScore: double (nullable = true)
+ | | | |-- polyphenPrediction: string (nullable = true)
+ | | | |-- siftScore: double (nullable = true)
+ | | | |-- siftPrediction: string (nullable = true)
+root
- |-- variantId: string (nullable = false)
- |-- chromosome: string (nullable = false)
- |-- position: integer (nullable = false)
- |-- referenceAllele: string (nullable = false)
- |-- alternateAllele: string (nullable = false)
- |-- chromosomeB37: string (nullable = true)
- |-- positionB37: integer (nullable = true)
- |-- alleleType: string (nullable = false)
- |-- alleleFrequencies: array (nullable = false)
- | |-- element: struct (containsNull = true)
- | | |-- populationName: string (nullable = true)
- | | |-- alleleFrequency: double (nullable = true)
- |-- cadd: struct (nullable = true)
- | |-- phred: float (nullable = true)
- | |-- raw: float (nullable = true)
- |-- mostSevereConsequence: string (nullable = true)
- |-- rsIds: array (nullable = true)
- | |-- element: string (containsNull = true)
- Bases: Dataset
Variant index dataset.
Variant index dataset is the result of intersecting the variant annotation dataset with the variants with V2D available information.
src/otg/dataset/variant_index.py20 + |
root
+ |-- variantId: string (nullable = false)
+ |-- chromosome: string (nullable = false)
+ |-- position: integer (nullable = false)
+ |-- referenceAllele: string (nullable = false)
+ |-- alternateAllele: string (nullable = false)
+ |-- chromosomeB37: string (nullable = true)
+ |-- positionB37: integer (nullable = true)
+ |-- alleleType: string (nullable = false)
+ |-- alleleFrequencies: array (nullable = false)
+ | |-- element: struct (containsNull = true)
+ | | |-- populationName: string (nullable = true)
+ | | |-- alleleFrequency: double (nullable = true)
+ |-- cadd: struct (nullable = true)
+ | |-- phred: float (nullable = true)
+ | |-- raw: float (nullable = true)
+ |-- mostSevereConsequence: string (nullable = true)
+ |-- rsIds: array (nullable = true)
+ | |-- element: string (containsNull = true)
+root
- |-- geneId: string (nullable = false)
- |-- variantId: string (nullable = false)
- |-- distance: long (nullable = true)
- |-- chromosome: string (nullable = false)
- |-- datatypeId: string (nullable = false)
- |-- datasourceId: string (nullable = false)
- |-- score: double (nullable = true)
- |-- resourceScore: double (nullable = true)
- |-- pmid: string (nullable = true)
- |-- biofeature: string (nullable = true)
- |-- variantFunctionalConsequenceId: string (nullable = true)
- |-- isHighQualityPlof: boolean (nullable = true)
- Bases: Dataset
Variant-to-gene (V2G) evidence dataset.
A variant-to-gene (V2G) evidence is understood as any piece of evidence that supports the association of a variant with a likely causal gene. The evidence can sometimes be context-specific and refer to specific biofeatures (e.g. cell types)
src/otg/dataset/v2g.py16 + |
root
+ |-- geneId: string (nullable = false)
+ |-- variantId: string (nullable = false)
+ |-- distance: long (nullable = true)
+ |-- chromosome: string (nullable = false)
+ |-- datatypeId: string (nullable = false)
+ |-- datasourceId: string (nullable = false)
+ |-- score: double (nullable = true)
+ |-- resourceScore: double (nullable = true)
+ |-- pmid: string (nullable = true)
+ |-- biofeature: string (nullable = true)
+ |-- variantFunctionalConsequenceId: string (nullable = true)
+ |-- isHighQualityPlof: boolean (nullable = true)
+TBC
TBC
Bases: StudyIndex
Study index dataset from FinnGen.
The following information is aggregated/extracted:
Some fields are also populated as constants, such as study type and the initial sample size.
src/otg/datasource/finngen/study_index.py14 + |
Importer of LD information from GnomAD.
The information comes from LD matrices made available by GnomAD in Hail's native format. We aggregate the LD information across 8 ancestries. The basic steps to generate the LDIndex are:
src/otg/datasource/gnomad/ld.py20 + |
GnomAD variants included in the GnomAD genomes dataset.
src/otg/datasource/gnomad/variants.py14 + |
Bases: StudyLocus
Study-locus dataset derived from GWAS Catalog.
src/otg/datasource/gwas_catalog/associations.py30 + |
Bases: StudyIndex
Study index from GWAS Catalog.
The following information is harmonised from the GWAS Catalog:
src/otg/datasource/gwas_catalog/study_index.py18 + |
Splitting multi-trait GWAS Catalog studies.
src/otg/datasource/gwas_catalog/study_splitter.py17 + |
Bases: SummaryStatistics
GWAS Catalog Summary Statistics reader.
src/otg/datasource/gwas_catalog/summary_statistics.py22 + |
TBC
Bases: Intervals
Interval dataset from Andersson et al. 2014.
src/otg/datasource/intervals/andersson.py21 + |
Bases: Intervals
Interval dataset from Javierre et al. 2016.
src/otg/datasource/intervals/javierre.py18 + |
Bases: Intervals
Interval dataset from Jung et al. 2019.
src/otg/datasource/intervals/jung.py18 + |
Bases: Intervals
Interval dataset from Thurman et al. 2012.
src/otg/datasource/intervals/thurman.py18 + |
The Open Targets Platform is a comprehensive resource that aims to aggregate and harmonise various types of data to facilitate the identification, prioritisation, and validation of drug targets. By integrating publicly available datasets including data generated by the Open Targets consortium, the Platform builds and scores target-disease associations to assist in drug target identification and prioritisation. It also integrates relevant annotation information about targets, diseases, phenotypes, and drugs, as well as their most relevant relationships.
Genomic data from Open Targets integrates human genome-wide association studies (GWAS) and functional genomics data including gene expression, protein abundance, chromatin interaction and conformation data from a wide range of cell types and tissues to make robust connections between GWAS-associated loci, variants and likely causal genes.
The Open Targets Platform is a comprehensive resource that aims to aggregate and harmonise various types of data to facilitate the identification, prioritisation, and validation of drug targets. By integrating publicly available datasets including data generated by the Open Targets consortium, the Platform builds and scores target-disease associations to assist in drug target identification and prioritisation. It also integrates relevant annotation information about targets, diseases, phenotypes, and drugs, as well as their most relevant relationships.
Genomic data from Open Targets integrates human genome-wide association studies (GWAS) and functional genomics data including gene expression, protein abundance, chromatin interaction and conformation data from a wide range of cell types and tissues to make robust connections between GWAS-associated loci, variants and likely causal genes.