Implement Preprocess DAG following recent architecture updates

[tskir]

Closes opentargets/issues#3106.

• Added three new functions to the common_airflow module: submit_step, read_yaml_config, and generate_dag.
• Updated the ETL DAG to use these functions, making it even more readable and focused on logic.
• Implemented a new separate DAG for the Preprocess part, as we discussed.
• Made some refactoring changes specific to FinnGen ingestion.

[codecov-commenter]

Codecov Report

Merging #216 (cc3a1e5) into tskir-3140-better-dag-import-fix (5a63093) will increase coverage by 0.10%.
The diff coverage is 86.84%.

@@                         Coverage Diff                          @@
##           tskir-3140-better-dag-import-fix     #216      +/-   ##
====================================================================
+ Coverage                             85.53%   85.64%   +0.10%     
====================================================================
  Files                                    80       81       +1     
  Lines                                  1839     1853      +14     
====================================================================
+ Hits                                   1573     1587      +14     
  Misses                                  266      266              

Files	Coverage Δ
src/airflow/dags/common_airflow.py	100.00% <100.00%> (ø)
src/airflow/dags/dag_genetics_etl.py	100.00% <100.00%> (ø)
src/airflow/dags/dag_preprocess.py	100.00% <100.00%> (ø)
src/otg/datasource/finngen/__init__.py	100.00% <ø> (ø)
src/otg/datasource/finngen/summary_stats.py	100.00% <100.00%> (ø)
src/otg/finngen.py	65.38% <0.00%> (ø)

[d0choa]

@ireneisdoomed and I failed to make the (commented) test test_no_import_errors in tests/airflow/test_dag.py run. The test is meant to validate that the DAG will be imported into Airflow without errors. It used to work before the split that created the common_airflow.py script, but I think it's now having issues with the fact that it needs to import the other script.

It feels like this is a critical test to have from a CI/CD point of view.

[tskir]

Thank you for the comment @d0choa! I wasn't aware of this commented out test. I'll look into it now.

[d0choa]

I know this is not part of this PR, but I felt you might know better what the issue might be. I reproduced the issue in a python console:

In [1]: import airflow
In [3]: from airflow.models import DagBag
In [4]: DagBag(dag_folder='src/airflow/dags', include_examples=False)
[2023-11-02T11:08:01.788+0000] {dagbag.py:536} INFO - Filling up the DagBag from src/airflow/dags
/Users/ochoa/Library/Caches/pypoetry/virtualenvs/otgenetics-_BWSZ0mB-py3.10/lib/python3.10/site-packages/airflow/providers/google/cloud/links/dataproc.py:79 AirflowProviderDeprecationWarning: This DataprocLink is deprecated.
/Users/ochoa/Library/Caches/pypoetry/virtualenvs/otgenetics-_BWSZ0mB-py3.10/lib/python3.10/site-packages/airflow/providers/google/cloud/links/dataproc.py:128 AirflowProviderDeprecationWarning: This DataprocListLink is deprecated.
[2023-11-02T11:08:04.379+0000] {dagbag.py:346} ERROR - Failed to import: src/airflow/dags/dag_preprocess.py
Traceback (most recent call last):
  File "/Users/ochoa/Library/Caches/pypoetry/virtualenvs/otgenetics-_BWSZ0mB-py3.10/lib/python3.10/site-packages/airflow/models/dagbag.py", line 342, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "src/airflow/dags/dag_preprocess.py", line 8, in <module>
    from . import common_airflow as common
ImportError: attempted relative import with no known parent package
[2023-11-02T11:08:04.384+0000] {dagbag.py:346} ERROR - Failed to import: src/airflow/dags/dag_genetics_etl.py
Traceback (most recent call last):
  File "/Users/ochoa/Library/Caches/pypoetry/virtualenvs/otgenetics-_BWSZ0mB-py3.10/lib/python3.10/site-packages/airflow/models/dagbag.py", line 342, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "src/airflow/dags/dag_genetics_etl.py", line 8, in <module>
    from . import common_airflow as common
ImportError: attempted relative import with no known parent package
Out[4]: <airflow.models.dagbag.DagBag at 0x108dbccd0>

[tskir]

@d0choa Thank you for the commands! I've created an issue to address this: opentargets/issues#3140

[tskir]

@d0choa I've now rebased this branch on top of the recently merged PR, so that we can take advantage of the re-enabled test to check these new changes (it passes here as well)

[ireneisdoomed]

@tskir Thanks! This fix is relevant if you want to test this PR #219

[tskir]

@ireneisdoomed Thank you for the fix! I've applied it and rebased this PR on top of it. Autoscaling PR is coming up separately as it's out of scope for this one 🙃

[ireneisdoomed]

Thanks! Looks good.
Just a comment outside the scope of this PR: I hadn't realised that Finngen's study table generation is coupled to SS processing. Right now Finngen's table is static, and we wouldn't need to produce one unless there's a new version out. However we haven't dealt with trait mappings yet, and this process could break the static nature of this step.

[d0choa]

+1 to @ireneisdoomed comments.
Let's try to merge this and we keep moving

[tskir]

Thank you @d0choa @ireneisdoomed for constructive and thought-provoking comments, as usual!