New DataRequest

[pgierz]

Rework of DataRequest

This introduces new DataRequest, DataRequestVariable, and DataRequestTable classes. The old ones are copies directly from Ruby and are not very nice to work with.

Here, we implement a generic abstract base class (ABC) for DataRequestVariable. This allows us to define how a DataRequestVariable should look like from the pymorize Python side, without caring about how the information is ultimately extracted from the JSON tables. CMIP6DataRequestVariable is the first direct implementation used for JSON tables. This has the key feature of allowing us to be flexible when new CMIP standards inevitably come out, for example with CMIP7.

A DataRequestVariable therefore must implement certain properties and methods to be correctly recognised, otherwise you get Python TypeErrors when subclassing.

TODO List

• DataRequestVariable (and concrete CMIP6DataRequestVariable)
• DataRequestTable (and concrete CMIP6DataRequestTable, specifically CMIP6JSONDataRequestTable -- yes, I want to build two subclass levels here, since it might happen in the future that the source of the DataRequestTable isn't a JSON file. Maybe it's a HTTP request. Maybe it's a post-it note on the coffee machine. I don't know)
• DataRequest (maybe named DataRequestCollection?) -- some gathering of various Variables and/or Tables into a bundle.
• Rework all the tests

Still to come

• Integration of @siligam's FlatTable design (those parts that fit together with this)

Copilot Summary

This pull request introduces a new abstract base class DataRequestVariable and its concrete implementation CMIP6DataRequestVariable in the src/pymorize/data_request/variable.py file. Additionally, it includes a unit test for the CMIP6DataRequestVariable class in tests/unit/test_drv.py.

Key changes include:

New Class Definitions:

• src/pymorize/data_request/variable.py:
  • Added DataRequestVariable abstract base class with various properties and methods to define a generic data request variable.
  • Added CMIP6DataRequestVariable class as a concrete implementation of DataRequestVariable for CMIP6 variables, utilizing the dataclass decorator for automatic method generation.
  • Implemented class methods for constructing instances from dictionaries and JSON files, and methods for converting instances to dictionary representations.

Unit Testing:

• tests/unit/test_drv.py:
  • Added a test case to verify the creation of a CMIP6DataRequestVariable instance from a CMIP6 JSON table file.

[pgierz]

Tests now fail at the integration level, which is what I had expected.

[pgierz]

This pull request includes significant updates to the testing configuration, dependency management, and the CMORizer class in the pymorize module. The most important changes are grouped by theme and summarized below:

Testing Configuration:

• Modified the pytest command in the CI workflow to include all Python files in the tests/unit directory. (.github/workflows/CI-test.yaml)
• Commented out unused imports and pytest hooks in conftest.py to clean up the code. (conftest.py) [1] [2]
• Added a warning filter to pytest.ini to ignore specific matplotlib-related warnings. (pytest.ini)

Dependency Management:

• Added semver to the list of required dependencies in setup.py. (setup.py)
• Added pytest-mock to the list of testing dependencies in setup.py. (setup.py)

CMORizer Class Enhancements:

• Introduced _SUPPORTED_CMOR_VERSIONS to the CMORizer class to specify supported CMOR versions. (src/pymorize/cmorizer.py)
• Added a destructor method to the CMORizer class to ensure the cluster is closed gracefully. (src/pymorize/cmorizer.py)
• Updated the _post_init_read_bare_tables method to use CMIP6IgnoreTableFiles instead of IgnoreTableFiles. (src/pymorize/cmorizer.py)
• Added validation for cmor_version in the _post_init_create_data_request method and updated the data request creation process. (src/pymorize/cmorizer.py)

Data Request Module:

• Introduced a new collection.py file with the DataRequest and CMIP6DataRequest classes, including methods for creating data requests from tables, directories, and git repositories. (src/pymorize/data_request/collection.py)
• Added a new factory.py file to implement the MetaFactory metaclass and a factory creation function for managing data request subclasses. (src/pymorize/data_request/factory.py)

These changes collectively enhance the functionality, maintainability, and testing capabilities of the pymorize module.

[mandresm]

Awesome @pgierz. I have a headache but it's very nice. I cannot say I understood everything but I could more or less follow. There are no many suggestions from my side, more questions, than anything. Once the suggestions are addressed feel free to merge right away.