Export/import of logs & comments - Recursive queries

[szoupanos]

We should verify why the nodes don't reference the comments and the logs in the metadata.json and it is the other way around

        "Log": {
            "time": {
                "convert_type": "date"
            },
            "loggername": {},
            "levelname": {},
            "message": {},
            "metadata": {},
            "dbnode": {
                "requires": "Node",
                "related_name": "dblogs"
            },
            "uuid": {}
        },
        "Comment": {
            "uuid": {},
            "ctime": {
                "convert_type": "date"
            },
            "mtime": {
                "convert_type": "date"
            },
            "content": {},
            "dbnode": {
                "related_name": "dbcomments",
                "requires": "Node"
            },
            "user": {
                "related_name": "dbcomments",
                "requires": "User"
            }
        }

If the nodes would reference the logs and the comments then, the fill_in_query method of the export would construct a query that would automatically export the logs and the comments of the given nodes.

If there is no serious reason why we don't follow this approach, we should proceed to this change to be uniform with the way other entities are exported.

[CasperWA]

When changing these it seems there is a reason why they are tied so tightly to the database schema:

Traceback (most recent call last):
  File "/home/candersen/virtualenv/test/aiida/aiida_core/aiida/orm/implementation/querybuilder.py", line 237, in get_column
    return getattr(alias, colname)
  File "/home/candersen/virtualenv/test/lib/python3.6/site-packages/sqlalchemy/orm/util.py", line 431, in __getattr__
    raise AttributeError(key)
AttributeError: dblog

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/candersen/virtualenv/test/aiida/aiida_core/aiida/backends/tests/test_export_and_import.py", line 505, in test_non_default_user_nodes
    export([sd2], outfile=filename1, silent=True)
  File "/home/candersen/virtualenv/test/aiida/aiida_core/aiida/orm/importexport.py", line 2948, in export
    export_tree(what, folder=folder, silent=silent, **kwargs)
  File "/home/candersen/virtualenv/test/aiida/aiida_core/aiida/orm/importexport.py", line 2365, in export_tree
    for temp_d in partial_query.iterdict():
  File "/home/candersen/virtualenv/test/aiida/aiida_core/aiida/orm/querybuilder.py", line 2050, in iterdict
    query = self.get_query()
  File "/home/candersen/virtualenv/test/aiida/aiida_core/aiida/orm/querybuilder.py", line 1907, in get_query
    query = self._build()
  File "/home/candersen/virtualenv/test/aiida/aiida_core/aiida/orm/querybuilder.py", line 1739, in _build
    self._build_projections(vertex['tag'])
  File "/home/candersen/virtualenv/test/aiida/aiida_core/aiida/orm/querybuilder.py", line 1051, in _build_projections
    self._add_to_projections(alias, entity_name, **extraspec)
  File "/home/candersen/virtualenv/test/aiida/aiida_core/aiida/orm/querybuilder.py", line 1007, in _add_to_projections
    entity_to_project = self._get_projectable_entity(alias, column_name, attr_key, cast=cast)
  File "/home/candersen/virtualenv/test/aiida/aiida_core/aiida/orm/querybuilder.py", line 983, in _get_projectable_entity
    entity = self._impl.get_column(column_name, alias)
  File "/home/candersen/virtualenv/test/aiida/aiida_core/aiida/orm/implementation/querybuilder.py", line 244, in get_column
    '\n'.join(alias._sa_class_manager.mapper.c.keys())  # pylint: disable=protected-access
aiida.common.exceptions.InputValidationError: dblog is not a column of <AliasedClass at 0x7fb9c51d2438; DbNode>
Valid columns are:
id
uuid
type
process_type
label
description
ctime
mtime
dbcomputer_id
user_id
public
nodeversion

This was after adding:

all_fields_info[NODE_ENTITY_NAME] = {
...
    "dblog": {
        "requires": LOG_ENTITY_NAME,
        "related_name": "dbnodes"
    }
...
}

[szoupanos]

Yes, I understand. It seems that it constructs in a wrong way the query in fill_in_query
It would be nice to see what it appends to the QB at every loop and see if the problem can be avoided.
I can help you debug it gets too complicated.

[CasperWA]

Yes, I understand. It seems that it constructs in a wrong way the query in fill_in_query
It would be nice to see what it appends to the QB at every loop and see if the problem can be avoided.
I can help you debug it gets too complicated.

It appends the keys as projections, which means when one tries to add a key that doesn't equal a database column it complains as above.
If you want to change this, we need to change the fill_in_query