Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a get_df_and_metadata_from_db function #8

Open
matthewcarbone opened this issue Mar 13, 2023 · 1 comment
Open

Implement a get_df_and_metadata_from_db function #8

matthewcarbone opened this issue Mar 13, 2023 · 1 comment

Comments

@matthewcarbone
Copy link

matthewcarbone commented Mar 13, 2023

In order to proceed with #7 I need a more detailed understanding of the API used to get the "data" and "metadata" to use in the Tiled pipeline. I have created a "dummy" scaffold function called get_df_and_metadata_from_db which I need implemented. @elistavitski can you do this? In the meantime, I will work on the code upstream to hook into this function (which we will treat as the endpoint).

The function should look something like this (started in ff529f3):

def get_df_and_metadata_from_db(...):
    # do stuff
    return PANDAS_DATAFRAME, PYTHON_DICT
@matthewcarbone
Copy link
Author

Eli, to add a bit more substance to what I'm saying, I have written another "dummy" function here:

def ingest_from_DataBroker(client, db, pbar=True):
    """Loads in all files matching the provided extension.
    NOTE: this will require some check at some point to ensure unique entries
    from databroker are not rewritten every time this function is called
    Parameters
    ----------
    db
    """

    # !!!PSEUDOCODE!!!!!!!!
    for uid in tqdm(db.uids, disable=not pbar):
        df, metadata = get_df_and_metadata_from_db(db, uid, ...)
        # !!!!!!!!!!!!!!!!!!!!!

        res = _process_df_and_metadata(df, metadata)
        _write_from_res(res, client)

This function should in principle crawl Data Broker and upload every (new) entry into Tiled. Note the line

df, metadata = get_df_and_metadata_from_db(db, uid, ...)

This is your endpoint- i.e. where I need your help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant