[DataCatalog]: Add functionality to search datasets in the catalog

[ElenaKhaustova]

Description

Users struggle to find datasets within the catalog, particularly when dealing with a large number of datasets. They express the need for search features to facilitate dataset discovery.

Context

"As a user in my list object, I can filter by name but I can't filter by what. So it would be good to be able to say give me all the sql datasets and then the names of the tables that are attached."

Comment form @astrojuanlu: Kedro Viz has an item in their roadmap to include a table view of all the metadata, could help with this.

Possible Implementation

Integrate search functionality into the catalog, enabling users to search for datasets based on keywords, patterns and by kind. Include support for regex search to accommodate users with advanced search requirements.

[datajoely]

Also search by kind - if I wanted to find all Parquet files today I'd have to get very creative. Retrieving paths associated with those would be super complicated.

[astrojuanlu]

@stephkaiser do you remember if we already opened an issue or discussion about the "metadata table view"?

(cc @rashidakanchwala for when you're back)

[yury-fedotov]

... enabling users to search for datasets based on keywords or patterns...

Isn't it what catalog.list() already does?

IIRC if you do e.g. catalog.load("compani"), it would raise an error with did you mean one of ["companies", "processed_companies"]?

[astrojuanlu]

Related #3312

[stephkaiser]

@stephkaiser do you remember if we already opened an issue or discussion about the "metadata table view"?

(cc @rashidakanchwala for when you're back)

@astrojuanlu we currently don't have an issue for this, I believe it was an idea we discussed when discussing this issue kedro-org/kedro-viz#1635

[astrojuanlu]

Notice that catalog.list() supports RegEx, see #3924

[astrojuanlu]

But

Integrate search functionality into the catalog, enabling users to search for datasets based on keywords, patterns and by kind. Include support for regex search to accommodate users with advanced search requirements.

this is a bit more advanced, I'd say

[merelcht]

When you say "search datasets in the catalog", what workflow are you talking about? Is this inside a notebook, on the CLI, directly in the IDE or on Kedro-Viz? Each of these user flows might have a different preferred solution.

[ElenaKhaustova]

After the discussion at backlog grooming, we've decided to:

1. Revisit the docs on filtering with regex - to check if they're clear enough (as this feature exists but users were not able to find it);
2. Revisit our filtering approach and decide on whether we want to extend it for example with "search by kind" method/ other (?)

[Galileo-Galilei]

(Just dropping a comment now for when this is ready to be tackled properly)

When implementing the dict like interface in #4218, several possibilities were proposed to filter on the values with a regex, ordered after by order of apparition in the PR:

1. ❌ enable to filter on KedroDataCatalog.keys(regex=...). the original proposition suggested the same interface for values and items but this was considered as confusing because for these 2 methods the filter would apply on the keys.
2. ❌ add a KedroDataCatalog.filter(regex=...) method
3. ✅ : keep the current KedroDataCatalog.list() method for regex filtering

Option 1 was overall considered as interesting, but several of us express concerns that it is not consistent with standard dict interface, hence it would affect discoverability.

Option 2 tends to be the leading choice, but option 3 jumped back a couple of minutes before merging after @idanov 's comment that it may be confusing because it's not clear what we are filtering on. I think we did not think enough about the implications and I want to reconsider it before the official release of KedroDataCatalog. I think the currently implemented option 3 is actually the worst of all three for several reasons :

• it has existed for a long time, and we've had a lot of evidence during the user interviews that people (including myself 😉) were not aware about it, so there's a clear signal we need to change something
• with the new implementation, we have list(catalog) and catalog.list() which do almost (but not exactly) the same thing. I think this will increase confusion about the method, because I'd bet we will document the two methods in different places and hurts discoverability.
• I personnally find less explicit that .list() is for filtering / searching that filter

I think it's worth rediscuting / voting for this specific method, and not consider #3931 done at this point.

PS : Overall this new implementation is 🔥, I just want to speak now instead "forever until a new major version holds my peace"

[ElenaKhaustova]

(Just dropping a comment now for when this is ready to be tackled properly)

When implementing the dict like interface in #4218, several possibilities were proposed to filter on the values with a regex, ordered after by order of apparition in the PR:

1. ❌ enable to filter on KedroDataCatalog.keys(regex=...). the original proposition suggested the same interface for values and items but this was considered as confusing because for these 2 methods the filter would apply on the keys.
2. ❌ add a KedroDataCatalog.filter(regex=...) method
3. ✅ : keep the current KedroDataCatalog.list() method for regex filtering

Option 1 was overall considered as interesting, but several of us express concerns that it is not consistent with standard dict interface, hence it would affect discoverability.

Option 2 tends to be the leading choice, but option 3 jumped back a couple of minutes before merging after @idanov 's comment that it may be confusing because it's not clear what we are filtering on. I think we did not think enough about the implications and I want to reconsider it before the official release of KedroDataCatalog. I think the currently implemented option 3 is actually the worst of all three for several reasons :

• it has existed for a long time, and we've had a lot of evidence during the user interviews that people (including myself 😉) were not aware about it, so there's a clear signal we need to change something
• with the new implementation, we have list(catalog) and catalog.list() which do almost (but not exactly) the same thing. I think this will increase confusion about the method, because I'd bet we will document the two methods in different places and hurts discoverability.
• I personnally find less explicit that .list() is for filtering / searching that filter

I think it's worth rediscuting / voting for this specific method, and not consider #3931 done at this point.

PS : Overall this new implementation is 🔥, I just want to speak now instead "forever until a new major version holds my peace"

Thanks for the summary, I do agree with you 🙂

For now, we agreed to think about the renaming when we work on this ticket. We wanted the name to be aligned with the filtering we will suggest, so we decided to keep the old name in the #4218