Using Kedro with existing project?

[ericmjl]

What are you trying to do?

I'm wondering if it's possible to use Kedro with an existing project that was not created with kedro new? For example, I already have a project that has a data/ directory, a src/ directory and more, and I'd like to start by only using the Pipelining capabilities.

I've been unsuccessful on my first two attempts; I installed Kedro into the project conda environment, but the only commands available to me are docs, info, and new.

[idanov]

@ericmjl It is very likely that most of Kedro's functionality will be available to you even with a custom project folder structure, however you might need to work a bit harder to get it to that point. The main way for Kedro to work is by using the load_context which has a base set of assumptions about your folder structure and all else should be free for modification.

To learn more about how Kedro's internal architecture and equip yourself with the skills needed to build your own framework on top of Kedro or simply use a new folder structure, this page in the documentation can give you a high-level overview and hopefully clarify some bits.

From your experience, I can see that it is possible that your project doesn't have a .kedro.yml file and thus the Kedro CLI cannot recognise your project as a Kedro project. You can find how you need to structure your .kedro.yml file here.

[limdauto]

Hi @ericmjl I'm closing this issue now as it's been a while. But please feel free to reopen it if you still need help.

[fmfreeze]

Hi to all on an old issue.
@idanov your links on more infos are not working anymore.

Is there an example or docs out there on how to integrate kedro into an existing project which I failed to find?
Thanks for any responses.

[astrojuanlu]

@fmfreeze have a look at https://kedro.readthedocs.io/en/stable/faq/architecture_overview.html. The second link does not apply anymore, .kedro.yml was gone in d8e6ace.

You might want to explore how to use some of the Kedro components, for example Kedro as a data registry.

In any case, "how to use Kedro with an existing project" was the very first question I had when I started using the project and I'd like to see us documenting it better. @idanov @stichbury @yetudada what do you think about reopening this issue?

[yetudada]

Hey everyone! We're going to start looking at this in two ways, how do I use Kedro when I have:

• A project that has a data/ directory, a src/ directory and more already e.g. what is the most minimal things I could add to my existing work to turn into a Kedro project
• A Jupyter notebook that I want to turn into a Kedro project

[stichbury]

I'm looking at this as it's the oldest docs ticket we have outstanding.

It seems to be addressed by #2512 which suggests adding a kedro init command (plus documentation) but in the absence of that work, there's still an opportunity to write about how to convert a basic project to Kedro. So I'm keeping this ticket alive and bumping its priority.

There's also #2461 to convert a notebook into a project. I didn't find the draft addressed the reader and suggested a new approach and ticket (kedro-org/kedro-devrel#80) so closed #2461 and things have stalled now. It's still important but not part of this issue, so I'll follow that rabbit down a separate hole.

[astrojuanlu]

If we were to write these docs now, the story would be something like "do kedro new somewhere and copy-paste your files over". It's worth pondering if that's better than nothing.

I'm leaving a comment on gh-2512.

[astrojuanlu]

We have identified this issue as a high priority work stream.

At the moment, our current advice for users who want to convert an existing project to Kedro is "run kedro new and copy-paste the files over". This is less than ideal, is perceived as too complex, and possibly constitutes a barrier of adoption.

Today in a training, after I showed how to do it, the consultant leading the engagement said "I didn’t know it was that involved [...] slightly underestimated the effort associated with setting up a Kedro pipeline".

More evidence of users trying to convert an existing project to Kedro: https://www.linen.dev/s/kedro/t/13221641/hi-everyone-i-wanted-first-to-thank-all-the-participants-of-#29e7e696-5078-4ccc-a28f-907a3f132696

Related issues:

• Run kedro new without creating a new directory #681
• Document workflow to incrementally create a minimal Kedro project from scratch #2512
• More control over folder structure #2553
• Do not fail if conf/local is not present #2593
• Publish a four-part mini-blog series about Notebook issues and how Kedro can resolve them  kedro-devrel#80

and possibly more.

Tangentially related: #2818

[astrojuanlu]

We now have

• Documentation that walks users through how to use Kedro's DataCatalog and OmegaConfigLoader from a notebook without ever using kedro new https://docs.kedro.org/en/latest/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.html
• Agreed to simplify the use of certain Kedro objects as library components Remove the environment default for ConfigLoader #2971 or document alternatives Make it easier to use the Config Loader #2819
• Plans to write more docs on how to go from there to a full-fledged Kedro project Document workflow to incrementally create a minimal Kedro project from scratch #2512

This is a work in progress that will require some API changes and end with documentation improvements. I will wait a bit to close this issue.

[stichbury]

+1 to closing this and thanks for the useful summary 🙌