Skip to content

Commit

Permalink
Merge branch 'main' into 3741-manually-create-sitemap-for-docs
Browse files Browse the repository at this point in the history
  • Loading branch information
DimedS authored Nov 25, 2024
2 parents b60209e + dfa0b79 commit 7e8ff08
Show file tree
Hide file tree
Showing 56 changed files with 1,271 additions and 535 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/all-checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
strategy:
matrix:
os: [ windows-latest, ubuntu-latest ]
python-version: [ "3.9", "3.10", "3.11", "3.12" ]
python-version: [ "3.9", "3.10", "3.11", "3.12", "3.13" ]
uses: ./.github/workflows/unit-tests.yml
with:
os: ${{ matrix.os }}
Expand All @@ -36,7 +36,7 @@ jobs:
strategy:
matrix:
os: [ windows-latest, ubuntu-latest ]
python-version: [ "3.9", "3.10", "3.11", "3.12" ]
python-version: [ "3.9", "3.10", "3.11", "3.12", "3.13" ]
uses: ./.github/workflows/e2e-tests.yml
with:
os: ${{ matrix.os }}
Expand All @@ -59,7 +59,7 @@ jobs:
strategy:
matrix:
os: [ windows-latest, ubuntu-latest ]
python-version: [ "3.9", "3.10", "3.11", "3.12" ]
python-version: [ "3.9", "3.10", "3.11", "3.12", "3.13" ]
uses: ./.github/workflows/pip-compile.yml
with:
os: ${{ matrix.os }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/check-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
python-version: '3.11'
- name: Install uv
run: |
python -m pip install "uv==0.1.32"
python -m pip install "uv==0.4.29"
- name: Install dependencies
run: |
uv pip install --system requests
Expand Down Expand Up @@ -52,7 +52,7 @@ jobs:
python-version: '3.11'
- name: Install uv
run: |
python -m pip install "uv==0.1.32"
python -m pip install "uv==0.4.29"
- name: Install dependencies
run: |
uv pip install --system build
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docs-only-checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
strategy:
matrix:
os: [ ubuntu-latest ]
python-version: [ "3.9", "3.10", "3.11", "3.12" ]
python-version: [ "3.9", "3.10", "3.11", "3.12", "3.13" ]
uses: ./.github/workflows/lint.yml
with:
os: ${{ matrix.os }}
Expand Down
48 changes: 48 additions & 0 deletions .github/workflows/label-community-issues.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: Label Community Issues

on:
issues:
types:
- opened

jobs:
label:
runs-on: ubuntu-latest
steps:
- name: Check if issue author is a member of Kedro org
uses: actions/github-script@v6
id: membership
with:
github-token: ${{ secrets.GH_TAGGING_TOKEN }}
result-encoding: string
script: |
try {
const result = await github.rest.orgs.getMembershipForUser({
org: "kedro-org",
username: '${{ github.actor }}'
})
console.log(result?.data?.state)
if (result?.data?.state == "active"){
console.log("%s: detected as an active member of Kedro org", '${{ github.actor }}')
return "member";
} else {
console.log("%s: not detected as active member of Kedro org", '${{ github.actor }}')
return "notMember";
}
} catch (error) {
console.log("%s: Error occured and marked user as notMember", '${{ github.actor }}')
console.log("Error", error.stack);
console.log("Error", error.name);
console.log("Error", error.message);
return "notMember";
}
- name: Label issue if author is from community
if: ${{ steps.membership.outputs.result == 'notMember' }}
uses: actions-ecosystem/action-add-labels@v1
with:
github_token: ${{ secrets.GH_TAGGING_TOKEN }}
labels: 'Community'
20 changes: 20 additions & 0 deletions .github/workflows/no-response.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: No Response

on:
issue_comment:
types: [created]
schedule:
# Run every day at 9am (UTC time)
- cron: '0 9 * * *'

jobs:
noResponse:
runs-on: ubuntu-latest
steps:
- uses: lee-dohm/[email protected]
with:
token: ${{ secrets.GITHUB_TOKEN }}
responseRequiredLabel: "support: needs more info"
daysUntilClose: 28
closeComment: >-
This issue has been closed due to lack of information. Feel free to re-open this issue if you're facing a similar problem. Please provide as much information as possible so we can help resolve your issue.
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ default_stages: [pre-commit, manual]

repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.6.1
rev: v0.7.1
hooks:
- id: ruff
name: "ruff on kedro/, tests/ and docs/"
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ package: clean install
python -m pip install build && python -m build

install-test-requirements:
python -m pip install "uv==0.1.32"
python -m pip install "uv==0.4.29"
uv pip install --system "kedro[test] @ ."

install-pre-commit:
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,3 +109,12 @@ If you're an academic, Kedro can also help you, for example, as a tool to solve
## Python version support policy
* The core [Kedro Framework](https://github.com/kedro-org/kedro) supports all Python versions that are actively maintained by the CPython core team. When a [Python version reaches end of life](https://devguide.python.org/versions/#versions), support for that version is dropped from Kedro. This is not considered a breaking change.
* The [Kedro Datasets](https://github.com/kedro-org/kedro-plugins/tree/main/kedro-datasets) package follows the [NEP 29](https://numpy.org/neps/nep-0029-deprecation_policy.html) Python version support policy. This means that `kedro-datasets` generally drops Python version support before `kedro`. This is because `kedro-datasets` has a lot of dependencies that follow NEP 29 and the more conservative version support approach of the Kedro Framework makes it hard to manage those dependencies properly.


## ☕️ Kedro Coffee Chat 🔶

We appreciate our community and want to stay connected. For that, we offer a public Coffee Chat format where we share updates and cool stuff around Kedro once every two weeks and give you time to ask your questions live.

Check out the upcoming demo topics and dates at the [Kedro Coffee Chat wiki page](https://github.com/kedro-org/kedro/wiki/Kedro-Coffee-Chat).

Follow our Slack [announcement channel](https://kedro-org.slack.com/archives/C03RKAQ0MGQ) to see Kedro Coffee Chat announcements and access demo recordings.
9 changes: 8 additions & 1 deletion RELEASE.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,23 @@
# Upcoming Release

## Major features and improvements
* Add official support for Python 3.13.
* Implemented dict-like interface for `KedroDataCatalog`.
* Implemented lazy dataset initializing for `KedroDataCatalog`.
* Project dependencies on both the default template and on starter templates are now explicitly declared on the `pyproject.toml` file, allowing Kedro projects to work with project management tools like `uv`, `pdm`, and `rye`.

**Note:** ``KedroDataCatalog`` is an experimental feature and is under active development. Therefore, it is possible we'll introduce breaking changes to this class, so be mindful of that if you decide to use it already. Let us know if you have any feedback about the ``KedroDataCatalog`` or ideas for new features.

## Bug fixes and other changes
* Added I/O support for Oracle Cloud Infrastructure (OCI) Object Storage filesystem
* Added I/O support for Oracle Cloud Infrastructure (OCI) Object Storage filesystem.
* Fixed `DatasetAlreadyExistsError` for `ThreadRunner` when Kedro project run and using runner separately.

## Breaking changes to the API
## Documentation changes
* Updated CLI autocompletion docs with new Click syntax.
* Standardised `.parquet` suffix in docs and tests.
* Added a new minimal Kedro project creation guide.
* Added example to explain how dataset factories work.

## Community contributions
* [Hyewon Choi](https://github.com/hyew0nChoi)
Expand Down Expand Up @@ -101,6 +107,7 @@ Many thanks to the following Kedroids for contributing PRs to this release:

## Upcoming deprecations for Kedro 0.20.0
* The utility method `get_pkg_version()` is deprecated and will be removed in Kedro 0.20.0.
* `LambdaDataset` is deprecated and will be removed in Kedro 0.20.0.

## Documentation changes
* Improved documentation for configuring dataset parameters in the data catalog
Expand Down
13 changes: 9 additions & 4 deletions asv.conf.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,21 @@
"project": "Kedro",
"project_url": "https://kedro.org/",
"repo": ".",
"install_command": ["pip install -e ."],
"branches": ["main"],
"install_command": [
"pip install -e . kedro-datasets[pandas-csvdataset]"
],
"branches": [
"main"
],
"environment_name": "kedro",
"environment_type": "virtualenv",
"show_commit_url": "http://github.com/kedro-org/kedro/commit/",
"results_dir": ".asv/results",
"benchmark_dir": "kedro_benchmarks",
"html_dir": ".asv/html",
"matrix": {
"req": {
"kedro-datasets": [],
"pandas": []
"kedro-datasets[pandas]": []
}
}
}
16 changes: 0 additions & 16 deletions benchmarks/benchmark_dummy.py

This file was deleted.

12 changes: 6 additions & 6 deletions docs/diagrams/installed-kedro-project.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
```mermaid
sequenceDiagram
title "Installed Kedro project"
title Installed Kedro project
participant script as "third-party Python script"
participant curr_dir as "Directory with Kedro conf/ in it"
participant session as "KedroSession"
participant script as Third-party Python script
participant curr_dir as Directory with Kedro conf/ in it
participant session as KedroSession
script->>script: run third-party script
script->>curr_dir: get path to the project config
script->>session: create a session with Kedro project name and project config dir
session->>run: run a pipeline and/or nodes
script->>session: create a session with Kedro project config dir
session->>session: run a pipeline and/or nodes
```
27 changes: 11 additions & 16 deletions docs/diagrams/kedro-ipython.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,25 @@
```mermaid
sequenceDiagram
title "$ kedro ipython"
title $ kedro ipython
participant cli as "$ kedro ipython"
participant env as "Environment variables"
participant ipython as "IPython"
participant entrypoint as "00-kedro-init.py\nreload_kedro"
participant hook_manager as "Hook manager"
participant project as "Kedro project directory"
participant session as "KedroSession"
participant context as "KedroContext"
participant cli as $ kedro ipython
participant env as Environment variables
participant ipython as IPython
participant entrypoint as ipython/__init__.py <br> reload_kedro
participant project as Kedro project directory
participant session as KedroSession
participant context as KedroContext
cli->>cli: Check if IPython is importable
cli->>env: Set IPYTHONDIR to metadata.project_path / ".ipython"
cli->>env: Set KEDRO_ENV to the chosen Kedro environment
cli->>cli: Print an info message
cli->>ipython: Start ipython
ipython->>entrypoint: load startup script
entrypoint->>entrypoint: import Kedro
entrypoint->>hook_manager: clear the hook manager
ipython->>entrypoint: load ipython extension
entrypoint->>project: find Kedro project
entrypoint->>project: bootstrap the project
entrypoint->>entrypoint: remove imported project package modules
entrypoint->>session: create a KedroSession
entrypoint->>session: activate the session
entrypoint->>session: load KedroContext
entrypoint->>context: get the data catalog
entrypoint->>entrypoint: expose session, context and catalog variables
entrypoint->>entrypoint: expose session, context, catalog and pipelines variables
entrypoint->>entrypoint: register reload_kedro line magic
```
16 changes: 8 additions & 8 deletions docs/diagrams/kedro-no-project.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
```mermaid
sequenceDiagram
title "$ kedro\ndirectory without Kedro project"
title $ kedro in directory without Kedro project
participant kedro as "$ kedro"
participant entrypoint as "setup.py\nkedro = kedro.framework.cli:main"
participant init_plugins as "Kedro Plugins\nentry_point = kedro.init"
participant kedro_cli as "Kedro CLI\nglobal commands\ninfo, new, docs, starter"
participant global_plugins as "Kedro Plugins\nentry_point = kedro.global_commands"
participant pyproject.toml as "Current directory\npyproject.toml"
participant click as "Click"
participant kedro as $ kedro
participant entrypoint as pyproject.toml <br> kedro = "kedro.framework.cli:main"
participant init_plugins as Kedro Plugins <br> [project.entry-points."kedro.init"]
participant kedro_cli as Kedro CLI <br> global commands <br> info, new, docs, starter
participant global_plugins as Kedro Plugins <br> [project.entry-points."kedro.global_commands"]
participant pyproject.toml as Current directory <br> pyproject.toml
participant click as Click
kedro->>entrypoint: Python calls this
Expand Down
14 changes: 7 additions & 7 deletions docs/diagrams/kedro-plugin.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
```mermaid
sequenceDiagram
title "$ kedro plugin"
title $ kedro plugin
participant cli as "$ kedro plugin"
participant prelude as "See kedro-with-project.puml for details"
participant project_plugin as "Kedro Plugin\nentry_point = kedro.project_commands"
participant click as "Click context"
participant session as "KedroSession"
participant cli as $ kedro plugin
participant prelude as See kedro-with-project.md for details
participant project_plugins as Kedro Plugins <br> [project.entry-points."kedro.project_commands"]
participant session as KedroSession
participant kedro_cli as KedroCLI
cli->>prelude: prepare click commands as prelude to this
prelude->>project_plugin: execute plugin click command
project_plugin->>click: get ProjectMetadata from the click context
project_plugin->>kedro_cli: get ProjectMetadata from the KedroCLI command collection group
project_plugin->>project_plugin: plugin code
project_plugin->>session: need to create KedroSession for all runtime config and info
```
22 changes: 11 additions & 11 deletions docs/diagrams/kedro-run.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
```mermaid
sequenceDiagram
title "$ kedro run"
title $ kedro run
participant cli as "$ kedro run"
participant prelude as "See kedro-with-project.puml for details"
participant project_cli as "Project directory\ncli.py"
participant session as "KedroSession"
participant context as "KedroContext"
participant runner as "Runner"
participant hooks as "Hook manager"
participant cli as $ kedro run
participant prelude as See kedro-with-project.md for details
participant project_cli as Project directory <br> cli.py
participant session as KedroSession
participant context as KedroContext
participant runner as Runner
participant hooks as Hook manager
cli->>prelude: prepare click commands as prelude to this
prelude->>project_cli: run
project_cli->>session: create KedroSession
session->>hooks: initialise hook manager
session->>session: run
session->>session: load KedroContext
session->>context: get the selected pipeline
context->>context: filter the pipeline based on command line arguments
session->>session: get the selected pipeline
session->>session: filter the pipeline based on command line arguments
session->>context: get catalog with load version / save version
session->>runner: create runner
session->>hooks: get hook manager
hooks->>hooks: before_pipeline_run
runner->>runner: run the filtered pipeline with the catalog
hooks->>hooks: on_pipeline_error (if runner fails)
Expand Down
Loading

0 comments on commit 7e8ff08

Please sign in to comment.