Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Lake][CLI] Implement lake update commnd #954

Closed
1 task done
idiom-bytes opened this issue Apr 29, 2024 · 3 comments
Closed
1 task done

[Lake][CLI] Implement lake update commnd #954

idiom-bytes opened this issue Apr 29, 2024 · 3 comments
Labels
Type: Enhancement New feature or request

Comments

@idiom-bytes
Copy link
Member

idiom-bytes commented Apr 29, 2024

Background / motivation

Parent epic: #685

Check TODO - Implement "update" command

TODOs / DoD

  • Implement "update" command
@idiom-bytes idiom-bytes added the Type: Enhancement New feature or request label Apr 29, 2024
@KatunaNorbert
Copy link
Member

KatunaNorbert commented May 3, 2024

Is the command supposed to only keep the lake up to date or should update previous rows? I suppose it should only keep the lake updated and should not take any cli params related to dates, it should use the elt logic for that.

@idiom-bytes
Copy link
Member Author

idiom-bytes commented May 7, 2024

Is the command supposed to only keep the lake up to date or should update previous rows? 

"Is the command supposed to only keep the lake up to date"

  • Yes, the command is supposed to keep the lake up-to-date by running the GQLDF + ETL loop
  • Every time it runs, it will start from the latest record inside pdr-predictions => ppss.end_ts

"updating previous rows"

  • Is a concept that exists inside the ETL logic
  • for now, records that were already processed will not be updated
  • we will have "null" entries in records until "incremental updates" are implemented
should not take any cli params related to dates

It should work like how we had previously.

  1. ppss.yaml drives everything
  2. ST + END can be passed in to override ppss.yaml lake_st and lake_end params

@idiom-bytes
Copy link
Member Author

This has been implemented in PR #956

We also decided to not allow for ST + END, so it doesn't take those parameters.
This is now merged into duckdb-integration PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants