Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#681 - Update raw save-to-disk #699

Closed
wants to merge 20 commits into from
Closed

#681 - Update raw save-to-disk #699

wants to merge 20 commits into from

Conversation

kdetry
Copy link
Contributor

@kdetry kdetry commented Feb 29, 2024

Fixes #681

Changes proposed in this PR:

  • CSV Data Store class is created
  • implemented to the table.py
  • tests are updated

PS: I didn't touch the OHLCV process because it looks like it has its own system and works properly

DoD

Load from + Write to CSVs...Just append

  • Please create a new function called append
  • After fetching new data, users will append data to a table.
  • Inside of append(), call a new function called append_to_csv. We just want to append a few rows at at time to a csv with as little latency as possible.
  • Inside of append(), call a new function called append_to_duckdb. Comment this out. Address this in the next ticket.

@kdetry kdetry requested a review from idiom-bytes February 29, 2024 13:19
@kdetry kdetry changed the title Issue681 - Update raw save-to-disk #681 - Update raw save-to-disk Mar 1, 2024
@idiom-bytes
Copy link
Member

This PR contains fixes & updates that are required for other PRs to move forward.

If your needs for CSVDataStore won't change @kdetry, then perhaps use this. Otherwise, I recommend you cherry-picking changes from as needed to continue with the other PRs and get the DuckDB Integration PR working.

I believe we should update CSVDataStore and have it "clean/simple".
There are updates required to clean up stuff in here that is part of "DuckDB Integration" PR.

@kdetry
Copy link
Contributor Author

kdetry commented Mar 4, 2024

We will continue on the following PR

#734

@kdetry kdetry closed this Mar 4, 2024
@kdetry kdetry deleted the issue681 branch May 8, 2024 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Lake][ETL] Update raw save-to-disk
2 participants