[Lake][ETL] Update raw save-to-disk

[idiom-bytes]

Motivation

Parquet can append efficiently w/ a batch of records , but isnt' good for single-row/records. We want to make the process as efficient as possible for now, without adding any additional overhead (i.e. live processing + message queue => dump).

Rather than re-creating a parquet file, simply append rows to the csv.

Outline

This was already partially implemented in the past (check git history).
I do not believe that it was saving out different csvs.
1 - Read last record from last csv
2 - Fetch all new records from subgraph
3 - Every value is validated
4 - *1000 lines per csv (please review feature)

DoD

• Update table.py (and consumers) to save/load to append/load from csv

Final Comments

We reviewed updating ohlcv_factory to integrate CSVDataStore, however the CSV writing + OHLCV part is tightly integrated so it's going to require a bit more time. For now, we left ohlcv_factory as-is, and will create a separate ticket for updating it, such that we can continue separating concerns.

[idiom-bytes]

Code merged into #734 - DuckDB E2E PR.
Closing Ticket.