Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add query write back capability to Python plugin API #25759

Open
pauldix opened this issue Jan 7, 2025 · 0 comments
Open

Add query write back capability to Python plugin API #25759

pauldix opened this issue Jan 7, 2025 · 0 comments
Labels

Comments

@pauldix
Copy link
Member

pauldix commented Jan 7, 2025

One of the common use cases for plugins is to do different kinds of downsampling into other tables. We should make this as easy and performant as possible with a special API. This will build on the work of #25757.

The Python side will look like:

def query_writeback(self, query: str, target_table: str, tags: List[str], parameters: : Optional[Dict[str, Any]] = None, query_database: Optional[str], target_database: Optional[str]) -> WriteBackResult

class WriteBackResult:
   def __init__(self, row_count: int, validation_errors: List[str], elapsed_time_ms: int):
       self.row_count = row_count
       self.validation_errors = validation_errors 
       self.elapsed_time_ms = elapsed_time_ms

The query and target databases default to whatever database the plugin trigger is tied to.

The Rust side should ensure that the query doesn't pull from the same table as the target, if the query and target databases are the same. It should execute the query and construct a WriteBatch from the results, updating the Catalog and writing the batch into the normal WAL write path.

@pauldix pauldix added the v3 label Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant