Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow running tasks on active schema (Big DANGER mode disclaimer) #192

Open
windiana42 opened this issue May 7, 2024 · 0 comments
Open
Labels
enhancement New feature or request speed usability

Comments

@windiana42
Copy link
Member

Sometimes schema swapping is causing large friction. Assume large tables were produced in a stage and we just want to replace one of them. Even the cache valid situation has to copy the content of a table to the temp schema if the stage is not 100% cache valid. #167 already addresses the problem that currently, work done in the temp schema is lost in case the stage code needs to be adjusted before schema can be swapped.

This issue has the idea that a flow run could be configured to work directly in the active schema. For example flow.run([task], in_active=True) could produce a table next to the target table and then swap tables by renaming (similar to the *__copy technique already implemented). This feature should output a big warning since transactional properties of the stage will not be working properly. Also @input_stage_versions tasks will have problems to cope with this run mode.

@windiana42 windiana42 added enhancement New feature or request usability speed labels May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request speed usability
Projects
None yet
Development

No branches or pull requests

1 participant