Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create redshift limit one macro #95

Merged
merged 7 commits into from
Jan 22, 2025
Merged

Conversation

fivetran-avinash
Copy link
Contributor

PR Overview

This PR will address the following Issue/Feature: Internal ticket

This PR will result in the following new package version:

v0.15.0 - staging model results will be different for redshift users potentially (1 row instead of 0 for empty rows)

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Under the Hood

  • (Affects Redshift only) Creates new shopify_union_data macro to accommodate Redshift's treatment of empty tables.
    • For each staging model, if the source table is not found in any of your schemas, the package will create a empty table with 0 rows for non-Redshift warehouses and a table with 1 all-null row for Redshift destinations.
    • This is necessary as Redshift will ignore explicit data casts when a table is completely empty and materialize every column as a varchar. This throws errors in downstream transformations in the shopify package. The 1 row will ensure that Redshift will respect the package's datatype casts.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

  • dbt run –full-refresh && dbt test
  • [NA] dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

  • [NA] The appropriate issue has been linked, tagged, and properly assigned
  • All necessary documentation and version upgrades have been applied
  • docs were regenerated (unless this PR does not include any code or yml updates)
  • BuildKite integration tests are passing
  • [NA] Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

See concurrent transform PR.

If you had to summarize this PR in an emoji, which would it be?

🇪🇺

@fivetran-avinash fivetran-avinash self-assigned this Jan 12, 2025
Copy link
Contributor

@fivetran-joemarkiewicz fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-avinash a few small change requests. Also, can you please provide some validations to show that the union data update is working as expected here. It would be great to see:

  • This properly fills 1 row of nulls when the table isn't present and the destination is Redshift and retains the datatype.
  • This properly fills 0 rows if the table isn't present and the destination is anything other than Redshift.
  • The model works as expected when the table is present and the destination is Redshift and any other warehouse.

CHANGELOG.md Outdated Show resolved Hide resolved
Copy link
Contributor Author

@fivetran-avinash fivetran-avinash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-joemarkiewicz Thanks for the validation call-out. See the ticket for validations. This PR is ready for re-review.

Copy link
Contributor

@fivetran-joemarkiewicz fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@fivetran-catfritz fivetran-catfritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@fivetran-avinash fivetran-avinash merged commit 9456612 into main Jan 22, 2025
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants