Bug 1922986 - Create script that exports BMO data as JSON suitable for import into a BigQuery instance in GCP #2354
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…r import into a BigQuery instance in GCP
- Fixed up docker-compose to use volumes instead of local directories for services - Fixed a couple of unrelated undefined warnings in two places
|
Closing to create a new pull request with cleaner changes. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request adds a new script and test suite changes to support exporting bug data from the DB to BigQuery. When this is merged, I will file a new PR to have a cron container run once daily to call export_bmo_etl.pl and import a snapshot of the bug data for that day. Each row of each table will have the same timestamp. The test suite does some basic tests against a bigquery emulator that does not require credentials. We have a bigquery instances already setup for production and dev and I am running a full import from bugzilla-dev currently to see how long it will take.