diff --git a/CHANGELOG.md b/CHANGELOG.md index 15d9b1df..cf3dc1da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,7 +29,7 @@ All notable changes to this project will be documented in this file. - Cleaned up argument passing and other repetitive code using dictionary and set comparisons for easy maintenance. (Converted variable names to a consistent snake_case from a mix of kebab-case and snake_case) - Fixed change history table processing to allow mixed case names when '"' are used in the name. - Moved most error, log and warning messages and query strings to global or class variables. -- Updated readme to cover new authentication methods +- Updated readme to cover new authentication methods ## [3.4.2] - 2022-10-24 ### Changed diff --git a/NOTICE b/NOTICE index efed0aae..ec1a72af 100644 --- a/NOTICE +++ b/NOTICE @@ -1,10 +1,9 @@ -This software includes the following python packages: - - Name License Author URL - Jinja2 BSD License Armin Ronacher https://palletsprojects.com/p/jinja/ - PyYAML MIT License Kirill Simonov https://pyyaml.org/ - pandas BSD License The Pandas Development Team https://pandas.pydata.org +This software includes the following python packages: + + Name License Author URL + Jinja2 BSD License Armin Ronacher https://palletsprojects.com/p/jinja/ + PyYAML MIT License Kirill Simonov https://pyyaml.org/ + pandas BSD License The Pandas Development Team https://pandas.pydata.org pytest MIT License Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt, https://docs.pytest.org/en/latest/ - Floris Bruynooghe, Brianna Laugher, Florian Bruhin and others - snowflake-connector-python Apache Software License Snowflake, Inc https://www.snowflake.com/ - \ No newline at end of file + Floris Bruynooghe, Brianna Laugher, Florian Bruhin and others + snowflake-connector-python Apache Software License Snowflake, Inc https://www.snowflake.com/ diff --git a/README.md b/README.md index ffca6c9c..68b33d88 100644 --- a/README.md +++ b/README.md @@ -228,7 +228,7 @@ Default [Password](https://docs.snowflake.com/en/user-guide/python-connector-exa If an authenticator is unsupported, then schemachange will default to `snowflake`. If the authenticator is `snowflake`, and both password and key pair values are provided then schemachange will use the password over the key pair values. ### Password Authentication -The Snowflake user password for `SNOWFLAKE_USER` is required to be set in the environment variable `SNOWFLAKE_PASSWORD` prior to calling the script. schemachange will fail if the `SNOWFLAKE_PASSWORD` environment variable is not set. The environment variable `SNOWFLAKE_AUTHENTICATOR` will be set to `snowflake` if it not explicitly set. +The Snowflake user password for `SNOWFLAKE_USER` is required to be set in the environment variable `SNOWFLAKE_PASSWORD` prior to calling the script. schemachange will fail if the `SNOWFLAKE_PASSWORD` environment variable is not set. The environment variable `SNOWFLAKE_AUTHENTICATOR` will be set to `snowflake` if it not explicitly set. _**DEPRECATION NOTICE**: The `SNOWSQL_PWD` environment variable is deprecated but currently still supported. Support for it will be removed in a later version of schemachange. Please use `SNOWFLAKE_PASSWORD` instead._ @@ -242,20 +242,20 @@ The URL of the authenticator resource that will be receive the POST request. * token-response-name The Expected name of the JSON element containing the Token in the return response from the authenticator resource. * token-request-payload -The Set of variables passed as a dictionary to the `data` element of the request. +The Set of variables passed as a dictionary to the `data` element of the request. * token-request-headers -The Set of variables passed as a dictionary to the `headers` element of the request. +The Set of variables passed as a dictionary to the `headers` element of the request. -It is recomended to use the YAML file and pass oauth secrets into the configuration using the templating engine instead of the command line option. +It is recomended to use the YAML file and pass oauth secrets into the configuration using the templating engine instead of the command line option. ### External Browser Authentication -External browser authentication can be used for local development by setting the environment variable `SNOWFLAKE_AUTHENTICATOR` to the value `externalbrowser` prior to calling schemachange. +External browser authentication can be used for local development by setting the environment variable `SNOWFLAKE_AUTHENTICATOR` to the value `externalbrowser` prior to calling schemachange. The client will be prompted to authenticate in a browser that pops up. Refer to the [documentation](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-use.html#setting-up-browser-based-sso) to cache the token to minimize the number of times the browser pops up to authenticate the user. ### Okta Authentication -For clients that do not have a browser, can use the popular SaaS Idp option to connect via Okta. This will require the Okta URL that you utilize for SSO. -Okta authentication can be used setting the environment variable `SNOWFLAKE_AUTHENTICATOR` to the value of your okta endpoint as a fully formed URL ( E.g. `https://.okta.com`) prior to calling schemachange. +For clients that do not have a browser, can use the popular SaaS Idp option to connect via Okta. This will require the Okta URL that you utilize for SSO. +Okta authentication can be used setting the environment variable `SNOWFLAKE_AUTHENTICATOR` to the value of your okta endpoint as a fully formed URL ( E.g. `https://.okta.com`) prior to calling schemachange. _** NOTE**: Please disable Okta MFA for the user who uses Native SSO authentication with client drivers. Please consult your Okta administrator for more information._ @@ -330,14 +330,14 @@ dry-run: false # A string to include in the QUERY_TAG that is attached to every SQL statement executed query-tag: 'QUERY_TAG' -# Information for Oauth token requests +# Information for Oauth token requests oauthconfig: # url Where token request are posted to token-provider-url: 'https://login.microsoftonline.com/{{ env_var('AZURE_ORG_GUID', 'default') }}/oauth2/v2.0/token' # name of Json entity returned by request token-response-name: 'access_token' # Headers needed for successful post or other security markings ( multiple labeled items permitted - token-request-headers: + token-request-headers: Content-Type: "application/x-www-form-urlencoded" User-Agent: "python/schemachange" # Request Payload for Token (it is recommended pass diff --git a/demo/citibike_jinja/modules/create_stage.j2 b/demo/citibike_jinja/modules/create_stage.j2 index f4704881..44750ab6 100644 --- a/demo/citibike_jinja/modules/create_stage.j2 +++ b/demo/citibike_jinja/modules/create_stage.j2 @@ -1,4 +1,4 @@ {% macro create_stage(stage_name, URL) -%} CREATE OR REPLACE STAGE {{stage_name}} URL = '{{URL}}'; -{%- endmacro %} \ No newline at end of file +{%- endmacro %} diff --git a/demo/citibike_jinja/schemachange-config.yml b/demo/citibike_jinja/schemachange-config.yml index c8c5c280..0a028a20 100644 --- a/demo/citibike_jinja/schemachange-config.yml +++ b/demo/citibike_jinja/schemachange-config.yml @@ -8,4 +8,3 @@ vars: # not a good example of secrets, just here to demo the secret filtering trips_s3_bucket: s3://snowflake-workshop-lab/citibike-trips weather_s3_bucket: s3://snowflake-workshop-lab/weather-nyc - diff --git a/pyproject.toml b/pyproject.toml index 4ba14420..9ac9b912 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -3,4 +3,4 @@ requires = [ "setuptools >= 40.9.0", "wheel", ] -build-backend = "setuptools.build_meta" \ No newline at end of file +build-backend = "setuptools.build_meta" diff --git a/requirements.txt b/requirements.txt index 5a40fa58..5e8d0440 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -snowflake-connector-python>=2.8,<4.0 +Jinja2~=3.0 pandas~=1.3 PyYAML~=6.0 -Jinja2~=3.0 +snowflake-connector-python>=2.8,<4.0 diff --git a/schemachange/cli.py b/schemachange/cli.py index 57fb6914..5221194d 100644 --- a/schemachange/cli.py +++ b/schemachange/cli.py @@ -19,7 +19,7 @@ from jinja2.loaders import BaseLoader from pandas import DataFrame -#region Global Variables +#region Global Variables # metadata _schemachange_version = '3.5.4' _config_file_name = 'schemachange-config.yml' @@ -50,7 +50,7 @@ + "{snowflake_role}\nUsing default warehouse {snowflake_warehouse}\nUsing default " \ + "database {snowflake_database}" _log_ch_use = "Using change history table {database_name}.{schema_name}.{table_name} " \ - + "(last altered {last_altered})" + + "(last altered {last_altered})" _log_ch_create = "Created change history table {database_name}.{schema_name}.{table_name}" _err_ch_missing = "Unable to find change history table {database_name}.{schema_name}.{table_name}" _log_ch_max_version = "Max applied change script version: {max_published_version_display}" @@ -58,17 +58,17 @@ + "applied change ({max_published_version})" _log_skip_r ="Skipping change script {script_name} because there is no change since the last " \ + "execution" -_log_apply = "Applying change script {script_name}" +_log_apply = "Applying change script {script_name}" _log_apply_set_complete = "Successfully applied {scripts_applied} change scripts (skipping " \ - + "{scripts_skipped}) \nCompleted successfully" + + "{scripts_skipped}) \nCompleted successfully" _err_vars_config = "vars did not parse correctly, please check its configuration" _err_vars_reserved = "The variable schemachange has been reserved for use by schemachange, " \ + "please use a different name" _err_invalid_folder = "Invalid {folder_type} folder: {path}" _err_dup_scripts = "The script name {script_name} exists more than once (first_instance " \ - + "{first_path}, second instance {script_full_path})" + + "{first_path}, second instance {script_full_path})" _err_dup_scripts_version = "The script version {script_version} exists more than once " \ - + "(second instance {script_full_path})" + + "(second instance {script_full_path})" _err_invalid_cht = 'Invalid change history table name: %s' _log_auth_type ="Proceeding with %s authentication" _log_pk_enc ="No private key passphrase provided. Assuming the key is not encrypted." @@ -232,7 +232,7 @@ def __init__(self, config): self.autocommit = config['autocommit'] self.verbose = config['verbose'] if self.set_connection_args(): - self.con = snowflake.connector.connect(**self.conArgs) + self.con = snowflake.connector.connect(**self.conArgs) if not self.autocommit: self.con.autocommit(False) else: @@ -268,7 +268,7 @@ def set_connection_args(self): default_authenticator = 'snowflake' if os.getenv("SNOWFLAKE_PASSWORD") is not None and os.getenv("SNOWFLAKE_PASSWORD"): snowflake_password = os.getenv("SNOWFLAKE_PASSWORD") - + # Check legacy/deprecated env variable if os.getenv("SNOWSQL_PWD") is not None and os.getenv("SNOWSQL_PWD"): if snowflake_password: @@ -276,15 +276,15 @@ def set_connection_args(self): else: warnings.warn(_warn_password, DeprecationWarning) snowflake_password = os.getenv("SNOWSQL_PWD") - + snowflake_authenticator = os.getenv("SNOWFLAKE_AUTHENTICATOR") if snowflake_authenticator: # Determine the type of Authenticator # OAuth based authentication - if snowflake_authenticator.lower() == 'oauth': + if snowflake_authenticator.lower() == 'oauth': oauth_token = self.get_oauth_token() - + if self.verbose: print( _log_auth_type % 'Oauth Access Token') self.conArgs['token'] = oauth_token @@ -296,7 +296,7 @@ def set_connection_args(self): print(_log_auth_type % 'External Browser') # IDP based Authentication, limited to Okta elif snowflake_authenticator.lower()[:8]=='https://': - + if self.verbose: print(_log_auth_type % 'Okta') print(_log_okta_ep % snowflake_authenticator) @@ -311,7 +311,7 @@ def set_connection_args(self): if self.verbose: print(_err_unsupported_auth_mthd.format(unsupported_authenticator=snowflake_authenticator) ) self.conArgs['authenticator'] = default_authenticator - else: + else: # default authenticator to snowflake self.conArgs['authenticator'] = default_authenticator @@ -348,7 +348,7 @@ def set_connection_args(self): self.conArgs['private_key'] = pkb else: raise NameError(_err_no_auth_mthd) - + return True def execute_snowflake_query(self, query): @@ -471,14 +471,14 @@ def apply_change_script(self, script, script_content, change_history_table): # Compose and execute the insert statement to the log file query = self._q_ch_log.format(**frmt_args) self.execute_snowflake_query(query) - + def deploy_command(config): # Make sure we have the required connection info, all of the below needs to be present. req_args = set(['snowflake_account','snowflake_user','snowflake_role','snowflake_warehouse']) provided_args = {k:v for (k,v) in config.items() if v} - missing_args = req_args -provided_args.keys() - if len(missing_args)>0: + missing_args = req_args -provided_args.keys() + if len(missing_args)>0: raise ValueError(_err_args_missing % ', '.join({s.replace('_', ' ') for s in missing_args})) #ensure an authentication method is specified / present. one of the below needs to be present. @@ -585,7 +585,7 @@ def render_command(config, script_path): # Validate the script file path script_path = os.path.abspath(script_path) if not os.path.isfile(script_path): - raise ValueError(_err_invalid_folder.format(folder_type='script_path', path=script_path)) + raise ValueError(_err_invalid_folder.format(folder_type='script_path', path=script_path)) # Always process with jinja engine jinja_processor = JinjaTemplateProcessor(project_root = config['root_folder'], \ modules_folder = config['modules_folder']) @@ -662,10 +662,10 @@ def get_schemachange_config(config_file_path, root_folder, modules_folder, snowf # Validate folder paths if 'root_folder' in config: - config['root_folder'] = os.path.abspath(config['root_folder']) + config['root_folder'] = os.path.abspath(config['root_folder']) if not os.path.isdir(config['root_folder']): raise ValueError(_err_invalid_folder.format(folder_type='root', path=config['root_folder'])) - + if config['modules_folder']: config['modules_folder'] = os.path.abspath(config['modules_folder']) if not os.path.isdir(config['modules_folder']): @@ -743,7 +743,7 @@ def get_all_scripts_recursively(root_directory, verbose): # Throw an error if the same version exists more than once if script_type == 'V': if script['script_version'] in all_versions: - raise ValueError(_err_dup_scripts_version.format(**script)) + raise ValueError(_err_dup_scripts_version.format(**script)) all_versions.append(script['script_version']) return all_files @@ -751,22 +751,22 @@ def get_all_scripts_recursively(root_directory, verbose): def get_change_history_table_details(change_history_table): # Start with the global defaults details = dict() - details['database_name'] = _metadata_database_name - details['schema_name'] = _metadata_schema_name - details['table_name'] = _metadata_table_name + details['database_name'] = _metadata_database_name + details['schema_name'] = _metadata_schema_name + details['table_name'] = _metadata_table_name # Then override the defaults if requested. The name could be in one, two or three part notation. if change_history_table is not None: table_name_parts = change_history_table.strip().split('.') - if len(table_name_parts) == 1: - details['table_name'] = table_name_parts[0] + if len(table_name_parts) == 1: + details['table_name'] = table_name_parts[0] elif len(table_name_parts) == 2: - details['table_name'] = table_name_parts[1] - details['schema_name'] = table_name_parts[0] + details['table_name'] = table_name_parts[1] + details['schema_name'] = table_name_parts[0] elif len(table_name_parts) == 3: - details['table_name'] = table_name_parts[2] - details['schema_name'] = table_name_parts[1] - details['database_name'] = table_name_parts[0] + details['table_name'] = table_name_parts[2] + details['schema_name'] = table_name_parts[1] + details['database_name'] = table_name_parts[0] else: raise ValueError(_err_invalid_cht % change_history_table) #if the object name does not include '"' raise to upper case on return @@ -885,7 +885,7 @@ def main(argv=sys.argv): if args.subcommand == 'render': render_command(config, args.script) else: - deploy_command(config) + deploy_command(config) if __name__ == "__main__": main() diff --git a/tests/test_JinjaEnvVar.py b/tests/test_JinjaEnvVar.py index 1df1e399..d67777a5 100644 --- a/tests/test_JinjaEnvVar.py +++ b/tests/test_JinjaEnvVar.py @@ -18,7 +18,7 @@ def test_env_var_with_no_default_and_no_environmental_variables_should_raise_exc @mock.patch.dict(os.environ, {}, clear=True) def test_env_var_with_default_and_no_environmental_variables_should_return_default(): - + print(os.environ) assert ('SF_DATABASE' in os.environ) is False @@ -28,13 +28,13 @@ def test_env_var_with_default_and_no_environmental_variables_should_return_defau @mock.patch.dict(os.environ, {"SF_DATABASE": "SCHEMACHANGE_DEMO_2"}, clear=True) def test_env_var_with_default_and_environmental_variables_should_return_environmental_variable_value(): - + result = JinjaEnvVar.env_var('SF_DATABASE', 'SCHEMACHANGE_DEMO') assert result == 'SCHEMACHANGE_DEMO_2' @mock.patch.dict(os.environ, {"SF_DATABASE": "SCHEMACHANGE_DEMO_3"}, clear=True) def test_JinjaEnvVar_with_jinja_template(): - + template = jinja2.Template("{{env_var('SF_DATABASE', 'SCHEMACHANGE_DEMO')}}", extensions=[JinjaEnvVar]) assert template.render() == "SCHEMACHANGE_DEMO_3" diff --git a/tests/test_JinjaTemplateProcessor.py b/tests/test_JinjaTemplateProcessor.py index 7c608f54..f6bcb1c5 100644 --- a/tests/test_JinjaTemplateProcessor.py +++ b/tests/test_JinjaTemplateProcessor.py @@ -49,13 +49,13 @@ def test_JinjaTemplateProcessor_render_simple_string_expecting_variable(): def test_JinjaTemplateProcessor_render_from_subfolder(tmp_path: pathlib.Path): root_folder = tmp_path / "MORE2" - + root_folder.mkdir() script_folder = root_folder/ "SQL" script_folder.mkdir() script_file = script_folder / "1.0.0_my_test.sql" script_file.write_text("Hello world!") - + processor = JinjaTemplateProcessor(str(root_folder), None) template_path = processor.relpath(str(script_file)) diff --git a/tests/test_SecretManager.py b/tests/test_SecretManager.py index 76ab169a..2ed70acb 100644 --- a/tests/test_SecretManager.py +++ b/tests/test_SecretManager.py @@ -80,4 +80,4 @@ def test_SecretManager_global_redact(): sm.add("Hello") SecretManager.set_global_manager(sm) - assert SecretManager.global_redact("Hello World!") == "***** World!" \ No newline at end of file + assert SecretManager.global_redact("Hello World!") == "***** World!" diff --git a/tests/test_get_all_scripts_recursively.py b/tests/test_get_all_scripts_recursively.py index ec1500bf..2d3baf60 100644 --- a/tests/test_get_all_scripts_recursively.py +++ b/tests/test_get_all_scripts_recursively.py @@ -278,4 +278,4 @@ def test_get_all_scripts_recursively__given_same_Repeatable_file_with_and_withou result = get_all_scripts_recursively("scripts", False) assert str(e.value).startswith( "The script name R__intial.sql exists more than once (first_instance " - ) \ No newline at end of file + ) diff --git a/tests/test_load_schemachange_config.py b/tests/test_load_schemachange_config.py index 01c1f3fe..04ad50bd 100644 --- a/tests/test_load_schemachange_config.py +++ b/tests/test_load_schemachange_config.py @@ -13,12 +13,12 @@ def test__load_schemachange_config__simple_config_file(tmp_path: pathlib.Path): config-version: 1 root-folder: scripts modules-folder: modules -vars: - database_name: SCHEMACHANGE_DEMO_JINJA +vars: + database_name: SCHEMACHANGE_DEMO_JINJA """ config_file = tmp_path / "schemachange-config.yml" config_file.write_text(config_contents) - + config = load_schemachange_config(str(config_file)) @@ -35,15 +35,15 @@ def test__load_schemachange_config__with_env_var_should_populate_value(tmp_path: config-version: 1.1 root-folder: {{env_var('TEST_VAR')}} modules-folder: modules -vars: - database_name: SCHEMACHANGE_DEMO_JINJA +vars: + database_name: SCHEMACHANGE_DEMO_JINJA """ config_file = tmp_path / "schemachange-config.yml" config_file.write_text(config_contents) config = load_schemachange_config(str(config_file)) - - assert config['root-folder'] == 'env_value' + + assert config['root-folder'] == 'env_value' def test__load_schemachange_config__requiring_env_var_but_env_var_not_set_should_raise_exception(tmp_path: pathlib.Path): @@ -52,13 +52,12 @@ def test__load_schemachange_config__requiring_env_var_but_env_var_not_set_should config-version: 1.1 root-folder: {{env_var('TEST_VAR')}} modules-folder: modules -vars: - database_name: SCHEMACHANGE_DEMO_JINJA +vars: + database_name: SCHEMACHANGE_DEMO_JINJA """ config_file = tmp_path / "schemachange-config.yml" config_file.write_text(config_contents) - + with pytest.raises(ValueError) as e: - config = load_schemachange_config(str(config_file)) + config = load_schemachange_config(str(config_file)) assert str(e.value) == "Could not find environmental variable TEST_VAR and no default value was provided" - diff --git a/tests/test_main.py b/tests/test_main.py index 67e400ad..7865b166 100644 --- a/tests/test_main.py +++ b/tests/test_main.py @@ -8,7 +8,7 @@ import schemachange.cli DEFAULT_CONFIG = { - 'root_folder': os.path.abspath('.'), + 'root_folder': os.path.abspath('.'), 'modules_folder': None, 'snowflake_account': None, 'snowflake_user': None,