chore: lint whitespace

CHANGELOG.md

@@ -29,7 +29,7 @@ All notable changes to this project will be documented in this file.
 - Cleaned up argument passing and other repetitive code using dictionary and set comparisons for easy maintenance. (Converted variable names to a consistent snake_case from a mix of kebab-case and snake_case)
 - Fixed change history table processing to allow mixed case names when '"' are used in the name.
 - Moved most error, log and warning messages and query strings to global or class variables.
-- Updated readme to cover new authentication methods 
+- Updated readme to cover new authentication methods
 
 ## [3.4.2] - 2022-10-24
 ### Changed

NOTICE

@@ -1,10 +1,9 @@
-This software includes the following python packages: 
- 
- Name                        License                  Author                                                              URL                                              
- Jinja2                      BSD License              Armin Ronacher                                                      https://palletsprojects.com/p/jinja/             
- PyYAML                      MIT License              Kirill Simonov                                                      https://pyyaml.org/                              
- pandas                      BSD License              The Pandas Development Team                                         https://pandas.pydata.org                        
+This software includes the following python packages:
+
+ Name                        License                  Author                                                              URL
+ Jinja2                      BSD License              Armin Ronacher                                                      https://palletsprojects.com/p/jinja/
+ PyYAML                      MIT License              Kirill Simonov                                                      https://pyyaml.org/
+ pandas                      BSD License              The Pandas Development Team                                         https://pandas.pydata.org
  pytest                      MIT License              Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt,                  https://docs.pytest.org/en/latest/
-                                                      Floris Bruynooghe, Brianna Laugher, Florian Bruhin and others  
- snowflake-connector-python  Apache Software License  Snowflake, Inc                                                      https://www.snowflake.com/                       
-
+                                                      Floris Bruynooghe, Brianna Laugher, Florian Bruhin and others
+ snowflake-connector-python  Apache Software License  Snowflake, Inc                                                      https://www.snowflake.com/

README.md

@@ -228,7 +228,7 @@ Default [Password](https://docs.snowflake.com/en/user-guide/python-connector-exa
 If an authenticator is unsupported, then schemachange will default to `snowflake`. If the authenticator is `snowflake`, and both password and key pair values are provided then schemachange will use the password over the key pair values.
 
 ### Password Authentication
-The Snowflake user password for `SNOWFLAKE_USER` is required to be set in the environment variable `SNOWFLAKE_PASSWORD` prior to calling the script. schemachange will fail if the `SNOWFLAKE_PASSWORD` environment variable is not set. The environment variable `SNOWFLAKE_AUTHENTICATOR` will be set to `snowflake` if it not explicitly set. 
+The Snowflake user password for `SNOWFLAKE_USER` is required to be set in the environment variable `SNOWFLAKE_PASSWORD` prior to calling the script. schemachange will fail if the `SNOWFLAKE_PASSWORD` environment variable is not set. The environment variable `SNOWFLAKE_AUTHENTICATOR` will be set to `snowflake` if it not explicitly set.
 
 _**DEPRECATION NOTICE**: The `SNOWSQL_PWD` environment variable is deprecated but currently still supported. Support for it will be removed in a later version of schemachange. Please use `SNOWFLAKE_PASSWORD` instead._
 
@@ -242,20 +242,20 @@ The URL of the authenticator resource that will be receive the POST request.
 * token-response-name
 The Expected name of the JSON element containing the Token in the return response from the authenticator resource.
 * token-request-payload
-The Set of variables passed as a dictionary to the `data` element of the request. 
+The Set of variables passed as a dictionary to the `data` element of the request.
 * token-request-headers
-The Set of variables passed as a dictionary to the `headers` element of the request. 
+The Set of variables passed as a dictionary to the `headers` element of the request.
 
-It is recomended to use the YAML file and pass oauth secrets into the configuration using the templating engine instead of the command line option.  
+It is recomended to use the YAML file and pass oauth secrets into the configuration using the templating engine instead of the command line option.
 
 
 ### External Browser Authentication
-External browser authentication can be used for local development by setting the environment variable `SNOWFLAKE_AUTHENTICATOR` to the value `externalbrowser` prior to calling schemachange. 
+External browser authentication can be used for local development by setting the environment variable `SNOWFLAKE_AUTHENTICATOR` to the value `externalbrowser` prior to calling schemachange.
 The client will be prompted to authenticate in a browser that pops up. Refer to the [documentation](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-use.html#setting-up-browser-based-sso) to cache the token to minimize the number of times the browser pops up to authenticate the user.
 
 ### Okta Authentication
-For clients that do not have a browser, can use the popular SaaS Idp option to connect via Okta. This will require the Okta URL that you utilize for SSO. 
-Okta authentication can be used setting the environment variable `SNOWFLAKE_AUTHENTICATOR` to the value of your okta endpoint as a fully formed URL ( E.g. `https://<org_name>.okta.com`) prior to calling schemachange. 
+For clients that do not have a browser, can use the popular SaaS Idp option to connect via Okta. This will require the Okta URL that you utilize for SSO.
+Okta authentication can be used setting the environment variable `SNOWFLAKE_AUTHENTICATOR` to the value of your okta endpoint as a fully formed URL ( E.g. `https://<org_name>.okta.com`) prior to calling schemachange.
 
 _** NOTE**: Please disable Okta MFA for the user who uses Native SSO authentication with client drivers. Please consult your Okta administrator for more information._
 
@@ -330,14 +330,14 @@ dry-run: false
 # A string to include in the QUERY_TAG that is attached to every SQL statement executed
 query-tag: 'QUERY_TAG'
 
-# Information for Oauth token requests 
+# Information for Oauth token requests
 oauthconfig:
   # url Where token request are posted to
   token-provider-url: 'https://login.microsoftonline.com/{{ env_var('AZURE_ORG_GUID', 'default') }}/oauth2/v2.0/token'
   # name of Json entity returned by request
   token-response-name: 'access_token'
   # Headers needed for successful post or other security markings ( multiple labeled items permitted
-  token-request-headers: 
+  token-request-headers:
     Content-Type: "application/x-www-form-urlencoded"
     User-Agent: "python/schemachange"
   # Request Payload for Token (it is recommended pass

demo/citibike_jinja/modules/create_stage.j2

@@ -1,4 +1,4 @@
 {% macro create_stage(stage_name, URL) -%}
 CREATE OR REPLACE STAGE {{stage_name}}
     URL = '{{URL}}';
-{%- endmacro %}
+{%- endmacro %}

demo/citibike_jinja/schemachange-config.yml

@@ -8,4 +8,3 @@ vars:
     # not a good example of secrets, just here to demo the secret filtering
     trips_s3_bucket: s3://snowflake-workshop-lab/citibike-trips
     weather_s3_bucket: s3://snowflake-workshop-lab/weather-nyc
-

pyproject.toml

@@ -3,4 +3,4 @@ requires = [
   "setuptools >= 40.9.0",
   "wheel",
 ]
-build-backend = "setuptools.build_meta"
+build-backend = "setuptools.build_meta"

requirements.txt

@@ -1,4 +1,4 @@
-snowflake-connector-python>=2.8,<4.0
+Jinja2~=3.0
 pandas~=1.3
 PyYAML~=6.0
-Jinja2~=3.0
+snowflake-connector-python>=2.8,<4.0

schemachange/cli.py

@@ -19,7 +19,7 @@
 from jinja2.loaders import BaseLoader
 from pandas import DataFrame
 
-#region Global Variables 
+#region Global Variables
 # metadata
 _schemachange_version = '3.5.4'
 _config_file_name = 'schemachange-config.yml'
@@ -50,25 +50,25 @@
   + "{snowflake_role}\nUsing default warehouse {snowflake_warehouse}\nUsing default " \
   + "database {snowflake_database}"
 _log_ch_use = "Using change history table {database_name}.{schema_name}.{table_name} " \
-  + "(last altered {last_altered})" 
+  + "(last altered {last_altered})"
 _log_ch_create = "Created change history table {database_name}.{schema_name}.{table_name}"
 _err_ch_missing = "Unable to find change history table {database_name}.{schema_name}.{table_name}"
 _log_ch_max_version = "Max applied change script version: {max_published_version_display}"
 _log_skip_v = "Skipping change script {script_name} because it's older than the most recently " \
   + "applied change ({max_published_version})"
 _log_skip_r ="Skipping change script {script_name} because there is no change since the last " \
   + "execution"
-_log_apply =  "Applying change script {script_name}"  
+_log_apply =  "Applying change script {script_name}"
 _log_apply_set_complete  =  "Successfully applied {scripts_applied} change scripts (skipping " \
-  + "{scripts_skipped}) \nCompleted successfully" 
+  + "{scripts_skipped}) \nCompleted successfully"
 _err_vars_config = "vars did not parse correctly, please check its configuration"
 _err_vars_reserved = "The variable schemachange has been reserved for use by schemachange, " \
   + "please use a different name"
 _err_invalid_folder  = "Invalid {folder_type} folder: {path}"
 _err_dup_scripts = "The script name {script_name} exists more than once (first_instance " \
-  + "{first_path}, second instance {script_full_path})" 
+  + "{first_path}, second instance {script_full_path})"
 _err_dup_scripts_version = "The script version {script_version} exists more than once " \
-  + "(second instance {script_full_path})" 
+  + "(second instance {script_full_path})"
 _err_invalid_cht  = 'Invalid change history table name: %s'
 _log_auth_type ="Proceeding with %s authentication"
 _log_pk_enc ="No private key passphrase provided. Assuming the key is not encrypted."
@@ -232,7 +232,7 @@ def __init__(self, config):
     self.autocommit = config['autocommit']
     self.verbose = config['verbose']
     if self.set_connection_args():
-      self.con = snowflake.connector.connect(**self.conArgs) 
+      self.con = snowflake.connector.connect(**self.conArgs)
       if not self.autocommit:
         self.con.autocommit(False)
     else:
@@ -268,23 +268,23 @@ def set_connection_args(self):
     default_authenticator = 'snowflake'
     if os.getenv("SNOWFLAKE_PASSWORD") is not None and os.getenv("SNOWFLAKE_PASSWORD"):
       snowflake_password = os.getenv("SNOWFLAKE_PASSWORD")
-    
+
     # Check legacy/deprecated env variable
     if os.getenv("SNOWSQL_PWD") is not None and os.getenv("SNOWSQL_PWD"):
       if snowflake_password:
         warnings.warn(_warn_password_dup, DeprecationWarning)
       else:
         warnings.warn(_warn_password, DeprecationWarning)
         snowflake_password = os.getenv("SNOWSQL_PWD")
-    
+
     snowflake_authenticator = os.getenv("SNOWFLAKE_AUTHENTICATOR")
 
     if snowflake_authenticator:
       # Determine the type of Authenticator
       # OAuth based authentication
-      if snowflake_authenticator.lower() == 'oauth':      
+      if snowflake_authenticator.lower() == 'oauth':
         oauth_token = self.get_oauth_token()
-      
+
         if self.verbose:
           print( _log_auth_type % 'Oauth Access Token')
         self.conArgs['token'] = oauth_token
@@ -296,7 +296,7 @@ def set_connection_args(self):
           print(_log_auth_type % 'External Browser')
       # IDP based Authentication, limited to Okta
       elif snowflake_authenticator.lower()[:8]=='https://':
-        
+
         if self.verbose:
           print(_log_auth_type % 'Okta')
           print(_log_okta_ep % snowflake_authenticator)
@@ -311,7 +311,7 @@ def set_connection_args(self):
         if self.verbose:
           print(_err_unsupported_auth_mthd.format(unsupported_authenticator=snowflake_authenticator) )
         self.conArgs['authenticator'] = default_authenticator
-    else: 
+    else:
         # default authenticator to snowflake
         self.conArgs['authenticator'] = default_authenticator
 
@@ -348,7 +348,7 @@ def set_connection_args(self):
         self.conArgs['private_key'] = pkb
       else:
         raise NameError(_err_no_auth_mthd)
-    
+
     return True
 
   def execute_snowflake_query(self, query):
@@ -471,14 +471,14 @@ def apply_change_script(self, script, script_content, change_history_table):
     # Compose and execute the insert statement to the log file
     query = self._q_ch_log.format(**frmt_args)
     self.execute_snowflake_query(query)
-  
+
 
 def deploy_command(config):
   # Make sure we have the required connection info, all of the below needs to be present.
   req_args = set(['snowflake_account','snowflake_user','snowflake_role','snowflake_warehouse'])
   provided_args = {k:v for (k,v) in config.items() if v}
-  missing_args = req_args -provided_args.keys()   
-  if len(missing_args)>0: 
+  missing_args = req_args -provided_args.keys()
+  if len(missing_args)>0:
     raise ValueError(_err_args_missing % ', '.join({s.replace('_', ' ') for s in missing_args}))
 
   #ensure an authentication method is specified / present. one of the below needs to be present.
@@ -585,7 +585,7 @@ def render_command(config, script_path):
   # Validate the script file path
   script_path = os.path.abspath(script_path)
   if not os.path.isfile(script_path):
-    raise ValueError(_err_invalid_folder.format(folder_type='script_path', path=script_path))   
+    raise ValueError(_err_invalid_folder.format(folder_type='script_path', path=script_path))
   # Always process with jinja engine
   jinja_processor = JinjaTemplateProcessor(project_root = config['root_folder'], \
      modules_folder = config['modules_folder'])
@@ -662,10 +662,10 @@ def get_schemachange_config(config_file_path, root_folder, modules_folder, snowf
 
   # Validate folder paths
   if 'root_folder' in config:
-    config['root_folder'] = os.path.abspath(config['root_folder'])   
+    config['root_folder'] = os.path.abspath(config['root_folder'])
   if not os.path.isdir(config['root_folder']):
     raise ValueError(_err_invalid_folder.format(folder_type='root', path=config['root_folder']))
- 
+
   if config['modules_folder']:
     config['modules_folder'] = os.path.abspath(config['modules_folder'])
     if not os.path.isdir(config['modules_folder']):
@@ -743,30 +743,30 @@ def get_all_scripts_recursively(root_directory, verbose):
       # Throw an error if the same version exists more than once
       if script_type == 'V':
         if script['script_version'] in all_versions:
-          raise ValueError(_err_dup_scripts_version.format(**script)) 
+          raise ValueError(_err_dup_scripts_version.format(**script))
         all_versions.append(script['script_version'])
 
   return all_files
 
 def get_change_history_table_details(change_history_table):
   # Start with the global defaults
   details = dict()
-  details['database_name'] = _metadata_database_name 
-  details['schema_name'] = _metadata_schema_name 
-  details['table_name'] = _metadata_table_name 
+  details['database_name'] = _metadata_database_name
+  details['schema_name'] = _metadata_schema_name
+  details['table_name'] = _metadata_table_name
 
   # Then override the defaults if requested. The name could be in one, two or three part notation.
   if change_history_table is not None:
     table_name_parts = change_history_table.strip().split('.')
-    if len(table_name_parts) == 1: 
-      details['table_name'] = table_name_parts[0] 
+    if len(table_name_parts) == 1:
+      details['table_name'] = table_name_parts[0]
     elif len(table_name_parts) == 2:
-      details['table_name'] = table_name_parts[1] 
-      details['schema_name'] = table_name_parts[0] 
+      details['table_name'] = table_name_parts[1]
+      details['schema_name'] = table_name_parts[0]
     elif len(table_name_parts) == 3:
-      details['table_name'] = table_name_parts[2] 
-      details['schema_name'] = table_name_parts[1] 
-      details['database_name'] = table_name_parts[0] 
+      details['table_name'] = table_name_parts[2]
+      details['schema_name'] = table_name_parts[1]
+      details['database_name'] = table_name_parts[0]
     else:
       raise ValueError(_err_invalid_cht % change_history_table)
   #if the object name does not include '"' raise to upper case on return
@@ -885,7 +885,7 @@ def main(argv=sys.argv):
   if args.subcommand == 'render':
     render_command(config, args.script)
   else:
-    deploy_command(config) 
+    deploy_command(config)
 
 if __name__ == "__main__":
   main()

tests/test_JinjaEnvVar.py

@@ -18,7 +18,7 @@ def test_env_var_with_no_default_and_no_environmental_variables_should_raise_exc
 
 @mock.patch.dict(os.environ, {}, clear=True)
 def test_env_var_with_default_and_no_environmental_variables_should_return_default():
-   
+
     print(os.environ)
     assert ('SF_DATABASE' in os.environ) is False
 
@@ -28,13 +28,13 @@ def test_env_var_with_default_and_no_environmental_variables_should_return_defau
 
 @mock.patch.dict(os.environ, {"SF_DATABASE": "SCHEMACHANGE_DEMO_2"}, clear=True)
 def test_env_var_with_default_and_environmental_variables_should_return_environmental_variable_value():
-   
+
     result = JinjaEnvVar.env_var('SF_DATABASE', 'SCHEMACHANGE_DEMO')
     assert result == 'SCHEMACHANGE_DEMO_2'
 
 
 @mock.patch.dict(os.environ, {"SF_DATABASE": "SCHEMACHANGE_DEMO_3"}, clear=True)
 def test_JinjaEnvVar_with_jinja_template():
-    
+
     template = jinja2.Template("{{env_var('SF_DATABASE', 'SCHEMACHANGE_DEMO')}}", extensions=[JinjaEnvVar])
     assert template.render() == "SCHEMACHANGE_DEMO_3"

tests/test_JinjaTemplateProcessor.py

@@ -49,13 +49,13 @@ def test_JinjaTemplateProcessor_render_simple_string_expecting_variable():
 def test_JinjaTemplateProcessor_render_from_subfolder(tmp_path: pathlib.Path):
 
     root_folder = tmp_path / "MORE2"
-    
+
     root_folder.mkdir()
     script_folder = root_folder/ "SQL"
     script_folder.mkdir()
     script_file = script_folder / "1.0.0_my_test.sql"
     script_file.write_text("Hello world!")
-    
+
     processor = JinjaTemplateProcessor(str(root_folder), None)
     template_path = processor.relpath(str(script_file))
 

tests/test_SecretManager.py

@@ -80,4 +80,4 @@ def test_SecretManager_global_redact():
     sm.add("Hello")
     SecretManager.set_global_manager(sm)
 
-    assert SecretManager.global_redact("Hello World!") == "***** World!"
+    assert SecretManager.global_redact("Hello World!") == "***** World!"

tests/test_get_all_scripts_recursively.py

@@ -278,4 +278,4 @@ def test_get_all_scripts_recursively__given_same_Repeatable_file_with_and_withou
             result = get_all_scripts_recursively("scripts", False)
         assert str(e.value).startswith(
             "The script name R__intial.sql exists more than once (first_instance "
-        )
+        )