Added Always First Functionality

README.md

@@ -24,6 +24,7 @@ For the complete list of changes made to schemachange check out the [CHANGELOG](
    1. [Versioned Script Naming](#versioned-script-naming)
    1. [Repeatable Script Naming](#repeatable-script-naming)
    1. [Always Script Naming](#always-script-naming)
+   1. [Always First Script Naming](#always-first-script-naming)
    1. [Script Requirements](#script-requirements)
    1. [Using Variables in Scripts](#using-variables-in-scripts)
       1. [Secrets filtering](#secrets-filtering)
@@ -38,7 +39,7 @@ For the complete list of changes made to schemachange check out the [CHANGELOG](
    1. [Okta Authentication](#okta-authentication)
 1. [Configuration](#configuration)
    1. [YAML Config File](#yaml-config-file)
-      1. [Yaml Jinja support](#yaml-jinja-support)
+      1. [YAML Jinja support](#yaml-jinja-support)
    1. [Command Line Arguments](#command-line-arguments)
 1. [Running schemachange](#running-schemachange)
    1. [Prerequisites](#prerequisites)
@@ -65,10 +66,12 @@ schemachange expects a directory structure like the following to exist:
     |-- V1.1.2__second_change.sql
     |-- R__sp_add_sales.sql
     |-- R__fn_get_timezone.sql
+    |-- F__clone_to_qa.sql
 |-- folder_2
     |-- folder_3
         |-- V1.1.3__third_change.sql
         |-- R__fn_sort_ascii.sql
+        |-- A__permissions.sql
 ```
 
 The schemachange folder structure is very flexible. The `project_root` folder is specified with the `-f` or `--root-folder` argument. schemachange only pays attention to the filenames, not the paths. Therefore, under the `project_root` folder you are free to arrange the change scripts any way you see fit. You can have as many subfolders (and nested subfolders) as you would like.
@@ -129,6 +132,20 @@ e.g.
 
 This type of change script is useful for an environment set up after cloning. Always scripts are applied always last.
 
+### Always First Script Naming
+
+Always First change scripts are executed with every run of schemachange if the configuration option is set to `True`; the default is `False`. This is an addition to the implementation of [Flyway Versioned Migrations](https://flywaydb.org/documentation/concepts/migrations.html#repeatable-migrations).
+The script name must following pattern:
+
+`F__Some_description.sql`
+
+e.g.
+
+* F__QA_Clone.sql
+* F__STG_Clone.sql
+
+This type of change script is useful for cloning an environment at the start of the CI/CD process. When a release is created, the first step is to recreate the QA clone off production, so the change scripts are applied to the most current version of the production environment. After QA approves the release, the cloning action is not needed, so the configuration option is set to `False` and the Always First scripts are skipped. Always First scripts are applied first when the configuration option is set to `True`.
+
 ### Script Requirements
 
 schemachange is designed to be very lightweight and not impose to many limitations. Each change script can have any number of SQL statements within it and must supply the necessary context, like database and schema names. The context can be supplied by using an explicit `USE <DATABASE>` command or by naming all objects with a three-part name (`<database name>.<schema name>.<object name>`). schemachange will simply run the contents of each script against the target Snowflake account, in the correct order.
@@ -329,6 +346,9 @@ autocommit: false
 # Display verbose debugging details during execution (the default is False)
 verbose: false
 
+# Execute Always First scripts which are executed before other script types (the default is False)
+always-first: false
+
 # Run schemachange in dry run mode (the default is False)
 dry-run: false
 
@@ -396,14 +416,15 @@ Parameter | Description
 --create-change-history-table | Create the change history table if it does not exist. The default is 'False'.
 -ac, --autocommit | Enable autocommit feature for DML commands. The default is 'False'.
 -v, --verbose | Display verbose debugging details during execution. The default is 'False'.
+-af, --always-first | Enable to execute Always First scripts. These will be executed before all other script types. The default is 'False'.
 --dry-run | Run schemachange in dry run mode. The default is 'False'.
 --query-tag | A string to include in the QUERY_TAG that is attached to every SQL statement executed.
 --oauth-config | Define values for the variables to Make Oauth Token requests  (e.g. {"token-provider-url": "https//...", "token-request-payload": {"client_id": "GUID_xyz",...},... })'
 
 #### render
 This subcommand is used to render a single script to the console. It is intended to support the development and troubleshooting of script that use features from the jinja template engine.
 
-`usage: schemachange render [-h] [--config-folder CONFIG_FOLDER] [-f ROOT_FOLDER] [-m MODULES_FOLDER] [--vars VARS] [-v] script`
+`usage: schemachange render [-h] [--config-folder CONFIG_FOLDER] [-f ROOT_FOLDER] [-m MODULES_FOLDER] [--vars VARS] [-v] [-af] script`
 
 Parameter | Description
 --- | ---
@@ -412,6 +433,7 @@ Parameter | Description
 -m MODULES_FOLDER, --modules-folder MODULES_FOLDER | The modules folder for jinja macros and templates to be used across multiple scripts
 --vars VARS | Define values for the variables to replaced in change scripts, given in JSON format (e.g. {"variable1": "value1", "variable2": "value2"})
 -v, --verbose | Display verbose debugging details during execution (the default is False)
+-af, --always-first | Enable to execute Always First scripts. These will be executed before all other script types. The default is 'False'.
 
 
 ## Running schemachange

schemachange/cli.py

@@ -126,7 +126,7 @@ def override_loader(self, loader: jinja2.BaseLoader):
     # to make unit testing easier
     self.__environment = jinja2.Environment(loader=loader, **self._env_args)
 
-  def render(self, script: str, vars: Dict[str, Any], verbose: bool) -> str:
+  def render(self, script: str, vars: Dict[str, Any], verbose: bool, always_first: bool) -> str:
     if not vars:
       vars = {}
     # jinja needs posix path
@@ -233,13 +233,16 @@ def __init__(self, config):
     self.oauth_config = config['oauth_config']
     self.autocommit = config['autocommit']
     self.verbose = config['verbose']
+
     if self.set_connection_args():
+      self.always_first = config['always_first']
       self.con = snowflake.connector.connect(**self.conArgs)
       if not self.autocommit:
         self.con.autocommit(False)
     else:
       print(_err_env_missing)
 
+
   def __del__(self):
     if hasattr(self, 'con'):
       self.con.close()
@@ -531,17 +534,30 @@ def deploy_command(config):
   print(_log_ch_max_version.format(max_published_version_display=max_published_version_display))
 
   # Find all scripts in the root folder (recursively) and sort them correctly
-  all_scripts = get_all_scripts_recursively(config['root_folder'], config['verbose'])
+  all_scripts = get_all_scripts_recursively(config['root_folder'], config['verbose'], config['always_first'])
   all_script_names = list(all_scripts.keys())
-  # Sort scripts such that versioned scripts get applied first and then the repeatable ones.
-  all_script_names_sorted =   sorted_alphanumeric([script for script in all_script_names if script[0] == 'V']) \
+  # Sort scripts such that always first scripts get executed first, then versioned scripts, and then the repeatable ones.
+  all_script_names_sorted = sorted_alphanumeric([script for script in all_script_names if script[0] == 'F']) \
+                            + sorted_alphanumeric([script for script in all_script_names if script[0] == 'V']) \
                             + sorted_alphanumeric([script for script in all_script_names if script[0] == 'R']) \
                             + sorted_alphanumeric([script for script in all_script_names if script[0] == 'A'])
 
   # Loop through each script in order and apply any required changes
   for script_name in all_script_names_sorted:
     script = all_scripts[script_name]
 
+    # Always process with jinja engine
+    jinja_processor = JinjaTemplateProcessor(project_root=config['root_folder'],
+                                             modules_folder=config['modules_folder'])
+    content = jinja_processor.render(jinja_processor.relpath(script['script_full_path']), config['vars'], \
+                                     config['verbose'], config['always_first'])
+
+    # Execute the Always First script(s) as long as the `always_first` configuration is True.
+    if script_name[0] == 'F' and config['always_first']:
+        if config['verbose']:
+            print(_log_skip_r.format(**script))
+        print(_log_apply.format(**script))
+
     # Apply a versioned-change script only if the version is newer than the most recent change in the database
     # Apply any other scripts, i.e. repeatable scripts, irrespective of the most recent change in the database
     if script_name[0] == 'V' and get_alphanum_key(script['script_version']) <= get_alphanum_key(max_published_version):
@@ -550,10 +566,6 @@ def deploy_command(config):
       scripts_skipped += 1
       continue
 
-    # Always process with jinja engine
-    jinja_processor = JinjaTemplateProcessor(project_root = config['root_folder'], modules_folder = config['modules_folder'])
-    content = jinja_processor.render(jinja_processor.relpath(script['script_full_path']), config['vars'], config['verbose'])
-
     # Apply only R scripts where the checksum changed compared to the last execution of snowchange
     if script_name[0] == 'R':
       # Compute the checksum for the script
@@ -572,6 +584,7 @@ def deploy_command(config):
         scripts_skipped += 1
         continue
 
+    # The Always scripts are applied in this step
     print(_log_apply.format(**script))
     if not config['dry_run']:
       session.apply_change_script(script, content, change_history_table)
@@ -594,7 +607,7 @@ def render_command(config, script_path):
   jinja_processor = JinjaTemplateProcessor(project_root = config['root_folder'], \
      modules_folder = config['modules_folder'])
   content = jinja_processor.render(jinja_processor.relpath(script_path), \
-    config['vars'], config['verbose'])
+    config['vars'], config['verbose'], config['always_first'])
 
   checksum = hashlib.sha224(content.encode('utf-8')).hexdigest()
   print("Checksum %s" % checksum)
@@ -635,7 +648,7 @@ def load_schemachange_config(config_file_path: str) -> Dict[str, Any]:
 def get_schemachange_config(config_file_path, root_folder, modules_folder, snowflake_account, \
   snowflake_user, snowflake_role, snowflake_warehouse, snowflake_database, snowflake_schema, \
   change_history_table, vars, create_change_history_table, autocommit, verbose, \
-  dry_run, query_tag, oauth_config, **kwargs):
+  dry_run, query_tag, oauth_config, always_first, **kwargs):
 
   # create cli override dictionary
   # Could refactor to just pass Args as a dictionary?
@@ -648,7 +661,7 @@ def get_schemachange_config(config_file_path, root_folder, modules_folder, snowf
     "change_history_table":change_history_table, "vars":vars, \
     "create_change_history_table":create_change_history_table, \
     "autocommit":autocommit, "verbose":verbose, "dry_run":dry_run,\
-    "query_tag":query_tag, "oauth_config":oauth_config}
+    "query_tag":query_tag, "oauth_config":oauth_config, "always_first":always_first}
   cli_inputs = {k:v for (k,v) in cli_inputs.items() if v}
 
   # load YAML inputs and convert kebabs to snakes
@@ -662,7 +675,7 @@ def get_schemachange_config(config_file_path, root_folder, modules_folder, snowf
     "snowflake_warehouse":None,  "snowflake_database":None, "snowflake_schema":None, \
     "change_history_table":None,  \
     "vars":{}, "create_change_history_table":False, "autocommit":False, "verbose":False,  \
-    "dry_run":False , "query_tag":None , "oauth_config":None }
+    "dry_run":False, "query_tag":None, "oauth_config":None, "always_first":False}
   #insert defualt values for items not populated
   config.update({ k:v for (k,v) in config_defaults.items() if not k in config.keys()})
 
@@ -687,7 +700,7 @@ def get_schemachange_config(config_file_path, root_folder, modules_folder, snowf
 
   return config
 
-def get_all_scripts_recursively(root_directory, verbose):
+def get_all_scripts_recursively(root_directory, verbose, always_first):
   all_files = dict()
   all_versions = list()
   # Walk the entire directory structure recursively
@@ -699,11 +712,17 @@ def get_all_scripts_recursively(root_directory, verbose):
         file_name.strip(), re.IGNORECASE)
       repeatable_script_name_parts = re.search(r'^([R])__(.+?)\.(?:sql|sql.jinja)$', \
         file_name.strip(), re.IGNORECASE)
+      always_first_script_name_parts = re.search(r'^([F])__(.+?)\.(?:sql|sql.jinja)$', \
+        file_name.strip(), re.IGNORECASE)
       always_script_name_parts = re.search(r'^([A])__(.+?)\.(?:sql|sql.jinja)$', \
         file_name.strip(), re.IGNORECASE)
 
       # Set script type depending on whether it matches the versioned file naming format
-      if script_name_parts is not None:
+      if always_first_script_name_parts is not None and always_first:
+        script_type = 'F'
+        if verbose:
+          print("Found Always First file " + file_full_path)
+      elif script_name_parts is not None:
         script_type = 'V'
         if verbose:
           print("Found Versioned file " + file_full_path)
@@ -732,11 +751,13 @@ def get_all_scripts_recursively(root_directory, verbose):
       script['script_name'] = script_name
       script['script_full_path'] = file_full_path
       script['script_type'] = script_type
-      script['script_version'] = '' if script_type in ['R', 'A'] else script_name_parts.group(2)
+      script['script_version'] = '' if script_type in ['R', 'A', 'F'] else script_name_parts.group(2)
       if script_type == 'R':
         script['script_description'] = repeatable_script_name_parts.group(2).replace('_', ' ').capitalize()
       elif script_type == 'A':
         script['script_description'] = always_script_name_parts.group(2).replace('_', ' ').capitalize()
+      elif script_type == 'F':
+        script['script_description'] = always_first_script_name_parts.group(2).replace('_', ' ').capitalize()
       else:
         script['script_description'] = script_name_parts.group(3).replace('_', ' ').capitalize()
 
@@ -831,6 +852,7 @@ def main(argv=sys.argv):
   parser_deploy.add_argument('--dry-run', action='store_true', help = 'Run schemachange in dry run mode (the default is False)', required = False)
   parser_deploy.add_argument('--query-tag', type = str, help = 'The string to add to the Snowflake QUERY_TAG session value for each query executed', required = False)
   parser_deploy.add_argument('--oauth-config', type = json.loads, help = 'Define values for the variables to Make Oauth Token requests  (e.g. {"token-provider-url": "https//...", "token-request-payload": {"client_id": "GUID_xyz",...},... })', required = False)
+  parser_deploy.add_argument('-af', '--always-first', action='store_true', help = 'Enable to execute Always First scripts. These will be executed before all other script types.', required = False)
    # TODO test CLI passing of args
 
   parser_render = subcommands.add_parser('render', description="Renders a script to the console, used to check and verify jinja output from scripts.")
@@ -839,6 +861,7 @@ def main(argv=sys.argv):
   parser_render.add_argument('-m', '--modules-folder', type = str, help = 'The modules folder for jinja macros and templates to be used across multiple scripts', required = False)
   parser_render.add_argument('--vars', type = json.loads, help = 'Define values for the variables to replaced in change scripts, given in JSON format (e.g. {"variable1": "value1", "variable2": "value2"})', required = False)
   parser_render.add_argument('-v', '--verbose', action='store_true', help = 'Display verbose debugging details during execution (the default is False)', required = False)
+  parser_render.add_argument('-af', '--always-first', action='store_true', help = 'Enable to execute Always First scripts. These will be executed before all other script types.', required = False)
   parser_render.add_argument('script', type = str, help = 'The script to render')
 
   # The original parameters did not support subcommands. Check if a subcommand has been supplied

tests/test_JinjaTemplateProcessor.py

@@ -14,7 +14,7 @@ def test_JinjaTemplateProcessor_render_simple_string():
     templates = {"test.sql": "some text"}
     processor.override_loader(DictLoader(templates))
 
-    context = processor.render("test.sql", None, True)
+    context = processor.render("test.sql", None, True, False)
 
     assert context == "some text"
 
@@ -27,7 +27,7 @@ def test_JinjaTemplateProcessor_render_simple_string_expecting_variable_that_doe
     processor.override_loader(DictLoader(templates))
 
     with pytest.raises(UndefinedError) as e:
-        context = processor.render("test.sql", None, True)
+        context = processor.render("test.sql", None, True, False)
 
     assert str(e.value) == "'myvar' is undefined"
 
@@ -41,7 +41,7 @@ def test_JinjaTemplateProcessor_render_simple_string_expecting_variable():
 
     vars = json.loads('{"myvar" : "world"}')
 
-    context = processor.render("test.sql", vars, True)
+    context = processor.render("test.sql", vars, True, False)
 
     assert context == "Hello world!"
 
@@ -59,6 +59,6 @@ def test_JinjaTemplateProcessor_render_from_subfolder(tmp_path: pathlib.Path):
     processor = JinjaTemplateProcessor(str(root_folder), None)
     template_path = processor.relpath(str(script_file))
 
-    context = processor.render(template_path, {}, True)
+    context = processor.render(template_path, {}, True, False)
 
     assert context == "Hello world!"