Skip to content

Commit

Permalink
[generic-config-updater] Adding non-strict mode (#1929)
Browse files Browse the repository at this point in the history
#### What I did
Added non-strict mode to patch sorting which is useful while building the E2E tests for the framework. It helps developers avoid the incompleteness of YANG models.

Was implementing by adding 2 options:
* Ignore tables without yang: This flag will ignore from validation all tables that does not have YANG model defined yet 
* Ignore path: This flag can explicitly ignore a path from validation, which can ignore any config. It will be useful for ignoring configs with YANG but the YANG is not up-to-date.

> ⚠️ **The non-strict mode is only meant for helping with E2E testing and not meant for production**

#### How I did it
Added flags `ignore-non-yang-tables` and `ignore-path` to the config commands:
```
config apply-patch
config replace
config rollback
```

Added NonStrictSorter, together with the older one which is the StrictSorter.

NonStrictSorter groups configs into 2 groups, YANG covered configs, and Non-YANG covered configs
- Non-YANG covered configs are the tables without YANG models, and the fields/tables ignored explicitly by the `-i` CLI option. The JsonPatch between Non-YANG current and Non-YANG target configs is generated and is clubbed together as a single JsonChange i.e. we will make a single call to the `ChangeApplier` for Non-YANG changes
- YANG covered configs are the rest of the configs. They are handled using the normal sorter. 

Check implementation for further details

#### How to verify it
unit-test

#### Previous command output (if the output of a command-line utility has changed)
```
admin@vlab-01:~$ sudo config apply-patch -h
Usage: config apply-patch [OPTIONS] PATCH_FILE_PATH

  Apply given patch of updates to Config. A patch is a JsonPatch which
  follows rfc6902. This command can be used do partial updates to the config
  with minimum disruption to running processes. It allows addition as well
  as deletion of configs. The patch file represents a diff of ConfigDb(ABNF)
  format or SonicYang format.

  <patch-file-path>: Path to the patch file on the file-system.

Options:
  -f, --format [CONFIGDB|SONICYANG]
                                  format of config of the patch is either
                                  ConfigDb(ABNF) or SonicYang
  -d, --dry-run                   test out the command without affecting
                                  config state
  -v, --verbose                   print additional details of what the
                                  operation is doing
  -h, -?, --help                  Show this message and exit.
admin@vlab-01:~$ sudo config replace -h
Usage: config replace [OPTIONS] TARGET_FILE_PATH

  Replace the whole config with the specified config. The config is replaced
  with minimum disruption e.g. if ACL config is different between current
  and target config only ACL config is updated, and other config/services
  such as DHCP will not be affected.

  **WARNING** The target config file should be the whole config, not just
  the part intended to be updated.

  <target-file-path>: Path to the target file on the file-system.

Options:
  -f, --format [CONFIGDB|SONICYANG]
                                  format of target config is either
                                  ConfigDb(ABNF) or SonicYang
  -d, --dry-run                   test out the command without affecting
                                  config state
  -v, --verbose                   print additional details of what the
                                  operation is doing
  -h, -?, --help                  Show this message and exit.
admin@vlab-01:~$ sudo config rollback -h
Usage: config rollback [OPTIONS] CHECKPOINT_NAME

  Rollback the whole config to the specified checkpoint. The config is
  rolled back with minimum disruption e.g. if ACL config is different
  between current and checkpoint config only ACL config is updated, and
  other config/services such as DHCP will not be affected.

  <checkpoint-name>: The checkpoint name, use `config list-checkpoints`
  command to see available checkpoints.

Options:
  -d, --dry-run   test out the command without affecting config state
  -v, --verbose   print additional details of what the operation is doing
  -?, -h, --help  Show this message and exit.
admin@vlab-01:~$ 
```

#### New command output (if the output of a command-line utility has changed)
Same as old command output since the options are hidden

The new added options are:
```
--n --ignore-non-yang-tables        ignore validation for tables without YANG models
--i --ignore-path                   ignore validation for config specified by given path which is a JsonPointer
```

#### Example of usages:
- KVM SONiC image that has multiple YANG errors in NEIGHBOR_BGP, DEVICE_METADATA, and VLAN tables.

Applying empty-patch without any non-strict mode flags
```sh
admin@vlab-01:~$ sudo config apply-patch empty.json-patch 
Patch Applier: Patch application starting.
Patch Applier: Patch: []
Patch Applier: Getting current config db.
Patch Applier: Simulating the target full config after applying the patch.
Patch Applier: Sorting patch updates.
Note: Below table(s) have no YANG models:
BGP_PEER_RANGE, BUFFER_PG, BUFFER_POOL, BUFFER_PROFILE, BUFFER_QUEUE, CABLE_LENGTH, CONSOLE_SWITCH, DEVICE_NEIGHBOR_METADATA, DHCP_RELAY, DHCP_SERVER, DSCP_TO_TC_MAP, FEATURE, KDUMP, MAP_PFC_PRIORITY_TO_QUEUE, PORT_QOS_MAP, QUEUE, RESTAPI, SCHEDULER, SNMP, SNMP_COMMUNITY, SYSLOG_SERVER, TC_TO_PRIORITY_GROUP_MAP, TC_TO_QUEUE_MAP, TELEMETRY, WRED_PROFILE, 
sonic_yang(3):All Keys are not parsed in BGP_NEIGHBOR
dict_keys(['10.0.0.57', '10.0.0.59', '10.0.0.61', '10.0.0.63', 'fc00::72', 'fc00::76', 'fc00::7a', 'fc00::7e'])
sonic_yang(3):exceptionList:["'admin_status'", "'admin_status'", "'admin_status'", "'admin_status'", "'admin_status'", "'admin_status'", "'admin_status'", "'admin_status'", 'Value not found for vrf_name neighbor in 10.0.0.57', 'Value not found for vrf_name neighbor in 10.0.0.59', 'Value not found for vrf_name neighbor in 10.0.0.61', 'Value not found for vrf_name neighbor in 10.0.0.63', 'Value not found for vrf_name neighbor in fc00::72', 'Value not found for vrf_name neighbor in fc00::76', 'Value not found for vrf_name neighbor in fc00::7a', 'Value not found for vrf_name neighbor in fc00::7e']
sonic_yang(3):Data Loading Failed:All Keys are not parsed in BGP_NEIGHBOR
dict_keys(['10.0.0.57', '10.0.0.59', '10.0.0.61', '10.0.0.63', 'fc00::72', 'fc00::76', 'fc00::7a', 'fc00::7e'])
Failed to apply patch
Usage: config apply-patch [OPTIONS] PATCH_FILE_PATH
Try "config apply-patch -h" for help.

Error: Given patch is not valid because it will result in an invalid config
admin@vlab-01:~
```

Applying command and ignoring the tables/fields with invalid YANG
```
admin@vlab-01:~$ sudo config apply-patch empty.json-patch -i /BGP_NEIGHBOR -i /DEVICE_METADATA -i /VLAN/Vlan1000/dhcpv6_servers -i /VLAN/Vlan1000/members
Patch Applier: Patch application starting.
Patch Applier: Patch: []
Patch Applier: Getting current config db.
Patch Applier: Simulating the target full config after applying the patch.
Patch Applier: Validating target config does not have empty tables, since they do not show up in ConfigDb.
Patch Applier: Sorting patch updates.
Note: Below table(s) have no YANG models:
BGP_PEER_RANGE, BUFFER_PG, BUFFER_POOL, BUFFER_PROFILE, BUFFER_QUEUE, CABLE_LENGTH, CONSOLE_SWITCH, DEVICE_NEIGHBOR_METADATA, DHCP_RELAY, DHCP_SERVER, DSCP_TO_TC_MAP, FEATURE, KDUMP, MAP_PFC_PRIORITY_TO_QUEUE, PORT_QOS_MAP, QUEUE, RESTAPI, SCHEDULER, SNMP, SNMP_COMMUNITY, SYSLOG_SERVER, TC_TO_PRIORITY_GROUP_MAP, TC_TO_QUEUE_MAP, TELEMETRY, WRED_PROFILE, 
Note: Below table(s) have no YANG models:
BGP_PEER_RANGE, BUFFER_PG, BUFFER_POOL, BUFFER_PROFILE, BUFFER_QUEUE, CABLE_LENGTH, CONSOLE_SWITCH, DEVICE_NEIGHBOR_METADATA, DHCP_RELAY, DHCP_SERVER, DSCP_TO_TC_MAP, FEATURE, KDUMP, MAP_PFC_PRIORITY_TO_QUEUE, PORT_QOS_MAP, QUEUE, RESTAPI, SCHEDULER, SNMP, SNMP_COMMUNITY, SYSLOG_SERVER, TC_TO_PRIORITY_GROUP_MAP, TC_TO_QUEUE_MAP, TELEMETRY, WRED_PROFILE, 
Note: Below table(s) have no YANG models:
BGP_PEER_RANGE, BUFFER_PG, BUFFER_POOL, BUFFER_PROFILE, BUFFER_QUEUE, CABLE_LENGTH, CONSOLE_SWITCH, DEVICE_NEIGHBOR_METADATA, DHCP_RELAY, DHCP_SERVER, DSCP_TO_TC_MAP, FEATURE, KDUMP, MAP_PFC_PRIORITY_TO_QUEUE, PORT_QOS_MAP, QUEUE, RESTAPI, SCHEDULER, SNMP, SNMP_COMMUNITY, SYSLOG_SERVER, TC_TO_PRIORITY_GROUP_MAP, TC_TO_QUEUE_MAP, TELEMETRY, WRED_PROFILE, 
Patch Applier: The patch was sorted into 0 changes.
Patch Applier: Applying 0 changes in order.
Patch Applier: Verifying patch updates are reflected on ConfigDB.
Patch Applier: Patch application completed.
Patch applied successfully.
admin@vlab-01:~$
```

- Patch updating a mix of tables with YANG models and others without YANG models

Applying command without any non-strict mode flags ... fails because it is updating tables without YANG
```
admin@vlab-01:~$ sudo config apply-patch mix.json-patch
Patch Applier: Patch application starting.
Patch Applier: Patch: [{"op": "remove", "path": "/ACL_TABLE/DATAACL"}, {"op": "remove", "path": "/SYSLOG_SERVER"}]
Patch Applier: Getting current config db.
Patch Applier: Simulating the target full config after applying the patch.
Patch Applier: Validating target config does not have empty tables, since they do not show up in ConfigDb.
Patch Applier: Sorting patch updates.
Note: Below table(s) have no YANG models:
SYSLOG_SERVER, 
Failed to apply patch
Usage: config apply-patch [OPTIONS] PATCH_FILE_PATH
Try "config apply-patch -h" for help.

Error: Given patch is not valid because it has changes to tables without YANG models
admin@vlab-01:~$ 
```

Adding `-n` also fails because the tables with YANG models have validation violations
```
admin@vlab-01:~$ sudo config apply-patch mix.json-patch -n
Patch Applier: Patch application starting.
Patch Applier: Patch: [{"op": "remove", "path": "/ACL_TABLE/DATAACL"}, {"op": "remove", "path": "/SYSLOG_SERVER"}]
Patch Applier: Getting current config db.
Patch Applier: Simulating the target full config after applying the patch.
Patch Applier: Validating target config does not have empty tables, since they do not show up in ConfigDb.
Patch Applier: Sorting patch updates.
Note: Below table(s) have no YANG models:
BGP_PEER_RANGE, BUFFER_PG, BUFFER_POOL, BUFFER_PROFILE, BUFFER_QUEUE, CABLE_LENGTH, CONSOLE_SWITCH, DEVICE_NEIGHBOR_METADATA, DHCP_RELAY, DHCP_SERVER, DSCP_TO_TC_MAP, FEATURE, KDUMP, MAP_PFC_PRIORITY_TO_QUEUE, PORT_QOS_MAP, QUEUE, RESTAPI, SCHEDULER, SNMP, SNMP_COMMUNITY, SYSLOG_SERVER, TC_TO_PRIORITY_GROUP_MAP, TC_TO_QUEUE_MAP, TELEMETRY, WRED_PROFILE, 
Note: Below table(s) have no YANG models:
BGP_PEER_RANGE, BUFFER_PG, BUFFER_POOL, BUFFER_PROFILE, BUFFER_QUEUE, CABLE_LENGTH, CONSOLE_SWITCH, DEVICE_NEIGHBOR_METADATA, DHCP_RELAY, DHCP_SERVER, DSCP_TO_TC_MAP, FEATURE, KDUMP, MAP_PFC_PRIORITY_TO_QUEUE, PORT_QOS_MAP, QUEUE, RESTAPI, SCHEDULER, SNMP, SNMP_COMMUNITY, TC_TO_PRIORITY_GROUP_MAP, TC_TO_QUEUE_MAP, TELEMETRY, WRED_PROFILE, 
sonic_yang(3):All Keys are not parsed in BGP_NEIGHBOR
dict_keys(['10.0.0.57', '10.0.0.59', '10.0.0.61', '10.0.0.63', 'fc00::72', 'fc00::76', 'fc00::7a', 'fc00::7e'])
sonic_yang(3):exceptionList:["'admin_status'", "'admin_status'", "'admin_status'", "'admin_status'", "'admin_status'", "'admin_status'", "'admin_status'", "'admin_status'", 'Value not found for vrf_name neighbor in 10.0.0.57', 'Value not found for vrf_name neighbor in 10.0.0.59', 'Value not found for vrf_name neighbor in 10.0.0.61', 'Value not found for vrf_name neighbor in 10.0.0.63', 'Value not found for vrf_name neighbor in fc00::72', 'Value not found for vrf_name neighbor in fc00::76', 'Value not found for vrf_name neighbor in fc00::7a', 'Value not found for vrf_name neighbor in fc00::7e']
sonic_yang(3):Data Loading Failed:All Keys are not parsed in BGP_NEIGHBOR
dict_keys(['10.0.0.57', '10.0.0.59', '10.0.0.61', '10.0.0.63', 'fc00::72', 'fc00::76', 'fc00::7a', 'fc00::7e'])
Failed to apply patch
Usage: config apply-patch [OPTIONS] PATCH_FILE_PATH
Try "config apply-patch -h" for help.

Error: Given patch is not valid because it will result in an invalid config
admin@vlab-01:~$ 
```

Adding `-n` option and ignoring the tables with YANG validation violations, works
```
admin@vlab-01:~$ sudo config apply-patch mix.json-patch -n -i /BGP_NEIGHBOR -i /DEVICE_METADATA -i /VLAN/Vlan1000/dhcpv6_servers -i /VLAN/Vlan1000/members
Patch Applier: Patch application starting.
Patch Applier: Patch: [{"op": "remove", "path": "/ACL_TABLE/DATAACL"}, {"op": "remove", "path": "/SYSLOG_SERVER"}]
Patch Applier: Getting current config db.
Patch Applier: Simulating the target full config after applying the patch.
Patch Applier: Validating target config does not have empty tables, since they do not show up in ConfigDb.
Patch Applier: Sorting patch updates.
Note: Below table(s) have no YANG models:
BGP_PEER_RANGE, BUFFER_PG, BUFFER_POOL, BUFFER_PROFILE, BUFFER_QUEUE, CABLE_LENGTH, CONSOLE_SWITCH, DEVICE_NEIGHBOR_METADATA, DHCP_RELAY, DHCP_SERVER, DSCP_TO_TC_MAP, FEATURE, KDUMP, MAP_PFC_PRIORITY_TO_QUEUE, PORT_QOS_MAP, QUEUE, RESTAPI, SCHEDULER, SNMP, SNMP_COMMUNITY, SYSLOG_SERVER, TC_TO_PRIORITY_GROUP_MAP, TC_TO_QUEUE_MAP, TELEMETRY, WRED_PROFILE, 
Note: Below table(s) have no YANG models:
BGP_PEER_RANGE, BUFFER_PG, BUFFER_POOL, BUFFER_PROFILE, BUFFER_QUEUE, CABLE_LENGTH, CONSOLE_SWITCH, DEVICE_NEIGHBOR_METADATA, DHCP_RELAY, DHCP_SERVER, DSCP_TO_TC_MAP, FEATURE, KDUMP, MAP_PFC_PRIORITY_TO_QUEUE, PORT_QOS_MAP, QUEUE, RESTAPI, SCHEDULER, SNMP, SNMP_COMMUNITY, TC_TO_PRIORITY_GROUP_MAP, TC_TO_QUEUE_MAP, TELEMETRY, WRED_PROFILE, 
libyang[0]: Invalid JSON data (unexpected value). (path: /sonic-acl:sonic-acl/ACL_TABLE/ACL_TABLE_LIST[ACL_TABLE_NAME='DATAACL']/ports)
sonic_yang(3):Data Loading Failed:Invalid JSON data (unexpected value).
libyang[0]: Invalid JSON data (unexpected value). (path: /sonic-acl:sonic-acl/ACL_TABLE/ACL_TABLE_LIST[ACL_TABLE_NAME='DATAACL']/ports)
sonic_yang(3):Data Loading Failed:Invalid JSON data (unexpected value).
libyang[0]: Missing required element "type" in "ACL_TABLE_LIST". (path: /sonic-acl:sonic-acl/ACL_TABLE/ACL_TABLE_LIST[ACL_TABLE_NAME='DATAACL'])
sonic_yang(3):Data Loading Failed:Missing required element "type" in "ACL_TABLE_LIST".
libyang[0]: Missing required element "type" in "ACL_TABLE_LIST". (path: /sonic-acl:sonic-acl/ACL_TABLE/ACL_TABLE_LIST[ACL_TABLE_NAME='DATAACL'])
sonic_yang(3):Data Loading Failed:Missing required element "type" in "ACL_TABLE_LIST".
Patch Applier: The patch was sorted into 8 changes:
Patch Applier:   * [{"op": "remove", "path": "/SYSLOG_SERVER"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL/policy_desc"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL/ports/0"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL/ports/0"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL/ports/0"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL/stage"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL/ports"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL"}]
Patch Applier: Applying 8 changes in order:
Patch Applier:   * [{"op": "remove", "path": "/SYSLOG_SERVER"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL/policy_desc"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL/ports/0"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL/ports/0"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL/ports/0"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL/stage"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL/ports"}]
Patch Applier:   * [{"op": "remove", "path": "/ACL_TABLE/DATAACL"}]
Patch Applier: Verifying patch updates are reflected on ConfigDB.
Patch Applier: Patch application completed.
Patch applied successfully.
admin@vlab-01:~$
```

## BONUS
We can ignore the validation for the whole config which practically mean skip directly to the change applier, can be useful for testing the change applier
```
admin@vlab-01:~$ sudo config apply-patch mix.json-patch -i ''
Patch Applier: Patch application starting.
Patch Applier: Patch: [{"op": "remove", "path": "/ACL_TABLE/DATAACL"}, {"op": "remove", "path": "/SYSLOG_SERVER"}]
Patch Applier: Getting current config db.
Patch Applier: Simulating the target full config after applying the patch.
Patch Applier: Validating target config does not have empty tables, since they do not show up in ConfigDb.
Patch Applier: Sorting patch updates.
Note: Below table(s) have no YANG models:
BGP_PEER_RANGE, BUFFER_PG, BUFFER_POOL, BUFFER_PROFILE, BUFFER_QUEUE, CABLE_LENGTH, CONSOLE_SWITCH, DEVICE_NEIGHBOR_METADATA, DHCP_RELAY, DHCP_SERVER, DSCP_TO_TC_MAP, FEATURE, KDUMP, MAP_PFC_PRIORITY_TO_QUEUE, PORT_QOS_MAP, QUEUE, RESTAPI, SCHEDULER, SNMP, SNMP_COMMUNITY, SYSLOG_SERVER, TC_TO_PRIORITY_GROUP_MAP, TC_TO_QUEUE_MAP, TELEMETRY, WRED_PROFILE, 
Note: Below table(s) have no YANG models:
BGP_PEER_RANGE, BUFFER_PG, BUFFER_POOL, BUFFER_PROFILE, BUFFER_QUEUE, CABLE_LENGTH, CONSOLE_SWITCH, DEVICE_NEIGHBOR_METADATA, DHCP_RELAY, DHCP_SERVER, DSCP_TO_TC_MAP, FEATURE, KDUMP, MAP_PFC_PRIORITY_TO_QUEUE, PORT_QOS_MAP, QUEUE, RESTAPI, SCHEDULER, SNMP, SNMP_COMMUNITY, SYSLOG_SERVER, TC_TO_PRIORITY_GROUP_MAP, TC_TO_QUEUE_MAP, TELEMETRY, WRED_PROFILE, 
Patch Applier: The patch was sorted into 1 change:
Patch Applier:   * [{"op": "remove", "path": "/SYSLOG_SERVER"}, {"op": "remove", "path": "/ACL_TABLE/DATAACL"}]
Patch Applier: Applying 1 change in order:
Patch Applier:   * [{"op": "remove", "path": "/SYSLOG_SERVER"}, {"op": "remove", "path": "/ACL_TABLE/DATAACL"}]
Patch Applier: Verifying patch updates are reflected on ConfigDB.
Patch Applier: Patch application completed.
Patch applied successfully.
admin@vlab-01:~$
```
  • Loading branch information
ghooo authored Dec 8, 2021
1 parent 2e462ef commit 7ceccd7
Show file tree
Hide file tree
Showing 8 changed files with 893 additions and 123 deletions.
19 changes: 12 additions & 7 deletions config/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -1170,9 +1170,11 @@ def load(filename, yes):
default=ConfigFormat.CONFIGDB.name,
help='format of config of the patch is either ConfigDb(ABNF) or SonicYang')
@click.option('-d', '--dry-run', is_flag=True, default=False, help='test out the command without affecting config state')
@click.option('-n', '--ignore-non-yang-tables', is_flag=True, default=False, help='ignore validation for tables without YANG models', hidden=True)
@click.option('-i', '--ignore-path', multiple=True, help='ignore validation for config specified by given path which is a JsonPointer', hidden=True)
@click.option('-v', '--verbose', is_flag=True, default=False, help='print additional details of what the operation is doing')
@click.pass_context
def apply_patch(ctx, patch_file_path, format, dry_run, verbose):
def apply_patch(ctx, patch_file_path, format, dry_run, ignore_non_yang_tables, ignore_path, verbose):
"""Apply given patch of updates to Config. A patch is a JsonPatch which follows rfc6902.
This command can be used do partial updates to the config with minimum disruption to running processes.
It allows addition as well as deletion of configs. The patch file represents a diff of ConfigDb(ABNF)
Expand All @@ -1186,8 +1188,7 @@ def apply_patch(ctx, patch_file_path, format, dry_run, verbose):
patch = jsonpatch.JsonPatch(patch_as_json)

config_format = ConfigFormat[format.upper()]

GenericUpdater().apply_patch(patch, config_format, verbose, dry_run)
GenericUpdater().apply_patch(patch, config_format, verbose, dry_run, ignore_non_yang_tables, ignore_path)

click.secho("Patch applied successfully.", fg="cyan", underline=True)
except Exception as ex:
Expand All @@ -1200,9 +1201,11 @@ def apply_patch(ctx, patch_file_path, format, dry_run, verbose):
default=ConfigFormat.CONFIGDB.name,
help='format of target config is either ConfigDb(ABNF) or SonicYang')
@click.option('-d', '--dry-run', is_flag=True, default=False, help='test out the command without affecting config state')
@click.option('-n', '--ignore-non-yang-tables', is_flag=True, default=False, help='ignore validation for tables without YANG models', hidden=True)
@click.option('-i', '--ignore-path', multiple=True, help='ignore validation for config specified by given path which is a JsonPointer', hidden=True)
@click.option('-v', '--verbose', is_flag=True, default=False, help='print additional details of what the operation is doing')
@click.pass_context
def replace(ctx, target_file_path, format, dry_run, verbose):
def replace(ctx, target_file_path, format, dry_run, ignore_non_yang_tables, ignore_path, verbose):
"""Replace the whole config with the specified config. The config is replaced with minimum disruption e.g.
if ACL config is different between current and target config only ACL config is updated, and other config/services
such as DHCP will not be affected.
Expand All @@ -1217,7 +1220,7 @@ def replace(ctx, target_file_path, format, dry_run, verbose):

config_format = ConfigFormat[format.upper()]

GenericUpdater().replace(target_config, config_format, verbose, dry_run)
GenericUpdater().replace(target_config, config_format, verbose, dry_run, ignore_non_yang_tables, ignore_path)

click.secho("Config replaced successfully.", fg="cyan", underline=True)
except Exception as ex:
Expand All @@ -1227,16 +1230,18 @@ def replace(ctx, target_file_path, format, dry_run, verbose):
@config.command()
@click.argument('checkpoint-name', type=str, required=True)
@click.option('-d', '--dry-run', is_flag=True, default=False, help='test out the command without affecting config state')
@click.option('-n', '--ignore-non-yang-tables', is_flag=True, default=False, help='ignore validation for tables without YANG models', hidden=True)
@click.option('-i', '--ignore-path', multiple=True, help='ignore validation for config specified by given path which is a JsonPointer', hidden=True)
@click.option('-v', '--verbose', is_flag=True, default=False, help='print additional details of what the operation is doing')
@click.pass_context
def rollback(ctx, checkpoint_name, dry_run, verbose):
def rollback(ctx, checkpoint_name, dry_run, ignore_non_yang_tables, ignore_path, verbose):
"""Rollback the whole config to the specified checkpoint. The config is rolled back with minimum disruption e.g.
if ACL config is different between current and checkpoint config only ACL config is updated, and other config/services
such as DHCP will not be affected.
<checkpoint-name>: The checkpoint name, use `config list-checkpoints` command to see available checkpoints."""
try:
GenericUpdater().rollback(checkpoint_name, verbose, dry_run)
GenericUpdater().rollback(checkpoint_name, verbose, dry_run, ignore_non_yang_tables, ignore_path)

click.secho("Config rolled back successfully.", fg="cyan", underline=True)
except Exception as ex:
Expand Down
72 changes: 34 additions & 38 deletions generic_config_updater/generic_updater.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
from enum import Enum
from .gu_common import GenericConfigUpdaterError, ConfigWrapper, \
DryRunConfigWrapper, PatchWrapper, genericUpdaterLogging
from .patch_sorter import PatchSorter
from .patch_sorter import StrictPatchSorter, NonStrictPatchSorter, ConfigSplitter, \
TablesWithoutYangConfigSplitter, IgnorePathsFromYangConfigSplitter
from .change_applier import ChangeApplier

CHECKPOINTS_DIR = "/etc/sonic/checkpoints"
Expand Down Expand Up @@ -32,18 +33,13 @@ def __init__(self,
self.logger = genericUpdaterLogging.get_logger(title="Patch Applier", print_all_to_console=True)
self.config_wrapper = config_wrapper if config_wrapper is not None else ConfigWrapper()
self.patch_wrapper = patch_wrapper if patch_wrapper is not None else PatchWrapper()
self.patchsorter = patchsorter if patchsorter is not None else PatchSorter(self.config_wrapper, self.patch_wrapper)
self.patchsorter = patchsorter if patchsorter is not None else StrictPatchSorter(self.config_wrapper, self.patch_wrapper)
self.changeapplier = changeapplier if changeapplier is not None else ChangeApplier()

def apply(self, patch):
self.logger.log_notice("Patch application starting.")
self.logger.log_notice(f"Patch: {patch}")

# validate patch is only updating tables with yang models
self.logger.log_notice("Validating patch is not making changes to tables without YANG models.")
if not(self.patch_wrapper.validate_config_db_patch_has_yang_models(patch)):
raise ValueError(f"Given patch is not valid because it has changes to tables without YANG models")

# Get old config
self.logger.log_notice("Getting current config db.")
old_config = self.config_wrapper.get_config_db_as_json()
Expand All @@ -62,11 +58,6 @@ def apply(self, patch):
"which is not allowed in ConfigDb. " \
f"Table{'s' if len(empty_tables) != 1 else ''}: {empty_tables_txt}")

# Validate target config according to YANG models
self.logger.log_notice("Validating target config according to YANG models.")
if not(self.config_wrapper.validate_config_db_config(target_config)):
raise ValueError(f"Given patch is not valid because it will result in an invalid config")

# Generate list of changes to apply
self.logger.log_notice("Sorting patch updates.")
changes = self.patchsorter.sort(patch)
Expand Down Expand Up @@ -102,10 +93,6 @@ def replace(self, target_config):
self.logger.log_notice("Config replacement starting.")
self.logger.log_notice(f"Target config length: {len(json.dumps(target_config))}.")

self.logger.log_notice("Validating target config according to YANG models.")
if not(self.config_wrapper.validate_config_db_config(target_config)):
raise ValueError(f"The given target config is not valid")

self.logger.log_notice("Getting current config db.")
old_config = self.config_wrapper.get_config_db_as_json()

Expand Down Expand Up @@ -156,11 +143,6 @@ def checkpoint(self, checkpoint_name):
self.logger.log_notice("Getting current config db.")
json_content = self.config_wrapper.get_config_db_as_json()

# if current config are not valid, we might not be able to rollback to it. So fail early by not taking checkpoint at all.
self.logger.log_notice("Validating current config according to YANG models.")
if not self.config_wrapper.validate_config_db_config(json_content):
raise ValueError(f"Running configs on the device are not valid.")

self.logger.log_notice("Getting checkpoint full-path.")
path = self._get_checkpoint_full_path(checkpoint_name)

Expand Down Expand Up @@ -314,14 +296,12 @@ def execute_write_action(self, action, *args):
self.config_lock.release_lock()

class GenericUpdateFactory:
def create_patch_applier(self, config_format, verbose, dry_run):
def create_patch_applier(self, config_format, verbose, dry_run, ignore_non_yang_tables, ignore_paths):
self.init_verbose_logging(verbose)

config_wrapper = self.get_config_wrapper(dry_run)

patch_applier = PatchApplier(config_wrapper=config_wrapper)

patch_wrapper = PatchWrapper(config_wrapper)
patch_sorter = self.get_patch_sorter(ignore_non_yang_tables, ignore_paths, config_wrapper, patch_wrapper)
patch_applier = PatchApplier(config_wrapper=config_wrapper, patchsorter=patch_sorter, patch_wrapper=patch_wrapper)

if config_format == ConfigFormat.CONFIGDB:
pass
Expand All @@ -336,14 +316,13 @@ def create_patch_applier(self, config_format, verbose, dry_run):

return patch_applier

def create_config_replacer(self, config_format, verbose, dry_run):
def create_config_replacer(self, config_format, verbose, dry_run, ignore_non_yang_tables, ignore_paths):
self.init_verbose_logging(verbose)

config_wrapper = self.get_config_wrapper(dry_run)

patch_applier = PatchApplier(config_wrapper=config_wrapper)

patch_wrapper = PatchWrapper(config_wrapper)
patch_sorter = self.get_patch_sorter(ignore_non_yang_tables, ignore_paths, config_wrapper, patch_wrapper)
patch_applier = PatchApplier(config_wrapper=config_wrapper, patchsorter=patch_sorter, patch_wrapper=patch_wrapper)

config_replacer = ConfigReplacer(patch_applier=patch_applier, config_wrapper=config_wrapper)
if config_format == ConfigFormat.CONFIGDB:
Expand All @@ -359,12 +338,14 @@ def create_config_replacer(self, config_format, verbose, dry_run):

return config_replacer

def create_config_rollbacker(self, verbose, dry_run=False):
def create_config_rollbacker(self, verbose, dry_run=False, ignore_non_yang_tables=False, ignore_paths=[]):
self.init_verbose_logging(verbose)

config_wrapper = self.get_config_wrapper(dry_run)
patch_wrapper = PatchWrapper(config_wrapper)
patch_sorter = self.get_patch_sorter(ignore_non_yang_tables, ignore_paths, config_wrapper, patch_wrapper)
patch_applier = PatchApplier(config_wrapper=config_wrapper, patchsorter=patch_sorter, patch_wrapper=patch_wrapper)

patch_applier = PatchApplier(config_wrapper=config_wrapper)
config_replacer = ConfigReplacer(config_wrapper=config_wrapper, patch_applier=patch_applier)
config_rollbacker = FileSystemConfigRollbacker(config_wrapper = config_wrapper, config_replacer = config_replacer)

Expand All @@ -382,21 +363,36 @@ def get_config_wrapper(self, dry_run):
else:
return ConfigWrapper()

def get_patch_sorter(self, ignore_non_yang_tables, ignore_paths, config_wrapper, patch_wrapper):
if not ignore_non_yang_tables and not ignore_paths:
return StrictPatchSorter(config_wrapper, patch_wrapper)

inner_config_splitters = []
if ignore_non_yang_tables:
inner_config_splitters.append(TablesWithoutYangConfigSplitter(config_wrapper))

if ignore_paths:
inner_config_splitters.append(IgnorePathsFromYangConfigSplitter(ignore_paths, config_wrapper))

config_splitter = ConfigSplitter(config_wrapper, inner_config_splitters)

return NonStrictPatchSorter(config_wrapper, patch_wrapper, config_splitter)

class GenericUpdater:
def __init__(self, generic_update_factory=None):
self.generic_update_factory = \
generic_update_factory if generic_update_factory is not None else GenericUpdateFactory()

def apply_patch(self, patch, config_format, verbose, dry_run):
patch_applier = self.generic_update_factory.create_patch_applier(config_format, verbose, dry_run)
def apply_patch(self, patch, config_format, verbose, dry_run, ignore_non_yang_tables, ignore_paths):
patch_applier = self.generic_update_factory.create_patch_applier(config_format, verbose, dry_run, ignore_non_yang_tables, ignore_paths)
patch_applier.apply(patch)

def replace(self, target_config, config_format, verbose, dry_run):
config_replacer = self.generic_update_factory.create_config_replacer(config_format, verbose, dry_run)
def replace(self, target_config, config_format, verbose, dry_run, ignore_non_yang_tables, ignore_paths):
config_replacer = self.generic_update_factory.create_config_replacer(config_format, verbose, dry_run, ignore_non_yang_tables, ignore_paths)
config_replacer.replace(target_config)

def rollback(self, checkpoint_name, verbose, dry_run):
config_rollbacker = self.generic_update_factory.create_config_rollbacker(verbose, dry_run)
def rollback(self, checkpoint_name, verbose, dry_run, ignore_non_yang_tables, ignore_paths):
config_rollbacker = self.generic_update_factory.create_config_rollbacker(verbose, dry_run, ignore_non_yang_tables, ignore_paths)
config_rollbacker.rollback(checkpoint_name)

def checkpoint(self, checkpoint_name, verbose):
Expand Down
12 changes: 12 additions & 0 deletions generic_config_updater/gu_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ def __init__(self, patch):
def apply(self, config):
return self.patch.apply(config)

def __repr__(self):
return str(self)

def __str__(self):
return f'{self.patch}'

Expand Down Expand Up @@ -141,6 +144,12 @@ def get_empty_tables(self, config):
empty_tables.append(key)
return empty_tables

def remove_empty_tables(self, config):
config_with_non_empty_tables = {}
for table in config:
if config[table]:
config_with_non_empty_tables[table] = copy.deepcopy(config[table])
return config_with_non_empty_tables

class DryRunConfigWrapper(ConfigWrapper):
# TODO: implement DryRunConfigWrapper
Expand Down Expand Up @@ -236,6 +245,9 @@ def get_path_tokens(self, path):
def create_path(self, tokens):
return JsonPointer.from_parts(tokens).path

def has_path(self, doc, path):
return JsonPointer(path).get(doc, default=None) is not None

def get_xpath_tokens(self, xpath):
"""
Splits the given xpath into tokens by '/'.
Expand Down
Loading

0 comments on commit 7ceccd7

Please sign in to comment.