[k8s-configuration] Update sourceControlConfiguration SDK Models to T…

…rack2 (#3676)

* Create pull.yml

* Update pull.yml

* Update azure-pipelines.yml

* Initial commit of k8s-extension

* Update CODEOWNERS

* Update azure-pipelines.yml

* Create pull.yml

* Update pull.yml

* Update pull.yml

* Update pipelines file

* Update k8s-configuration name

* Update test script params

* Update pipeline file

* Remove codeowners

* Update pipelines file

* Update CODEOWNERS

* Update private preview pipelines

* Remove open service mesh from public release

* Update pipeline files

* Update custom pipelines files

* Add publish step to k8s-configuration

* Update pipeline to publish extension

* Update public extension pipeline

* Change condition variable

* Update pipeline naming

* Add version to public preview/private preview

* Update pipelines

* Add different testing based on private branch

* Add annotations to extension model

* Update k8s-custom-pipelines.yml

* Update SDKs with Updated Swagger Spec for 2020-07-01-preview (#13)

* Update sdks with updated swagger spec

* Update version and history rst

* Reorder release history timeline

* Fix ExtensionInstanceForCreate for import

* remove py2 bdist support

* Add custom table formatting

* Remove unnecessary files

* Fix style issues

* Fix branch based on comments

* Update identity piece manually

* Don't handle defaults at the CLI level

* Remove defaults from CLI client

* Check null target namespace with namespace scope

* Update style

* Add cassandra operator and location to model

* Stage Public Version of k8s-extension 0.2.0 for official release (#15)

* Create pull.yml

* Update pull.yml

* Update azure-pipelines.yml

* Initial commit of k8s-extension

* Update pipelines file

* Update CODEOWNERS

* Update private preview pipelines

* Remove open service mesh from public release

* Update pipeline files

* Update public extension pipeline

* Change condition variable

* Add version to public preview/private preview

* Update pipelines

* Add different testing based on private branch

* Add annotations to extension model

* Update k8s-custom-pipelines.yml

* Update SDKs with Updated Swagger Spec for 2020-07-01-preview (#13)

* Update sdks with updated swagger spec

* Update version and history rst

* Reorder release history timeline

* Fix ExtensionInstanceForCreate for import

* remove py2 bdist support

* Add custom table formatting

* Remove unnecessary files

* Fix style issues

* Fix branch based on comments

* Update identity piece manually

* Don't handle defaults at the CLI level

* Remove defaults from CLI client

* Check null target namespace with namespace scope

* Update style

* Add cassandra operator and location to model

Co-authored-by: action@github.com <Action - Fork Sync>

* Remove custom pipelines file

* Update extension description, remove private const

* Update pipeline file

* Disable check ref docs

* Disable refs docs

* Update to include better create warning logs and remove update context (#20)

* Update to include better create warning logs and remove update context

* Remove help text for update

* Fix spelling error

* Update message

* Fix k8s-extension conflict with private version

* Fix style errors

* Fix filename

* add customization for microsoft.azureml.kubernetes (#23)

* add customization for microsoft.azureml.kubernetes

* Update release history

Co-authored-by: Yue Yu <yuyu3@microsoft.com>
Co-authored-by: jonathan-innis <jonathan.innis.ji@gmail.com>

* Add E2E Testing from Separate branch into internal code (#26)

* Add internal e2e testing

* Change to testing folder

* Inference CLI validation for Scoring FE (#24)

* cli validation starter

* added the call to the fe validation function

* nodeport validation not required

* test fix

Co-authored-by: Jonathan Innis <jonathan.innis.ji@gmail.com>

* legal warning added (#27)

* Remove deprecated method logger.warn

* Update k8s-custom-pipelines.yml for Azure Pipelines

* Update k8s-custom-pipelines.yml for Azure Pipelines

* Add Azure Defender to E2E testing (#28)

* Add azure defender testing to e2e

* Remove the debug flag

* Add configuration testing

* Fix pipeline failures

* Make test script more intuitive

* Remove parameter from testing

* Add some debug

* Fix wrong location for k8s config whl

* Fix pip install upgrade issue

* Fix pip install upgrade issue

* Add Check for Provider Registration and Refactor (#19)

* Add check for provider registration and refactor

* Fix bug in checking registration

* Add license header to utils

* Update private key check and error messaging

* Update based on refactoring

* Fix failing tests

* Add provider registration check

* Create a test for uppercase url, address comments

* Add blank line to fix style check

* Testing increase to ubuntu-latest

* Update k8s-configuration Models to Track2 (#63)

* Update models to track2

* Increase k8s-configuration version number

* Update kind version

* Change error to warning because of DSA failure

* Remove unneeded files

Co-authored-by: action@github.com <Action - Fork Sync>
Co-authored-by: yuyue9284 <15863499+yuyue9284@users.noreply.github.com>
Co-authored-by: Yue Yu <yuyu3@microsoft.com>
Co-authored-by: Lia Kazakova <58274127+liakaz@users.noreply.github.com>

src/k8s-configuration/HISTORY.rst

@@ -3,6 +3,14 @@
 Release History
 ===============
 
+1.1.0
+++++++++++++++++++
+* Update sourceControlConfiguration resource models to Track2
+
+1.0.1
+++++++++++++++++++
+* Add provider registration check
+
 1.0.0
 ++++++++++++++++++
 * Support api-version 2021-03-01

src/k8s-configuration/azext_k8s_configuration/_client_factory.py

@@ -3,13 +3,18 @@
 # Licensed under the MIT License. See License.txt in the project root for license information.
 # --------------------------------------------------------------------------------------------
 
+from azure.cli.core.commands.client_factory import get_mgmt_service_client
 
-def cf_k8s_configuration(cli_ctx, *_):
 
-    from azure.cli.core.commands.client_factory import get_mgmt_service_client
+def cf_k8s_configuration(cli_ctx, *_):
     from azext_k8s_configuration.vendored_sdks import SourceControlConfigurationClient
     return get_mgmt_service_client(cli_ctx, SourceControlConfigurationClient)
 
 
 def cf_k8s_configuration_operation(cli_ctx, _):
     return cf_k8s_configuration(cli_ctx).source_control_configurations
+
+
+def _resource_providers_client(cli_ctx):
+    from azure.mgmt.resource import ResourceManagementClient
+    return get_mgmt_service_client(cli_ctx, ResourceManagementClient).providers

src/k8s-configuration/azext_k8s_configuration/_consts.py

@@ -0,0 +1,7 @@
+# --------------------------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# --------------------------------------------------------------------------------------------
+
+PROVIDER_NAMESPACE = 'Microsoft.KubernetesConfiguration'
+REGISTERED = "Registered"

src/k8s-configuration/azext_k8s_configuration/_params.py

@@ -13,7 +13,7 @@
 )
 
 from azure.cli.core.commands.validators import get_default_location_from_resource_group
-from ._validators import validate_configuration_type, validate_operator_namespace, validate_operator_instance_name
+from ._validators import _validate_configuration_type, _validate_operator_namespace, _validate_operator_instance_name
 
 
 def load_arguments(self, _):
@@ -38,7 +38,7 @@ def load_arguments(self, _):
                    arg_type=get_enum_type(['namespace', 'cluster']),
                    help='''Specify scope of the operator to be 'namespace' or 'cluster' ''')
         c.argument('configuration_type',
-                   validator=validate_configuration_type,
+                   validator=_validate_configuration_type,
                    arg_type=get_enum_type(['sourceControlConfiguration']),
                    help='Type of the configuration')
         c.argument('enable_helm_operator',
@@ -60,11 +60,11 @@ def load_arguments(self, _):
         c.argument('operator_instance_name',
                    arg_group="Operator",
                    help='Instance name of the Operator',
-                   validator=validate_operator_instance_name)
+                   validator=_validate_operator_instance_name)
         c.argument('operator_namespace',
                    arg_group="Operator",
                    help='Namespace in which to install the Operator',
-                   validator=validate_operator_namespace)
+                   validator=_validate_operator_namespace)
         c.argument('operator_type',
                    arg_group="Operator",
                    help='''Type of the operator. Valid value is 'flux' ''')

src/k8s-configuration/azext_k8s_configuration/_utils.py

@@ -0,0 +1,61 @@
+# --------------------------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# --------------------------------------------------------------------------------------------
+
+import base64
+from azure.cli.core.azclierror import MutuallyExclusiveArgumentError, InvalidArgumentValueError
+
+
+def _get_cluster_type(cluster_type):
+    if cluster_type.lower() == 'connectedclusters':
+        return 'Microsoft.Kubernetes'
+    # Since cluster_type is an enum of only two values, if not connectedClusters, it will be managedClusters.
+    return 'Microsoft.ContainerService'
+
+
+def _fix_compliance_state(config):
+    # If we get Compliant/NonCompliant as compliance_sate, change them before returning
+    if config.compliance_status.compliance_state.lower() == 'noncompliant':
+        config.compliance_status.compliance_state = 'Failed'
+    elif config.compliance_status.compliance_state.lower() == 'compliant':
+        config.compliance_status.compliance_state = 'Installed'
+
+    return config
+
+
+def _get_data_from_key_or_file(key, filepath):
+    if key != '' and filepath != '':
+        raise MutuallyExclusiveArgumentError(
+            'Error! Both textual key and key filepath cannot be provided',
+            'Try providing the file parameter without providing the plaintext parameter')
+    data = ''
+    if filepath != '':
+        data = _read_key_file(filepath)
+    elif key != '':
+        data = key
+    return data
+
+
+def _read_key_file(path):
+    try:
+        with open(path, "r") as myfile:  # user passed in filename
+            data_list = myfile.readlines()  # keeps newline characters intact
+            data_list_len = len(data_list)
+            if (data_list_len) <= 0:
+                raise Exception("File provided does not contain any data")
+            raw_data = ''.join(data_list)
+        return _to_base64(raw_data)
+    except Exception as ex:
+        raise InvalidArgumentValueError(
+            'Error! Unable to read key file specified with: {0}'.format(ex),
+            'Verify that the filepath specified exists and contains valid utf-8 data') from ex
+
+
+def _from_base64(base64_str):
+    return base64.b64decode(base64_str)
+
+
+def _to_base64(raw_data):
+    bytes_data = raw_data.encode('utf-8')
+    return base64.b64encode(bytes_data).decode('utf-8')

src/k8s-configuration/azext_k8s_configuration/_validators.py

@@ -4,34 +4,48 @@
 # --------------------------------------------------------------------------------------------
 
 import re
-from azure.cli.core.azclierror import InvalidArgumentValueError
+import io
+from azure.cli.core.azclierror import InvalidArgumentValueError, MutuallyExclusiveArgumentError
+
+from knack.log import get_logger
+from azext_k8s_configuration._client_factory import _resource_providers_client
+from azext_k8s_configuration._utils import _from_base64
+import azext_k8s_configuration._consts as consts
+from urllib.parse import urlparse
+from paramiko.hostkeys import HostKeyEntry
+from paramiko.ed25519key import Ed25519Key
+from paramiko.ssh_exception import SSHException
+from Crypto.PublicKey import RSA, ECC, DSA
+
+
+logger = get_logger(__name__)
 
 
 # Parameter-Level Validation
-def validate_configuration_type(configuration_type):
+def _validate_configuration_type(configuration_type):
     if configuration_type.lower() != 'sourcecontrolconfiguration':
         raise InvalidArgumentValueError(
             'Invalid configuration-type',
             'Try specifying the valid value "sourceControlConfiguration"')
 
 
-def validate_operator_namespace(namespace):
+def _validate_operator_namespace(namespace):
     if namespace.operator_namespace:
-        __validate_k8s_name(namespace.operator_namespace, "--operator-namespace", 23)
+        _validate_k8s_name(namespace.operator_namespace, "--operator-namespace", 23)
 
 
-def validate_operator_instance_name(namespace):
+def _validate_operator_instance_name(namespace):
     if namespace.operator_instance_name:
-        __validate_k8s_name(namespace.operator_instance_name, "--operator-instance-name", 23)
+        _validate_k8s_name(namespace.operator_instance_name, "--operator-instance-name", 23)
 
 
 # Create Parameter Validation
-def validate_configuration_name(configuration_name):
-    __validate_k8s_name(configuration_name, "--name", 63)
+def _validate_configuration_name(configuration_name):
+    _validate_k8s_name(configuration_name, "--name", 63)
 
 
 # Helper
-def __validate_k8s_name(param_value, param_name, max_len):
+def _validate_k8s_name(param_value, param_name, max_len):
     if len(param_value) > max_len:
         raise InvalidArgumentValueError(
             'Error! Invalid {0}'.format(param_name),
@@ -44,3 +58,86 @@ def __validate_k8s_name(param_value, param_name, max_len):
         raise InvalidArgumentValueError(
             'Error! Invalid {0}'.format(param_name),
             'Parameter {0} can only contain lowercase alphanumeric characters and hyphens'.format(param_name))
+
+
+def _validate_url_with_params(repository_url, ssh_private_key_set, known_hosts_contents_set, https_auth_set):
+    scheme = urlparse(repository_url).scheme
+
+    if scheme.lower() in ('http', 'https'):
+        if ssh_private_key_set:
+            raise MutuallyExclusiveArgumentError(
+                'Error! An --ssh-private-key cannot be used with an http(s) url',
+                'Verify the url provided is a valid ssh url and not an http(s) url')
+        if known_hosts_contents_set:
+            raise MutuallyExclusiveArgumentError(
+                'Error! --ssh-known-hosts cannot be used with an http(s) url',
+                'Verify the url provided is a valid ssh url and not an http(s) url')
+        if not https_auth_set and scheme == 'https':
+            logger.warning('Warning! https url is being used without https auth params, ensure the repository '
+                           'url provided is not a private repo')
+    else:
+        if https_auth_set:
+            raise MutuallyExclusiveArgumentError(
+                'Error! https auth (--https-user and --https-key) cannot be used with a non-http(s) url',
+                'Verify the url provided is a valid http(s) url and not an ssh url')
+
+
+def _validate_known_hosts(knownhost_data):
+    try:
+        knownhost_str = _from_base64(knownhost_data).decode('utf-8')
+    except Exception as ex:
+        raise InvalidArgumentValueError(
+            'Error! ssh known_hosts is not a valid utf-8 base64 encoded string',
+            'Verify that the string provided safely decodes into a valid utf-8 format') from ex
+    lines = knownhost_str.split('\n')
+    for line in lines:
+        line = line.strip(' ')
+        line_len = len(line)
+        if (line_len == 0) or (line[0] == "#"):
+            continue
+        try:
+            host_key = HostKeyEntry.from_line(line)
+            if not host_key:
+                raise Exception('not enough fields found in known_hosts line')
+        except Exception as ex:
+            raise InvalidArgumentValueError(
+                'Error! ssh known_hosts provided in wrong format',
+                'Verify that all lines in the known_hosts contents are provided in a valid sshd(8) format') from ex
+
+
+def _validate_private_key(ssh_private_key_data):
+    try:
+        RSA.import_key(_from_base64(ssh_private_key_data))
+        return
+    except ValueError:
+        try:
+            ECC.import_key(_from_base64(ssh_private_key_data))
+            return
+        except ValueError:
+            try:
+                DSA.import_key(_from_base64(ssh_private_key_data))
+                return
+            except ValueError:
+                try:
+                    key_obj = io.StringIO(_from_base64(ssh_private_key_data).decode('utf-8'))
+                    Ed25519Key(file_obj=key_obj)
+                    return
+                except SSHException:
+                    raise InvalidArgumentValueError(
+                        'Error! --ssh-private-key provided in invalid format',
+                        'Verify the key provided is a valid PEM-formatted key of type RSA, ECC, DSA, or Ed25519')
+
+
+# pylint: disable=broad-except
+def _validate_cc_registration(cmd):
+    try:
+        rp_client = _resource_providers_client(cmd.cli_ctx)
+        registration_state = rp_client.get(consts.PROVIDER_NAMESPACE).registration_state
+
+        if registration_state.lower() != consts.REGISTERED.lower():
+            logger.warning("'Source Control Configuration' cannot be used because '%s' provider has not been "
+                           "registered. More details for registering this provider can be found here - "
+                           "https://aka.ms/RegisterKubernetesConfigurationProvider", consts.PROVIDER_NAMESPACE)
+    except Exception:
+        logger.warning("Unable to fetch registration state of '%s' provider. "
+                       "Failed to enable 'source control configuration' feature...", consts.PROVIDER_NAMESPACE)