WIP PvC Prereqs and Control Plane merge (cloudera-labs#61)

* Add control keys for autotls, pvc_type, free_ipa to control deployment behavior more simply.
* Standardise the host group name for ECS nodes to be 'ecs_nodes' to match the other standard groups we use in Ansible inventory
* Deprecate duplicate filter_null_configs filter from api_client
* Add handler to restart cloudera management service
* Migrate autotls implementation
* Migrate cms_tls setup 
* Add default external_auth configuration to generally handle freeipa or mit setup
* Update importAdminCredentials command to not fail when already imported, but report other errors
* Add new role cloudera_manager.services_info to perform useful service discovery on existing clusters. 
* Migrate role to set session_timeout for cloudera_manager
* Migrate role to set hue_ticket_lifetime for PvC-DS deployments
* Migrate role to setup TLS for KMS
* Migrate role to fix some libs for the Oozie UI in some PvC-DS deployments
* Migrate role to setup some default Ranger policies for some PvC-DS deployments
* Migrate role to setup a SOLR role in Knox for some PvC-DS deployments
* Migrate role to ensure a Ranger plugin for SOLR is deployed in some PvC-DS deployments
* Update the defaults for database type and version to respond to el7 or el8 appropriately.
* Migrate role to setup WXM.
* Update krb5_client deployment for FreeIPA setup, including a patch for dbus_session config and specific configs for when running PvC-DS.
* Default krb5_domain to krb5_realm.lower automatically.
* Add default kerberos configuration to krb5_common, including simple defaults for when MIT KDC or Red Hat IPA are selected. Passwords default to the cloudera_manager_admin_password instead of hardcoded values like 'changeme'
* Add fixes for FreeIPA server deployment
* Fix refresh_ranger_kms_repo role to function correctly when determining the Ranger URL in modern Ansible.
* Add operation to restart a given cluster or a given cluster's services or cluster management services for user convenience. They could be handlers, but this felt more useful as more people know how to use roles than handlers.
* Migrate role to setup iptables or nftables for PvC ECS deployment on Rhel7 or Rhel8
* Add firewalld to unwanted services during automated os prereq setup
* Add fix where setting up postgresql_connector sometimes requires python3-psycopg2 to be setup for SSB.
* Migrate role to set up a subset of necessary local accounts on ecs_nodes
* Add check to ensure that FreeIPA and a custom repo are not on the same host as they both try to hardcode port 8443.
* Enhance error message when TLS setup is only being partially applied to hosts in the cluster definition
* Update ecs cluster template to set version to DATA_SERVICES1 to reflect current Cloudera Manager 7.6.5 requirements
* Modifie ecs services Jinja template to seek host groups by long name. 
* Explicitly set default database_type to postgresql to avoid user confusion
* Add nfs-utils to OS prereqs when installing ECS
* Add control for whether or not embedded database mode for ECS is implemented
* Remove unused deployment.j2 template
* Add controlPlaneValuesEmbedded.j2 for embedded database values
* Fix bug in services.j2 for ECS deployment where it would look for the wrong host template name
* Rename free_ipa switch to freeipa_activated to match other top level switches
* Allow Cloudera Manager version and distro to be set explicitly for repo setup
* Update default cloudera-manager version to 7.6.5
* Fix custom_repo to recognise ecs_nodes as valid
* Update dbus patch for freeipa client to only restart services if something is changed
* Add autodns support to freeipa clients
* Add autodns function to freeipa server setup, including creating required zones and records for PvC-DS ECS if that is being deployed
* Add task to Flush and Delete IPTables when setting up ECS
* Set default Cloudera Manager version to 7.6.1 for base deployments. (7.6.5 is primarily for PvC-DS.)
* Add draft ECS teardown processes
* Add cloudera.cluster.operations.stop_cluster as a convenience method, as ECS needs to be stopped and cleaned in a specific sequence.
* Provide additional wildcard DNS records for ECS in FreeIPA Autodns setup

Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com>

roles/cloudera_manager/config/filter_plugins/filters.py → roles/cloudera_manager/api_client/handlers/main.yml

@@ -1,4 +1,3 @@
-#!/usr/bin/python
 # Copyright 2021 Cloudera, Inc.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -13,17 +12,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+---
 
-class FilterModule(object):
-
-  def filters(self):
-    return {
-      'filter_null_configs': self.filter_null_configs
-    }
-
-  def filter_null_configs(self, configs, existing_configs):
-    filtered_configs = dict(configs)
-    for item, value in configs.items():
-      if item not in existing_configs and not value:
-        del filtered_configs[item]
-    return filtered_configs
+- name: restart cloudera management service
+  cm_api:
+    endpoint: /cm/service/commands/restart
+    method: POST
+    timeout: "{{ cluster_restart_timeout | default(3000) }}"

roles/cloudera_manager/autotls/files/cert.py_patch

@@ -0,0 +1,11 @@
+--- cert.py	2020-12-02 00:54:05.000000000 +0100
++++ cert.py_2	2021-02-18 09:09:38.095192730 +0100
+@@ -1949,7 +1949,7 @@
+       LOG.info("Could not find JKS truststore at location: %s. Converting "
+                "PEM truststore to JKS." % cluster_ca_jks)
+       generate_truststore(self.cfg.keytool, cluster_ca_jks, truststore_password,
+-                          cluster_ca_pem)
++                          cluster_ca_pem, self.cfg.keystore_type)
+
+     global_ca_pem = self.trust_files[GLOBAL_TLS_SET][PEM_TLS_TYPE]
+     copied_cluster_to_global = False

roles/cloudera_manager/autotls/tasks/main.yml

@@ -23,23 +23,48 @@
     msg: This playbook requires Cloudera Manager 7.1+
   when: response.json.version is version('7.1', '<')
 
+- name: Patch Cloudera Manager older than 7.3
+  include_tasks:
+    file: patch_old_cm
+  when: response.json.version is version('7.3.0', '<')
+
+- name: Check if password or key is used to connect to machines
+  set_fact:
+    use_password: "{{ true if node_password is defined and node_password|length > 0 else false }}"
+
+- name: DEBUG Auto-TLS using password
+  debug:
+    msg: "{{ lookup('template', 'auto-tls.json') }}"
+  when: use_password and debug | default(false)
+
 - name: Enable Auto-TLS
-  cloudera.cluster.cm_api:
-    endpoint: /cm/commands/generateCmca
+  cm_api:
+    endpoint: "/cm/commands/generateCmca"
     method: POST
-    body: "{{ lookup('template', 'request.j2', convert_data=False) }}"
+    body: "{{ lookup('template', 'auto-tls.json') }}"
+    timeout: 360
+  ignore_errors: true
+  when: use_password
 
-- name: Restart Cloudera Manager Server
-  service:
-    name: cloudera-scm-server
-    state: restarted
-  notify:
-    - wait cloudera-scm-server
+- name: Set node_key on one line
+  set_fact:
+    node_key_one_line: "{{ lookup('file', '~/node_key' ) | replace('\n', '\\n') | replace('\"', '\\\"' ) }}"
+  when: not use_password
 
-- name: Wait for Cloudera Manager Server to come back up
-  meta: flush_handlers
+- name: DEBUG Auto-TLS using key
+  debug:
+    msg: "{{ lookup('template', 'auto-tls-key.json') }}"
+  when: not use_password
 
-- name: Restart Cloudera Management Service
-  cloudera.cluster.cm_api:
-    endpoint: /cm/service/commands/restart
+- name: Enable Auto-TLS
+  cm_api:
+    endpoint: "/cm/commands/generateCmca"
     method: POST
+    body: "{{ lookup('template', 'auto-tls-key.json') }}"
+  ignore_errors: true
+  when: not use_password
+  notify:
+    - restart cloudera-scm-server
+    - restart cloudera management service
+    - restart cloudera-scm-agent
+

roles/cloudera_manager/autotls/tasks/patch_old_cm.yml

@@ -0,0 +1,16 @@
+---
+- name: Copy patch to machines
+  copy:
+    src: "{{ role_path}}/files/cert.py_patch"
+    dest: /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/tools/cert.py_patch
+    owner: cloudera-scm
+    group: cloudera-scm
+    mode: '0644'
+
+- name: Backup cert.py
+  shell: cp /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/tools/cert.py /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/tools/cert.py.backup
+
+- name: Fix cert.py
+  ansible.posix.patch:
+    src: "{{ role_path}}/patch/cert.py_patch"
+    dest: /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/tools/cert.py

roles/cloudera_manager/autotls/templates/auto-tls-key.json

@@ -0,0 +1,9 @@
+{
+    "customCA" : false,
+    "configureAllServices" : "true",
+    "sshPort" : 22,
+    {% if freeipa_activated %}"trustedCaCerts" : "/etc/ipa/ca.crt",{% endif %}
+    "userName" : "root",
+    "privateKey": "{{ node_key_one_line }}"
+} 
+

roles/cloudera_manager/autotls/templates/auto-tls.json

@@ -0,0 +1,9 @@
+{
+    "customCA" : false,
+    "configureAllServices" : "true",
+    "sshPort" : 22,
+    {% if freeipa_activated %}"trustedCaCerts" : "/etc/ipa/ca.crt",{% endif %}
+    "userName" : "root",
+    "password": "{{ node_password }}"
+} 
+

roles/cloudera_manager/cms_tls/files/cms_keystore_tls.json

@@ -0,0 +1,16 @@
+{
+    "items": [
+      {
+        "name": "ssl_server_keystore_location",
+        "value": "{{CM_AUTO_TLS}}"
+      },
+      {
+        "name": "ssl_server_keystore_password",
+        "value": "{{CM_AUTO_TLS}}"
+      },
+      {
+        "name": "ssl_enabled",
+        "value": "true"
+      }
+    ]
+  }

roles/cloudera_manager/cms_tls/files/cms_navigator_keystore_tls.json

@@ -0,0 +1,28 @@
+{
+    "items": [
+      {
+        "name": "navigator_truststore_file",
+        "value": "{{CM_AUTO_TLS}}"
+      },
+      {
+        "name": "navigator_truststore_password",
+        "value": "{{CM_AUTO_TLS}}"
+      },
+      {
+        "name": "ssl_server_keystore_location",
+        "value": "{{CM_AUTO_TLS}}"
+      },
+      {
+        "name": "ssl_server_keystore_password",
+        "value": "{{CM_AUTO_TLS}}"
+      },
+      {
+        "name": "ssl_server_keystore_keypassword",
+        "value": "{{CM_AUTO_TLS}}"
+      },
+      {
+        "name": "ssl_enabled",
+        "value": "true"
+      }
+    ]
+  }

roles/cloudera_manager/cms_tls/files/cms_navigator_metaserver_keystore_tls.json

@@ -0,0 +1,20 @@
+{
+    "items": [
+      {
+        "name": "ssl_server_keystore_location",
+        "value": "{{CM_AUTO_TLS}}"
+      },
+      {
+        "name": "ssl_server_keystore_password",
+        "value": "{{CM_AUTO_TLS}}"
+      },
+      {
+        "name": "ssl_server_keystore_keypassword",
+        "value": "{{CM_AUTO_TLS}}"
+      },
+      {
+        "name": "ssl_enabled",
+        "value": "true"
+      }
+    ]
+  }

roles/cloudera_manager/cms_tls/files/cms_truststore_tls.json

@@ -0,0 +1,12 @@
+{
+    "items": [
+      {
+        "name": "ssl_client_truststore_location",
+        "value": "{{CM_AUTO_TLS}}"
+      },
+      {
+        "name": "ssl_client_truststore_password",
+        "value": "{{CM_AUTO_TLS}}"
+      }
+    ]
+  }

roles/cloudera_manager/cms_tls/meta/main.yml

@@ -0,0 +1,3 @@
+---
+dependencies:
+  - role: cloudera.cluster.cloudera_manager.api_client

roles/cloudera_manager/cms_tls/tasks/main.yml

@@ -0,0 +1,40 @@
+---
+- name: Setup TLS for Activity Monitor
+  cm_api:
+    method: PUT 
+    endpoint: /cm/service/roleConfigGroups/mgmt-ACTIVITYMONITOR-BASE/config
+    body: "{{ lookup('file', 'cms_keystore_tls.json', convert_data=False) }}"
+
+- name: Setup TLS for Host Monitor
+  cm_api:
+    method: PUT 
+    endpoint: /cm/service/roleConfigGroups/mgmt-HOSTMONITOR-BASE/config
+    body: "{{ lookup('file', 'cms_keystore_tls.json', convert_data=False) }}"
+
+- name: Setup TLS for Service Monitor
+  cm_api:
+    method: PUT 
+    endpoint: /cm/service/roleConfigGroups/mgmt-SERVICEMONITOR-BASE/config
+    body: "{{ lookup('file', 'cms_keystore_tls.json', convert_data=False) }}"
+
+- name: Setup TLS for Navigator
+  cm_api:
+    method: PUT 
+    endpoint: /cm/service/roleConfigGroups/mgmt-NAVIGATOR-BASE/config
+    body: "{{ lookup('file', 'cms_navigator_keystore_tls.json', convert_data=False) }}"
+  when: cloudera_manager_version is version('7.0.0','<')
+
+- name: Setup TLS for Navigator Meta Server 
+  cm_api:
+    method: PUT 
+    endpoint: /cm/service/roleConfigGroups/mgmt-NAVIGATORMETASERVER-BASE/config
+    body: "{{ lookup('file', 'cms_navigator_metaserver_keystore_tls.json', convert_data=False) }}"
+  when: cloudera_manager_version is version('7.0.0','<')
+
+- name: Setup TLS for CMS
+  cm_api:
+    method: PUT 
+    endpoint: /cm/service/config
+    body: "{{ lookup('file', 'cms_truststore_tls.json', convert_data=False) }}"
+  notify:
+    - restart cloudera management service

roles/cloudera_manager/config/meta/main.yml

@@ -14,6 +14,6 @@
 
 ---
 dependencies:
-  - role: cloudera_manager/api_client
+  - role: cloudera.cluster.cloudera_manager.api_client
 
 

roles/cloudera_manager/external_auth/defaults/main.yml

@@ -13,7 +13,31 @@
 # limitations under the License.
 
 ---
-
 cloudera_manager_external_auth:
+  provider: "{{ 'FreeIPA' if freeipa_activated == true else omit }}"
   external_first: no
   external_only: no
+  external_set: "{{ 'yes' if freeipa_activated == true else 'no' }}"
+  role_mappings: "{{ default_free_ipa_role_mappings if freeipa_activated == true else omit }}"
+
+default_free_ipa_role_mappings:
+  - group: admins
+    roles: [ ROLE_ADMIN ]
+  - group: auditors
+    roles: [ ROLE_AUDITOR ]
+  - group: users
+    roles: [ ROLE_USER ]
+
+auth_providers:
+  FreeIPA:
+    type: LDAP
+    ldap_url: "{{ ipa_ldap_url }}"
+    ldap_base_dn:
+    ldap_bind_user_dn: "{{ ipa_ldap_user_bind_dn }}"
+    ldap_bind_password: "{{ ipa_ldap_user_bind_password }}"
+    ldap_search_base:
+      user: "{{ ipa_ldap_user_search_base }}"
+      group: "{{ ipa_ldap_group_search_base }}"
+    ldap_search_filter:
+      user: "{{ ipa_ldap_user_search_filter }}"
+      group: "{{ ipa_ldap_user_group_filter }}"

roles/cloudera_manager/external_auth/meta/main.yml

@@ -16,3 +16,4 @@
 
 dependencies:
   - role: cloudera.cluster.cloudera_manager.api_client
+  - role: cloudera.cluster.infrastructure.krb5_common

roles/cloudera_manager/kerberos/tasks/main.yml

@@ -26,6 +26,9 @@
     endpoint: /cm/commands/importAdminCredentials?username={{ krb5_kdc_admin_user | urlencode }}&password={{ krb5_kdc_admin_password | urlencode }}
     method: POST
   register: result
+  failed_when:
+    - result is failed
+    - "'already exists' not in result.content"
   until: result is not failed
   retries: 3
   delay: 10

roles/cloudera_manager/repo/defaults/main.yml

@@ -14,6 +14,8 @@
 
 ---
 cloudera_archive_base_url: https://archive.cloudera.com
-cloudera_manager_version: 7.4.4
+cloudera_manager_version: 7.6.1
+cloudera_manager_distro_name: "{{ ansible_os_family | lower }}"
+cloudera_manager_distro_version: "{{ ansible_distribution_major_version }}"
 
-install_repo_on_host: yes
+install_repo_on_host: yes

roles/cloudera_manager/repo/vars/RedHat.yml

@@ -13,12 +13,11 @@
 # limitations under the License.
 
 ---
-__cloudera_manager_distro_name: "{{ ansible_os_family | lower }}{{ ansible_distribution_major_version }}"
 __cloudera_manager_major_version: "{{ cloudera_manager_version.split('.')[0] }}"
 __cloudera_manager_cm5_path: "{{ ansible_os_family | lower }}/{{ ansible_distribution_major_version }}/x86_64/cm/{{ cloudera_manager_version }}"
-__cloudera_manager_cm6_path: "{{ cloudera_manager_version }}/{{ __cloudera_manager_distro_name }}/yum"
+__cloudera_manager_cm6_path: "{{ cloudera_manager_version }}/{{ cloudera_manager_distro_name }}{{ cloudera_manager_distro_version }}/yum"
 
-__cloudera_manager_repo_url_trial: "{{ cloudera_archive_base_url | regex_replace('/?$','') }}/cm{{ __cloudera_manager_major_version }}/{{ cloudera_manager_version }}/{{ __cloudera_manager_distro_name }}/yum"
+__cloudera_manager_repo_url_trial: "{{ cloudera_archive_base_url | regex_replace('/?$','') }}/cm{{ __cloudera_manager_major_version }}/{{ cloudera_manager_version }}/{{ cloudera_manager_distro_name }}{{ cloudera_manager_distro_version }}/yum"
 __cloudera_manager_repo_url_paywall: "{{ cloudera_archive_base_url | regex_replace('/?$','') }}/p/cm{{ __cloudera_manager_major_version }}/{{ (__cloudera_manager_major_version == '5' ) | ternary(__cloudera_manager_cm5_path, __cloudera_manager_cm6_path) }}"
 
 __cloudera_manager_repo_key_filename: "RPM-GPG-KEY-cloudera"

roles/cloudera_manager/services_info/defaults/main.yml

@@ -0,0 +1,6 @@
+cluster_name: Default
+ranger_user: "{{ ranger_rangeradmin_user | default('admin') }}"
+ranger_password: "{{ ranger_rangeradmin_user_password | default(cloudera_manager_admin_password) }}"
+solr_admin_password: "{{ solr_solradmin_user_password | default(cloudera_manager_admin_password) }}"
+
+wxm_api_port: 12022

roles/operations/restart_agents/tasks/main.yml → roles/cloudera_manager/services_info/meta/main.yml

@@ -13,7 +13,6 @@
 # limitations under the License.
 
 ---
-- name: restart cloudera-scm-agent
-  service:
-    name: cloudera-scm-agent
-    state: restarted
+
+dependencies:
+  - role: cloudera.cluster.cloudera_manager.api_client