WIP PvC Prereqs and Control Plane merge #61
Conversation
…ent behavior more simply. Some of these are inherited from the FRish fork Standardised the host group name for ECS nodes to be 'ecs_nodes' to match the other standard groups we use in Ansible inventory deprecated duplicate filter_null_configs filter from api_client Added handler to restart cloudera management service Migrated autotls implementation from FRisch fork for PvC-CP setup Migrated cms_tls setup from FRisch fork for PvC-CP setup Added default external_auth configuration to generally handle freeipa or mit setup Updated importAdminCredentials command to not fail when already imported, but report other errors Added new role cloudera_manager.services_info to perform useful service discovery on existing clusters. Should probably be a module. WIP. Migrated role to set session_timeout for cloudera_manager from FRisch fork for PvC-CP setup Migrated role from FRisch fork to set hue_ticket_lifetime for PvC-DS deployments Migrated role from FRisch fork to setup tls for kms Migrated role from FRisch fork to fix some libs for the Oozie UI in some PvC-DS deployments Migrated role from FRisch fork to setup some default Ranger policies for some PvC-DS deployments Migrated role from FRisch fork to setup a solr role in knox for some PvC-DS deployments Migrated role from FRisch fork to ensure a ranger plugin for solr is deployed in some PvC-DS deployments Updated the defaults for database type and version to respond to el7 or el8 appropriately. Could use defaults for more OS and versions. WIP. Migrated role from FRisch fork to setup WXM. Needs more testing. WIP Updated krb5_client deployment with patches from FRisch fork for FreeIPA setup, including a patch for dbus_session config and specific configs for when running PvC-DS. krb5_domain is now defaulted to krb5_realm.lower automatically. Finally. Default kerberos configuration added to krb5_common, including simple defaults for when MIT KDC or Red Hat IPA are selected. Passwords are defaulted to the cloudera_manager_admin_password instead of hardcoded values like changeme Migrated in fixes for FreeIPA server deployment from FRisch fork Fixed role to refresh_ranger_kms_repo to function correctly when determining the Ranger URL in modern Ansible. Really should be a module. WIP. Added operation to restart a given cluster or a given cluster's services or cluster management services for user convenience. They could be handlers, but this felt more useful as more people know how to use roles than handlers. Migrated role to setup iptables or nftables for PvC ECS deployment on Rhel7 or Rhel8 from Chuck Levesque Added firewalld to unwanted services during automated os prereq setup, as it is unwelcome for PvC-DS deployments Migrated fix from FRisch fork where setting up postgresql_connector sometimes requires python3-psycopg2 to be setup for SSB. Put behind a default False switch of install_py3_psycopg2 Migrated role from Chuck Levesque to set up a subset of necessary local accounts on ecs_nodes Added check to ensure that FreeIPA and a Custom Repo are not on the same host as they both try to hardcode port 8443. Honestly who does this. Enhanced error message when tls setup is only being partially applied to hosts in the cluster definition Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com>
…ect current cloudera manager 7.6.5 requirements Modified ecs services jinja template to seek host groups by long name. This appears to correct a bug, but these submission templates should be migrated from Jinja to something more sustainable. Explicitly set default database_type to postgresql to avoid user confusion Added nfs-utils to OS prereqs when installing ECS Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com>
…emented Remove unused deployment.j2 template Added controlPlaneValuesEmbedded.j2 for embedded database values Fixed bug in services.j2 for ECS deployment where it would look for the wrong host template name Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com>
…switches Allow Cloudera Manager version and distro to be set explicitly for repo setup Update default cloudera-manager version to 7.6.5 fix custom_repo to recognise ecs_nodes as valid Update dbus patch for freeipa client to only restart services if something is changed add autodns support to freeipa clients Add autodns function to freeipa server setup, including creating required zones and records for PvC-DS ECS if that is being deployed Add task to Flush and Delete IPTables when setting up ECS Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com>
….6.5 is primarily for PvC-DS. Add draft ECS teardown processes Add cloudera.cluster.operations.stop_cluster as a convenience method, as ECS needs to be stopped and cleaned in a specific sequence. Provide additional wildcard DNS records for ECS in FreeIPA Autodns setup Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com>
|
IPA Server install fails on Rhel8 TASK [freeipa.ansible_freeipa.ipaserver : Install - Setup DS] ****************** TASK [freeipa.ansible_freeipa.ipaserver : Cleanup temporary files] ************* PLAY RECAP ********************************************************************* Saturday 06 August 2022 16:11:51 +0000 (0:00:05.842) 0:13:34.722 *******cloudera.cluster.user_accounts : Create local user accounts ------------ 88.58s |
|
cloudera.cluster.tls_generate_csr In ECS world, we correctly create only one user "cloudera-scm", we dont have any hadoopy users. When the tls_generate runs, it does not respect that this host does not want any new users. |
|
cloudera.cluster.databases : Create database roles This creates ECS/OCP DS roles and schemas unnecessarily (and its slow and annoying), the 11 DS schema's are never used when we deploy the using the DataServices "embedded-db" container. These schemas are: |
|
Please note that this PR contains breaking changes as we begin development on the 4.x release. |
|
@clevesque - please submit your bug fixes as PRs against the update |
* Add control keys for autotls, pvc_type, free_ipa to control deployment behavior more simply. * Standardise the host group name for ECS nodes to be 'ecs_nodes' to match the other standard groups we use in Ansible inventory * Deprecate duplicate filter_null_configs filter from api_client * Add handler to restart cloudera management service * Migrate autotls implementation * Migrate cms_tls setup * Add default external_auth configuration to generally handle freeipa or mit setup * Update importAdminCredentials command to not fail when already imported, but report other errors * Add new role cloudera_manager.services_info to perform useful service discovery on existing clusters. * Migrate role to set session_timeout for cloudera_manager * Migrate role to set hue_ticket_lifetime for PvC-DS deployments * Migrate role to setup TLS for KMS * Migrate role to fix some libs for the Oozie UI in some PvC-DS deployments * Migrate role to setup some default Ranger policies for some PvC-DS deployments * Migrate role to setup a SOLR role in Knox for some PvC-DS deployments * Migrate role to ensure a Ranger plugin for SOLR is deployed in some PvC-DS deployments * Update the defaults for database type and version to respond to el7 or el8 appropriately. * Migrate role to setup WXM. * Update krb5_client deployment for FreeIPA setup, including a patch for dbus_session config and specific configs for when running PvC-DS. * Default krb5_domain to krb5_realm.lower automatically. * Add default kerberos configuration to krb5_common, including simple defaults for when MIT KDC or Red Hat IPA are selected. Passwords default to the cloudera_manager_admin_password instead of hardcoded values like 'changeme' * Add fixes for FreeIPA server deployment * Fix refresh_ranger_kms_repo role to function correctly when determining the Ranger URL in modern Ansible. * Add operation to restart a given cluster or a given cluster's services or cluster management services for user convenience. They could be handlers, but this felt more useful as more people know how to use roles than handlers. * Migrate role to setup iptables or nftables for PvC ECS deployment on Rhel7 or Rhel8 * Add firewalld to unwanted services during automated os prereq setup * Add fix where setting up postgresql_connector sometimes requires python3-psycopg2 to be setup for SSB. * Migrate role to set up a subset of necessary local accounts on ecs_nodes * Add check to ensure that FreeIPA and a custom repo are not on the same host as they both try to hardcode port 8443. * Enhance error message when TLS setup is only being partially applied to hosts in the cluster definition * Update ecs cluster template to set version to DATA_SERVICES1 to reflect current Cloudera Manager 7.6.5 requirements * Modifie ecs services Jinja template to seek host groups by long name. * Explicitly set default database_type to postgresql to avoid user confusion * Add nfs-utils to OS prereqs when installing ECS * Add control for whether or not embedded database mode for ECS is implemented * Remove unused deployment.j2 template * Add controlPlaneValuesEmbedded.j2 for embedded database values * Fix bug in services.j2 for ECS deployment where it would look for the wrong host template name * Rename free_ipa switch to freeipa_activated to match other top level switches * Allow Cloudera Manager version and distro to be set explicitly for repo setup * Update default cloudera-manager version to 7.6.5 * Fix custom_repo to recognise ecs_nodes as valid * Update dbus patch for freeipa client to only restart services if something is changed * Add autodns support to freeipa clients * Add autodns function to freeipa server setup, including creating required zones and records for PvC-DS ECS if that is being deployed * Add task to Flush and Delete IPTables when setting up ECS * Set default Cloudera Manager version to 7.6.1 for base deployments. (7.6.5 is primarily for PvC-DS.) * Add draft ECS teardown processes * Add cloudera.cluster.operations.stop_cluster as a convenience method, as ECS needs to be stopped and cleaned in a specific sequence. * Provide additional wildcard DNS records for ECS in FreeIPA Autodns setup Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com>
* Update with Private Cloud prerequisite and Control Plane changes (#61) * Add control keys for autotls, pvc_type, free_ipa to control deployment behavior more simply. * Standardise the host group name for ECS nodes to be 'ecs_nodes' to match the other standard groups we use in Ansible inventory * Deprecate duplicate filter_null_configs filter from api_client * Add handler to restart cloudera management service * Migrate autotls implementation * Migrate cms_tls setup * Add default external_auth configuration to generally handle freeipa or mit setup * Update importAdminCredentials command to not fail when already imported, but report other errors * Add new role cloudera_manager.services_info to perform useful service discovery on existing clusters. * Migrate role to set session_timeout for cloudera_manager * Migrate role to set hue_ticket_lifetime for PvC-DS deployments * Migrate role to setup TLS for KMS * Migrate role to fix some libs for the Oozie UI in some PvC-DS deployments * Migrate role to setup some default Ranger policies for some PvC-DS deployments * Migrate role to setup a SOLR role in Knox for some PvC-DS deployments * Migrate role to ensure a Ranger plugin for SOLR is deployed in some PvC-DS deployments * Update the defaults for database type and version to respond to el7 or el8 appropriately. * Migrate role to setup WXM. * Update krb5_client deployment for FreeIPA setup, including a patch for dbus_session config and specific configs for when running PvC-DS. * Default krb5_domain to krb5_realm.lower automatically. * Add default kerberos configuration to krb5_common, including simple defaults for when MIT KDC or Red Hat IPA are selected. Passwords default to the cloudera_manager_admin_password instead of hardcoded values like 'changeme' * Add fixes for FreeIPA server deployment * Fix refresh_ranger_kms_repo role to function correctly when determining the Ranger URL in modern Ansible. * Add operation to restart a given cluster or a given cluster's services or cluster management services for user convenience. They could be handlers, but this felt more useful as more people know how to use roles than handlers. * Migrate role to setup iptables or nftables for PvC ECS deployment on Rhel7 or Rhel8 * Add firewalld to unwanted services during automated os prereq setup * Add fix where setting up postgresql_connector sometimes requires python3-psycopg2 to be setup for SSB. * Migrate role to set up a subset of necessary local accounts on ecs_nodes * Add check to ensure that FreeIPA and a custom repo are not on the same host as they both try to hardcode port 8443. * Enhance error message when TLS setup is only being partially applied to hosts in the cluster definition * Update ecs cluster template to set version to DATA_SERVICES1 to reflect current Cloudera Manager 7.6.5 requirements * Modifiy ecs services Jinja template to seek host groups by long name. * Explicitly set default database_type to postgresql to avoid user confusion * Add nfs-utils to OS prereqs when installing ECS * Add control for whether or not embedded database mode for ECS is implemented * Remove unused deployment.j2 template * Add controlPlaneValuesEmbedded.j2 for embedded database values * Fix bug in services.j2 for ECS deployment where it would look for the wrong host template name * Rename free_ipa switch to freeipa_activated to match other top level switches * Allow Cloudera Manager version and distro to be set explicitly for repo setup * Update default cloudera-manager version to 7.6.5 * Fix custom_repo to recognise ecs_nodes as valid * Update dbus patch for freeipa client to only restart services if something is changed * Add autodns support to freeipa clients * Add autodns function to freeipa server setup, including creating required zones and records for PvC-DS ECS if that is being deployed * Add task to Flush and Delete IPTables when setting up ECS * Set default Cloudera Manager version to 7.6.1 for base deployments. (7.6.5 is primarily for PvC-DS.) * Add draft ECS teardown processes * Add cloudera.cluster.operations.stop_cluster as a convenience method, as ECS needs to be stopped and cleaned in a specific sequence. * Provide additional wildcard DNS records for ECS in FreeIPA Autodns setup * Fixes for RHEL8.6 support and custom_repo with Cloudera Manager (#83) * Fix download and reuse of Cloudera Manager repo-as-tarball * Switch to using custom_repo as base url for archive.cloudera.com by default if custom_repo is specified in build without a specific base_url being supplied. * Fix kts setup for RHEL8+ where gpg 2.1+ is used which has changed the default file set - look for kbx files. * Setup of automatic DNS on the freeipa server now supports running on ec2 instances with RHEL8.6, and EL8 generally * Automatically set selinux to permissive on the krb5_server for RHEL8, as otherwise setup is blocked * Separate Py2 and Py3 setup on RHEL8 in preparation for final Py2 deprecation in Cloudera products * Set default admins group as configurable variable in freeipa config * Add option to ensure creation of a 'superuser' in FreeIPA as the default 'admin' user can clash with system users within CDP in some circumstances. This user is then useful to PvC-DS installs and not created by default otherwise. * Create Cloudera Manager module framework (#62) * Create common cm_utils.py for managing CM API interactions, unit and integration test frameworks, and cm_version_info and cm_endpoint_info modules. * Clean up error messages and remove unused imports * Create ad-hoc/unimplemented API resource module, cm_resource_info * Fix JSON decode bug for ApiExceptions * Centralize call_api method * Add documentation and document fragment for cm_resource * Create cm_resource module for ad-hoc CM API endpoint calls * Update to handle private IP installations (#93) * Handle delegation to CM server when needed * Handle remote CA management for sidecar/embedded FreeIPA installs Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com> Signed-off-by: Webster Mudge <wmudge@cloudera.com> Co-authored-by: Dan Chaffelson <chaffelson@gmail.com>
Added control keys for autotls, pvc_type, free_ipa to control deployment behavior more simply. Some of these are inherited from the FRish fork
Standardised the host group name for ECS nodes to be 'ecs_nodes' to match the other standard groups we use in Ansible inventory
deprecated duplicate filter_null_configs filter from api_client
Added handler to restart cloudera management service
Migrated autotls implementation from FRisch fork for PvC-CP setup
Migrated cms_tls setup from FRisch fork for PvC-CP setup
Added default external_auth configuration to generally handle freeipa or mit setup
Updated importAdminCredentials command to not fail when already imported, but report other errors
Added new role cloudera_manager.services_info to perform useful service discovery on existing clusters. Should probably be a module. WIP.
Migrated role to set session_timeout for cloudera_manager from FRisch fork for PvC-CP setup
Migrated role from FRisch fork to set hue_ticket_lifetime for PvC-DS deployments
Migrated role from FRisch fork to setup tls for kms
Migrated role from FRisch fork to fix some libs for the Oozie UI in some PvC-DS deployments
Migrated role from FRisch fork to setup some default Ranger policies for some PvC-DS deployments
Migrated role from FRisch fork to setup a solr role in knox for some PvC-DS deployments
Migrated role from FRisch fork to ensure a ranger plugin for solr is deployed in some PvC-DS deployments
Updated the defaults for database type and version to respond to el7 or el8 appropriately. Could use defaults for more OS and versions. WIP.
Migrated role from FRisch fork to setup WXM. Needs more testing. WIP
Updated krb5_client deployment with patches from FRisch fork for FreeIPA setup, including a patch for dbus_session config and specific configs for when running PvC-DS.
krb5_domain is now defaulted to krb5_realm.lower automatically. Finally.
Default kerberos configuration added to krb5_common, including simple defaults for when MIT KDC or Red Hat IPA are selected. Passwords are defaulted to the cloudera_manager_admin_password instead of hardcoded values like changeme
Migrated in fixes for FreeIPA server deployment from FRisch fork
Fixed role to refresh_ranger_kms_repo to function correctly when determining the Ranger URL in modern Ansible. Really should be a module. WIP.
Added operation to restart a given cluster or a given cluster's services or cluster management services for user convenience. They could be handlers, but this felt more useful as more people know how to use roles than handlers.
Migrated role to setup iptables or nftables for PvC ECS deployment on Rhel7 or Rhel8 from Chuck Levesque
Added firewalld to unwanted services during automated os prereq setup, as it is unwelcome for PvC-DS deployments
Migrated fix from FRisch fork where setting up postgresql_connector sometimes requires python3-psycopg2 to be setup for SSB. Put behind a default False switch of install_py3_psycopg2
Migrated role from Chuck Levesque to set up a subset of necessary local accounts on ecs_nodes
Added check to ensure that FreeIPA and a Custom Repo are not on the same host as they both try to hardcode port 8443. Honestly who does this.
Enhanced error message when tls setup is only being partially applied to hosts in the cluster definition
Signed-off-by: Daniel Chaffelson chaffelson@gmail.com