Use cluster_labels to centrally define more hard-coded strings

[darrenjaneczek]

Related to the cluster_labels configuration value added in #319:

There are a few remaining instances of cluster_namespace and cluster, namespace in Recording Rules and Scaling which are not yet derived from the new configuration list cluster_labels. Only the Dashboards were updated to make use of the group_by_cluster and group_prefix_clusters strings which are generated from cluster_labels.

Acceptance criteria:

• Recording rules and Scale are updated to make use of the centrally-overridable cluster_labels
  • Use group_prefix_clusters variable from $._group_config instead of hard-coded cluster_namespace
  • Use group_by_cluster variable from $._group_config instead of hard-coded cluster, namespace
• Evaluate if it is appropriate to have other hard-coded strings that "start with" above two hard-coded string ('cluster_namespace_deployment', 'cluster, namespace, deployment', etc.) and update them accordingly

Question:

@pracucci , with the [cluster, namespace] list (cluster_labels) defining which labels identify a cluster, does it make sense to seek out references to cluster_namespace* string patterns and replace them with the corresponding prefix (group_prefix_clusters)?

Response:

I would start as smallest as possible because the potential replacement is huge. If this PR solves your use case, I would start from here and work on it on incremental PRs.
Originally posted by @pracucci in #319 (comment)