Configuration to Set Up Crossplane, Provider and Resource Observability with open source software integrations such as Prometheus and Grafana.
Observability is a measure of how well platform performance can be inferred from knowledge of its metrics, logs and traces outputs.
Warning
Prometheus and Grafana may require significant cluster resources in relation to the amount of metrics scraped, processed and visualized. This may impact cluster operations. Consult the respective Prometheus Operator and Grafana documentation for set up guidance prior to using this configuration on mission critical Crossplane management clusters.
Warning
Crossplane has no concept of metric stability. This implies that metrics used in this configuration may be absent in future versions of Crossplane and / or its providers.
The goal for configuration-observability-oss is to complement other configurations such as configuration-caas. See the Upbound Marketplace for additional configurations.
Run make e2e directly to exercise end to end tests
for the observability integrations. After running the
tests, the kind cluster will remain but the tests will
clean up the operator namespace and delete the pods in it
at the conclusion of the tests by default.
Apply the resource claim as follows to re-create the namespace, Prometheus, Grafana and dependencies for further exploration.
kubectl apply -f examples/oss.yaml
To load dashboards that are part of this configuration repository, please apply the following dashboard resource claims.
kubectl apply -f examples/dashboards/folder-grafana.yaml
kubectl apply -f examples/dashboards/dashboard-grafana-crossplane-health.yaml
kubectl apply -f examples/dashboards/dashboard-grafana-crossplane-mr.yaml
kubectl apply -f examples/dashboards/dashboard-grafana-crossplane-resources-ttr.yaml
kubectl apply -f examples/dashboards/dashboard-grafana-crossplane-sli-metrics.yaml
Use the following to forward localhost:9090 to the Prometheus pod.
PROMETHEUS_POD_NAME=$(k -n operators get pods|\
awk '{print $1}'|\
tail +2|\
grep prometheus-0)
kubectl -n operators port-forward ${PROMETHEUS_POD_NAME} 9090
Use the following to forward localhost:3000 to the Grafana pod.
GRAFANA_POD_NAME=$(k -n operators get pods|\
awk '{print $1}'|\
tail +2|\
grep grafana)
kubectl -n operators port-forward ${GRAFANA_POD_NAME} 3000
Log in to Grafana at http://localhost:3000 with the credentials obtained from running the following.
scripts/grafana-get-creds.sh
See example dashboards below.
Once your cluster has been bootstrapped, and that prometheus and grafana endpoints have been forwarded, what's next?
Install a kubernetes secret with your provider credentials or use IRSA or your own preferred way to provide the providers with the permissions to create and reconcile cloud resources.
If you use AWS, One way would be to add your credentials to
~/.aws/credentials, and to run
kubectl create secret generic aws-creds \
-n upbound-system \
--from-file=credentials=~/.aws/credentials
Note that your shell may need a fully qualified path versus ~ above.
Apply a provider configuration as follows.
cat <<EOF | kubectl -f -
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
name: default
spec:
credentials:
source: Secret
secretRef:
namespace: upbound-system
name: aws-creds
key: credentials
EOF
Apply resource claims and see information on the loaded dashboards. For example you can create an AWS VPC as follows, and you can use your own compositions and any of our marketplace configurations.
apiVersion: ec2.aws.upbound.io/v1beta1
kind: VPC
metadata:
name: sample-vpc
annotations:
meta.upbound.io/example-id: ec2/v1beta1/vpc
spec:
forProvider:
region: us-west-1
cidrBlock: 172.16.0.0/16
tags:
Name: SampleVpc
Feel free to join the SIG Observability Slack Channel to participate in the Crossplane observability journey.