tidb dashboard 监控指标采集问题

[soulmz]

Bug Report

What version of Kubernetes are you using?

k8s v1.27.8
tidb-cluster 7.5.0

What version of TiDB Operator are you using?

TiDB Operator Version: version.Info{GitVersion:"v1.5.2", GitCommit:"456a0273f67ac61212da78956f49f0a4a07e21d8", GitTreeState:"clean", BuildDate:"2024-01-19T03:50:22Z", GoVersion:"go1.21.5", Compiler:"gc", Platform:"linux/amd64"}
What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

openEBS localPV

What's the status of the TiDB cluster pods?

What did you do?

What did you expect to see?

dashboard 有关 tidb 组件监控数据存在问题。

这个指标有问题，rate(process_cpu_seconds_total{job="tidb"}[120s])

通过 prometheus 查看，发现 job 不匹配。

dashboard grafana url 链接地址错误。

部分监控指标不显示，初步排查，发现都是 promQL 表达式有问题。

What did you see instead?

[soulmz]

请问这个表达式是否是代码写死的？

如果不是写死的，能否通过数据库手动改正？

查阅官方文档，没发现修改的方式。

[csuzhangxc]

This is caused by the job_name mismatch of TidbMonitor (Prometheus) and Dashboard promQL.

You can try to use configMapRef of TidbMonitor to mount an external ConfigMap to overwrite the default prometheus.yml in ClusterName-monitor ConfigMap with Dashboard expected job_name (e.g tidb, not ns-clusterName-tidb).

ref https://github.com/pingcap/tidb-operator/blob/v1.5.2/examples/advanced/tidb-monitor.yaml#L192-L197

also ref https://github.com/pingcap/tidb-dashboard/blob/b8552047eebc8d34e911cf1e1505347599482ac1/ui/packages/tidb-dashboard-for-op/src/apps/Monitoring/metricsQueries.ts#L612

[soulmz]

This is caused by the job_name mismatch of TidbMonitor (Prometheus) and Dashboard promQL.

You can try to use configMapRef of TidbMonitor to mount an external ConfigMap to overwrite the default prometheus.yml in ClusterName-monitor ConfigMap with Dashboard expected job_name (e.g tidb, not ns-clusterName-tidb).

ref https://github.com/pingcap/tidb-operator/blob/v1.5.2/examples/advanced/tidb-monitor.yaml#L192-L197

also ref https://github.com/pingcap/tidb-dashboard/blob/b8552047eebc8d34e911cf1e1505347599482ac1/ui/packages/tidb-dashboard-for-op/src/apps/Monitoring/metricsQueries.ts#L612

tidb-dashboard 这个是写死了在 ts 代码里吗？我使用 外部的configmap 方式没找到具体的覆盖 tidb-dashboard 面板上的监控指标。

[csuzhangxc]

This is caused by the job_name mismatch of TidbMonitor (Prometheus) and Dashboard promQL.
You can try to use configMapRef of TidbMonitor to mount an external ConfigMap to overwrite the default prometheus.yml in ClusterName-monitor ConfigMap with Dashboard expected job_name (e.g tidb, not ns-clusterName-tidb).
ref https://github.com/pingcap/tidb-operator/blob/v1.5.2/examples/advanced/tidb-monitor.yaml#L192-L197
also ref https://github.com/pingcap/tidb-dashboard/blob/b8552047eebc8d34e911cf1e1505347599482ac1/ui/packages/tidb-dashboard-for-op/src/apps/Monitoring/metricsQueries.ts#L612

tidb-dashboard 这个是写死了在 ts 代码里吗？我使用 外部的configmap 方式没找到具体的覆盖 tidb-dashboard 面板上的监控指标。

I mean, use an external ConfigMap to overwite the metrics labels (job_name) in Prometheus so that they can match the hardcoded PromQL in tidb-dashboard.

[soulmz]

This is caused by the job_name mismatch of TidbMonitor (Prometheus) and Dashboard promQL.
You can try to use configMapRef of TidbMonitor to mount an external ConfigMap to overwrite the default prometheus.yml in ClusterName-monitor ConfigMap with Dashboard expected job_name (e.g tidb, not ns-clusterName-tidb).
ref https://github.com/pingcap/tidb-operator/blob/v1.5.2/examples/advanced/tidb-monitor.yaml#L192-L197
also ref https://github.com/pingcap/tidb-dashboard/blob/b8552047eebc8d34e911cf1e1505347599482ac1/ui/packages/tidb-dashboard-for-op/src/apps/Monitoring/metricsQueries.ts#L612

tidb-dashboard 这个是写死了在 ts 代码里吗？我使用 外部的configmap 方式没找到具体的覆盖 tidb-dashboard 面板上的监控指标。

I mean, use an external ConfigMap to overwite the metrics labels (job_name) in Prometheus so that they can match the hardcoded PromQL in tidb-dashboard.

我通过 修改了旧的 prometheus ，增加了 job_name: tidb

global:
  evaluation_interval: 15s
  scrape_interval: 15s
  external_labels:
    prometheus_replica: tidb-cluster_ex-monitor-0
scrape_configs:
- job_name: tidb-cluster-ex-pd
  honor_labels: true
  scrape_interval: 15s
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names:
      - tidb-cluster
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: keep
    regex: ex
  - source_labels:
    - __meta_kubernetes_namespace
    action: keep
    regex: tidb-cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    action: keep
    regex: "true"
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: keep
    regex: pd
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: $1.$2-pd-peer.$3:$4
    target_label: __address__
    source_labels:
    - __meta_kubernetes_pod_name
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_annotation_prometheus_io_port
  - source_labels:
    - __meta_kubernetes_namespace
    action: replace
    target_label: kubernetes_namespace
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: replace
    target_label: cluster
  - source_labels:
    - __meta_kubernetes_pod_name
    action: replace
    target_label: instance
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: replace
    target_label: component
  - source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    separator: '-'
    target_label: tidb_cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels:
    - __address__
    action: hashmod
    target_label: __tmp_hash
    modulus: 1
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
- job_name: tidb-cluster-ex-tidb
  honor_labels: true
  scrape_interval: 15s
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names:
      - tidb-cluster
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: keep
    regex: ex
  - source_labels:
    - __meta_kubernetes_namespace
    action: keep
    regex: tidb-cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    action: keep
    regex: "true"
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: keep
    regex: tidb
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: $1.$2-tidb-peer.$3:$4
    target_label: __address__
    source_labels:
    - __meta_kubernetes_pod_name
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_annotation_prometheus_io_port
  - source_labels:
    - __meta_kubernetes_namespace
    action: replace
    target_label: kubernetes_namespace
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: replace
    target_label: cluster
  - source_labels:
    - __meta_kubernetes_pod_name
    action: replace
    target_label: instance
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: replace
    target_label: component
  - source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    separator: '-'
    target_label: tidb_cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels:
    - __address__
    action: hashmod
    target_label: __tmp_hash
    modulus: 1
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
# add tidb
- job_name: tidb
  honor_labels: true
  scrape_interval: 15s
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names:
      - tidb-cluster
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: keep
    regex: ex
  - source_labels:
    - __meta_kubernetes_namespace
    action: keep
    regex: tidb-cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    action: keep
    regex: "true"
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: keep
    regex: tidb
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: $1.$2-tidb-peer.$3:$4
    target_label: __address__
    source_labels:
    - __meta_kubernetes_pod_name
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_annotation_prometheus_io_port
  - source_labels:
    - __meta_kubernetes_namespace
    action: replace
    target_label: kubernetes_namespace
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: replace
    target_label: cluster
  - source_labels:
    - __meta_kubernetes_pod_name
    action: replace
    target_label: instance
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: replace
    target_label: component
  - source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    separator: '-'
    target_label: tidb_cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels:
    - __address__
    action: hashmod
    target_label: __tmp_hash
    modulus: 1
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
- job_name: tidb-cluster-ex-tikv
  honor_labels: true
  scrape_interval: 15s
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names:
      - tidb-cluster
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: keep
    regex: ex
  - source_labels:
    - __meta_kubernetes_namespace
    action: keep
    regex: tidb-cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    action: keep
    regex: "true"
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: keep
    regex: tikv
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: $1.$2-tikv-peer.$3:$4
    target_label: __address__
    source_labels:
    - __meta_kubernetes_pod_name
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_annotation_prometheus_io_port
  - source_labels:
    - __meta_kubernetes_namespace
    action: replace
    target_label: kubernetes_namespace
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: replace
    target_label: cluster
  - source_labels:
    - __meta_kubernetes_pod_name
    action: replace
    target_label: instance
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: replace
    target_label: component
  - source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    separator: '-'
    target_label: tidb_cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels:
    - __address__
    action: hashmod
    target_label: __tmp_hash
    modulus: 1
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
- job_name: tidb-cluster-ex-tiproxy
  honor_labels: true
  scrape_interval: 15s
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names:
      - tidb-cluster
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: keep
    regex: ex
  - source_labels:
    - __meta_kubernetes_namespace
    action: keep
    regex: tidb-cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    action: keep
    regex: "true"
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: keep
    regex: tiproxy
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: $1.$2-tiproxy-peer.$3:$4
    target_label: __address__
    source_labels:
    - __meta_kubernetes_pod_name
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_annotation_prometheus_io_port
  - source_labels:
    - __meta_kubernetes_namespace
    action: replace
    target_label: kubernetes_namespace
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: replace
    target_label: cluster
  - source_labels:
    - __meta_kubernetes_pod_name
    action: replace
    target_label: instance
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: replace
    target_label: component
  - source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    separator: '-'
    target_label: tidb_cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels:
    - __address__
    action: hashmod
    target_label: __tmp_hash
    modulus: 1
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
- job_name: tidb-cluster-ex-tiflash
  honor_labels: true
  scrape_interval: 15s
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names:
      - tidb-cluster
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: keep
    regex: ex
  - source_labels:
    - __meta_kubernetes_namespace
    action: keep
    regex: tidb-cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    action: keep
    regex: "true"
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: keep
    regex: tiflash
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: $1.$2-tiflash-peer.$3:$4
    target_label: __address__
    source_labels:
    - __meta_kubernetes_pod_name
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_annotation_prometheus_io_port
  - source_labels:
    - __meta_kubernetes_namespace
    action: replace
    target_label: kubernetes_namespace
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: replace
    target_label: cluster
  - source_labels:
    - __meta_kubernetes_pod_name
    action: replace
    target_label: instance
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: replace
    target_label: component
  - source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    separator: '-'
    target_label: tidb_cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels:
    - __address__
    action: hashmod
    target_label: __tmp_hash
    modulus: 1
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
- job_name: tidb-cluster-ex-tiflash-proxy
  honor_labels: true
  scrape_interval: 15s
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names:
      - tidb-cluster
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: keep
    regex: ex
  - source_labels:
    - __meta_kubernetes_namespace
    action: keep
    regex: tidb-cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    action: keep
    regex: "true"
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: keep
    regex: tiflash
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: $1.$2-tiflash-peer.$3:$4
    target_label: __address__
    source_labels:
    - __meta_kubernetes_pod_name
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_annotation_tiflash_proxy_prometheus_io_port
  - source_labels:
    - __meta_kubernetes_namespace
    action: replace
    target_label: kubernetes_namespace
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: replace
    target_label: cluster
  - source_labels:
    - __meta_kubernetes_pod_name
    action: replace
    target_label: instance
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: replace
    target_label: component
  - source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    separator: '-'
    target_label: tidb_cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels:
    - __address__
    action: hashmod
    target_label: __tmp_hash
    modulus: 1
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
- job_name: tidb-cluster-ex-pump
  honor_labels: true
  scrape_interval: 15s
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names:
      - tidb-cluster
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: keep
    regex: ex
  - source_labels:
    - __meta_kubernetes_namespace
    action: keep
    regex: tidb-cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    action: keep
    regex: "true"
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: keep
    regex: pump
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: $1.$2-pump.$3:$4
    target_label: __address__
    source_labels:
    - __meta_kubernetes_pod_name
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_annotation_prometheus_io_port
  - source_labels:
    - __meta_kubernetes_namespace
    action: replace
    target_label: kubernetes_namespace
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: replace
    target_label: cluster
  - source_labels:
    - __meta_kubernetes_pod_name
    action: replace
    target_label: instance
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: replace
    target_label: component
  - source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    separator: '-'
    target_label: tidb_cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels:
    - __address__
    action: hashmod
    target_label: __tmp_hash
    modulus: 1
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
- job_name: tidb-cluster-ex-drainer
  honor_labels: true
  scrape_interval: 15s
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names:
      - tidb-cluster
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: keep
    regex: ex
  - source_labels:
    - __meta_kubernetes_namespace
    action: keep
    regex: tidb-cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    action: keep
    regex: "true"
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: keep
    regex: drainer
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: $1.$2.$3:$4
    target_label: __address__
    source_labels:
    - __meta_kubernetes_pod_name
    - __meta_kubernetes_pod_label_app_kubernetes_io_name
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_annotation_prometheus_io_port
  - source_labels:
    - __meta_kubernetes_namespace
    action: replace
    target_label: kubernetes_namespace
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: replace
    target_label: cluster
  - source_labels:
    - __meta_kubernetes_pod_name
    action: replace
    target_label: instance
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: replace
    target_label: component
  - source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    separator: '-'
    target_label: tidb_cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels:
    - __address__
    action: hashmod
    target_label: __tmp_hash
    modulus: 1
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
- job_name: tidb-cluster-ex-ticdc
  honor_labels: true
  scrape_interval: 15s
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names:
      - tidb-cluster
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: keep
    regex: ex
  - source_labels:
    - __meta_kubernetes_namespace
    action: keep
    regex: tidb-cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    action: keep
    regex: "true"
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: keep
    regex: ticdc
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: $1.$2-ticdc-peer.$3:$4
    target_label: __address__
    source_labels:
    - __meta_kubernetes_pod_name
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_annotation_prometheus_io_port
  - source_labels:
    - __meta_kubernetes_namespace
    action: replace
    target_label: kubernetes_namespace
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: replace
    target_label: cluster
  - source_labels:
    - __meta_kubernetes_pod_name
    action: replace
    target_label: instance
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: replace
    target_label: component
  - source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    separator: '-'
    target_label: tidb_cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels:
    - __address__
    action: hashmod
    target_label: __tmp_hash
    modulus: 1
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
- job_name: tidb-cluster-ex-lightning
  honor_labels: true
  scrape_interval: 15s
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names:
      - tidb-cluster
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: keep
    regex: ex
  - source_labels:
    - __meta_kubernetes_namespace
    action: keep
    regex: tidb-cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    action: keep
    regex: "true"
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: keep
    regex: tidb-lightning
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: $2.$3:$4
    target_label: __address__
    source_labels:
    - __meta_kubernetes_pod_name
    - __meta_kubernetes_pod_label_app_kubernetes_io_name
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_annotation_prometheus_io_port
  - source_labels:
    - __meta_kubernetes_namespace
    action: replace
    target_label: kubernetes_namespace
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    action: replace
    target_label: cluster
  - source_labels:
    - __meta_kubernetes_pod_name
    action: replace
    target_label: instance
  - source_labels:
    - __meta_kubernetes_pod_label_app_kubernetes_io_component
    action: replace
    target_label: component
  - source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    separator: '-'
    target_label: tidb_cluster
  - source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels:
    - __address__
    action: hashmod
    target_label: __tmp_hash
    modulus: 1
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
rule_files:
  - '/prometheus-rules/rules/*.rules.yml'

ok, 现在已经有数据了。但是部分指标名称已经更换了

例如

QPS 数据 tidb_executor_statement_total -> tidb_executor_statement_db_total

现在可以展示 tidb 部分数据了。指标命名更换了这个有办法吗？

[csuzhangxc]

QPS 数据 tidb_executor_statement_total -> tidb_executor_statement_db_total

What's the metrics name of getting from TiDB Pod's /metrics path directly?

[soulmz]

QPS 数据 tidb_executor_statement_total -> tidb_executor_statement_db_total

What's the metrics name of getting from TiDB Pod's /metrics path directly?

dashboard 写的是 tidb_executor_statement_total 。
但是，tidb metrics 指标已经更换了 tidb_executor_statement_db_total 。

[csuzhangxc]

could you open an issue in https://github.com/pingcap/tidb

[csuzhangxc]

@soulmz Both of them should exist, https://github.com/pingcap/tidb/blob/v7.5.0/pkg/metrics/executor.go#L55-L69