Skip to content

Commit

Permalink
Added Support for Alloy Module Imports
Browse files Browse the repository at this point in the history
Resolves #537
  • Loading branch information
bentonam committed May 21, 2024
1 parent 4fcddeb commit 91e52a9
Show file tree
Hide file tree
Showing 12 changed files with 52,861 additions and 0 deletions.
7 changes: 7 additions & 0 deletions charts/k8s-monitoring/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -373,6 +373,13 @@ The Prometheus and Loki services may be hosted on the same cluster, or remotely
| metrics.alloy.metricsTuning.useIntegrationAllowList | bool | `false` | Filter the list of metrics from Grafana Alloy to the minimal set required for Kubernetes Monitoring as well as the Grafana Alloy integration. |
| metrics.alloy.scrapeInterval | string | 60s | How frequently to scrape metrics from Grafana Alloy. Overrides metrics.scrapeInterval |

### Metrics Job: Alloy Modules

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| metrics.alloyModules.connections | list | `[]` | List of connection configurations used by modules. Configures the import.git component ([docs](https://grafana.com/docs/alloy/latest/reference/components/import.git/) <br>- `alias: ""` the alias of the connection <br>- `repository: ""` URL of the Git repository containing the module. <br>- `revision: ""` Branch, tag, or commit to be checked out. <br>- `pull_frequency: 15m` How often the module should check for updates. <br>- `default: true` If true, this connection is used as the default when none is specified. <br>- `basic_auth: {}` Credentials for basic authentication if needed. ([docs](https://grafana.com/docs/alloy/latest/reference/config-blocks/import.git/#basic_auth-block)) <br>- `ssh_key: {}` Provides SSH key details for secure connections. ([docs](https://grafana.com/docs/alloy/latest/reference/config-blocks/import.git/#ssh_key-block)) |
| metrics.alloyModules.modules | list | `[]` | List of Modules to import. Each module is expected to have a "kubernetes" module and a "scrape" component. Each module can have the following properties: <br>- `path: ""` the path to the alloy module <br>- `connection: ""` (optional) the alias of the connection to use, if not specified the default connection is used <br>- `targets: {}` (optional) Additional arguments to be passed to the modules kubernetes component <br>- `scrape: {}` (optional) Additional arguments to be passed to the modules scrape component <br>- `extraRelabelingRules: ""` additional relabeling rules for the discovery.relabel component <br>- `extraMetricRelabelingRules:` additional relabeling rules for the prometheus.relabel component |

### Metrics Job: ApiServer

| Key | Type | Default | Description |
Expand Down
4 changes: 4 additions & 0 deletions charts/k8s-monitoring/templates/_configs.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,10 @@
{{- include "alloy.config.service_monitors" . }}
{{- end }}

{{- if len .Values.metrics.alloyModules.modules }}
{{- include "alloy.config.alloyMetricModules" . }}
{{- end }}

{{- include "alloy.config.metricsService" . }}
{{- end }}

Expand Down
103 changes: 103 additions & 0 deletions charts/k8s-monitoring/templates/alloy_config/_alloy-modules.alloy.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
{{ define "alloy.config.alloyMetricModules" }}
// Alloy Modules
{{- $connections := .Values.metrics.alloyModules.connections }}
{{- /* Determine the default connection */}}
{{- $defaultConnection := dict }}
{{- range $connections }}
{{- if .default }}
{{- $defaultConnection = . }}
{{- end }}
{{- end }}
{{- $extraRelabelingRules := .Values.metrics.extraRelabelingRules | default "" }}
{{- $extraMetricRelabelingRules := .Values.metrics.extraMetricRelabelingRules | default "" }}
{{- $defaultMaxCacheSize := .Values.metrics.maxCacheSize }}
{{- $defaultScrapeInterval := .Values.metrics.scrapeInterval }}
{{- $clustering := .Values.alloy.alloy.clustering.enabled }}

{{- range $module := .Values.metrics.alloyModules.modules }}
{{- $moduleName := $module.alias }}
{{- /* Find the connection for the module or use the default */}}
{{- $moduleConnection := $defaultConnection }}
{{- range $conn := $connections }}
{{- if and $module.connection (eq $conn.alias $module.connection) }}
{{- $moduleConnection = $conn }}
{{- end }}
{{- end }}

// {{ $moduleName }}: import the module
import.git "module_{{ $moduleName }}" {
repository = "{{ $moduleConnection.repository }}"
revision = "{{ $moduleConnection.revision }}"
path = "{{ $module.path }}"
pull_frequency = "{{ $moduleConnection.pull_frequency }}"

{{- with $moduleConnection.basic_auth }}
basic_auth {
{{ if .username }}username = "{{ .username }}"{{ end }}
{{ if .password }}password = "{{ .password }}"{{ end }}
{{ if .password_file }}password_file = "{{ .password_file }}"{{ end }}
}
{{- end }}

{{- with $moduleConnection.ssh_key }}
ssh_key {
{{ if .username }}username = "{{ .username }}"{{ end }}
{{ if .key }}key = "{{ .key }}"{{ end }}
{{ if .key_file }}key_file = "{{ .key_file }}"{{ end }}
{{ if .passphrase }}passphrase = "{{ .passphrase }}"{{ end }}
}
{{- end }}
}

// {{ $moduleName }}: call the targets component
module_{{ $moduleName }}.kubernetes "targets" {
{{- if $module.targets }}
{{- range $key, $value := $module.targets }}
{{ $key }} = {{ if kindIs "slice" $value }}[{{ range $index, $item := $value }}"{{ $item }}"{{ if lt $index (sub (len $value) 1) }}, {{ end }}{{ end }}]{{ else }}"{{ $value }}"{{ end }}
{{- end }}
{{- end }}
}

// {{ $moduleName }}: additional service discovery relabelings
discovery.relabel "module_{{ $moduleName }}" {
targets = module_{{ $moduleName }}.kubernetes.targets.output
// global discovery relabelings
{{- if $extraRelabelingRules }}
{{ $extraRelabelingRules | indent 2 }}
{{- end }}
// {{ $moduleName }}: additional discovery relabelings
{{- if $module.extraRelabelingRules }}
{{ $module.extraRelabelingRules | indent 2 }}
{{- end }}
}

// {{ $moduleName }}: call the scrape component
module_{{ $moduleName }}.scrape "metrics" {
targets = discovery.relabel.module_{{ $moduleName }}.output
forward_to = [prometheus.relabel.module_{{ $moduleName }}.receiver]
clustering = {{ $clustering }}
scrape_interval = "{{ if and (hasKey $module "scrape") (hasKey $module.scrape "scrape_interval") }}{{ index $module.scrape "scrape_interval" }}{{- else }}{{ $defaultScrapeInterval }}{{- end }}"
{{- if $module.scrape }}
{{- range $key, $value := $module.scrape }}
{{- if not (eq $key "scrape_interval") }}
{{ $key }} = {{ if kindIs "slice" $value }}[{{ range $index, $item := $value }}"{{ $item }}"{{ if lt $index (sub (len $value) 1) }}, {{ end }}{{ end }}]{{ else }}"{{ $value }}"{{ end }}
{{- end }}
{{- end }}
{{- end }}
}

// {{ $moduleName }}: additional metric relabelings
prometheus.relabel "module_{{ $moduleName }}" {
max_cache_size = {{ if and (hasKey $module "scrape") (hasKey $module.scrape "max_cache_size") }}{{ index $module.scrape "max_cache_size" | int }}{{- else }}{{ $defaultMaxCacheSize | int }}{{- end }}
// global metric relabelings
{{- if $extraMetricRelabelingRules }}
{{ $extraMetricRelabelingRules | indent 2 }}
{{- end }}
// additional {{ $moduleName }} module specific metric relabelings
{{- if $module.extraMetricRelabelingRules }}
{{ $module.extraMetricRelabelingRules | indent 2 }}
{{- end }}
forward_to = [prometheus.relabel.metrics_service.receiver]
}
{{- end }}
{{ end }}
11 changes: 11 additions & 0 deletions charts/k8s-monitoring/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -796,6 +796,17 @@
}
}
},
"alloyModules": {
"type": "object",
"properties": {
"connections": {
"type": "array"
},
"modules": {
"type": "array"
}
}
},
"apiserver": {
"type": "object",
"properties": {
Expand Down
36 changes: 36 additions & 0 deletions charts/k8s-monitoring/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1158,6 +1158,41 @@ metrics:
# @section -- Metrics Job: Kubernetes Monitoring Telemetry
enabled: true

# Alloy Modules
# Modules can be invoked using metrics.extraConfig, this block is consuming opinionated modules from the grafana/alloy-modules repository
# or any other repository that follows the same module structure. Each module is expected to have a "kubernetes" module and a "scrape" module.
alloyModules:
# -- List of connection configurations used by modules. Configures the import.git component
# ([docs](https://grafana.com/docs/alloy/latest/reference/components/import.git/)
# <br>- `alias: ""` the alias of the connection
# <br>- `repository: ""` URL of the Git repository containing the module.
# <br>- `revision: ""` Branch, tag, or commit to be checked out.
# <br>- `pull_frequency: 15m` How often the module should check for updates.
# <br>- `default: true` If true, this connection is used as the default when none is specified.
# <br>- `basic_auth: {}` Credentials for basic authentication if needed. ([docs](https://grafana.com/docs/alloy/latest/reference/config-blocks/import.git/#basic_auth-block))
# <br>- `ssh_key: {}` Provides SSH key details for secure connections. ([docs](https://grafana.com/docs/alloy/latest/reference/config-blocks/import.git/#ssh_key-block))
# @section -- Metrics Job: Alloy Modules
connections: []
# - alias: grafana
# repository: https://github.com/grafana/alloy-modules.git
# revision: main
# pull_frequency: 15m
# default: true

# -- List of Modules to import. Each module is expected to have a "kubernetes" module and a "scrape" component.
# Each module can have the following properties:
# <br>- `path: ""` the path to the alloy module
# <br>- `connection: ""` (optional) the alias of the connection to use, if not specified the default connection is used
# <br>- `targets: {}` (optional) Additional arguments to be passed to the modules kubernetes component
# <br>- `scrape: {}` (optional) Additional arguments to be passed to the modules scrape component
# <br>- `extraRelabelingRules: ""` additional relabeling rules for the discovery.relabel component
# <br>- `extraMetricRelabelingRules:` additional relabeling rules for the prometheus.relabel component
# @section -- Metrics Job: Alloy Modules
modules: []
# - alias: memcached
# path: modules/databases/kv/memcached/metrics.alloy


# Settings related to metrics ingested via receivers
# @section -- Metrics -> OTEL Receivers
receiver:
Expand Down Expand Up @@ -1274,6 +1309,7 @@ logs:
# -- Stage blocks to be added to the loki.process component for cluster events.
# ([docs](https://grafana.com/docs/alloy/latest/reference/components/loki.process/#blocks))
# This value is templated so that you can refer to other values from this file.
# @section -- Logs Scrape: Cluster Events
extraStageBlocks: ""

# -- Logs the cluster events to stdout. Useful for debugging.
Expand Down
43 changes: 43 additions & 0 deletions examples/metric-module-imports/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Alloy Modules

This example shows a how to leverage the [Alloy Modules](https://github.com/grafana/alloy-modules) for collecting metrics. These modules are opinionated, where each module has at least the following two components defined:

1. `kubernetes` Used to find targets in the cluster
2. `scrape` Used to scrape the found targets.

```yaml
cluster:
name: metric-module-imports

externalServices:
prometheus:
host: https://prometheus.example.com
basicAuth:
username: 12345
password: "It's a secret to everyone"
loki:
host: https://loki.example.com
basicAuth:
username: 12345
password: "It's a secret to everyone"

metrics:
alloyModules:
connections:
- alias: grafana
repository: https://github.com/grafana/alloy-modules.git
revision: main
pull_frequency: 15m
default: true
modules:
- alias: memcached
path: modules/databases/kv/memcached/metrics.alloy
- alias: loki
path: modules/databases/timeseries/loki/metrics.alloy
- alias: mimir
path: modules/databases/timeseries/mimir/metrics.alloy
- alias: tempo
path: modules/databases/timeseries/tempo/metrics.alloy
- alias: grafana
path: modules/ui/grafana/metrics.alloy
```
48 changes: 48 additions & 0 deletions examples/metric-module-imports/events.alloy

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 91e52a9

Please sign in to comment.