Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DOC] Add missing doc for metrics #3726

Merged
merged 5 commits into from
May 30, 2024
Merged

[DOC] Add missing doc for metrics #3726

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2e4d032
Add missing doc for metrics
knylander-grafana May 28, 2024
e48da07
Update docs/sources/tempo/operations/traceql-metrics.md
knylander-grafana May 28, 2024
169e176
Apply suggestions from code review
knylander-grafana May 29, 2024
18a4f80
Apply suggestions from code review
knylander-grafana May 30, 2024
66ca504
Apply suggestions from code review
knylander-grafana May 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 10 additions & 31 deletions docs/sources/tempo/operations/traceql-metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,62 +1,41 @@
---
aliases: []
title: TraceQL metrics
menuTitle: TraceQL metrics
description: Learn about using TraceQL metrics.
title: Configure TraceQL metrics
menuTitle: Configure TraceQL metrics
description: Learn about configuring TraceQL metrics.
weight: 550
keywords:
- Prometheus
- TraceQL
- TraceQL metrics
---

# TraceQL metrics
# Configure TraceQL metrics

{{< docs/experimental product="TraceQL metrics" >}}

Tempo 2.4 introduces the addition of metrics queries to the TraceQL language as an experimental feature.
TraceQL language provides metrics queries as an experimental feature.
Metric queries extend trace queries by applying a function to trace query results.
This powerful feature creates metrics from traces, much in the same way that LogQL metric queries create metrics from logs.
Initially, only `count_over_time` and `rate` are supported.

For example:
```
{ resource.service.name = "foo" && status = error } | rate()
```

In this case, we are calculating the rate of the erroring spans coming from the service `foo`. Rate is a `spans/sec` quantity.
Combined with the `by()` operator, this can be even more powerful!

```
{ resource.service.name = "foo" && status = error } | rate() by (span.http.route)
```
For more information about available queries, refer to [TraceQL metrics queries]({{< relref "../traceql/metrics-queries/" >}}).

Check failure on line 21 in docs/sources/tempo/operations/traceql-metrics.md

View workflow job for this annotation

GitHub Actions / doc-validator 

[doc-validator] reported by reviewdog 🐶
The `relref` shortcode argument '{{< relref "../traceql/metrics-queries/" >}}' has a trailing slash, which can break the resolution.
You can reference an `index.md` file either by its path or by its containing folder without the ending `/`.
You can reference an `_index.md` file only by its containing folder.

Remove the trailing slash to make sure that changing the index type doesn't break the link.

Raw Output:
{"message":"The `relref` shortcode argument '{{< relref \"../traceql/metrics-queries/\" >}}' has a trailing slash, which can break the resolution.\nYou can reference an `index.md` file either by its path or by its containing folder without the ending `/`.\nYou can reference an `_index.md` file only by its containing folder.\n\nRemove the trailing slash to make sure that changing the index type doesn't break the link.","location":{"path":"docs/sources/tempo/operations/traceql-metrics.md","range":{"start":{"line":21,"column":56},"end":{"line":21,"column":127}}},"severity":1,"source":{"name":"doc-validator"},"code":{"value":"relref-has-trailing-slash"},"suggestions":[{"range":{"start":{"line":21,"column":56},"end":{"line":21,"column":127}},"text":"[TraceQL metrics queries]({{< relref \"../traceql/metrics-queries\" >}})"}]}
knylander-grafana marked this conversation as resolved.
Show resolved Hide resolved

Now, we are still rating the erroring spans in the service `foo` but the metrics have been broken
down by HTTP endpoint. This might let you determine that `/api/sad` had a higher rate of erroring
spans than `/api/happy`, for example.

## Enable and use TraceQL metrics

You can use the TraceQL metrics in Grafana with any existing or new Tempo data source.
This capability is available in Grafana Cloud and Grafana (10.4 and newer).

![Metrics visualization in Grafana](/media/docs/tempo/metrics-explore-sample-2.4.png)

### Before you begin
## Before you begin

To use the metrics generated from traces, you need to:

* Set the `local-blocks` processor to active in your `metrics-generator` configuration
* Configure a Tempo data source configured in Grafana or Grafana Cloud
* Access Grafana Cloud or Grafana 10.4

### Configure the `local-blocks` processor
## Configure the `local-blocks` processor

Once the `local-blocks` processor is enabled in your `metrics-generator`
configuration, you can configure it using the following block to make sure
it records all spans for TraceQL metrics.

Here is an example configuration:

```yaml
metrics_generator:
processor:
Expand All @@ -70,7 +49,7 @@

Refer to the [metrics-generator configuration]({{< relref "../configuration#metrics-generator" >}}) documentation for more information.

### Evaluate query timeouts
## Evaluate query timeouts

Because of their expensive nature, these queries can take a long time to run in different systems.
As such, consider increasing the timeouts in various places of
Expand Down
86 changes: 86 additions & 0 deletions docs/sources/tempo/traceql/metrics-queries.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
title: TraceQL metrics queries
menuTitle: TraceQL metrics queries
description: Learn about TraceQL metrics queries
weight: 600
keywords:
- metrics query
- TraceQL metrics
---

# TraceQL metrics queries

{{< docs/experimental product="TraceQL metrics" >}}

TraceQL metrics is an experimental feature in Grafana Tempo that creates metrics from traces.

Metric queries extend trace queries by applying a function to trace query results.
This powerful feature allows for adhoc aggregation of any existing TraceQL query by any dimension available in your traces, much in the same way that LogQL metric queries create metrics from logs.

Traces are a unique observability signal that contain causal relationships between the components in your system.
Do you want to know how many database calls across all systems are downstream of your application?
What services beneath a given endpoint are currently failing?
What services beneath an endpoint are currently slow? TraceQL metrics can answer all these questions by parsing your traces in aggregate.

![Metrics visualization in Grafana](/media/docs/tempo/metrics-explore-sample-2.4.png)

## Enable and use TraceQL metrics

You can use the TraceQL metrics in Grafana with any existing or new Tempo data source.
This capability is available in Grafana Cloud and Grafana (10.4 and newer).

To enable TraceQL metrics, refer to [Configure TraceQL metrics](https://grafana.com/docs/tempo/latest/operations/traceql-metrics/) for more information.

## Functions

TraceQL supports include `rate`, `quantile_over_time`, and `histogram_over_time` functions.
knylander-grafana marked this conversation as resolved.
Show resolved Hide resolved
These functions can be added as an operator at the end of any TraceQL query.

`rate`
: Provides the percentage of
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does rate actually do? I was trying to figure out the best way to define it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rate is like count_over_time, but normalized to spans per second. They are based on the granularity of the step parameter in the request. If your step is 1 minute, then rate = count_over_time / 60 (there are 60 seconds in a minute).

knylander-grafana marked this conversation as resolved.
Show resolved Hide resolved

## The `rate` function

The following query shows the rate of errors by service and span name.

```
{ status = error } | rate() by (resource.service.name, name)
```

This example calculates the rate of the erroring spans coming from the service `foo`. Rate is a `spans/sec` quantity.

```
{ resource.service.name = "foo" && status = error } | rate()
```

Combined with the `by()` operator, this can be even more powerful.

```
{ resource.service.name = "foo" && status = error } | rate() by (span.http.route)
```

This example still rates the erroring spans in the service `foo` but the metrics have been broken
down by HTTP endpoint. This might let you determine that `/api/sad` had a higher rate of erroring
knylander-grafana marked this conversation as resolved.
Show resolved Hide resolved
spans than `/api/happy`, for example.

### The `quantile_over_time` and `histogram_over_time` functions

The `quantile_over_time()` and `histogram_over_time()` functions let you aggregate numerical values, such as the all important span duration. Notice that you can specify multiple quantiles in the same query.

```
{ name = "GET /:endpoint" } | quantile_over_time(duration, .99, .9, .5)`
```

You can group by any span or resource attribute.

```
{ name = "GET /:endpoint" } | quantile_over_time(duration, .99) by (span.http.target)
```

Quantiles aren't limited to span duration.
Any numerical attribute on the span is fair game.
To demonstrate this flexibility, consider this nonsensical quantile on `span.http.status_code`:

```
{ name = "GET /:endpoint" } | quantile_over_time(span.http.status_code, .99, .9, .5)
```
Loading