Search Autocomplete Produces Random Numbers

[GabrielMulcahy]

Describe the bug
Using the search option for Tempo in Grafana's explore window, when filtering by one attribute of a trace, all other attribute dropdowns change from suggesting real values to seemingly random numbers (but still the correct number of options). Filtering by those number values doesn't produce anything.

This is an issue with my test and production environments. The former only has a handful of spans going in to it, the latter far more. There are no error logs produced by my test instance of Tempo, so it doesn't seem to be logging any errors to do with this issue.

This doesn't seem to be a Grafana issue as I get the same behaviour querying the HTTP API directly at GET /api/search/tag/<tag>/values

To Reproduce
Steps to reproduce the behavior:

1. Import a span into Tempo
2. Configure Tempo search in the explore window to filter by multiple resource attriubutes, such as service, environment and instance.
3. Restrict by one attribute value and watch the others be replaced by number values.

Expected behavior
Like with Loki search, picking one attribute filters all other attribute dropdowns to a possible subset of applicable values. So if MyService only has a TEST environment, picking that as the service attribute results in only TEST showing up in the environment dropdown without any corrupted(?) number values in its place.

Environment:
Deployed on Docker Swarm currently using a local backend storage. Using latest verson of Tempo and Grafana.

Additional Context
Screenshots of the faulty behaviour:
instance dropdown working correctly with no other restrictions:

instance dropdown now showing a random number since restricting by environment=TEST:

Tempo config:

target: all

server:
  http_listen_port: 3200
  log_level: info

distributor:
  receivers:
    otlp:
      protocols:
        http:
        grpc:
        
compactor:        
  compaction:
    block_retention: 72h
    compacted_block_retention: 15m

storage:
  trace:
    backend: local              
    block:
      v2_encoding: zstd  
    wal:
      path: /tmp/tempo/wal 
      v2_encoding: snappy
    local:
      path: /tmp/tempo/blocks

overrides:
  max_traces_per_user: 50000
  max_bytes_per_tag_values_query: 0

[mapno]

Hi! Thanks for submitting the issue. Could you share what version of Tempo you're running and a span/trace that reproduces the behaviour? I wasn't able to reproduce it

[GabrielMulcahy]

Hi @mapno! Thanks for taking a look. I've been using grafana/tempo:latest. Locking to 2.3.1 does resolve this issue though.

[mapno]

This issue has been fixed in this PR #3339