[TSDB] Fields with metric type counter not supported when TSDB is enabled

[ritalwar]

Elasticsearch Version

v 8.7.0

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin Kernel Version 21.3.0

Problem Description

There is an observation for 2 different scenarios:

Scenario 1: Adding integration with enabling TSDB flag

Behaviour

Error in Dashboards for the fields of metric type counter.

Index Mapping is like:

{
        "mssql": {
          "properties": {
            "metrics": {
              "properties": {
                "active_log_size": {
                  "type": "long",
                  "meta": {
                    "unit": "byte"
                  },
                  "time_series_metric": "counter"
                },
                "database_id": {
                  "type": "long"
                },
                "database_name": {
                  "type": "keyword",
                  "time_series_dimension": true
                },
                "instance_name": {
                  "type": "keyword",
                  "time_series_dimension": true
                },
                "log_backup_time": {
                  "type": "date"
                },
                "log_recovery_size": {
                  "type": "long",
                  "meta": {
                    "unit": "byte"
                  },
                  "time_series_metric": "gauge"
                },
          }
        },
}

Scenario 2: Adding Integration without enabling TSDB flag

Behaviour

Dashboards woking fine:

Index Mapping is like:

{
        "mssql": {
          "properties": {
            "metrics": {
              "properties": {
                "active_log_size": {
                  "type": "long",
                  "meta": {
                    "unit": "byte"
                  }
                },
                "database_id": {
                  "type": "long"
                },
                "database_name": {
                  "type": "keyword"
                },
                "instance_name": {
                  "type": "keyword"
                },
                "log_backup_time": {
                  "type": "date"
                },
                "log_recovery_size": {
                  "type": "long",
                  "meta": {
                    "unit": "byte"
                  }
                },
            }
          }
        },
}

Enable TSDB Flag:
Dashboard still works fine and index mapping look like:

{
        "mssql": {
          "properties": {
            "metrics": {
              "properties": {
                "active_log_size": {
                  "type": "long",
                  "meta": {
                    "unit": "byte"
                  },
                  "time_series_metric": "counter"
                },
                "database_id": {
                  "type": "long"
                },
                "database_name": {
                  "type": "keyword",
                  "time_series_dimension": true
                },
                "instance_name": {
                  "type": "keyword",
                  "time_series_dimension": true
                },
                "log_backup_time": {
                  "type": "date"
                },
                "log_recovery_size": {
                  "type": "long",
                  "meta": {
                    "unit": "byte"
                  },
                  "time_series_metric": "gauge"
                },
            }
          }
        },
}

Steps to Reproduce

Issue appears when we add any TSDB migrated integration with TSDB flag enabled.
Tested it with 2 integrations i.e MSSQL(Microsoft SQL Server) and RabbitMQ.
Dashboards are distorted with field not found error, for the fields of metric type counter.

Related Issue: #93539

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[ritalwar]

Also, Please refer:

[martijnvg]

Is it possible to retrieve the error that occurred in the last screenshot?
I'm trying to figure out what is going on here, and I don't follow why the error in the first screenshot is returned.
With the tsdb is enabled, max aggregation is supported on counter fields. When tsdb is disabled then all aggregations are allowed on counter fields.

[ritalwar]

There is nothing more to the last screenshot.
Just Inspected the Request for the errored lens visualisation shared along and got this.

[lalit-satapathy]

Duplicate of elastic/kibana#152467

[martijnvg]

I don't think that this is an Elasticsearch issue. Note that due to integration issue (#93539), the decision was made to not autocomplete counter fields in Kibana. But iirc if counter were to be used directly, then visualizations should be able to work. Work on its way so that in 8.8, counter fields can be autocompleted (elastic/kibana#152912).

[lalit-satapathy]

Closing as elastic/kibana#152467 is closed