[TSDB] Dimension limit override settings blocking the build

[agithomas]

Refer : https://fleet-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-5312/12/pipeline

The configuration to override the default limit is provided as

elasticsearch:
  index_template:
    settings:
      index.mapping.dimension_fields.limit: 32

Inspite this is a valid configuration, the integration test error appears. This may have an impact on more than one packages , not just limited to the influxdb.

[jsoriano]

@agithomas could you try to install the package and check in the index templates if the setting is there?

[agithomas]

@agithomas could you try to install the package and check in the index templates if the setting is there?

Yes, i can find that in the index template.

    "mapping": {
      "total_fields": {
        "limit": "10000"
      },
      "dimension_fields": {
        "limit": "32"
      }
    },

[jsoriano]

Is this in the main/composed index template, or in the component template with the package fields? If it is in the main one maybe it is applying the limits per component template 🤔

[jsoriano]

Looks related: elastic/elasticsearch#93564

[jsoriano]

Is this in the main/composed index template, or in the component template with the package fields? If it is in the main one maybe it is applying the limits per component template thinking

It is in the same component template where the package fields are defined.

@martijnvg is this expected? Should we install the dimension fields limit in the composed index template?

Find here a dump of the templates installed for this package: package-dump.zip

[agithomas]

I think, i understood what is happening.

• The above issue is found when i tried to enable TSDB for influxdb package, having 8.3.0 as the minimum kibana version also the ES version.

• When the test is run, 8.3.0 ES is created and the tests are executed on 8.3.0 stack.

• In 8.3.0, it appears, 16 is the default dimension limit and it cannot be overwritten by the settings such as

elasticsearch:
  index_template:
    settings:
      index.mapping.dimension_fields.limit: 32

@martijnvg , from which version of the ES, the ability to override the dimension limit (ie: index.mapping.dimension_fields.limit: <new limit> )exist in ES?

[martijnvg]

What is the actual error that occurs here? This particular setting does seem to exist in 8.3.0 and earlier versions.

However note that TSDB became available as tech preview (elastic/elasticsearch#90116)) with version 8.5.0 and many other tsdb related settings were not available before version 8.5.0.

[agithomas]

What is the actual error that occurs here? This particular setting does seem to exist in 8.3.0 and earlier versions.

[illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded"

[martijnvg]

When does this error occur? When adding a component template, index template or when a data stream gets created?

[agithomas]

When does this error occur? When adding a component template, index template or when a data stream gets created?

• You face the error during the integration testing. Logs : https://fleet-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-5312/12/pipeline

• You face the error when creating an integration. Screenshot below. Soon after the error, i check if there exist

• a datastream

• an index template

• a component template

• an index

having a matching regex `influx', in kibana. But, i do not find anything that match.

So, i can't really tell the stage at where the failure occur in 8.3.0 stack.

Any other steps i can run to get the exact stage of failure?

[agithomas]

When does this error occur? When adding a component template, index template or when a data stream gets created?

The extended error log line from the integration test log is as below

Error: error running package system tests: could not complete test run: could not add data stream config to policy: could not add package to policy; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"Error installing influxdb 0.3.0: illegal_argument_exception: [illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded"}

[lalit-satapathy]

When the test is run, 8.3.0 ES is created and the tests are executed on 8.3.0 stack.

The issue could be because of "system test" is running an older version of the ES.

[martijnvg]

I tried installing the templates in the package dump here locally and everything seems to work ok. No validation errors.
The index.mapping.dimension_fields.limit index setting exists for a long time already (from 8.0 if I'm not mistaken) and has always been a dynamic setting, meaning that it could be updated.

The mentioned Limit of total dimension fields [16] has been exceeded error indicates that somehow limit has not been increased to 48 or 32 which both component templates are doing. It would be helpful to know which Elasticsearch api call returns this error and then check the associated data stream or template. From what I can see this shouldn't occur if any of the templates in the package dump get installed.

[agithomas]

Testing with 8.6 stack . Gets a successful run of system test

Testing with 8.4 Stack . Do not get a successful run of system test

** Testing with 8.5 Stack. Do not get a successful run of the system test **

** Tested again with 8.6 Stack. Gets a successful run **

** Tested with 8.7.0-Snapshot. Gets a successful run **

In all the above tests, only one thing that differs in the ES stack version.

Are two different APIs getting invoked before and after 8.6.0 ? @ruflin , @jsoriano , do you know? Or any means of knowing / verifying this?

[jsoriano]

The failure happens in the request to Kibana to add the package policy (POST kbn:/api/fleet/fleet_policies). It can be also reproduced when installing a package using POST kbn:/api/fleet/epm/packages/influxdb-0.3.0.

I don't know exactly on what request to ES it fails. It only logs:

[2023-03-07T19:08:54.844+00:00][WARN ][plugins.fleet] Failure to install package [influxdb]: [ResponseError: illegal_argument_exception: [illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded]

The dump I did in #1174 (comment) was with 8.6.1, actually if I try to dump on previous versions I cannot because the package cannot be installed.

In principle elastic-package is doing exactly the same requests in 8.5 and 8.6, @martijnvg could it be that something related has changed on how these settings are handled in 8.6?

Or @kpollich, could it be that something related changed in how Kibana installs packages in 8.6?

@agithomas the quickest path for you to unblock your progress will probably be to update the kibana version constraint to ^8.6.0 if this is an option.

[lalit-satapathy]

The kibana.version issue was discussed as a feedback from @ruflin earlier.

Ideal behaviour for us that index.mapping.dimension_fields.limit check is ignored on older versions and not throw an error. Is that feasible?

[martijnvg]

In principle elastic-package is doing exactly the same requests in 8.5 and 8.6, @martijnvg could it be that something related has changed on how these settings are handled in 8.6?

Not that I know of. The mentioned setting (index.mapping.dimension_fields.limit) and the ability to define fields as dimensions hasn't changed to my knowledge in the mentioned versions.

[kpollich]

[2023-03-07T19:08:54.844+00:00][WARN ][plugins.fleet] Failure to install package [influxdb]: [ResponseError: illegal_argument_exception: [illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded]

Ideal behaviour for us that index.mapping.dimension_fields.limit check is ignored on older versions and not throw an error. Is that feasible?

Or @kpollich, could it be that something related changed in how Kibana installs packages in 8.6?

Docs for this setting are here: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html#mapping-settings-limit

Fleet does not provide this setting in any way, so we're just getting the default value of 16 here. If we're exceeding the limit then I view this as an expected error. We'd need a way for the package to export a different limit or for Fleet to hardcode a higher one into all index templates.

As for why this fails in 8.6 - Fleet didn't support TSDB in any way prior to 8.6. As of elastic/kibana#144974, Fleet actually honors dimension fields and enables time series indexing on appropriate data streams. So, in earlier versions all time series metrics, dimensions, etc are ignored when installing the package. In 8.6, Fleet will actually honor these settings.

[jsoriano]

Docs for this setting are here: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html#mapping-settings-limit

Fleet does not provide this setting in any way, so we're just getting the default value of 16 here. If we're exceeding the limit then I view this as an expected error. We'd need a way for the package to export a different limit or for Fleet to hardcode a higher one into all index templates.

Packages can already override this limit through elasticsearch.index_template.settings, as is the case here (from this PR). And Fleet is installing it (at least from 8.6), as seen on the dump on this comment #1174 (comment).

As for why this fails in 8.6 - Fleet didn't support TSDB in any way prior to 8.6. As of elastic/kibana#144974, Fleet actually honors dimension fields and enables time series indexing on appropriate data streams. So, in earlier versions all time series metrics, dimensions, etc are ignored when installing the package. In 8.6, Fleet will actually honor these settings.

It seems that dimensions are installed before 8.6, even if TSDB is not enabled yet. The error about dimensions limit exceeded appears on previous versions.

@martijnvg could it be that dimension_fields.limit setting is only leveraged if tsdb mode is enabled?

[lalit-satapathy]

Need a closure on this. Although currently only InfluxDB is impacted, but that package is already on 8.6. We will see this issue impacting more packages once they start migrating.

[jsoriano]

I am afraid that the only solution at this point is to increase the kibana version constraint to at least ^8.6.0 for packages that need to increase the dimensions limit.
The alternative would be to change the behaviour in older versions of kibana and/or elasticsearch, and I don't think this is feasible at this point 😕

[ruflin]

@lalit-satapathy How many packages are affected at the moment by requiring going above the default limit?

[lalit-satapathy]

@lalit-satapathy How many packages are affected at the moment by requiring going above the default limit?

InfluxDB and may be few other prometheus type packages. The current understanding is, we may force kibana.version for TSDB packages >= 8.8. as per that plan, this issue may not be considered critical.

[lalit-satapathy]

@lalit-satapathy How many packages are affected at the moment by requiring going above the default limit?

InfluxDB and may be few other prometheus type packages. The current understanding is, we may force kibana.version for TSDB packages >= 8.8. as per that plan, this issue may not be considered critical.

@agithomas, Should we keep this open?

[agithomas]

Closing this because it is decided that 8.8.x will be the min version for TSDB enablement for integration packages.