TSDB: Facing issue while reindexing a TSDB enabled data stream

[ali786XI]

Elastic Stack version :- 8.8.0

I am trying to reindex a TSDB enabled data stream ( server data stream of CouchDB ) data but I am facing an error at the last step of reindexing. It seems like while reindexing, my existing settings of a TSDB enabled index are not applied on the destination index.

Step 1 Firstly, I am storing my existing data into a temp_index using the Reindex API.

POST _reindex
{
  "source": {
    "index": "metrics-couchdb.server-default"
  },
  "dest": {
    "index": "temp_index"
  }
}

Step 2 Now I need to delete the data stream and it's index templates so that after upgrading the package with new mappings I can reinstall my integration and get those new templates

DELETE /_data_stream/metrics-couchdb.server-default

DELETE _index_template/metrics-couchdb.server

Step 3 After installing the new version of the integration and upgrading the integration policies, I am running the last step of reindexing API.

POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "index": "temp_index"
  },
  "dest": {
    "index": "metrics-couchdb.server-default",
    "op_type": "create"
  }
}

At this step, I am facing an error as below.

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "[index.mode=time_series] requires a non-empty [index.routing_path]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "[index.mode=time_series] requires a non-empty [index.routing_path]"
  },
  "status": 400
}

Let me know what is the solution for this.

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[salvatore-campagna]

We need settings and templates applied to involved indices and data streams.

I can guess a possible reason for the failure here and my guess will also explain why I am asking you to provide settings and templates. Consider that given the information I have right now I might be wrong. Applying settings on data streams and their indices is done using templates which normally match indices by means of a pattern. I guess here, due to the naming of the indices (temp_index), templates do not apply and settings are not applied as a consequence. For this reason I suspect temp_index is missing all tsdb specific settings, including the routing path fields. One of the settings required for a tsdb index is the routing path (routing path fields are a subset of dimension fields).

As a side note, if my guess is correct, a possible workaround would be to get the settings from the original index and apply them manually to temp_index after reindexing is complete but before indexing data into the destination data stream.

[ali786XI]

Here the settings applied to the metrics-couchdb.server template

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "metrics"
        },
        "mode": "time_series",
        "codec": "best_compression",
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "mapping": {
          "total_fields": {
            "limit": "10000"
          }
        },
        "time_series": {
          "end_time": "2023-08-03T13:47:32.000Z",
          "start_time": "2023-08-03T09:47:32.000Z"
        },
        "final_pipeline": ".fleet_final_pipeline-1",
        "query": {
          "default_field": [
            "ecs.version",
            "error.message",
            "event.category",
            "event.dataset",
            "event.kind",
            "event.module",
            "event.type",
            "host.name",
            "service.address",
            "service.type",
            "tags",
            "agent.id",
            "cloud.account.id",
            "cloud.region",
            "cloud.instance.id",
            "cloud.provider",
            "cloud.availability_zone",
            "container.id"
          ]
        },
        "default_pipeline": "metrics-couchdb.server-0.8.0",
        "routing_path": [
          "cloud.instance.id",
          "service.address",
          "agent.id",
          "container.id",
          "cloud.account.id",
          "cloud.region",
          "cloud.availability_zone",
          "host.name",
          "cloud.provider"
        ]
      }
    }
  }
}

[salvatore-campagna]

We need to check if they are applied to temp_index. Could you please also provide me with the settings on the temp_index after reindexing into temp_index is complete?

[ali786XI]

We need to check if they are applied to temp_index. Could you please also provide me with the settings on the temp_index after reindexing into temp_index is complete?

Yes correct. Those settings are not being applied to the temp_index. Here is the setting of the temp_index after reindexing. My doubt is now these TSDB related settings are applied at runtime now or I can update some of the settings beforehand?

{
  "settings": {
    "index": {
      "routing": {
        "allocation": {
          "include": {
            "_tier_preference": "data_content"
          }
        }
      },
      "version": {
        "created": "8090099"
      }
    }
  },
  "defaults": {
    "index": {
      "final_pipeline": "_none",
      "max_inner_result_window": "100",
      "unassigned": {
        "node_left": {
          "delayed_timeout": "1m"
        }
      },
      "max_terms_count": "65536",
      "rollup": {
        "source": {
          "name": "",
          "uuid": ""
        }
      },
      "lifecycle": {
        "prefer_ilm": "true",
        "rollover_alias": "",
        "origination_date": "-1",
        "name": "",
        "parse_origination_date": "false",
        "step": {
          "wait_time_threshold": "12h"
        },
        "indexing_complete": "false"
      },
      "merge": {
        "scheduler": {
          "max_thread_count": "4",
          "auto_throttle": "true",
          "max_merge_count": "9"
        },
        "policy": {
          "merge_factor": "32",
          "floor_segment": "2mb",
          "max_merge_at_once_explicit": "30",
          "max_merge_at_once": "10",
          "max_merged_segment": "0b",
          "expunge_deletes_allowed": "10.0",
          "segments_per_tier": "10.0",
          "type": "UNSET",
          "deletes_pct_allowed": "20.0"
        }
      },
      "max_refresh_listeners": "1000",
      "max_regex_length": "1000",
      "load_fixed_bitset_filters_eagerly": "true",
      "number_of_routing_shards": "1",
      "write": {
        "wait_for_active_shards": "1"
      },
      "mapping": {
        "coerce": "false",
        "nested_fields": {
          "limit": "50"
        },
        "depth": {
          "limit": "20"
        },
        "field_name_length": {
          "limit": "9223372036854775807"
        },
        "total_fields": {
          "limit": "1000"
        },
        "nested_objects": {
          "limit": "10000"
        },
        "ignore_malformed": "false",
        "dimension_fields": {
          "limit": "21"
        }
      },
      "source_only": "false",
      "soft_deletes": {
        "enabled": "true",
        "retention": {
          "operations": "0"
        },
        "retention_lease": {
          "period": "12h"
        }
      },
      "max_script_fields": "32",
      "query": {
        "default_field": [
          "*"
        ],
        "parse": {
          "allow_unmapped_fields": "true"
        }
      },
      "format": "0",
      "frozen": "false",
      "sort": {
        "missing": [],
        "mode": [],
        "field": [],
        "order": []
      },
      "routing_path": [],
      "version": {
        "compatibility": "8090099"
      }
    }
  }
}

[salvatore-campagna]

I think you can try manually applying the missing settings to temp_index before using it to reindex data into the data stream. Probably you can just overwrite temp_index settings with the ones from the original index. Note, I am assuming that templates apply correctly with all the settings into the data stream index you are using as a destination. I am trying to understand if the error is happening while reindexing and to be more precise when reading the source index temp_index in time series mode without having all expected settings.

[ali786XI]

If I try to update the setting for the temp_index using the update index API. (tried updating the settings only related to TSDB ones)

PUT /temp_index/_settings
{
  "settings": {
    "index": {
      "routing_path": [
        "cloud.provider",
        "host.name",
        "cloud.account.id",
        "agent.id",
        "cloud.instance.id",
        "service.address",
        "cloud.region",
        "container.id",
        "cloud.availability_zone"
      ]
    }
  }
}

then it throws the following error

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "final temp_index setting [index.routing_path], not updateable"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "final temp_index setting [index.routing_path], not updateable"
  },
  "status": 400
}

[salvatore-campagna]

Yes, some of the settings are valid only when the index is created. Maybe you can try manually creating the destination index before reindexing and applying all the settings at creation time.

[salvatore-campagna]

I didn't notice before that you are manually deleting the data stream and the template...Why are you doing that?
I guess if the template is deleted and later on you create the destination index again (manually I guess), settings are not applied. Probably applying the settings to temp_index is not strictly required...what is required for sure is that the destination index is a time series index and includes all time series related settings (see here for more details: https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds-index-settings.html)

As you can see here: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
the reindex API does not copy settings:

Reindex requires [_source](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html) to be enabled for all documents in the source.

The destination should be configured as wanted before calling _reindex. Reindex does not copy the settings from the source or its associated template.

Mappings, shard counts, replicas, and so on must be configured ahead of time.

[ali786XI]

I didn't notice before that you are manually deleting the data stream and the template...Why are you doing that?

Actually, I can give you an overview of the issue. The problem here is the a field host.ip which is showing as a conflicting field in Kibana. The solution to this is providing the mapping in ecs.yml. Now after the mapping is applied, we need to update the templates as well so as to reflect the mapping. For this after the data is copied to a temporary index, we have to delete the index template related to the data stream which in turn also requires to delete the data stream as well.

Probably applying the settings to temp_index is not strictly required...what is required for sure is that the destination index is a time series index and includes all time series related settings

Thanks for this I am able to apply the settings to the temp_index. But again the data stream is deleted right so at the 3rd step, when it will try to create a data_stream it again requires those TSDB settings. So what can be the take for this?

[salvatore-campagna]

Yes you are right you need to have those settings applied to the data stream indices. I would say that applying those to temp_index is not really required (I was wrong in suggesting that at the beginning). I think the missing settings need to be applied by means of a template. You can't create the index and later apply settings because settings need to be applied at index creation time. As you already experienced trying to update settings after the index is created is not possible since for some settings updates are not allowed.

[salvatore-campagna]

I realized now that you might need to create the data stream index manually including all the settings at creation time, paying attention to the start_time and end_time time. When reindexing, indeed the timestamp field will be there and it needs to fall into the time series index (start_time, end_time) range to avoid errors or documents being dropped.

I know this might be a bit tedious to do manually...but we do not have a better way...at least at the moment.

[agithomas]

@salvatore-campagna / @martijnvg , presently as many as 10 integrations need to have the steps as part of the documentation. Considering the large scope, it is possible to have the steps documented as part of ElasticSearch documentation.

Once this documentation is available, we can have the incorrect information removed from the Integration readme and instead provide a link to the documentation. Can this be taken up in the upcoming release?

cc @lalit-satapathy

[agithomas]

@aliabbas-elastic , can you follow the steps mentioned in the issue and following them the TSDB re-indexing could be performed?

[ali786XI]

Sure @agithomas I ll have a look. Thanks !!

[ali786XI]

Tested the reindexing steps on TSDB enabled integrations and they are resolving the conflicts.