Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use composable index templates for metric indices #1811

Merged
merged 7 commits into from
Dec 21, 2023

Conversation

inqueue
Copy link
Member

@inqueue inqueue commented Dec 2, 2023

In this commit, Rally is being switched from using legacy index templates to newer composable templates for metrics indices. Legacy index templates have been deprecated since ES 7.8 in favor of composable templates, and are not supported in ES Serverless. Rally metrics index templates have just one modification to make them composable templates. No other modifications have been made.

  • Elasticsearch favors composable templates over legacy index templates when both have matching index patterns. After this change, new metrics indices will use the new index templates.

Closes #1810.

@inqueue inqueue added the enhancement Improves the status quo label Dec 2, 2023
@inqueue inqueue self-assigned this Dec 2, 2023
@inqueue inqueue changed the title Use put_index_template for metrics index templates Use composable index templates for metric indices Dec 4, 2023
@inqueue inqueue marked this pull request as ready for review December 4, 2023 16:41
@inqueue inqueue added the :Metrics How metrics are stored, calculated or aggregated label Dec 4, 2023
Copy link
Contributor

@gbanasiak gbanasiak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

What do you think about adding a migration guide entry explaining the consequences of this change to existing ES metric stores:

  • legacy index templates are not removed automatically but composable index templates take precedence (applies to races, results and metrics),
  • index template for annotations only gets created when rally-annotations index is missing, so full migration from legacy to composable would require cumbersome renaming of rally-annotations index, followed by Rally run (esrally add annotation ...), followed by reindexing of old data to new rally-annotations index.

esrally/metrics.py Outdated Show resolved Hide resolved
@inqueue
Copy link
Member Author

inqueue commented Dec 20, 2023

LGTM.

What do you think about adding a migration guide entry explaining the consequences of this change to existing ES metric stores:

  • legacy index templates are not removed automatically but composable index templates take precedence (applies to races, results and metrics),
  • index template for annotations only gets created when rally-annotations index is missing, so full migration from legacy to composable would require cumbersome renaming of rally-annotations index, followed by Rally run (esrally add annotation ...), followed by reindexing of old data to new rally-annotations index.

Hi @gbanasiak, the requested changes have been applied. Would you mind taking another look?

@inqueue inqueue added the :Docs Changes to the documentation label Dec 20, 2023
Copy link
Contributor

@gbanasiak gbanasiak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for iterating. LGTM.

docs/migrate.rst Outdated Show resolved Hide resolved
Co-authored-by: Grzegorz Banasiak <grzegorz.banasiak@elastic.co>
@inqueue inqueue merged commit 4078777 into elastic:master Dec 21, 2023
15 checks passed
@gareth-ellis gareth-ellis added this to the 2.11 milestone May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Docs Changes to the documentation enhancement Improves the status quo :Metrics How metrics are stored, calculated or aggregated
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use the _index_template API for metrics templates
3 participants