Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize index settings for synthetics #2196

Closed
andrewvc opened this issue Nov 19, 2021 · 1 comment · Fixed by #2319
Closed

Optimize index settings for synthetics #2196

andrewvc opened this issue Nov 19, 2021 · 1 comment · Fixed by #2319
Assignees
Labels
enhancement New feature or request Team:obs-ux-infra_services Label for the Obs UX: Infra & Services team [elastic/obs-ux-infra_services-team]

Comments

@andrewvc
Copy link
Contributor

The current mappings for this integration consume far too much space for network data in particular. Network data takes up the vast majority of Elasticsearch storage space in most cases. There are a number of sub-optimal things about our mappings, for network data in particular. Fixing these has yielded a 50% reduction in storage space in some ad-hoc tests.

The main problems with the current mapping are:

  1. Dynamic mappings are enabled, and there are a number of fields sent by heartbeat that are not part of the explicit mapping. Auto-mapping these takes up a large quantity of space. Some such fields are individual HTTP headers.
  2. Compression can be bumped up to best_compression since this is a write-heavy, read-light index
  3. Index sorting can be enabled. Sorting the index by similar data puts similar docs in adjoining spots on disk, which makes them wind up grouped together in compression blocks making compression vastly more efficient.
  4. In the future we should curate the list of fields more thoroughly and index only the minimum number, but to start with let's pick the low hanging fruit.

The following index settings should be sufficient. I tried adding them to synthetics/manifest.yml but that did not seem to actually affect the templates.

elasticsearch:
  index_template:
    mappings:
      dynamic: false
    settings:
      index:
        codec: best_compression
        sort.field: 
        - "url.full"
        - "http.request.url"
        - "http.response.headers.etag"
        - "monitor.id"
@andrewvc andrewvc added enhancement New feature or request Team:obs-ux-infra_services Label for the Obs UX: Infra & Services team [elastic/obs-ux-infra_services-team] labels Nov 19, 2021
@elasticmachine
Copy link

Pinging @elastic/uptime (Team:Uptime)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Team:obs-ux-infra_services Label for the Obs UX: Infra & Services team [elastic/obs-ux-infra_services-team]
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants