Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Limit concurrent snapshot file restores in recovery per node #79316

Merged
merged 5 commits into from
Oct 18, 2021
Merged

Limit concurrent snapshot file restores in recovery per node #79316

merged 5 commits into from
Oct 18, 2021

Commits on Oct 18, 2021

  1. Limit the number of concurrent snapshot file restores per node during… 

    … recoveries

Today we limit the max number of concurrent snapshot file restores
per recovery. This works well when the default
node_concurrent_recoveries is used (which is 2). When this limit is
increased, it is possible to exahust the underlying repository
connection pool, affecting other workloads.

This commit adds a new setting
`indices.recovery.max_concurrent_snapshot_file_downloads_per_node` that
allows to limit the max number of snapshot file downloads per node
during recoveries. When a recovery starts in the target node it tries
to acquire a permit that allows it to download snapshot files when it is
granted. This is communicated to the source node in the
StartRecoveryRequest. This is a rather conservative approach since it is
possible that a recovery that gets a permit to use snapshot files
doesn't recover any snapshot file while there's a concurrent recovery
that doesn't get a permit could take advantage of recovering from a
snapshot.

Closes elastic#79044
    @fcofdez
    fcofdez committed Oct 18, 2021
    Configuration menu
    Copy the full SHA
    8a8b13d View commit details
    Browse the repository at this point in the history

  2. Review comments

    @fcofdez
    fcofdez committed Oct 18, 2021
    Configuration menu
    Copy the full SHA
    ed9c4ef View commit details
    Browse the repository at this point in the history

  3. Fix test

    @fcofdez
    fcofdez committed Oct 18, 2021
    Configuration menu
    Copy the full SHA
    89ff1ed View commit details
    Browse the repository at this point in the history

  4. More review comments

    @fcofdez
    fcofdez committed Oct 18, 2021
    Configuration menu
    Copy the full SHA
    7182bf7 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/master' into limit-concurrent-re… 

    …covery-connections
    @fcofdez
    fcofdez committed Oct 18, 2021
    Configuration menu
    Copy the full SHA
    5325c6e View commit details
    Browse the repository at this point in the history