Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move index sealing terminology to synced flush #11336

Merged
merged 3 commits into from
May 29, 2015
Merged

Move index sealing terminology to synced flush #11336

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b376a3f
Move index sealing terminology to synced flush
bleskes May 23, 2015
6d269cb
feedback
bleskes May 27, 2015
37bdbe0
doc feedback
bleskes May 27, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion docs/reference/indices.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,6 @@ and warmers.
* <<indices-refresh>>
* <<indices-flush>>
* <<indices-optimize>>
* <<indices-seal>>
* <<indices-upgrade>>

--
Expand Down
153 changes: 150 additions & 3 deletions docs/reference/indices/flush.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,9 @@ trigger flush operations as required in order to clear memory.

[source,js]
--------------------------------------------------
$ curl -XPOST 'http://localhost:9200/twitter/_flush'
POST /twitter/_flush
--------------------------------------------------
// AUTOSENSE

[float]
[[flush-parameters]]
Expand Down Expand Up @@ -39,7 +40,153 @@ or even on `_all` the indices.

[source,js]
--------------------------------------------------
$ curl -XPOST 'http://localhost:9200/kimchy,elasticsearch/_flush'
POST /kimchy,elasticsearch/_flush

$ curl -XPOST 'http://localhost:9200/_flush'
POST /_flush
--------------------------------------------------
// AUTOSENSE

[[indices-synced-flush]]
=== Synced Flush

Elasticsearch tracks the indexing activity of each shards. Shards that have not
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

of each shards^H

received any indexing operations for 30 minutes (configurable) are automatically marked as inactive. This presents
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

configurable, but where? do we document this anywhere? I couldn't find it. I'm not sure we should, but I'd either link "configurable" to the page with the setting, or remove it

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah - probably a good call. Just making it a link to an anchor tag would be great!

an opportunity for Elasticsearch to reduce shard resources and also perform
a special kind of flush, called `synced flush`. A synced flush performs normal
flushing and adds a special uniquely generated marker (`sync_id`) to all shards.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A synced flush performs a normal flush, then adds a generated unique marker (sync_id) to all shards.


Since the sync id marker was added when there were no ongoing indexing operations, it can
be used as a quick way to check if two shards indices are identical. This quick sync id
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two shards indices -> two shard copies

comparison (if present) is used during recovery or restarts to skip the first and
most costly phase of the process. In that case, no segment files need to be copied and
the transaction log replay phase of the recovery can start immediately. Note that since the sync id
marker was applied together with a flush, it is highly likely that the transaction log will be empty,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

highly -> very

speeding up recoveries even more.

This is particularly useful for use cases having lots of indices which are
never or very rarely updated, such as time based data. This use case typically generates lots of indices whose
recovery without the synced flush marker would take a long time.

To check whether a shard has a marker or not, one can use the `commit` section of shard stats returned by
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"one can use" (v formal) -> "look for the sync_id in the commit section..."

the <<indices-stats,indices stats>> API:

[source,bash]
--------------------------------------------------
GET /twitter/_stats/commit?level=shards
--------------------------------------------------
// AUTOSENSE

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps an example of the output?

[float]
=== Synced Flush API

The Synced Flush API allows an administrator to initiate a synced flush manually. This can be particularly useful for
a planned (rolling) cluster restart where one can stop indexing and doesn't want to wait for the default 30 minutes to pass
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"where ..." -> "you don't want to wait the default 30 minutes for idle indices to be sync-flushed automatically."

when the synced flush will be performed automatically.

While handy, there are a couple of caveats for this API:

1. Synced flush is a best effort operation. Any ongoing indexing operations will cause
the synced flush to fail. This means that some shards may be synced flushed while others aren't. See below for more.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"to fail"+ "on that shard".

2. The `sync_id` marker is removed as soon as the shard is flushed again. That is because a flush replaces the low level
lucene commit point where the marker is stored. Uncommitted operations in the transaction log do not remove the marker.
In practice, one should consider any indexing operation on an index as removing the marker as a flush can be triggered by Elasticsearch
at any time.


Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add a note: "It is harmless to request a synced flush while there is ongoing indexing. Shards that are idle will succeed and shards that are not will fail. Any shards that succeeded will have faster recovery times."

[source,bash]
--------------------------------------------------
POST /twitter/_flush/synced
--------------------------------------------------
// AUTOSENSE

The response contains details about how many shards were successfully sync-flushed and information about any failure.

Here is what it looks like when all shards of a two shards and one replica index successfully
sync-flushed:

[source,js]
--------------------------------------------------
{
"_shards": {
"total": 4,
"successful": 4,
"failed": 0
},
"twitter": {
"total": 4,
"successful": 4,
"failed": 0
}
}
--------------------------------------------------


Here is what it looks like when one shard group failed due to pending operations:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a callout with the returned HTTP status. In fact, should we include the status in the body, like we do for a number of other APIs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not consistent - seems to be API dependent. Can do, but then I'll have to wrote the index section under indices and that makes _shards weird. I'll add the note.


[source,js]
--------------------------------------------------
{
"_shards": {
"total": 4,
"successful": 2,
"failed": 2
},
"twitter": {
"total": 4,
"successful": 2,
"failed": 2,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"failed": 1 ?

I only see one failure message. :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, that's tricky - the failure is a failure of an entire shard group (for which we don't have a good name in ES), see text above. Thus it fails multiple shards. I agree it can be confusing, but I don't have a better way, which is consistent with other APIs and doesn't take lots of programming. suggestions welcome :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah - I see - its the primary and all the replicas. Makes sense. I don't have a better way to write it though.

"failures": [
{
"shard": 1,
"reason": "[2] ongoing operations on primary"
}
]
}
}
--------------------------------------------------


Sometimes the failures are specific to a shard copy. The copies that failed will not be eligible for
fast recovery but those that succeeded still will be. This case is reported as follows:

[source,js]
--------------------------------------------------
{
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a callout with the HTTP status

"_shards": {
"total": 4,
"successful": 1,
"failed": 1
},
"twitter": {
"total": 4,
"successful": 3,
"failed": 1,
"failures": [
{
"shard": 1,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably the other copies succeeded in this case. Is it a good idea to output these in the same total/successful/failed/failures format? Also, it might be useful to mention what this means from a recovery standpoint. Something like "those copies that failed will not be eligible for fast recovery but those that succeeded still will be."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the extra clarification for recoveries- it's great. I don't follow the suggestion to change the output format?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made that comment when I thought you had replica groups as the highest level so its not valid any more. Please ignore it.

"reason": "unexpected error",
"routing": {
"state": "STARTED",
"primary": false,
"node": "SZNr2J_ORxKTLUCydGX4zA",
"relocating_node": null,
"shard": 1,
"index": "twitter"
}
}
]
}
}
--------------------------------------------------


The synced flush API can be applied to more than one index with a single call,
or even on `_all` the indices.

[source,js]
--------------------------------------------------
POST /kimchy,elasticsearch/_flush/synced

POST /_flush/synced
--------------------------------------------------
// AUTOSENSE
91 changes: 0 additions & 91 deletions docs/reference/indices/seal.asciidoc
View file
Open in desktop

This file was deleted.

56 changes: 35 additions & 21 deletions docs/reference/setup/upgrade.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,13 +85,27 @@ This syntax applies to Elasticsearch 1.0 and later:

[source,sh]
--------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.enable" : "none"
}
}'
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.enable" : "none"
}
}'
--------------------------------------------------

* There is no problem continuing to index while doing the upgrade. However, you can speed the process considerably
by *temporarily* stopping non-essential indexing and issuing a manual <<indices-synced-flush, synced flush>>.
A synced flush is special kind of flush which can seriously speed up recovery of shards. Elasticsearch automatically
uses it when an index has been inactive for a while (default is `30m`) but you can manuallky trigger it using the following command:

[source,sh]
--------------------------------------------------
curl -XPOST localhost:9200/_all/_flush/synced
--------------------------------------------------

Note that a synced flush call is a best effort operation. It will fail there are any pending indexing operations. It is safe to issue
it multiple times if needed.


* Shut down a single node within the cluster.

* Confirm that all shards are correctly reallocated to the remaining running nodes.
Expand All @@ -110,11 +124,11 @@ This syntax applies to Elasticsearch 1.0 and later:

[source,sh]
--------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.enable" : "all"
}
}'
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.enable" : "all"
}
}'
--------------------------------------------------

* Observe that all shards are properly allocated on all nodes. Balancing may take some time.
Expand Down Expand Up @@ -150,11 +164,11 @@ This syntax is from versions prior to 1.0:

[source,sh]
--------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"cluster.routing.allocation.disable_allocation" : true
}
}'
curl -XPUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"cluster.routing.allocation.disable_allocation" : true
}
}'
--------------------------------------------------

* Stop all Elasticsearch services on all nodes in the cluster.
Expand All @@ -169,12 +183,12 @@ This syntax is from versions prior to 1.0:
This syntax is from release 1.0 and later:
[source,sh]
------------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"cluster.routing.allocation.disable_allocation": false,
"cluster.routing.allocation.enable" : "all"
}
}'
curl -XPUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"cluster.routing.allocation.disable_allocation": false,
"cluster.routing.allocation.enable" : "all"
}
}'
------------------------------------------------------

The cluster upgrade can be streamlined by installing the software before stopping cluster services. If this is done, testing must be performed to ensure that no production data or configuration files are overwritten prior to restart.
39 changes: 39 additions & 0 deletions rest-api-spec/api/indices.flush_synced.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"indices.flush.synced": {
"documentation": "http://www.elastic.co/guide/en/elasticsearch/reference/master/indices-flush.html",
"methods": ["POST", "GET"],
"url": {
"path": "/_flush/synced",
"paths": [
"/_flush/synced",
"/{index}/_flush/synced"
],
"parts": {
"index": {
"type" : "list",
"description" : "A comma-separated list of index names; use `_all` or empty string for all indices"
},
"ignore_unavailable": {
"type": "boolean",
"description": "Whether specified concrete indices should be ignored when unavailable (missing or closed)"
},
"allow_no_indices": {
"type": "boolean",
"description": "Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes `_all` string or when no indices have been specified)"
},
"expand_wildcards": {
"type": "enum",
"options": [
"open",
"closed",
"none",
"all"
],
"default": "open",
"description": "Whether to expand wildcard expression to concrete indices that are open, closed or both."
}
}
},
"body": null
}
}
Loading