Move index sealing terminology to synced flush #11336

bleskes · 2015-05-25T19:34:23Z

#10032 introduced the notion of sealing an index by marking it with a special read only marker, allowing for a couple of optimization to happen. The most important one was to speed up recoveries of shards where we know nothing has changed since they were online by skipping the file based sync phase. During the implementation we came up with a light notion which achieves the same recovery benefits but without the read only aspects which we dubbed synced flush. The fact that it was light weight and didn't put the index in read only mode, allowed us to do it automatically in the background which has great advantage. However we also felt the need to allow users to manually trigger this operation.

The implementation at #11179 added the sync flush internal logic and the manual (rest) rest API. The name of the API was modeled after the sealing terminology which may end up being confusing. This commit changes the API name to match the internal synced flush naming, namely `{index}/_flush/synced'.

On top of that it contains a couple other changes:

Remove all java client API. This feature is not supposed to be called programtically by applications but rather by admins.
Improve rest responses making structure similar to other (flush) API
Change IndexShard#getOperationsCount to exclude the internal +1 on open shard . it's confusing to get 1 while there are actually no ongoing operations
Some minor other clean ups

Closes #11251

elastic#10032 introduced the notion of sealing an index by marking it with a special read only marker, allowing for a couple of optimization to happen. The most important one was to speed up recoveries of shards where we know nothing has changed since they were online by skipping the file based sync phase. During the implementation we came up with a light notion which achieves the same recovery benefits but without the read only aspects which we dubbed synced flush. The fact that it was light weight and didn't put the index in read only mode, allowed us to do it automatically in the background which has great advantage. However we also felt the need to allow users to manually trigger this operation. The implementation at elastic#11179 added the sync flush internal logic and the manual (rest) rest API. The name of the API was modeled after the sealing terminology which may end up being confusing. This commit changes the API name to match the internal synced flush naming, namely `{index}/_flush/synced'. On top of that it contains a couple other changes: - Remove all java client API. This feature is not supposed to be called programtically by applications but rather by admins. - Improve rest responses making structure similar to other (flush) API - Change IndexShard#getOperationsCount to exclude the internal +1 on open shard . it's confusing to get 1 while there are actually no ongoing operations - Some minor other clean ups

bleskes · 2015-05-25T19:34:41Z

@brwe @clintongormley can you please review?

clintongormley · 2015-05-25T19:46:27Z

docs/reference/indices/flush.asciidoc

+1. Synced flush is a best effort operation. Any ongoing indexing operations will cause
+the synced flush to fail. This means that some shards may be synced flushed while others aren't. See below for more.
+2. The `sync_id` marker is removed as soon as the shard is flushed again. Uncommitted
+operations in the transaction log do not remove the marker. That is because the marker is store as part


the marker is storeD

clintongormley · 2015-05-25T19:48:26Z

Docs look great! (two tiny changes)

bleskes · 2015-05-25T19:55:59Z

docs/reference/indices/flush.asciidoc

+[float]
+=== Synced Flush API
+
+The Synced Flush API allows an administrator to initiate a synced flush manually. This can particularly useful for


This can BE particularly useful for

nik9000 · 2015-05-26T12:55:38Z

I don't mind the "seal" name - I just stopped thinking of it as a hermetic seal and started thinking of it as a wax seal on an envelope. You break it when you stuff more documents in it. I forget the other half of the metaphor about having to break the seal to read the documents.

brwe · 2015-05-26T13:09:16Z

src/main/java/org/elasticsearch/rest/action/admin/indices/flush/RestSyncedFlushAction.java

                builder.endObject();
-                return new BytesRestResponse(response.status(), builder);
+                return new BytesRestResponse(RestStatus.OK, builder);


ok for now but we need to figure out what to return (#11251)

nik9000 · 2015-05-26T13:38:48Z

I don't mind the "seal" name - I just stopped thinking of it as a hermetic seal and started thinking of it as a wax seal on an envelope. You break it when you stuff more documents in it. I forget the other half of the metaphor about having to break the seal to read the documents.

I take it back - after reviewing the documentation this way makes more sense to me. No fun metaphor though.

bleskes · 2015-05-27T07:36:19Z

@brwe @clintongormley @nik9000 thx for all the feedback. I pushed a new commit. Also assumed will end up with a 409 for #11251 ..

clintongormley · 2015-05-27T10:12:48Z

docs/reference/indices/flush.asciidoc

+[[indices-synced-flush]]
+=== Synced Flush
+
+Elasticsearch tracks the indexing activity of each shards. Shards that have not


of each shards^H

clintongormley · 2015-05-27T10:24:12Z

docs/reference/indices/flush.asciidoc

+GET /twitter/_stats/commit?level=shards
+--------------------------------------------------
+// AUTOSENSE
+


Perhaps an example of the output?

clintongormley · 2015-05-27T10:34:07Z

Minor doc comments, but looking good!

brwe · 2015-05-27T11:30:36Z

this is tagged for 2.0 but should it not also go in 1.6?

nik9000 · 2015-05-27T12:39:58Z

this is tagged for 2.0 but should it not also go in 1.6?

Yeah, I was just about to ask that.

bleskes · 2015-05-27T12:40:57Z

pushed another update. I was planning the back port PR (which is likely to happen) with 1.6., but can mark this one as well...

brwe · 2015-05-28T09:34:44Z

LGTM but not sure if that counts as a go

s1monw · 2015-05-28T09:41:49Z

src/test/java/org/elasticsearch/indices/flush/FlushTest.java

@@ -140,26 +147,7 @@ public void testSyncedFlush() throws ExecutionException, InterruptedException, I
    }

    @TestLogging("indices:TRACE")


remove the trace here

s1monw · 2015-05-28T09:55:27Z

LGTM too

To better distribute the memory allocating to indexing, the IndexingMemoryController periodically checks the different shard for their last indexing activity. If no activity has happened for a while, the controller marks the shards as in active and allocated it's memory buffer budget (but a small minimal budget) to other active shards. The recently added synced flush feature (elastic#11179, elastic#11336) uses this inactivity trigger to attempt as a trigger to attempt adding a sync id marker (which will speed up future recoveries). We wait for 30m before declaring a shard inactive. However, these days the operation just requires a refresh and is light. We can be stricter (and 5m) increase the chance a synced flush will be triggered.

To better distribute the memory allocating to indexing, the IndexingMemoryController periodically checks the different shard for their last indexing activity. If no activity has happened for a while, the controller marks the shards as in active and allocated it's memory buffer budget (but a small minimal budget) to other active shards. The recently added synced flush feature (#11179, #11336) uses this inactivity trigger to attempt as a trigger to attempt adding a sync id marker (which will speed up future recoveries). We wait for 30m before declaring a shard inactive. However, these days the operation just requires a refresh and is light. We can be stricter (and 5m) increase the chance a synced flush will be triggered. Closes #11479

bleskes added >non-issue v2.0.0-beta1 review labels May 25, 2015

bleskes mentioned this pull request May 25, 2015

Response codes for index sealing #11251

Closed

clintongormley reviewed May 25, 2015
View reviewed changes

bleskes reviewed May 25, 2015
View reviewed changes

brwe reviewed May 26, 2015
View reviewed changes

bleskes mentioned this pull request May 26, 2015

Allow to seal an index #10032

Closed

feedback

6d269cb

clintongormley reviewed May 27, 2015
View reviewed changes

doc feedback

37bdbe0

bleskes added the v1.6.0 label May 27, 2015

s1monw reviewed May 28, 2015
View reviewed changes

brwe merged commit 37bdbe0 into elastic:master May 29, 2015

kevinkluge removed the review label May 29, 2015

brwe added a commit that referenced this pull request May 29, 2015

[doc] remove reference to seal, was removed in #11336

a031232

brwe added a commit to brwe/elasticsearch that referenced this pull request May 29, 2015

[doc] remove reference to seal, was removed in elastic#11336

d7cb1b9

clintongormley mentioned this pull request May 29, 2015

The restart node join cluster very slow #11415

Closed

clintongormley added >feature release highlight :Distributed/Recovery Anything around constructing a new shard, either from a local or a remote source. and removed >non-issue labels May 29, 2015

clintongormley mentioned this pull request May 31, 2015

Make cluster recovery near instantaneous if all shards are present and accounted for #6069

Closed

bleskes mentioned this pull request Jun 3, 2015

Reduce shard inactivity timeout to 5m #11479

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move index sealing terminology to synced flush #11336

Move index sealing terminology to synced flush #11336

bleskes commented May 25, 2015

bleskes commented May 25, 2015

clintongormley May 25, 2015

clintongormley commented May 25, 2015

bleskes May 25, 2015

nik9000 commented May 26, 2015

brwe May 26, 2015

nik9000 commented May 26, 2015

bleskes commented May 27, 2015

clintongormley May 27, 2015

clintongormley May 27, 2015

clintongormley commented May 27, 2015

brwe commented May 27, 2015

nik9000 commented May 27, 2015

bleskes commented May 27, 2015

brwe commented May 28, 2015

s1monw May 28, 2015

s1monw commented May 28, 2015

		@@ -140,26 +147,7 @@ public void testSyncedFlush() throws ExecutionException, InterruptedException, I
		}

		@TestLogging("indices:TRACE")

Move index sealing terminology to synced flush #11336

Move index sealing terminology to synced flush #11336

Conversation

bleskes commented May 25, 2015

bleskes commented May 25, 2015

clintongormley May 25, 2015

Choose a reason for hiding this comment

clintongormley commented May 25, 2015

bleskes May 25, 2015

Choose a reason for hiding this comment

nik9000 commented May 26, 2015

brwe May 26, 2015

Choose a reason for hiding this comment

nik9000 commented May 26, 2015

bleskes commented May 27, 2015

clintongormley May 27, 2015

Choose a reason for hiding this comment

clintongormley May 27, 2015

Choose a reason for hiding this comment

clintongormley commented May 27, 2015

brwe commented May 27, 2015

nik9000 commented May 27, 2015

bleskes commented May 27, 2015

brwe commented May 28, 2015

s1monw May 28, 2015

Choose a reason for hiding this comment

s1monw commented May 28, 2015