Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add doc's sequence number + primary term to GetResult and use it for updates #36680

Merged
merged 7 commits into from
Dec 17, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions docs/plugins/ingest-attachment.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ Returns this:
"_type": "_doc",
"_id": "my_id",
"_version": 1,
"_seq_no": 22,
"_primary_term": 1,
"_source": {
"data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
"attachment": {
Expand All @@ -74,7 +76,7 @@ Returns this:
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]


To specify only some fields to be extracted:
Expand Down Expand Up @@ -146,6 +148,8 @@ Returns this:
"_type": "_doc",
"_id": "my_id",
"_version": 1,
"_seq_no": 35,
"_primary_term": 1,
"_source": {
"data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
"attachment": {
Expand All @@ -157,7 +161,7 @@ Returns this:
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]


[source,js]
Expand Down Expand Up @@ -194,6 +198,8 @@ Returns this:
"_type": "_doc",
"_id": "my_id_2",
"_version": 1,
"_seq_no": 40,
"_primary_term": 1,
"_source": {
"data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
"max_size": 5,
Expand All @@ -206,7 +212,7 @@ Returns this:
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]


[[ingest-attachment-with-arrays]]
Expand Down Expand Up @@ -285,6 +291,8 @@ Returns this:
"_type" : "_doc",
"_id" : "my_id",
"_version" : 1,
"_seq_no" : 50,
"_primary_term" : 1,
"found" : true,
"_source" : {
"attachments" : [
Expand Down Expand Up @@ -312,7 +320,7 @@ Returns this:
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]


Note that the `target_field` needs to be set, otherwise the
Expand Down
12 changes: 9 additions & 3 deletions docs/plugins/ingest-geoip.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@ Which returns:
"_type": "_doc",
"_id": "my_id",
"_version": 1,
"_seq_no": 55,
"_primary_term": 1,
"_source": {
"ip": "8.8.8.8",
"geoip": {
Expand All @@ -85,7 +87,7 @@ Which returns:
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term":1/"_primary_term" : $body._primary_term/]

Here is an example that uses the default country database and adds the
geographical information to the `geo` field based on the `ip` field`. Note that
Expand Down Expand Up @@ -124,6 +126,8 @@ returns this:
"_type": "_doc",
"_id": "my_id",
"_version": 1,
"_seq_no": 65,
"_primary_term": 1,
"_source": {
"ip": "8.8.8.8",
"geo": {
Expand All @@ -133,7 +137,7 @@ returns this:
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]


Not all IP addresses find geo information from the database, When this
Expand Down Expand Up @@ -174,13 +178,15 @@ Which returns:
"_type" : "_doc",
"_id" : "my_id",
"_version" : 1,
"_seq_no" : 71,
"_primary_term": 1,
"found" : true,
"_source" : {
"ip" : "80.231.5.0"
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]

[[ingest-geoip-mappings-note]]
===== Recognizing Location as a Geopoint
Expand Down
4 changes: 3 additions & 1 deletion docs/plugins/ingest-user-agent.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ Which returns
"_type": "_doc",
"_id": "my_id",
"_version": 1,
"_seq_no": 22,
"_primary_term": 1,
"_source": {
"agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
"user_agent": {
Expand All @@ -73,7 +75,7 @@ Which returns
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term": 1/"_primary_term" : $body._primary_term/]

===== Using a custom regex file
To use a custom regex file for parsing the user agents, that file has to be put into the `config/ingest-user-agent` directory and
Expand Down
12 changes: 9 additions & 3 deletions docs/reference/docs/get.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ The result of the above get operation is:
"_type" : "_doc",
"_id" : "0",
"_version" : 1,
"_seq_no" : 10,
"_primary_term" : 1,
"found": true,
"_source" : {
"user" : "kimchy",
Expand All @@ -30,7 +32,7 @@ The result of the above get operation is:
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]

The above result includes the `_index`, `_type`, `_id` and `_version`
of the document we wish to retrieve, including the actual `_source`
Expand Down Expand Up @@ -156,6 +158,8 @@ The result of the above get operation is:
"_type": "_doc",
"_id": "1",
"_version": 1,
"_seq_no" : 22,
"_primary_term" : 1,
"found": true,
"fields": {
"tags": [
Expand All @@ -164,7 +168,7 @@ The result of the above get operation is:
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]


Field values fetched from the document itself are always returned as an array.
Expand Down Expand Up @@ -199,6 +203,8 @@ The result of the above get operation is:
"_type": "_doc",
"_id": "2",
"_version": 1,
"_seq_no" : 13,
"_primary_term" : 1,
"_routing": "user1",
"found": true,
"fields": {
Expand All @@ -208,7 +214,7 @@ The result of the above get operation is:
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]

Also only leaf fields can be returned via the `stored_field` option. So object fields can't be returned and such requests
will fail.
Expand Down
4 changes: 3 additions & 1 deletion docs/reference/docs/reindex.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -870,13 +870,15 @@ which will return:
"_index": "test2",
"_type": "_doc",
"_version": 1,
"_seq_no": 44,
"_primary_term": 1,
"_source": {
"text": "words words",
"tag": "foo"
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term": 1/"_primary_term" : $body._primary_term/]

[float]
[[docs-reindex-slice]]
Expand Down
6 changes: 4 additions & 2 deletions docs/reference/getting-started.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -421,7 +421,7 @@ And the response:
"_primary_term" : 1
}
--------------------------------------------------
// TESTRESPONSE[s/"_seq_no" : 0/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]
// TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]

From the above, we can see that a new customer document was successfully created inside the customer index. The document also has an internal id of 1 which we specified at index time.

Expand All @@ -445,11 +445,13 @@ And the response:
"_type" : "_doc",
"_id" : "1",
"_version" : 1,
"_seq_no" : 25,
"_primary_term" : 1,
"found" : true,
"_source" : { "name": "John Doe" }
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]

Nothing out of the ordinary here other than a field, `found`, stating that we found a document with the requested ID 1 and another field, `_source`, which returns the full JSON document that we indexed from the previous step.

Expand Down
24 changes: 15 additions & 9 deletions docs/reference/ingest/ingest-node.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -730,13 +730,15 @@ GET test/_doc/2
"_type": "_doc",
"_id": "2",
"_version": 1,
"_seq_no": 22,
"_primary_term": 1,
"found": true,
"_source": {
"foo": "bar"
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term": 1/"_primary_term" : $body._primary_term/]
////

The source document can also use dot delimited fields to represent nested fields.
Expand Down Expand Up @@ -967,6 +969,8 @@ GET test/_doc/2
"_type": "_doc",
"_id": "2",
"_version": 1,
"_seq_no": 34,
"_primary_term": 1,
"found": true,
"_source": {
"tags": [
Expand All @@ -976,7 +980,7 @@ GET test/_doc/2
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]
////


Expand Down Expand Up @@ -1088,6 +1092,8 @@ GET test/_doc/1
"_type": "_doc",
"_id": "1",
"_version": 1,
"_seq_no": 60,
"_primary_term": 1,
"found": true,
"_source": {
"href": {
Expand All @@ -1097,7 +1103,7 @@ GET test/_doc/1
}
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]


Regular expressions can be expensive and should be avoided if viable
Expand Down Expand Up @@ -1548,11 +1554,11 @@ PUT /myindex/_doc/1?pipeline=monthlyindex
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_seq_no" : 55,
"_primary_term" : 1
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]


The above request will not index this document into the `myindex` index, but into the `myindex-2016-04-01` index because
Expand Down Expand Up @@ -2787,11 +2793,11 @@ Response from the index request:
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_seq_no": 66,
"_primary_term": 1,
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]

Indexed document:

Expand Down Expand Up @@ -2963,11 +2969,11 @@ The response from the above index request:
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_seq_no": 89,
"_primary_term": 1,
}
--------------------------------------------------
// TESTRESPONSE
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]

In the above response, you can see that our document was actually indexed into `my_index` instead of
`any_index`. This type of manipulation is often convenient in pipelines that have various branches of transformation,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
import java.util.Map;
import java.util.Set;

import static org.elasticsearch.index.seqno.SequenceNumbers.UNASSIGNED_SEQ_NO;
import static org.hamcrest.Matchers.equalTo;

public class PercolateQueryBuilderTests extends AbstractQueryTestCase<PercolateQueryBuilder> {
Expand Down Expand Up @@ -152,12 +153,13 @@ protected GetResponse executeGet(GetRequest getRequest) {
assertThat(getRequest.version(), Matchers.equalTo(indexedDocumentVersion));
if (indexedDocumentExists) {
return new GetResponse(
new GetResult(indexedDocumentIndex, indexedDocumentType, indexedDocumentId, 0L, true,
new GetResult(indexedDocumentIndex, indexedDocumentType, indexedDocumentId, 0, 1, 0L, true,
documentSource.iterator().next(), Collections.emptyMap())
);
} else {
return new GetResponse(
new GetResult(indexedDocumentIndex, indexedDocumentType, indexedDocumentId, -1, false, null, Collections.emptyMap())
new GetResult(indexedDocumentIndex, indexedDocumentType, indexedDocumentId, UNASSIGNED_SEQ_NO, 0, -1,
false, null, Collections.emptyMap())
);
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,7 @@ static BulkItemResponse processUpdateResponse(final UpdateRequest updateRequest,
final Tuple<XContentType, Map<String, Object>> sourceAndContent =
XContentHelper.convertToMap(indexSourceAsBytes, true, updateIndexRequest.getContentType());
updateResponse.setGetResult(UpdateHelper.extractGetResult(updateRequest, concreteIndex,
indexResponse.getSeqNo(), indexResponse.getPrimaryTerm(),
indexResponse.getVersion(), sourceAndContent.v2(), sourceAndContent.v1(), indexSourceAsBytes));
}
} else if (translatedResult == DocWriteResponse.Result.DELETED) {
Expand All @@ -315,7 +316,8 @@ static BulkItemResponse processUpdateResponse(final UpdateRequest updateRequest,
deleteResponse.getType(), deleteResponse.getId(), deleteResponse.getSeqNo(), deleteResponse.getPrimaryTerm(),
deleteResponse.getVersion(), deleteResponse.getResult());

final GetResult getResult = UpdateHelper.extractGetResult(updateRequest, concreteIndex, deleteResponse.getVersion(),
final GetResult getResult = UpdateHelper.extractGetResult(updateRequest, concreteIndex,
deleteResponse.getSeqNo(), deleteResponse.getPrimaryTerm(), deleteResponse.getVersion(),
translate.updatedSourceAsMap(), translate.updateSourceContentType(), null);

updateResponse.setGetResult(getResult);
Expand Down
14 changes: 14 additions & 0 deletions server/src/main/java/org/elasticsearch/action/get/GetResponse.java
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,20 @@ public long getVersion() {
return getResult.getVersion();
}

/**
* The sequence number assigned to the last operation to have changed this document, if found.
*/
public long getSeqNo() {
return getResult.getSeqNo();
}

/**
* The primary term of the last primary that has changed this document, if found.
*/
public long getPrimaryTerm() {
return getResult.getPrimaryTerm();
}

/**
* The source of the document if exists.
*/
Expand Down
Loading