Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding RefCounted method implementations to BulkShardRequest #103161

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
b3bf791
Adding RefCounted method implementations to BulkRequest
masseyke Nov 29, 2023
3fe2e53
Update docs/changelog/102778.yaml
masseyke Nov 29, 2023
e917100
merging main
masseyke Nov 29, 2023
a164a46
Fixing tests (except ml)
masseyke Nov 29, 2023
9b698e1
Merge branch 'making-BulkRequest-implement-RefCounted' of github.com:…
masseyke Nov 29, 2023
99f2cd2
fixing ml tests
masseyke Nov 30, 2023
3680075
fixing tests
masseyke Nov 30, 2023
6292b13
fixing BasicEnrichTests
masseyke Nov 30, 2023
d45722f
fixing BasicEnrichTests
masseyke Nov 30, 2023
c671543
fixing tests
masseyke Nov 30, 2023
d3177f7
fixing more tests
masseyke Nov 30, 2023
3bfe9a5
fixing more tests
masseyke Nov 30, 2023
ab77f83
fixing EnrichMultiNodeIT
masseyke Nov 30, 2023
fa59215
fixing tests
masseyke Nov 30, 2023
fb80410
cleaning up ml code
masseyke Nov 30, 2023
08eebc5
merging main
masseyke Nov 30, 2023
2a71c53
Fixing tests
masseyke Dec 1, 2023
2b42354
Merge branch 'main' into making-BulkRequest-implement-RefCounted
masseyke Dec 1, 2023
249220b
fixing tests
masseyke Dec 1, 2023
c454aff
fixing test
masseyke Dec 1, 2023
5244077
fixing tests
masseyke Dec 1, 2023
c96d2b2
reverting accidental commit
masseyke Dec 1, 2023
208fe64
fixing ml tests
masseyke Dec 1, 2023
74be24a
fixing tests
masseyke Dec 1, 2023
7f66d12
fixing tests
masseyke Dec 1, 2023
3f76581
reverting accidental commit
masseyke Dec 1, 2023
1f425e5
fixing tests
masseyke Dec 1, 2023
5619b39
fixing tests
masseyke Dec 1, 2023
2847f7e
fixing tests
masseyke Dec 4, 2023
d1c6e19
fixing TransportOpenIdConnectLogoutActionTests
masseyke Dec 4, 2023
4f43e4b
fixing TransportSamlLogoutActionTests
masseyke Dec 4, 2023
5bcbb92
fixing tests
masseyke Dec 4, 2023
d4183ad
fixing tests
masseyke Dec 4, 2023
67f6647
fixing compilation error
masseyke Dec 4, 2023
ce71b8f
fixing watcher
masseyke Dec 4, 2023
77e55da
fixing ML tests
masseyke Dec 5, 2023
0d7495b
merging main
masseyke Dec 5, 2023
f143de6
fixing ml tests
masseyke Dec 6, 2023
890259a
formatting
masseyke Dec 6, 2023
c3a783f
formatting
masseyke Dec 6, 2023
c1ce720
fixing some ml tests
masseyke Dec 6, 2023
3de223b
fixing TestFeatureResetIT
masseyke Dec 6, 2023
9ff9b33
fixing RollupIndexerStateTests
masseyke Dec 6, 2023
47d2dcf
fixing AutodetectProcessManager
masseyke Dec 7, 2023
0c6fa78
merging main
masseyke Dec 7, 2023
e1351c5
Merge branch 'main' into making-BulkRequest-implement-RefCounted
masseyke Dec 7, 2023
2a2829d
fixing TransportDeleteFilterAction
masseyke Dec 7, 2023
5a9f08f
Merge branch 'main' into making-BulkRequest-implement-RefCounted
masseyke Dec 7, 2023
1fb746b
Adding leak detection to BulkShardRequest
masseyke Dec 7, 2023
c703fd2
Update docs/changelog/103161.yaml
masseyke Dec 7, 2023
1e0bc1b
fixing ResultsPersisterService
masseyke Dec 7, 2023
88e00fe
merging main
masseyke Dec 7, 2023
bd7cbe3
merging
masseyke Dec 7, 2023
c040391
Merge branch 'making-BulkShardRequest-implement-RefCounted' of github…
masseyke Dec 7, 2023
a8617d2
fixing tests
masseyke Dec 7, 2023
02239c6
Merge branch 'main' into making-BulkRequest-implement-RefCounted
masseyke Dec 8, 2023
404b01f
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Dec 8, 2023
ab6dc42
fixing test
masseyke Dec 8, 2023
1ed8296
fixing DownsampleDataStreamTests
masseyke Dec 8, 2023
2e5c7c8
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Dec 8, 2023
e46c83f
fixing TransportReplicationAction
masseyke Dec 8, 2023
399ca37
merging main
masseyke Dec 12, 2023
cd179e2
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Dec 12, 2023
a36fec8
merging main
masseyke Dec 14, 2023
3ec6c99
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Dec 14, 2023
aeba60d
merging main
masseyke Jan 3, 2024
a190c7d
merging
masseyke Jan 3, 2024
ece05a9
fixing AbstractPausableIntegTestCase
masseyke Jan 3, 2024
7c9b19d
fixing CrossClustersCancellationIT
masseyke Jan 3, 2024
05e2bac
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 3, 2024
511c154
fixing TransportShardBulkActionTests
masseyke Jan 3, 2024
4a02df7
fixing TransportShardBulkActionTests
masseyke Jan 3, 2024
09eca96
Merge branch 'main' into making-BulkRequest-implement-RefCounted
masseyke Jan 3, 2024
990684c
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 3, 2024
e26004a
Merge branch 'main' into making-BulkRequest-implement-RefCounted
masseyke Jan 4, 2024
f584f2b
fixing TimeBasedIndicesIT
masseyke Jan 4, 2024
f3ad2e7
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 4, 2024
0082909
Merge branch 'main' into making-BulkRequest-implement-RefCounted
masseyke Jan 4, 2024
6155fa8
fixing Retry
masseyke Jan 4, 2024
a81e1fc
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 4, 2024
8c301f7
merging main
masseyke Jan 8, 2024
0ed242a
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 8, 2024
1e19506
fixing ResultsPersisterServiceTests
masseyke Jan 8, 2024
2745f0b
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 8, 2024
d69940e
merging main
masseyke Jan 8, 2024
2642ab7
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 8, 2024
162b566
Merge branch 'main' into making-BulkRequest-implement-RefCounted
masseyke Jan 10, 2024
8778731
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 10, 2024
dc7d52e
Merge branch 'main' into making-BulkRequest-implement-RefCounted
masseyke Jan 11, 2024
724c374
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 11, 2024
62f9063
merging main
masseyke Jan 12, 2024
626d708
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 12, 2024
ea82baf
merging main
masseyke Jan 16, 2024
cbe3582
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 16, 2024
63fb268
merging main
masseyke Jan 16, 2024
23a7a42
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 16, 2024
9d94a2e
Merge branch 'main' into making-BulkRequest-implement-RefCounted
masseyke Jan 18, 2024
25a9c4f
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 18, 2024
9be85c6
merging main
masseyke Jan 18, 2024
c80ed2f
fixing a test
masseyke Jan 18, 2024
bc8e0e7
merging
masseyke Jan 18, 2024
225eaa0
merging main
masseyke Jan 30, 2024
f8c6dcb
Merge branch 'making-BulkRequest-implement-RefCounted' into making-Bu…
masseyke Jan 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/changelog/102778.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pr: 102778
summary: Adding `RefCounted` method implementations to `BulkRequest`
area: Ingest Node
type: enhancement
issues: []
5 changes: 5 additions & 0 deletions docs/changelog/103161.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pr: 103161
summary: Adding `RefCounted` method implementations to `BulkShardRequest`
area: Ingest Node
type: enhancement
issues: []
Original file line number Diff line number Diff line change
Expand Up @@ -77,21 +77,22 @@ public void testCancellationDuringTimeSeriesAggregation() throws Exception {

for (int i = 0; i < numberOfRefreshes; i++) {
// Make sure we sometimes have a few segments
BulkRequestBuilder bulkRequestBuilder = client().prepareBulk().setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
for (int j = 0; j < numberOfDocsPerRefresh; j++) {
bulkRequestBuilder.add(
prepareIndex("test").setOpType(DocWriteRequest.OpType.CREATE)
.setSource(
"@timestamp",
now + (long) i * numberOfDocsPerRefresh + j,
"val",
(double) j,
"dim",
String.valueOf(j % 100)
)
);
try (BulkRequestBuilder bulkRequestBuilder = client().prepareBulk().setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE)) {
for (int j = 0; j < numberOfDocsPerRefresh; j++) {
bulkRequestBuilder.add(
prepareIndex("test").setOpType(DocWriteRequest.OpType.CREATE)
.setSource(
"@timestamp",
now + (long) i * numberOfDocsPerRefresh + j,
"val",
(double) j,
"dim",
String.valueOf(j % 100)
)
);
}
assertNoFailures(bulkRequestBuilder.get());
}
assertNoFailures(bulkRequestBuilder.get());
}

logger.info("Executing search");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import org.elasticsearch.action.DocWriteRequest;
import org.elasticsearch.action.admin.indices.alias.Alias;
import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
import org.elasticsearch.action.bulk.BulkRequestBuilder;
import org.elasticsearch.action.index.IndexRequestBuilder;
import org.elasticsearch.action.support.WriteRequest;
import org.elasticsearch.aggregations.AggregationIntegTestCase;
Expand Down Expand Up @@ -502,23 +503,26 @@ public void testGetHitsFailure() throws Exception {
).setMapping("key", "type=keyword,time_series_dimension=true", "val", "type=double")
);

client().prepareBulk()
.add(prepareIndex("test").setId("2").setSource("key", "bar", "val", 2, "@timestamp", "2021-01-01T00:00:10Z"))
.add(prepareIndex("test").setId("1").setSource("key", "bar", "val", 10, "@timestamp", "2021-01-01T00:00:00Z"))
.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE)
.get();
client().prepareBulk()
.add(prepareIndex("test").setId("4").setSource("key", "bar", "val", 50, "@timestamp", "2021-01-01T00:00:30Z"))
.add(prepareIndex("test").setId("3").setSource("key", "bar", "val", 40, "@timestamp", "2021-01-01T00:00:20Z"))
.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE)
.get();
client().prepareBulk()
.add(prepareIndex("test").setId("7").setSource("key", "foo", "val", 20, "@timestamp", "2021-01-01T00:00:00Z"))
.add(prepareIndex("test").setId("8").setSource("key", "foo", "val", 30, "@timestamp", "2021-01-01T00:10:00Z"))
.add(prepareIndex("test").setId("5").setSource("key", "baz", "val", 20, "@timestamp", "2021-01-01T00:00:00Z"))
.add(prepareIndex("test").setId("6").setSource("key", "baz", "val", 30, "@timestamp", "2021-01-01T00:10:00Z"))
.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE)
.get();
try (BulkRequestBuilder bulkRequestBuilder = client().prepareBulk()) {
bulkRequestBuilder.add(prepareIndex("test").setId("2").setSource("key", "bar", "val", 2, "@timestamp", "2021-01-01T00:00:10Z"))
.add(prepareIndex("test").setId("1").setSource("key", "bar", "val", 10, "@timestamp", "2021-01-01T00:00:00Z"))
.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE)
.get();
}
try (BulkRequestBuilder bulkRequestBuilder = client().prepareBulk()) {
bulkRequestBuilder.add(prepareIndex("test").setId("4").setSource("key", "bar", "val", 50, "@timestamp", "2021-01-01T00:00:30Z"))
.add(prepareIndex("test").setId("3").setSource("key", "bar", "val", 40, "@timestamp", "2021-01-01T00:00:20Z"))
.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE)
.get();
}
try (BulkRequestBuilder bulkRequestBuilder = client().prepareBulk()) {
bulkRequestBuilder.add(prepareIndex("test").setId("7").setSource("key", "foo", "val", 20, "@timestamp", "2021-01-01T00:00:00Z"))
.add(prepareIndex("test").setId("8").setSource("key", "foo", "val", 30, "@timestamp", "2021-01-01T00:10:00Z"))
.add(prepareIndex("test").setId("5").setSource("key", "baz", "val", 20, "@timestamp", "2021-01-01T00:00:00Z"))
.add(prepareIndex("test").setId("6").setSource("key", "baz", "val", 30, "@timestamp", "2021-01-01T00:10:00Z"))
.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE)
.get();
}

QueryBuilder queryBuilder = QueryBuilders.rangeQuery("@timestamp").lte("2021-01-01T00:10:00Z");

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,15 +64,16 @@ public void setup() throws Exception {
assertTrue(prepareTimeSeriesIndex(mapping, startMillis, endMillis, routingDimensions).isAcknowledged());
logger.info("Dimensions: " + numberOfDimensions + " docs: " + numberOfDocuments + " start: " + startMillis + " end: " + endMillis);

final BulkRequestBuilder bulkIndexRequest = client().prepareBulk();
for (int docId = 0; docId < numberOfDocuments; docId++) {
final XContentBuilder document = timeSeriesDocument(FOO_DIM_VALUE, BAR_DIM_VALUE, BAZ_DIM_VALUE, docId, timestamps::next);
bulkIndexRequest.add(prepareIndex("index").setOpType(DocWriteRequest.OpType.CREATE).setSource(document));
try (BulkRequestBuilder bulkIndexRequest = client().prepareBulk()) {
for (int docId = 0; docId < numberOfDocuments; docId++) {
final XContentBuilder document = timeSeriesDocument(FOO_DIM_VALUE, BAR_DIM_VALUE, BAZ_DIM_VALUE, docId, timestamps::next);
bulkIndexRequest.add(prepareIndex("index").setOpType(DocWriteRequest.OpType.CREATE).setSource(document));
}

final BulkResponse bulkIndexResponse = bulkIndexRequest.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE).get();
assertFalse(bulkIndexResponse.hasFailures());
assertEquals(RestStatus.OK.getStatus(), client().admin().indices().prepareFlush("index").get().getStatus().getStatus());
}

final BulkResponse bulkIndexResponse = bulkIndexRequest.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE).get();
assertFalse(bulkIndexResponse.hasFailures());
assertEquals(RestStatus.OK.getStatus(), client().admin().indices().prepareFlush("index").get().getStatus().getStatus());
}

private static XContentBuilder timeSeriesDocument(
Expand Down
Loading