[CI] CircuitBreakingException in PagedBytesStringFieldDataTests.testDeletedDocs

[cbuescher]

Failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/174/consoleFull

I wasn't able to reproduce this on master, even with running several hundred iterations. I also used Java 11, still no game. Here's the incantation anyway:

REPRODUCE WITH
./gradlew :server:test \
  -Dtests.seed=83098F6182FD3A5A \
  -Dtests.class=org.elasticsearch.index.fielddata.PagedBytesStringFieldDataTests \
  -Dtests.method="testDeletedDocs" \
  -Dtests.security.manager=true \
  -Dtests.locale=et-EE \
  -Dtests.timezone=America/Costa_Rica

(and currently needs to add the -x :server:testScriptDocValuesMissingV6Behaviour, see #31964)

16:04:28 ERROR   0.15s J3 | PagedBytesStringFieldDataTests.testDeletedDocs <<< FAILURES!
16:04:28    > Throwable #1: ElasticsearchException[java.util.concurrent.ExecutionException: CircuitBreakingException[[parent] Data too large, data for [value] would be [518120594/494.1mb], which is larger than the limit of [510027366/486.3mb]]]; nested: ExecutionException[CircuitBreakingException[[parent] Data too large, data for [value] would be [518120594/494.1mb], which is larger than the limit of [510027366/486.3mb]]]; nested: CircuitBreakingException[[parent] Data too large, data for [value] would be [518120594/494.1mb], which is larger than the limit of [510027366/486.3mb]];
16:04:28    > 	at __randomizedtesting.SeedInfo.seed([83098F6182FD3A5A:3982E8FFD520658D]:0)
16:04:28    > 	at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
16:04:28    > 	at org.elasticsearch.index.fielddata.AbstractFieldDataImplTestCase.testDeletedDocs(AbstractFieldDataImplTestCase.java:79)
16:04:28    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16:04:28    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16:04:28    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:04:28    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
16:04:28    > 	at java.base/java.lang.Thread.run(Thread.java:834)
16:04:28    > Caused by: java.util.concurrent.ExecutionException: CircuitBreakingException[[parent] Data too large, data for [value] would be [518120594/494.1mb], which is larger than the limit of [510027366/486.3mb]]
16:04:28   2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA/java11/ES_RUNTIME_JAVA/java11/nodes/virtual&&linux/server/build/testrun/test/J3/temp/org.elasticsearch.index.fielddata.PagedBytesStringFieldDataTests_83098F6182FD3A5A-001
16:04:28   2> NOTE: test params are: codec=Asserting(Lucene70): {field=PostingsFormat(name=Asserting), _id=PostingsFormat(name=Asserting), text=PostingsFormat(name=Asserting), type=PostingsFormat(name=Asserting), value=PostingsFormat(name=Asserting)}, docValues:{_id=DocValuesFormat(name=Lucene70), text=DocValuesFormat(name=Asserting), type=DocValuesFormat(name=Lucene70), value=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1325, maxMBSortInHeap=7.490371495226552, sim=RandomSimilarity(queryNorm=true): {}, locale=et-EE, timezone=America/Costa_Rica
16:04:28    > 	at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:436)
16:04:28   2> NOTE: Linux 4.4.104-39-default amd64/Oracle Corporation 11-ea (64-bit)/cpus=16,threads=1,free=390612560,total=536870912
16:04:28   2> NOTE: All tests run in this JVM: [TDigestPercentileRanksAggregatorTests, ClusterStateHealthTests, InternalDerivativeTests, RefreshListenersTests, UpdateThreadPoolSettingsTests, DocumentFieldMapperTests, RestClearScrollActionTests, MultiPointBuilderTests, KeyStoreWrapperTests, FieldCapabilitiesTests, ClusterStatsNodesTests, IndexServiceTests, StreamTests, TransportBulkActionTests, PendingClusterStatesQueueTests, StreamsTests, QueueResizingEsThreadPoolExecutorTests, ClusterUpdateSettingsRequestTests, PluginsTests, ResourceWatcherServiceTests, JsonXContentTests, SearchModuleTests, GeoCentroidTests, ShardVersioningTests, ScriptServiceTests, PagedBytesStringFieldDataTests]
16:04:28    > 	at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:136)
16:04:28    > 	at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:68)
16:04:28    > 	... 37 more
16:04:28    > Caused by: CircuitBreakingException[[parent] Data too large, data for [value] would be [518120594/494.1mb], which is larger than the limit of [510027366/486.3mb]]
16:04:28    > 	at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:257)
16:04:28    > 	at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128)
16:04:28    > 	at org.elasticsearch.index.fielddata.RamAccountingTermsEnum.flush(RamAccountingTermsEnum.java:71)
16:04:28    > 	at org.elasticsearch.index.fielddata.RamAccountingTermsEnum.next(RamAccountingTermsEnum.java:85)
16:04:28    > 	at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:113)
16:04:28    > 	at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:52)
16:04:28    > 	at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.lambda$load$0(IndicesFieldDataCache.java:139)
16:04:28    > 	at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)
16:04:28    > 	... 39 more

[elasticmachine]

Pinging @elastic/es-search-aggs

[elasticmachine]

Pinging @elastic/es-core-infra

[dakrone]

I opened #32056 which might also help with the error message on this