Add Boxplot Aggregation

docs/reference/aggregations/metrics/boxplot-aggregation.asciidoc

@@ -0,0 +1,204 @@
+[role="xpack"]
+[testenv="basic"]
+[[search-aggregations-metrics-boxplot-aggregation]]
+=== Boxplot Aggregation
+
+A `boxplot` metrics aggregation that computes boxplot of numeric values extracted from the aggregated documents.
+These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.
+
+The `boxplot` aggregation returns essential information for making a https://en.wikipedia.org/wiki/Box_plot[box plot]: minimum, maximum
+median, first quartile (25th percentile)  and third quartile (75th percentile) values.
+
+==== Syntax
+
+A `boxplot` aggregation looks like this in isolation:
+
+[source,js]
+--------------------------------------------------
+{
+    "boxplot": {
+        "buckets_path": "my_cardinality_agg"
+    }
+}
+--------------------------------------------------
+// NOTCONSOLE
+
+Let's look at a boxplot representing load time:
+
+[source,console]
+--------------------------------------------------
+GET latency/_search
+{
+    "size": 0,
+    "aggs" : {
+        "load_time_boxplot" : {
+            "boxplot" : {
+                "field" : "load_time" <1>
+            }
+        }
+    }
+}
+--------------------------------------------------
+// TEST[setup:latency]
+<1> The field `load_time` must be a numeric field
+
+The response will look like this:
+
+[source,console-result]
+--------------------------------------------------
+{
+    ...
+
+   "aggregations": {
+      "load_time_boxplot": {
+         "min": 0.0,
+         "max": 990.0,
+         "q1": 165.0,
+         "q2": 445.0,
+         "q3": 725.0
+      }
+   }
+}
+--------------------------------------------------
+// TESTRESPONSE[s/\.\.\./"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/]
+
+As you can see, the aggregation will return a calculated value for each percentile
+in the default range.  If we assume response times are in milliseconds, it is
+immediately obvious that the webpage normally loads in 10-725ms, but occasionally
+spikes to 945-985ms.
+
+==== Script
+
+The boxplot metric supports scripting.  For example, if our load times
+are in milliseconds but we want percentiles calculated in seconds, we could use
+a script to convert them on-the-fly:
+
+[source,console]
+--------------------------------------------------
+GET latency/_search
+{
+    "size": 0,
+    "aggs" : {
+        "load_time_boxplot" : {
+            "boxplot" : {
+                "script" : {
+                    "lang": "painless",
+                    "source": "doc['load_time'].value / params.timeUnit", <1>
+                    "params" : {
+                        "timeUnit" : 1000   <2>
+                    }
+                }
+            }
+        }
+    }
+}
+--------------------------------------------------
+// TEST[setup:latency]
+
+<1> The `field` parameter is replaced with a `script` parameter, which uses the
+script to generate values which percentiles are calculated on
+<2> Scripting supports parameterized input just like any other script
+
+This will interpret the `script` parameter as an `inline` script with the `painless` script language and no script parameters. To use a
+stored script use the following syntax:
+
+[source,console]
+--------------------------------------------------
+GET latency/_search
+{
+    "size": 0,
+    "aggs" : {
+        "load_time_boxplot" : {
+            "boxplot" : {
+                "script" : {
+                    "id": "my_script",
+                    "params": {
+                        "field": "load_time"
+                    }
+                }
+            }
+        }
+    }
+}
+--------------------------------------------------
+// TEST[setup:latency,stored_example_script]
+
+[[search-aggregations-metrics-boxplot-aggregation-approximation]]
+==== Boxplot values are (usually) approximate
+
+The algorithm used by the `boxplot` metric is called TDigest (introduced by
+Ted Dunning in
+https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf[Computing Accurate Quantiles using T-Digests]).
+
+[WARNING]
+====
+Boxplot as other percentile aggregations are also
+https://en.wikipedia.org/wiki/Nondeterministic_algorithm[non-deterministic].
+This means you can get slightly different results using the same data.
+====
+
+[[search-aggregations-metrics-boxplot-aggregation-compression]]
+==== Compression
+
+Approximate algorithms must balance memory utilization with estimation accuracy.
+This balance can be controlled using a `compression` parameter:
+
+[source,console]
+--------------------------------------------------
+GET latency/_search
+{
+    "size": 0,
+    "aggs" : {
+        "load_time_boxplot" : {
+            "boxplot" : {
+                "field" : "load_time",
+                "compression" : 200 <1>
+            }
+        }
+    }
+}
+--------------------------------------------------
+// TEST[setup:latency]
+
+<1> Compression controls memory usage and approximation error
+
+The TDigest algorithm uses a number of "nodes" to approximate percentiles -- the
+more nodes available, the higher the accuracy (and large memory footprint) proportional
+to the volume of data.  The `compression` parameter limits the maximum number of
+nodes to `20 * compression`.
+
+Therefore, by increasing the compression value, you can increase the accuracy of
+your percentiles at the cost of more memory.  Larger compression values also
+make the algorithm slower since the underlying tree data structure grows in size,
+resulting in more expensive operations.  The default compression value is
+`100`.
+
+A "node" uses roughly 32 bytes of memory, so under worst-case scenarios (large amount
+of data which arrives sorted and in-order) the default settings will produce a
+TDigest roughly 64KB in size.  In practice data tends to be more random and
+the TDigest will use less memory.
+
+==== Missing value
+
+The `missing` parameter defines how documents that are missing a value should be treated.
+By default they will be ignored but it is also possible to treat them as if they
+had a value.
+
+[source,console]
+--------------------------------------------------
+GET latency/_search
+{
+    "size": 0,
+    "aggs" : {
+        "grade_boxplot" : {
+            "boxplot" : {
+                "field" : "grade",
+                "missing": 10 <1>
+            }
+        }
+    }
+}
+--------------------------------------------------
+// TEST[setup:latency]
+
+<1> Documents without a value in the `grade` field will fall into the same bucket as documents that have the value `10`.

server/src/test/java/org/elasticsearch/search/aggregations/metrics/MinAggregatorTests.java

@@ -378,7 +378,7 @@ public void testGetProperty() throws IOException {
             iw.addDocument(singleton(new NumericDocValuesField("number", 7)));
             iw.addDocument(singleton(new NumericDocValuesField("number", 1)));
         }, (Consumer<InternalGlobal>) global -> {
-            assertEquals(1.0, global.getDocCount(), 2);
+            assertEquals(2, global.getDocCount());
             assertTrue(AggregationInspectionHelper.hasValue(global));
             assertNotNull(global.getAggregations().asMap().get("min"));
 

x-pack/plugin/analytics/src/main/java/org/elasticsearch/xpack/analytics/AnalyticsPlugin.java

@@ -7,12 +7,15 @@
 
 import org.elasticsearch.action.ActionRequest;
 import org.elasticsearch.action.ActionResponse;
+import org.elasticsearch.common.xcontent.ContextParser;
 import org.elasticsearch.index.mapper.Mapper;
 import org.elasticsearch.license.XPackLicenseState;
 import org.elasticsearch.plugins.ActionPlugin;
 import org.elasticsearch.plugins.MapperPlugin;
 import org.elasticsearch.plugins.Plugin;
 import org.elasticsearch.plugins.SearchPlugin;
+import org.elasticsearch.search.aggregations.AggregationBuilder;
+import org.elasticsearch.xpack.analytics.boxplot.InternalBoxplot;
 import org.elasticsearch.xpack.analytics.mapper.HistogramFieldMapper;
 import org.elasticsearch.xpack.core.XPackPlugin;
 import org.elasticsearch.xpack.core.action.XPackInfoFeatureAction;
@@ -21,6 +24,7 @@
 import org.elasticsearch.xpack.analytics.action.AnalyticsInfoTransportAction;
 import org.elasticsearch.xpack.analytics.action.AnalyticsUsageTransportAction;
 import org.elasticsearch.xpack.analytics.action.TransportAnalyticsStatsAction;
+import org.elasticsearch.xpack.analytics.boxplot.BoxplotAggregationBuilder;
 import org.elasticsearch.xpack.analytics.cumulativecardinality.CumulativeCardinalityPipelineAggregationBuilder;
 import org.elasticsearch.xpack.analytics.cumulativecardinality.CumulativeCardinalityPipelineAggregator;
 import org.elasticsearch.xpack.analytics.stringstats.InternalStringStats;
@@ -56,11 +60,16 @@ public List<PipelineAggregationSpec> getPipelineAggregations() {
 
     @Override
     public List<AggregationSpec> getAggregations() {
-        return singletonList(
+        return Arrays.asList(
             new AggregationSpec(
                 StringStatsAggregationBuilder.NAME,
                 StringStatsAggregationBuilder::new,
-                StringStatsAggregationBuilder::parse).addResultReader(InternalStringStats::new)
+                StringStatsAggregationBuilder::parse).addResultReader(InternalStringStats::new),
+            new AggregationSpec(
+                BoxplotAggregationBuilder.NAME,
+                BoxplotAggregationBuilder::new,
+                (ContextParser<String, AggregationBuilder>) (p, c) -> BoxplotAggregationBuilder.parse(c, p))
+                .addResultReader(InternalBoxplot::new)
         );
     }
 

x-pack/plugin/analytics/src/main/java/org/elasticsearch/xpack/analytics/boxplot/Boxplot.java

@@ -0,0 +1,63 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+package org.elasticsearch.xpack.analytics.boxplot;
+
+import org.elasticsearch.search.aggregations.metrics.NumericMetricsAggregation;
+
+public interface Boxplot extends NumericMetricsAggregation.MultiValue {
+
+    /**
+     * @return The minimum value of all aggregated values.
+     */
+    double getMin();
+
+    /**
+     * @return The maximum value of all aggregated values.
+     */
+    double getMax();
+
+    /**
+     * @return The first quartile of all aggregated values.
+     */
+    double getQ1();
+
+    /**
+     * @return The second quartile of all aggregated values.
+     */
+    double getQ2();
+
+    /**
+     * @return The third quartile of all aggregated values.
+     */
+    double getQ3();
+
+    /**
+     * @return The minimum value of all aggregated values as a String.
+     */
+    String getMinAsString();
+
+    /**
+     * @return The maximum value of all aggregated values as a String.
+     */
+    String getMaxAsString();
+
+    /**
+     * @return The first quartile of all aggregated values as a String.
+     */
+    String getQ1AsString();
+
+    /**
+     * @return The second quartile of all aggregated values as a String.
+     */
+    String getQ2AsString();
+
+    /**
+     * @return The third quartile of all aggregated values as a String.
+     */
+    String getQ3AsString();
+
+}