[Metricbeat] Move statsd metricbeat module to GA (#16447) (#16449)

* Update docs
* Add system test
* Update changelog

(cherry picked from commit c6701dd)

CHANGELOG.next.asciidoc

@@ -243,6 +243,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add support for NATS 2.1. {pull}16317[16317]
 - Improve the `haproxy` module to support metrics exposed via HTTPS. {issue}14579[14579] {pull}16333[16333]
 - Add metricsets based on Ceph Manager Daemon to the `ceph` module. {issue}7723[7723] {pull}16254[16254]
+- Release `statsd` module as GA. {pull}16447[16447] {issue}14280[14280]
 
 *Packetbeat*
 

metricbeat/docs/modules/statsd.asciidoc

@@ -6,10 +6,41 @@ This file is generated! See scripts/mage/docs_collector.go
 [role="xpack"]
 == Statsd module
 
-beta[]
+The `statsd` module is a Metricbeat module which spawns a UDP server and listens for metrics in StatsD compatible
+format.
 
-The Statsd module is a Metricbeat module which opens a UDP port and listens for statsd metrics.
+[float]
+=== Metric types
+
+The module supports the following types of metrics:
+
+*Counter (c)*:: Measurement which accumulates over period of time until flushed (value set to 0).
+
+*Gauge (g)*:: Measurement which can increase, decrease or be set to a value.
+
+*Timer (ms)*:: Time measurement (in milliseconds) of an event.
+
+*Histogram (h)*:: Time measurement, alias for timer.
+
+*Set (s)*:: Measurement which counts unique occurrences until flushed (value set to 0).
+
+[float]
+=== Module-specific configuration notes
+
+The `statsd` module has these additional config options:
+
+*`ttl`*:: It defines how long a metric will be reported after it was last recorded.
+Irrespective of the given ttl, metrics will be reported at least once.
+A ttl of zero means metrics will never expire.
+
+[float]
+=== Metricsets
 
+Currently, there is only `server` metricset in `statsd` module.
+
+[float]
+==== `server`
+The metricset collects metric data sent using UDP and publishes them under the `statsd` prefix.
 
 
 [float]

metricbeat/docs/modules/statsd/server.asciidoc

@@ -5,8 +5,6 @@ This file is generated! See scripts/mage/docs_collector.go
 [[metricbeat-metricset-statsd-server]]
 === Statsd server metricset
 
-beta[]
-
 include::../../../../x-pack/metricbeat/module/statsd/server/_meta/docs.asciidoc[]
 
 This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.

metricbeat/docs/modules_list.asciidoc

@@ -204,8 +204,8 @@ This file is generated! See scripts/mage/docs_collector.go
 .3+| .3+|  |<<metricbeat-metricset-stan-channels,channels>> beta[]  
 |<<metricbeat-metricset-stan-stats,stats>> beta[]  
 |<<metricbeat-metricset-stan-subscriptions,subscriptions>> beta[]  
-|<<metricbeat-module-statsd,Statsd>>  beta[]   |image:./images/icon-no.png[No prebuilt dashboards]    |  
-.1+| .1+|  |<<metricbeat-metricset-statsd-server,server>> beta[]  
+|<<metricbeat-module-statsd,Statsd>>     |image:./images/icon-no.png[No prebuilt dashboards]    |  
+.1+| .1+|  |<<metricbeat-metricset-statsd-server,server>>   
 |<<metricbeat-module-system,System>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
 .17+| .17+|  |<<metricbeat-metricset-system-core,core>>   
 |<<metricbeat-metricset-system-cpu,cpu>>   

x-pack/metricbeat/module/statsd/README.md

@@ -1,37 +1,32 @@
-For manual testing and development of this module, start metricbeat with the standard configuration for the module:
+### Development
 
-```
-------------------------------------------------------------------------------
+Run metricbeat locally with configured `statsd` module:
+
+```yaml
 - module: statsd
   metricsets: ["server"]
   host: "localhost"
   port: "8125"
   enabled: true
   #ttl: "30s"
-------------------------------------------------------------------------------
-```
-
-Look for a log line to this effect:
-```
-Started listening for UDP on: 127.0.0.1:8125
 ```
+
+Use favorite statsd client to emit metrics, e.g.:
 
-then use a statsd client to test the features. In an empty directory do the following:
-
-```
+```bash
 $ npm install statsd-client
 $ node
-> var SDC = require('statsd-client'),
-    sdc = new SDC({host: 'localhost', port: 8125});
+```
 
-> sdc.increment('systemname.subsystem.value'); // Increment by one
-> sdc.gauge('what.you.gauge', 100);
-> sdc.gaugeDelta('what.you.gauge', -70); // Will now count 50
-> sdc.gauge('gauge.with.tags', 100, {foo: 'bar'});
-> sdc.set('set.with.tags', 100, {foo: 'bar'});
-> sdc.set('set.with.tags', 200, {foo: 'bar'});
-> sdc.set('set.with.tags', 100, {foo: 'baz'});
-....
+Emit some metrics:
 
-<CTRL+D>
+```javascript
+let SDC = require('statsd-client'), sdc = new SDC({host: 'localhost', port: 8125});
+sdc.increment('systemname.subsystem.value');
+sdc.gauge('what.you.gauge', 100);
+sdc.gaugeDelta('what.you.gauge', -70);
+sdc.gauge('gauge.with.tags', 100, {foo: 'bar'});
+sdc.set('set.with.tags', 100, {foo: 'bar'});
+sdc.set('set.with.tags', 200, {foo: 'bar'});
+sdc.set('set.with.tags', 100, {foo: 'baz'});
 ```

x-pack/metricbeat/module/statsd/_meta/docs.asciidoc

@@ -1,2 +1,35 @@
-The Statsd module is a Metricbeat module which opens a UDP port and listens for statsd metrics.
+The `statsd` module is a Metricbeat module which spawns a UDP server and listens for metrics in StatsD compatible
+format.
 
+[float]
+=== Metric types
+
+The module supports the following types of metrics:
+
+*Counter (c)*:: Measurement which accumulates over period of time until flushed (value set to 0).
+
+*Gauge (g)*:: Measurement which can increase, decrease or be set to a value.
+
+*Timer (ms)*:: Time measurement (in milliseconds) of an event.
+
+*Histogram (h)*:: Time measurement, alias for timer.
+
+*Set (s)*:: Measurement which counts unique occurrences until flushed (value set to 0).
+
+[float]
+=== Module-specific configuration notes
+
+The `statsd` module has these additional config options:
+
+*`ttl`*:: It defines how long a metric will be reported after it was last recorded.
+Irrespective of the given ttl, metrics will be reported at least once.
+A ttl of zero means metrics will never expire.
+
+[float]
+=== Metricsets
+
+Currently, there is only `server` metricset in `statsd` module.
+
+[float]
+==== `server`
+The metricset collects metric data sent using UDP and publishes them under the `statsd` prefix.

x-pack/metricbeat/module/statsd/_meta/fields.yml

@@ -2,7 +2,7 @@
   title: "Statsd"
   description: >
     Statsd module
-  release: beta
+  release: ga
   fields:
     - name: statsd
       type: group

x-pack/metricbeat/module/statsd/server/_meta/docs.asciidoc

@@ -1,17 +1,3 @@
-This is the server metricset of the statsd module.
-
-Events sent to the stats endpoint will be put by default under the `statsd` prefix.
-
-_ttl_ defines how long a metric will be reported after it was last recorded.
-Irrespective of the given ttl, metrics will be reported at least once.
-A ttl of zero means metrics will never expire.
-
-["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------
-- module: statsd
-  metricsets: ["server"]
-  host: "localhost"
-  port: "8125"
-  #ttl: "30s"
-------------------------------------------------------------------------------
+[role="xpack"]
 
+This is the `server` metricset of the `statsd` module.

x-pack/metricbeat/module/statsd/server/_meta/fields.yml

@@ -1 +1 @@
-- release: beta
+- release: ga

x-pack/metricbeat/module/statsd/test_statsd.py

@@ -0,0 +1,46 @@
+import os
+import socket
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '../../tests/system'))
+from xpack_metricbeat import XPackTest, metricbeat
+
+STATSD_HOST = '127.0.0.1'
+STATSD_PORT = 8125
+
+METRIC_MESSAGE = bytes('metric1:777.0|g|#k1:v1,k2:v2', 'utf-8')
+
+
+class Test(XPackTest):
+
+    def test_server(self):
+        """
+        statsd server metricset test
+        """
+
+        # Start the application
+        self.render_config_template(modules=[{
+            "name": "statsd",
+            "metricsets": ["server"],
+            "period": "5s",
+            "host": STATSD_HOST,
+            "port": STATSD_PORT,
+        }])
+        proc = self.start_beat()
+        self.wait_until(lambda: self.log_contains("Started listening for UDP"))
+
+        # Send UDP packet with metric
+        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        sock.sendto(METRIC_MESSAGE, (STATSD_HOST, STATSD_PORT))
+        sock.close()
+
+        self.wait_until(lambda: self.output_lines() > 0)
+        proc.check_kill_and_wait()
+        self.assert_no_logged_warnings(replace='use of closed network connection')
+
+        # Verify output
+        output = self.read_output_json()
+        self.assertGreater(len(output), 0)
+        evt = output[0]
+        assert evt["statsd"]["metric1"]["value"] == 777
+        self.assert_fields_are_documented(evt)