Allow auto-configuration of Metricbeat stack modules for stack monito…

…ring (elastic#17609)

* Add ability to reconfigure a module

* Reconfigure Logstash module with required metricsets for xpack.enabled

* Replace assert with require

* Adding CHANGELOG entry

* Update default configuration files

* Auto-configure beat module metricsets when xpack.enabled = true

* Refactoring common code into helper function

* Adding tests for ReConfigure() / making it part of Module interface

* Moving comments

* Fixing infinite recursion 🤦

* Implement for kibana module

* Implementing for elasticsearch module

* Moving ReConfigure method to BaseModule from Module

* Fixing test function name

* Use errors.Wrapf

* Logging config change

* Adding comment about intent of use.

* s/ReConfigure/Reconfigure/

* Don't pass registry

* Return copy of reconfigured module

* Updating module docs to clarify auto-configuration

* Fixing test

* Trying out docs for `beat` module

* Fixing tests

* Adding tests for ReConfigure() / making it part of Module interface

* Moving comments

* Fixing infinite recursion 🤦

* Implement for kibana module

* Implementing for elasticsearch module

* Moving ReConfigure method to BaseModule from Module

* Logging config change

* Return copy of reconfigured module

* Updating module docs to clarify auto-configuration

* Fixing test

* Trying out docs for `beat` module

* Update metricbeat/docs/modules/beat.asciidoc

Co-Authored-By: DeDe Morton <dede.morton@elastic.co>

* Uppercasing start of log message

* Updating all stack modules' docs

* Reodering imports

* Fixing rebase error

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

CHANGELOG.next.asciidoc

@@ -416,6 +416,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Reference kubernetes manifests mount data directory from the host when running metricbeat as daemonset, so data persist between executions in the same node. {pull}17429[17429]
 - Add more detailed error messages, system tests and small refactoring to the service metricset in windows. {pull}17725[17725]
 - Move the perfmon metricset to GA. {issue}16608[16608] {pull}17879[17879]
+- Stack Monitoring modules now auto-configure required metricsets when `xpack.enabled: true` is set. {issue}16471[[16471] {pull}17609[17609]
 
 *Packetbeat*
 

metricbeat/docs/modules/beat.asciidoc

@@ -5,16 +5,20 @@ This file is generated! See scripts/mage/docs_collector.go
 [[metricbeat-module-beat]]
 == Beat module
 
-The Beat module contains a minimal set of metrics to enable monitoring of any Beat or other software based on libbeat across
-multiple versions. To monitor more Beat metrics, use our {stack}
-{monitor-features}.
-
-The default metricsets are `state` and `stats`.
+The `beat` module collects metrics about any Beat or other software based on libbeat.
 
 [float]
 === Compatibility
 
-The Beat module works with Beats 7.3.0 and later.
+The `beat` module works with {beats} 7.3.0 and later.
+
+[float]
+=== Usage for Stack Monitoring
+
+The `beat` module can be used to collect metrics shown in our {stack} {monitor-features}
+UI in {kib}. To enable this usage, set `xpack.enabled: true` and remove any `metricsets`
+from the module's configuration. Alternatively, run `metricbeat modules disable beat` and
+`metricbeat modules enable beat-xpack`.
 
 
 [float]

metricbeat/docs/modules/elasticsearch.asciidoc

@@ -5,20 +5,20 @@ This file is generated! See scripts/mage/docs_collector.go
 [[metricbeat-module-elasticsearch]]
 == Elasticsearch module
 
-There are two modules that collect metrics about {es}: 
-
-* The Elasticsearch module contains a minimal set of metrics to enable
-monitoring of Elasticsearch across multiple versions. The default metricsets in
-this module are `node` and `node_stats`.
-* The Elasticsearch X-Pack module enables you to monitor more Elasticsearch
-metrics with our {stack} {monitor-features}. The default metricsets in this
-module are `ccr`, `cluster_stats`, `enrich`, ``index`, `index_recovery`,
-`index_summary`, `ml_job`, `node_stats`, and `shard`.
+The `elasticsearch` module collects metrics about {es}.
 
 [float]
 === Compatibility
 
-The Elasticsearch module works with Elasticsearch 6.7.0 and later.
+The `elasticsearch` module works with {es} 6.7.0 and later.
+
+[float]
+=== Usage for Stack Monitoring
+
+The `elasticsearch` module can be used to collect metrics shown in our {stack} {monitor-features}
+UI in {kib}. To enable this usage, set `xpack.enabled: true` and remove any `metricsets`
+from the module's configuration. Alternatively, run `metricbeat modules disable elasticsearch` and
+`metricbeat modules enable elasticsearch-xpack`.
 
 
 [float]

metricbeat/docs/modules/kibana.asciidoc

@@ -5,17 +5,20 @@ This file is generated! See scripts/mage/docs_collector.go
 [[metricbeat-module-kibana]]
 == Kibana module
 
-There are two modules that collect metrics about {kib}: 
-
-* The Kibana module tracks only the high-level metrics. The default metricset in
-this module is `status`.
-* The Kibana X-Pack module enables you to monitor more Kibana metrics with our
-{stack} {monitor-features}. The default metricset in this module is `stats`.
+The `kibana` module collects metrics about {kib}.
 
 [float]
 === Compatibility
 
-The Kibana module works with Kibana 6.7.0 and later.
+The `kibana` module works with {kib} 6.7.0 and later.
+
+[float]
+=== Usage for Stack Monitoring
+
+The `kibana` module can be used to collect metrics shown in our {stack} {monitor-features}
+UI in {kib}. To enable this usage, set `xpack.enabled: true` and remove any `metricsets`
+from the module's configuration. Alternatively, run `metricbeat modules disable kibana` and
+`metricbeat modules enable kibana-xpack`.
 
 
 [float]

metricbeat/docs/modules/logstash.asciidoc

@@ -5,14 +5,20 @@ This file is generated! See scripts/mage/docs_collector.go
 [[metricbeat-module-logstash]]
 == Logstash module
 
-This is the Logstash module.
-
-The default metricsets are `node` and `node_stats`.
+The `logstash` module collects metrics about {ls}.
 
 [float]
 === Compatibility
 
-The Logstash module works with Logstash 7.3.0 and later.
+The `logstash` module works with {ls} 7.3.0 and later.
+
+[float]
+=== Usage for Stack Monitoring
+
+The `logstash` module can be used to collect metrics shown in our {stack} {monitor-features}
+UI in {kib}. To enable this usage, set `xpack.enabled: true` and remove any `metricsets`
+from the module's configuration. Alternatively, run `metricbeat modules disable logstash` and
+`metricbeat modules enable logstash-xpack`.
 
 
 [float]

metricbeat/helper/elastic/elastic.go

@@ -21,6 +21,8 @@ import (
 	"fmt"
 	"strings"
 
+	"github.com/pkg/errors"
+
 	"github.com/elastic/beats/v7/libbeat/logp"
 
 	"github.com/elastic/beats/v7/libbeat/common"
@@ -127,3 +129,42 @@ func FixTimestampField(m common.MapStr, field string) error {
 	}
 	return nil
 }
+
+// NewModule returns a new Elastic stack module with the appropriate metricsets configured.
+func NewModule(base *mb.BaseModule, xpackEnabledMetricsets []string, logger *logp.Logger) (*mb.BaseModule, error) {
+	moduleName := base.Name()
+
+	config := struct {
+		XPackEnabled bool `config:"xpack.enabled"`
+	}{}
+	if err := base.UnpackConfig(&config); err != nil {
+		return nil, errors.Wrapf(err, "could not unpack configuration for module %v", moduleName)
+	}
+
+	// No special configuration is needed if xpack.enabled != true
+	if !config.XPackEnabled {
+		return base, nil
+	}
+
+	var raw common.MapStr
+	if err := base.UnpackConfig(&raw); err != nil {
+		return nil, errors.Wrapf(err, "could not unpack configuration for module %v", moduleName)
+	}
+
+	// These metricsets are exactly the ones required if xpack.enabled == true
+	raw["metricsets"] = xpackEnabledMetricsets
+
+	newConfig, err := common.NewConfigFrom(raw)
+	if err != nil {
+		return nil, errors.Wrapf(err, "could not create new configuration for module %v", moduleName)
+	}
+
+	newModule, err := base.WithConfig(*newConfig)
+	if err != nil {
+		return nil, errors.Wrapf(err, "could not reconfigure module %v", moduleName)
+	}
+
+	logger.Debugf("Configuration for module %v modified because xpack.enabled was set to true", moduleName)
+
+	return newModule, nil
+}

metricbeat/helper/elastic/elastic_test.go

@@ -22,7 +22,10 @@ import (
 	"testing"
 
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 
+	"github.com/elastic/beats/v7/libbeat/common"
+	"github.com/elastic/beats/v7/libbeat/logp"
 	"github.com/elastic/beats/v7/metricbeat/mb"
 )
 
@@ -97,7 +100,7 @@ func TestFixTimestampField(t *testing.T) {
 		{
 			"converts float64s in scientific notation to ints",
 			map[string]interface{}{
-				"foo": 1.571284349E12,
+				"foo": 1.571284349e12,
 			},
 			map[string]interface{}{
 				"foo": 1571284349000,
@@ -140,3 +143,86 @@ func TestFixTimestampField(t *testing.T) {
 		})
 	}
 }
+
+func TestConfigureModule(t *testing.T) {
+	mockRegistry := mb.NewRegister()
+
+	const moduleName = "test_module"
+
+	err := mockRegistry.AddMetricSet(moduleName, "foo", mockMetricSetFactory)
+	require.NoError(t, err)
+	err = mockRegistry.AddMetricSet(moduleName, "bar", mockMetricSetFactory)
+	require.NoError(t, err)
+	err = mockRegistry.AddMetricSet(moduleName, "qux", mockMetricSetFactory)
+	require.NoError(t, err)
+	err = mockRegistry.AddMetricSet(moduleName, "baz", mockMetricSetFactory)
+	require.NoError(t, err)
+
+	tests := map[string]struct {
+		initConfig             metricSetConfig
+		xpackEnabledMetricsets []string
+		newConfig              metricSetConfig
+	}{
+		"no_xpack_enabled": {
+			metricSetConfig{
+				Module:     moduleName,
+				MetricSets: []string{"foo", "bar"},
+			},
+			[]string{"baz", "qux", "foo"},
+			metricSetConfig{
+				Module:     moduleName,
+				MetricSets: []string{"foo", "bar"},
+			},
+		},
+		"xpack_enabled": {
+			metricSetConfig{
+				Module:       moduleName,
+				XPackEnabled: true,
+				MetricSets:   []string{"foo", "bar"},
+			},
+			[]string{"baz", "qux", "foo"},
+			metricSetConfig{
+				Module:       moduleName,
+				XPackEnabled: true,
+				MetricSets:   []string{"baz", "qux", "foo"},
+			},
+		},
+	}
+
+	for name, test := range tests {
+		t.Run(name, func(t *testing.T) {
+			cfg := common.MustNewConfigFrom(test.initConfig)
+			m, _, err := mb.NewModule(cfg, mockRegistry)
+			require.NoError(t, err)
+
+			bm, ok := m.(*mb.BaseModule)
+			if !ok {
+				require.Fail(t, "expecting module to be base module")
+			}
+
+			newM, err := NewModule(bm, test.xpackEnabledMetricsets, logp.L())
+			require.NoError(t, err)
+
+			var newConfig metricSetConfig
+			err = newM.UnpackConfig(&newConfig)
+			require.NoError(t, err)
+			require.Equal(t, test.newConfig, newConfig)
+		})
+	}
+}
+
+type mockMetricSet struct {
+	mb.BaseMetricSet
+}
+
+func (m *mockMetricSet) Fetch(r mb.ReporterV2) error { return nil }
+
+type metricSetConfig struct {
+	Module       string   `config:"module"`
+	MetricSets   []string `config:"metricsets"`
+	XPackEnabled bool     `config:"xpack.enabled"`
+}
+
+func mockMetricSetFactory(base mb.BaseMetricSet) (mb.MetricSet, error) {
+	return &mockMetricSet{base}, nil
+}

metricbeat/mb/mb.go

@@ -27,6 +27,8 @@ import (
 	"net/url"
 	"time"
 
+	"github.com/pkg/errors"
+
 	"github.com/elastic/beats/v7/libbeat/common"
 	"github.com/elastic/beats/v7/libbeat/logp"
 	"github.com/elastic/beats/v7/libbeat/monitoring"
@@ -94,6 +96,40 @@ func (m *BaseModule) UnpackConfig(to interface{}) error {
 	return m.rawConfig.Unpack(to)
 }
 
+// WithConfig re-configures the module with the given raw configuration and returns a
+// copy of the module.
+// Intended to be called from module factories. Note that if metricsets are specified
+// in the new configuration, those metricsets must already be registered with
+// mb.Registry.
+func (m *BaseModule) WithConfig(config common.Config) (*BaseModule, error) {
+	var chkConfig struct {
+		Module string `config:"module"`
+	}
+	if err := config.Unpack(&chkConfig); err != nil {
+		return nil, errors.Wrap(err, "error parsing new module configuration")
+	}
+
+	// Don't allow module name change
+	if chkConfig.Module != "" && chkConfig.Module != m.name {
+		return nil, fmt.Errorf("cannot change module name from %v to %v", m.name, chkConfig.Module)
+	}
+
+	if err := config.SetString("module", -1, m.name); err != nil {
+		return nil, errors.Wrap(err, "unable to set existing module name in new configuration")
+	}
+
+	newBM := &BaseModule{
+		name:      m.name,
+		rawConfig: &config,
+	}
+
+	if err := config.Unpack(&newBM.config); err != nil {
+		return nil, errors.Wrap(err, "error parsing new module configuration")
+	}
+
+	return newBM, nil
+}
+
 // MetricSet interfaces
 
 // MetricSet is the common interface for all MetricSet implementations. In