Validate metrics using metadata.csv

[AlexandreYang]

What does this PR do?

Validate metrics using metadata.csv.

See examples below:

tests/test_check.py:79: in test_e2e
    aggregator.assert_metrics_using_metadata(exclude=['lighttpd.performance.busy_servers'])
../datadog_checks_base/datadog_checks/base/stubs/aggregator.py:336: in assert_metrics_using_metadata
    assert not errors, "Metadata assertion errors using metadata.csv: " + "\n\t- ".join([""] + sorted(list(errors)))
E   AssertionError: Metadata assertion errors using metadata.csv:
E     	- Expect `lighttpd.net.bytes` to be in metadata.csv.
E     	- Expect `lighttpd.net.request_per_s` to be in metadata.csv.
E     	- Expect `lighttpd.performance.idle_server` to have type `count` but got `gauge`.
E     	- Expect `lighttpd.performance.uptime` to be in metadata.csv.
E   assert not {'Expect `lighttpd.net.bytes` to be in metadata.csv.', 'Expect `lighttpd.net.request_per_s` to be in metadata.csv.', '...nce.idle_server` to have type `count` but got `gauge`.', 'Expect `lighttpd.performance.uptime` to be in metadata.csv.'}

Motivation

That will help validating that metadata.csv metric types and actual types match.

Additional Notes

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached

[codecov]

Codecov Report

Merging #6027 into master will decrease coverage by 0.37%.
The diff coverage is 21.42%.

Impacted Files	Coverage Δ
...hecks_base/datadog_checks/base/stubs/aggregator.py	67.21% <21.05%> (-4.21%)	⬇️
datadog_checks_dev/datadog_checks/dev/utils.py	51.53% <22.22%> (-1.42%)	⬇️
datadog_checks_base/tests/conftest.py	39.13% <0.00%> (-60.87%)	⬇️
datadog_checks_dev/tests/test_docker.py	48.14% <0.00%> (-51.86%)	⬇️
disk/tests/common.py	61.53% <0.00%> (-38.47%)	⬇️
datadog_checks_dev/tests/tooling/test_utils.py	73.40% <0.00%> (-26.60%)	⬇️
datadog_checks_dev/datadog_checks/dev/env.py	22.72% <0.00%> (-20.46%)	⬇️
datadog_checks_dev/datadog_checks/dev/docker.py	25.80% <0.00%> (-18.55%)	⬇️
...atadog_checks_dev/datadog_checks/dev/structures.py	68.85% <0.00%> (-18.04%)	⬇️
datadog_checks_dev/tests/test_conditions.py	82.25% <0.00%> (-17.75%)	⬇️
... and 579 more

[ofek]

This is a bad idea to have as the default behavior since most environments would need to be extremely specialized to produce even the majority of metrics.

Instead, let's add a new method that asserts everything in metadata.csv and add the ability to pass in a list of metrics to exclude.

WDYT?

[ofek]

Let's use

integrations-core/datadog_checks_dev/datadog_checks/dev/utils.py

Line 226 in 15334a6

def find_check_root(depth=0):

[AlexandreYang]

On hold, need to move stubs to datadog_checks_dev first.

Related: #3700 (comment)

[AlexandreYang]

@ofek

I changed the implementation to avoid aggregator. assert_metrics_using_metadata depending on the metadata.csv file internally.

Now aggregator. assert_metrics_using_metadata received the metadata_metrics as an argument. Better decoupling and reusability.

[ofek]

wdyt? since more integrations would have integration tests and not e2e

Suggested change

	def assert_metrics_using_metadata(self, metadata_metrics, check_metric_type=True, exclude=None):
	def assert_metrics_using_metadata(self, metadata_metrics, check_metric_type=False, exclude=None):

[AlexandreYang]

@ofek
I think this method has the most value for e2e since check the type is quite valuable.

Having check_metric_type=True by default will encourage that. WDYT?