Add definition for logging.metrics.enabled, pass config from agent to beats #26828

michel-laterman · 2021-07-09T16:51:08Z

What does this PR do?

Define logging.metrics.enabled and logging.metrics.period.
When logging.metrics.enabled is set to false the elastic-agent will
pass the setting to the underlying beats through a command line variable.

Checklist

My code follows the style guidelines of this project
~~I have commented my code, particularly in hard-to-understand areas~~
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
~~I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Add agent.monitoring.log_metrics: false to elastic-agent.yml (in stand-alone mode) and ensure that the beat logs do not have metrics entries.

i.e.: default behaviour with agent.monitoring.log_metrics missing:

elastic-agent-7.14.0-SNAPSHOT-darwin-x86_64|⇒ ps aux | grep beat
laterman          4742   0.0  0.0  4408696    828 s000  R+    9:50am   0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox beat
root              4645   0.0  0.1  5747796  82824   ??  S     9:49am   0:00.49 /Library/Elastic/Agent/data/elastic-agent-6e0888/install/metricbeat-7.14.0-SNAPSHOT-darwin-x86_64/metricbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E gc_percent=${METRICBEAT_GOGC:100} -E logging.level=info -E http.enabled=true -E http.host=unix:///Library/Elastic/Agent/data/tmp/default/metricbeat/metricbeat.sock_monitor -E logging.json=true -E logging.ecs=true -E logging.files.path=/Library/Elastic/Agent/data/elastic-agent-6e0888/logs/default -E logging.files.name=metricbeat_monitor-json.log -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E path.data=/Library/Elastic/Agent/data/elastic-agent-6e0888/run/default/metricbeat--7.14.0-SNAPSHOT--36643631373035623733363936343635
root              4644   0.0  0.1  6020232  68824   ??  S     9:49am   0:00.41 /Library/Elastic/Agent/data/elastic-agent-6e0888/install/filebeat-7.14.0-SNAPSHOT-darwin-x86_64/filebeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E gc_percent=${FILEBEAT_GOGC:100} -E logging.level=info -E http.enabled=true -E http.host=unix:///Library/Elastic/Agent/data/tmp/default/filebeat/filebeat.sock_monitor -E logging.json=true -E logging.ecs=true -E logging.files.path=/Library/Elastic/Agent/data/elastic-agent-6e0888/logs/default -E logging.files.name=filebeat_monitor-json.log -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E path.data=/Library/Elastic/Agent/data/elastic-agent-6e0888/run/default/filebeat--7.14.0-SNAPSHOT--36643631373035623733363936343635
root              4643   0.0  0.1  5808224 100492   ??  S     9:49am   0:06.82 /Library/Elastic/Agent/data/elastic-agent-6e0888/install/metricbeat-7.14.0-SNAPSHOT-darwin-x86_64/metricbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E gc_percent=${METRICBEAT_GOGC:100} -E logging.level=info -E http.enabled=true -E http.host=unix:///Library/Elastic/Agent/data/tmp/default/metricbeat/metricbeat.sock -E logging.json=true -E logging.ecs=true -E logging.files.path=/Library/Elastic/Agent/data/elastic-agent-6e0888/logs/default -E logging.files.name=metricbeat-json.log -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E path.data=/Library/Elastic/Agent/data/elastic-agent-6e0888/run/default/metricbeat--7.14.0-SNAPSHOT
root              4641   0.0  0.1  5184168  68088   ??  S     9:48am   0:00.43 /Library/Elastic/Agent/data/elastic-agent-6e0888/install/filebeat-7.14.0-SNAPSHOT-darwin-x86_64/filebeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E gc_percent=${FILEBEAT_GOGC:100} -E logging.level=info -E http.enabled=true -E http.host=unix:///Library/Elastic/Agent/data/tmp/default/filebeat/filebeat.sock -E logging.json=true -E logging.ecs=true -E logging.files.path=/Library/Elastic/Agent/data/elastic-agent-6e0888/logs/default -E logging.files.name=filebeat-json.log -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E path.data=/Library/Elastic/Agent/data/elastic-agent-6e0888/run/default/filebeat--7.14.0-SNAPSHOT

setting log_metrics: false:

elastic-agent-7.14.0-SNAPSHOT-darwin-x86_64|⇒ ps aux | grep beat
root              4816   8.5  0.1  5941648  94980   ??  S     9:51am   0:01.27 /Library/Elastic/Agent/data/elastic-agent-6e0888/install/metricbeat-7.14.0-SNAPSHOT-darwin-x86_64/metricbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E gc_percent=${METRICBEAT_GOGC:100} -E logging.level=info -E http.enabled=true -E http.host=unix:///Library/Elastic/Agent/data/tmp/default/metricbeat/metricbeat.sock -E logging.json=true -E logging.ecs=true -E logging.files.path=/Library/Elastic/Agent/data/elastic-agent-6e0888/logs/default -E logging.files.name=metricbeat-json.log -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E logging.metrics.enabled=false -E path.data=/Library/Elastic/Agent/data/elastic-agent-6e0888/run/default/metricbeat--7.14.0-SNAPSHOT
root              4825   3.7  0.1  5132456  61964   ??  S     9:51am   0:00.28 /Library/Elastic/Agent/data/elastic-agent-6e0888/install/filebeat-7.14.0-SNAPSHOT-darwin-x86_64/filebeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E gc_percent=${FILEBEAT_GOGC:100} -E logging.level=info -E http.enabled=true -E http.host=unix:///Library/Elastic/Agent/data/tmp/default/filebeat/filebeat.sock_monitor -E logging.json=true -E logging.ecs=true -E logging.files.path=/Library/Elastic/Agent/data/elastic-agent-6e0888/logs/default -E logging.files.name=filebeat_monitor-json.log -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E logging.metrics.enabled=false -E path.data=/Library/Elastic/Agent/data/elastic-agent-6e0888/run/default/filebeat--7.14.0-SNAPSHOT--36643631373035623733363936343635
laterman          4829   0.0  0.0  4408696    832 s000  R+    9:51am   0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox beat
root              4826   0.0  0.1  5139548  75556   ??  S     9:51am   0:00.26 /Library/Elastic/Agent/data/elastic-agent-6e0888/install/metricbeat-7.14.0-SNAPSHOT-darwin-x86_64/metricbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E gc_percent=${METRICBEAT_GOGC:100} -E logging.level=info -E http.enabled=true -E http.host=unix:///Library/Elastic/Agent/data/tmp/default/metricbeat/metricbeat.sock_monitor -E logging.json=true -E logging.ecs=true -E logging.files.path=/Library/Elastic/Agent/data/elastic-agent-6e0888/logs/default -E logging.files.name=metricbeat_monitor-json.log -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E logging.metrics.enabled=false -E path.data=/Library/Elastic/Agent/data/elastic-agent-6e0888/run/default/metricbeat--7.14.0-SNAPSHOT--36643631373035623733363936343635
root              4818   0.0  0.1  5137808  59964   ??  S     9:51am   0:00.24 /Library/Elastic/Agent/data/elastic-agent-6e0888/install/filebeat-7.14.0-SNAPSHOT-darwin-x86_64/filebeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E gc_percent=${FILEBEAT_GOGC:100} -E logging.level=info -E http.enabled=true -E http.host=unix:///Library/Elastic/Agent/data/tmp/default/filebeat/filebeat.sock -E logging.json=true -E logging.ecs=true -E logging.files.path=/Library/Elastic/Agent/data/elastic-agent-6e0888/logs/default -E logging.files.name=filebeat-json.log -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E logging.metrics.enabled=false -E path.data=/Library/Elastic/Agent/data/elastic-agent-6e0888/run/default/filebeat--7.14.0-SNAPSHOT

Related issues

Closes [Elastic Agent] Sending monitoring metrics to logs datastream #26758

Add a log_metrics config option that defaults to true. When set to false the agent will pass -E logging.metrice.enabled=false to disable the underlying beats from adding metrics to their log streams.

elasticmachine · 2021-07-09T18:04:19Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-12T02:34:06.795+0000
Duration: 139 min 48 sec
Commit: 6caf83b

Test stats 🧪

Test	Results
Failed	0
Passed	53000
Skipped	5318
Total	58318

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	53000
Skipped	5318
Total	58318

elasticmachine · 2021-07-12T17:48:46Z

Pinging @elastic/agent (Team:Agent)

ruflin · 2021-07-14T13:31:19Z

Can you share your thoughts on if we should set the default to true (as it is) or false?

michel-laterman · 2021-07-14T15:01:55Z

@urso and I briefly discussed this, setting it to true does not change existing behaviour and allows us to use the logs when providing support to users.

urso · 2021-07-15T14:28:32Z

x-pack/elastic-agent/_meta/config/common.p2.yml.tmpl

@@ -33,6 +33,8 @@ inputs:
 #   logs: true
 #   # enables metrics monitoring
 #   metrics: true
+#   # running processes will emit metrics into logs
+#   log_metrics: true


What is the difference between metrics and logs_metrics?

We already have the agent.logging.metrics.enabled settings. Should we rather use that setting?

@michalpristas WDYT?

need to check if these settings are passed to beats

i'm not against of reusing what we have. these settings are not forwarded to beats but used for constructing our logger.

mergify · 2021-07-22T14:44:34Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b log-metrics-toggle upstream/log-metrics-toggle
git merge upstream/master
git push upstream log-metrics-toggle

michalpristas · 2021-07-29T10:37:23Z

where are we on this. any updates?

Change the log metrics enablement config from a new option under metrics to agent.logging.metrics.enabled.

michel-laterman · 2021-07-31T06:01:47Z

libbeat/logp/config.go

+//
+// Currently these options are not used through this object in beats (as monitoring is setup elsewhere).
+type MetricsConfig struct {
+	Enabled bool          `config:"enabled" yaml:"enabled"`


@urso, I noticed that the beats load through

beats/libbeat/cmd/instance/beat.go

Line 453 in 60e9e34

if b.Config.MetricLogging == nil || b.Config.MetricLogging.Enabled() {

but the logging config does define these variables, do you think it makes sense to define it as part of the logging config so it can be retrieved by the agent?

michel-laterman · 2021-08-03T17:35:41Z

/test

michel-laterman · 2021-08-12T01:47:14Z

libbeat/tests/system/config/mockbeat.yml.j2

-{% if metrics_period -%}
-logging.metrics.period: {{ metrics_period }}
+{% if file_name -%}
+logging.files.name: {{ file_name }}


The export command being tested disables environment variable resolution.

beats/libbeat/cmd/export/config.go

Line 42 in 4727470

settings.DisableConfigResolver = true

I think it's so export config can be used to copy the config and ignore the local environment.
Originally the test was for logging.metrics.period but seeing as how this pr adds the definition for this attribute as a time.Duration the export fails because the string could not be parsed as a duration.
I instead changed the test to use a string type attribute so the export does not fail.

However going forward the export config command will run into this issue if logging.metrics.period or logging.metrics.enabled are set to environment variables.
This should not effect other commands (which resolve environment variables).

tldr; attribute+test has changed because environment variable expansion is disabled for the export commands

blakerouse

Looks good to me.

… beats (#26828) Define logging.metrics.enabled and logging.metrics.period. When logging.metrics.enabled is set to false the elastic-agent will pass the setting to the underlying beats through a command line variable. (cherry picked from commit 393b2f0)

… beats (#26828) (#27343) Define logging.metrics.enabled and logging.metrics.period. When logging.metrics.enabled is set to false the elastic-agent will pass the setting to the underlying beats through a command line variable. (cherry picked from commit 393b2f0) Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>

michel-laterman added enhancement Team:Elastic-Agent Label for the Agent team v7.15.0 backport-v7.15.0 Automated backport with mergify labels Jul 9, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 9, 2021

Add agent.monitoring.log_metrics config option.

2246233

Add a log_metrics config option that defaults to true. When set to false the agent will pass -E logging.metrice.enabled=false to disable the underlying beats from adding metrics to their log streams.

michel-laterman force-pushed the log-metrics-toggle branch from 3d635dd to 2246233 Compare July 9, 2021 16:51

michel-laterman marked this pull request as ready for review July 12, 2021 17:48

michel-laterman requested review from michalpristas and blakerouse July 12, 2021 17:49

urso reviewed Jul 15, 2021

View reviewed changes

michel-laterman added 4 commits July 30, 2021 16:33

Merge remote-tracking branch 'origin/master' into log-metrics-toggle

47bb266

Change the source of log metrics config

15e674c

Change the log metrics enablement config from a new option under metrics to agent.logging.metrics.enabled.

correct missing references in config

75699ac

Fix typo

360f5fe

michel-laterman commented Jul 31, 2021

View reviewed changes

Merge branch 'master' into log-metrics-toggle

bfe0c05

michel-laterman added 4 commits August 9, 2021 08:53

Merge branch 'master' into log-metrics-toggle

5d55e09

Merge branch 'master' into log-metrics-toggle

784851b

remove yaml tags

4442041

Change export test variable to a string attribute

462ce60

michel-laterman commented Aug 12, 2021

View reviewed changes

add metrics.period back to config to fix test

6caf83b

blakerouse approved these changes Aug 12, 2021

View reviewed changes

michel-laterman changed the title ~~Add agent.monitoring.log_metrics config option.~~ Add definition for logging.metrics.enabled, pass config from agent to beats Aug 12, 2021

michel-laterman merged commit 393b2f0 into elastic:master Aug 12, 2021

michel-laterman deleted the log-metrics-toggle branch August 12, 2021 17:41

mergify bot mentioned this pull request Aug 12, 2021

[7.x](backport #26828) Add definition for logging.metrics.enabled, pass config from agent to beats #27343

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add definition for logging.metrics.enabled, pass config from agent to beats #26828

Add definition for logging.metrics.enabled, pass config from agent to beats #26828

michel-laterman commented Jul 9, 2021 •

edited

Loading

elasticmachine commented Jul 9, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

elasticmachine commented Jul 12, 2021

ruflin commented Jul 14, 2021

michel-laterman commented Jul 14, 2021

urso Jul 15, 2021

michel-laterman Jul 15, 2021

michalpristas Jul 19, 2021

mergify bot commented Jul 22, 2021

michalpristas commented Jul 29, 2021

michel-laterman Jul 31, 2021

urso Aug 2, 2021

michel-laterman commented Aug 3, 2021

michel-laterman Aug 12, 2021

blakerouse left a comment

Add definition for logging.metrics.enabled, pass config from agent to beats #26828

Add definition for logging.metrics.enabled, pass config from agent to beats #26828

Conversation

michel-laterman commented Jul 9, 2021 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Jul 9, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Jul 12, 2021

ruflin commented Jul 14, 2021

michel-laterman commented Jul 14, 2021

urso Jul 15, 2021

Choose a reason for hiding this comment

michel-laterman Jul 15, 2021

Choose a reason for hiding this comment

michalpristas Jul 19, 2021

Choose a reason for hiding this comment

mergify bot commented Jul 22, 2021

michalpristas commented Jul 29, 2021

michel-laterman Jul 31, 2021

Choose a reason for hiding this comment

urso Aug 2, 2021

Choose a reason for hiding this comment

michel-laterman commented Aug 3, 2021

michel-laterman Aug 12, 2021

Choose a reason for hiding this comment

blakerouse left a comment

Choose a reason for hiding this comment

michel-laterman commented Jul 9, 2021 •

edited

Loading

elasticmachine commented Jul 9, 2021 •

edited by jenkins-beats-ci bot

Loading