From 6e79f12b8bf5a3567017bf0e585facef5717d7d7 Mon Sep 17 00:00:00 2001 From: DeDe Morton Date: Wed, 8 Jan 2020 15:11:19 -0800 Subject: [PATCH] [DOCS] Clarify changes in monitoring settings (#15003) * [DOCS] Clarify changes in monitoring settings * Change topic names and order * Add more fixes from the review * Update libbeat/docs/monitoring/monitoring-beats.asciidoc --- .../docs/monitoring/monitoring-beats.asciidoc | 24 +- ...toring-internal-collection-legacy.asciidoc | 33 +++ .../monitoring-internal-collection.asciidoc | 9 +- .../monitoring/monitoring-metricbeat.asciidoc | 252 ------------------ .../shared-monitor-config-legacy.asciidoc | 138 ++++++++++ .../monitoring/shared-monitor-config.asciidoc | 37 ++- libbeat/docs/security/users.asciidoc | 49 +--- 7 files changed, 202 insertions(+), 340 deletions(-) create mode 100644 libbeat/docs/monitoring/monitoring-internal-collection-legacy.asciidoc delete mode 100644 libbeat/docs/monitoring/monitoring-metricbeat.asciidoc create mode 100644 libbeat/docs/monitoring/shared-monitor-config-legacy.asciidoc diff --git a/libbeat/docs/monitoring/monitoring-beats.asciidoc b/libbeat/docs/monitoring/monitoring-beats.asciidoc index d13842aa072..28fa7d72d9d 100644 --- a/libbeat/docs/monitoring/monitoring-beats.asciidoc +++ b/libbeat/docs/monitoring/monitoring-beats.asciidoc @@ -9,18 +9,14 @@ You can use the {stack} {monitor-features} to gain insight into the health of {beatname_uc} agents running in your environment. To monitor {beatname_uc}, make sure monitoring is enabled on your {es} cluster, -then configure the method used to collect {beatname_uc} metrics. You -ifndef::serverless[] -can use one of following methods: -endif::[] -ifdef::serverless[] -can use the following method: -endif::[] - -* <> -ifndef::serverless[] -* <> -endif::[] +then configure the method used to collect {beatname_uc} metrics. You can use one +of following methods: + +* <> - Internal +collectors send monitoring data directly to your monitoring cluster. +* <> - +Legacy internal collectors send monitoring data to your production cluster. + To learn about monitoring in general, see {stack-ov}/xpack-monitoring.html[Monitoring the {stack}]. @@ -29,6 +25,4 @@ To learn about monitoring in general, see include::monitoring-internal-collection.asciidoc[] -ifndef::serverless[] -include::monitoring-metricbeat.asciidoc[] -endif::[] +include::monitoring-internal-collection-legacy.asciidoc[] diff --git a/libbeat/docs/monitoring/monitoring-internal-collection-legacy.asciidoc b/libbeat/docs/monitoring/monitoring-internal-collection-legacy.asciidoc new file mode 100644 index 00000000000..ab3c0123c88 --- /dev/null +++ b/libbeat/docs/monitoring/monitoring-internal-collection-legacy.asciidoc @@ -0,0 +1,33 @@ +////////////////////////////////////////////////////////////////////////// +//// This content is shared by all Elastic Beats. Make sure you keep the +//// descriptions here generic enough to work for all Beats that include +//// this file. When using cross references, make sure that the cross +//// references resolve correctly for any files that include this one. +//// Use the appropriate variables defined in the index.asciidoc file to +//// resolve Beat names: beatname_uc and beatname_lc. +//// Use the following include to pull this content into a doc file: +//// include::../../libbeat/docs/monitoring/monitoring-internal-collection-legacy.asciidoc[] +////////////////////////////////////////////////////////////////////////// + +[role="xpack"] +[[monitoring-internal-collection-legacy]] +== Use legacy internal collection to send monitoring data +++++ +Legacy internal collection (deprecated) +++++ + +deprecated[7.2.0] + +In {beatname_uc} version 7.1 and earlier, you configured internal collectors +that sent monitoring data to the production cluster, which would either index +the data locally, or forward the data to a dedicated monitoring cluster via HTTP +exporters. + +Starting in {beatname_uc} version 7.2, the legacy settings for internal +collection are deprecated and will be removed in version 8.0.0. Instead of +sending monitoring data to your production cluster, it's recommended that you +use the configuration described under +<> to route +monitoring data directly to your monitoring cluster. + +include::shared-monitor-config-legacy.asciidoc[] diff --git a/libbeat/docs/monitoring/monitoring-internal-collection.asciidoc b/libbeat/docs/monitoring/monitoring-internal-collection.asciidoc index 5715fd14d7f..c1d23f34b49 100644 --- a/libbeat/docs/monitoring/monitoring-internal-collection.asciidoc +++ b/libbeat/docs/monitoring/monitoring-internal-collection.asciidoc @@ -11,16 +11,13 @@ [role="xpack"] [[monitoring-internal-collection]] -== Collect {beatname_uc} monitoring data with internal collectors +== Use internal collection to send monitoring data ++++ Internal collection ++++ -The following method involves sending the metrics to the production cluster, -which ultimately routes them to the monitoring cluster. -ifndef::serverless[] -For an alternative method, see <>. -endif::[] +Use internal collectors to send {beats} monitoring data directly to your +monitoring cluster. To learn about monitoring in general, see {stack-ov}/xpack-monitoring.html[Monitoring the {stack}]. diff --git a/libbeat/docs/monitoring/monitoring-metricbeat.asciidoc b/libbeat/docs/monitoring/monitoring-metricbeat.asciidoc deleted file mode 100644 index d6f675bcaae..00000000000 --- a/libbeat/docs/monitoring/monitoring-metricbeat.asciidoc +++ /dev/null @@ -1,252 +0,0 @@ -[role="xpack"] -[[monitoring-metricbeat-collection]] -== Collect {beatname_uc} monitoring data with {metricbeat} -[subs="attributes"] -++++ -{metricbeat} collection -++++ - -In 7.3 and later, you can use {metricbeat} to collect data about {beatname_uc} -and ship it to the monitoring cluster, rather than routing it through the -production cluster as described in <>. - -ifeval::["{beatname_lc}"=="metricbeat"] -Because you'll be using {metricbeat} to _monitor_ {beatname_uc}, you'll need to -run two instances of {beatname_uc}: a main instance that collects metrics from -the system and services running on the server, and a second instance that -collects metrics from {beatname_uc} only. Using a separate instance as a -monitoring agent allows you to send monitoring data to a dedicated monitoring -cluster. If the main agent goes down, the monitoring agent remains active. - -If you're running {beatname_uc} as a service, this approach requires extra work -because you need to run two instances of the same installed service -concurrently. If you don't want to run two instances concurrently, use -<> instead of using -{metricbeat}. -endif::[] - -To learn about monitoring in general, see -{stack-ov}/xpack-monitoring.html[Monitoring the {stack}]. - -//NOTE: The tagged regions are re-used in the Stack Overview. - -To collect and ship monitoring data: - -. <> - -. <> - -[float] -[[configure-shipper]] -=== Configure the shipper you want to monitor - -. Enable the HTTP endpoint to allow external collection of monitoring data: -+ --- -// tag::enable-http-endpoint[] -Add the following setting in the {beatname_uc} configuration file -(+{beatname_lc}.yml+): - -[source,yaml] ----------------------------------- -http.enabled: true ----------------------------------- - -By default, metrics are exposed on port 5066. If you need to monitor multiple -{beats} shippers running on the same server, set `http.port` to expose metrics -for each shipper on a different port number: - -[source,yaml] ----------------------------------- -http.port: 5067 ----------------------------------- -// end::enable-http-endpoint[] --- - -. Disable the default collection of {beatname_uc} monitoring metrics. + -+ --- -// tag::disable-beat-collection[] -Add the following setting in the {beatname_uc} configuration file -(+{beatname_lc}.yml+): - -[source,yaml] ----------------------------------- -monitoring.enabled: false ----------------------------------- -// end::disable-beat-collection[] - -For more information, see -<>. --- - -ifndef::serverless[] -. <<{beatname_lc}-starting,Start {beatname_uc}>>. -endif::serverless[] - -[float] -[[configure-metricbeat]] -=== Install and configure {metricbeat} to collect monitoring data - -ifeval::["{beatname_lc}"!="metricbeat"] -. {metricbeat-ref}/metricbeat-installation.html[Install {metricbeat}] on the -same server as {beatname_uc}. If you already have {metricbeat} installed on the -server, skip this step. -endif::[] -ifeval::["{beatname_lc}"=="metricbeat"] -. The next step depends on how you want to run {metricbeat}: -* If you're running as a service and want to run a separate monitoring instance, -take the the steps required for your environment to run two instances of -{metricbeat} as a service. The steps for doing this vary by platform and are -beyond the scope of this documentation. -* If you're running the binary directly in the foreground and want to run a -separate monitoring instance, -{metricbeat-ref}/metricbeat-installation.html[install {metricbeat}] to a -different path. If necessary, set `path.config`, `path.data`, and `path.log` -to point to the correct directories. See <> for the default -locations. -endif::[] - -. Enable the `beat-xpack` module in {metricbeat}. + -endif::[] -+ --- -// tag::enable-beat-module[] -For example, to enable the default configuration in the `modules.d` directory, -run the following command, using the correct command syntax for your OS: - -["source","sh",subs="attributes,callouts"] ----------------------------------------------------------------------- -metricbeat modules enable beat-xpack ----------------------------------------------------------------------- - -For more information, see -{metricbeat-ref}/configuration-metricbeat.html[Specify which modules to run] and -{metricbeat-ref}/metricbeat-module-beat.html[beat module]. -// end::enable-beat-module[] --- - -. Configure the `beat-xpack` module in {metricbeat}. + -+ --- -// tag::configure-beat-module[] -The `modules.d/beat-xpack.yml` file contains the following settings: - -[source,yaml] ----------------------------------- -- module: beat - metricsets: - - stats - - state - period: 10s - hosts: ["http://localhost:5066"] - #username: "user" - #password: "secret" - xpack.enabled: true ----------------------------------- - -Set the `hosts`, `username`, and `password` settings as required by your -environment. For other module settings, it's recommended that you accept the -defaults. - -By default, the module collects {beatname_uc} monitoring data from -`localhost:5066`. If you exposed the metrics on a different host or port when -you enabled the HTTP endpoint, update the `hosts` setting. - -To monitor multiple {beats} agents, specify a list of hosts, for example: -[source,yaml] ----------------------------------- -hosts: ["http://localhost:5066","http://localhost:5067","http://localhost:5068"] ----------------------------------- - -If you configured {beatname_uc} to use encrypted communications, you must access -it via HTTPS. For example, use a `hosts` setting like `https://localhost:5066`. -// end::configure-beat-module[] - -// tag::remote-monitoring-user[] -If the Elastic {security-features} are enabled, you must also provide a user -ID and password so that {metricbeat} can collect metrics successfully: - -.. Create a user on the production cluster that has the -`remote_monitoring_collector` {stack-ov}/built-in-roles.html[built-in role]. -Alternatively, use the `remote_monitoring_user` -{stack-ov}/built-in-users.html[built-in user]. - -.. Add the `username` and `password` settings to the beat module configuration -file. -// end::remote-monitoring-user[] --- - -. Optional: Disable the system module in the {metricbeat}. -+ --- -// tag::disable-system-module[] -By default, the {metricbeat-ref}/metricbeat-module-system.html[system module] is -enabled. The information it collects, however, is not shown on the -*Stack Monitoring* page in {kib}. Unless you want to use that information for -other purposes, run the following command: - -["source","sh",subs="attributes,callouts"] ----------------------------------------------------------------------- -metricbeat modules disable system ----------------------------------------------------------------------- -// end::disable-system-module[] --- - -. Identify where to send the monitoring data. + -+ --- -TIP: In production environments, we strongly recommend using a separate cluster -(referred to as the _monitoring cluster_) to store the data. Using a separate -monitoring cluster prevents production cluster outages from impacting your -ability to access your monitoring data. It also prevents monitoring activities -from impacting the performance of your production cluster. - -For example, specify the {es} output information in the {metricbeat} -configuration file (`metricbeat.yml`): - -[source,yaml] ----------------------------------- -output.elasticsearch: - # Array of hosts to connect to. - hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"] <1> - - # Optional protocol and basic auth credentials. - #protocol: "https" - #username: "elastic" - #password: "changeme" ----------------------------------- -<1> In this example, the data is stored on a monitoring cluster with nodes -`es-mon-1` and `es-mon-2`. - -If you configured the monitoring cluster to use encrypted communications, you -must access it via HTTPS. For example, use a `hosts` setting like -`https://es-mon-1:9200`. - -IMPORTANT: The {es} {monitor-features} use ingest pipelines, therefore the -cluster that stores the monitoring data must have at least one ingest node. - -If the {es} {security-features} are enabled on the monitoring cluster, you -must provide a valid user ID and password so that {metricbeat} can send metrics -successfully: - -.. Create a user on the monitoring cluster that has the -`remote_monitoring_agent` {stack-ov}/built-in-roles.html[built-in role]. -Alternatively, use the `remote_monitoring_user` -{stack-ov}/built-in-users.html[built-in user]. -+ -TIP: If you're using index lifecycle management, the remote monitoring user -requires additional privileges to create and read indices. For more -information, see <>. - -.. Add the `username` and `password` settings to the {es} output information in -the {metricbeat} configuration file. - -For more information about these configuration options, see -{metricbeat-ref}/elasticsearch-output.html[Configure the {es} output]. --- - -. {metricbeat-ref}/metricbeat-starting.html[Start {metricbeat}] to begin -collecting monitoring data. - -. {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}]. diff --git a/libbeat/docs/monitoring/shared-monitor-config-legacy.asciidoc b/libbeat/docs/monitoring/shared-monitor-config-legacy.asciidoc new file mode 100644 index 00000000000..d00778f8602 --- /dev/null +++ b/libbeat/docs/monitoring/shared-monitor-config-legacy.asciidoc @@ -0,0 +1,138 @@ +////////////////////////////////////////////////////////////////////////// +//// This content is shared by all Elastic Beats. Make sure you keep the +//// descriptions here generic enough to work for all Beats that include +//// this file. When using cross references, make sure that the cross +//// references resolve correctly for any files that include this one. +//// Use the appropriate variables defined in the index.asciidoc file to +//// resolve Beat names: beatname_uc and beatname_lc. +//// Use the following include to pull this content into a doc file: +//// include::../../libbeat/docs/monitoring/shared-monitor-config.asciidoc[] +//// Make sure this content appears below a level 2 heading. +////////////////////////////////////////////////////////////////////////// + +[role="xpack"] +[[configuration-monitor-legacy]] +=== Settings for legacy internal collection + +deprecated::[7.2.0,These settings are deprecated and will be removed in version 8.0.0. Instead of sending monitoring data to your production cluster it's recommended that you use the configuration described under <> to route monitoring data directly to your monitoring cluster.] + +[float] +=== `xpack.monitoring.enabled` deprecated:[7.2] + +The `enabled` config is a boolean setting to enable or disable {monitoring}. +If set to `true`, monitoring is enabled. + +The default value is `false`. + +[float] +=== `xpack.monitoring.elasticsearch` deprecated:[7.2] + +The {es} instances that you want to ship your {beatname_uc} metrics to. This +configuration option contains the following fields: + +[float] +==== `bulk_max_size` + +The maximum number of metrics to bulk in a single {es} bulk API index request. +The default is `50`. For more information, see <>. + +[float] +==== `backoff.init` + +The number of seconds to wait before trying to reconnect to Elasticsearch after +a network error. After waiting `backoff.init` seconds, {beatname_uc} tries to +reconnect. If the attempt fails, the backoff timer is increased exponentially up +to `backoff.max`. After a successful connection, the backoff timer is reset. The +default is 1s. + +[float] +===== `backoff.max` + +The maximum number of seconds to wait before attempting to connect to +Elasticsearch after a network error. The default is 60s. + +[float] +==== `compression_level` + +The gzip compression level. Setting this value to `0` disables compression. The +compression level must be in the range of `1` (best speed) to `9` (best +compression). The default value is `0`. Increasing the compression level +reduces the network usage but increases the CPU usage. + +[float] +==== `headers` + +Custom HTTP headers to add to each request. For more information, see +<>. + +[float] +==== `hosts` + +The list of {es} nodes to connect to. Monitoring metrics are distributed to +these nodes in round-robin order. For more information, see +<>. + +[float] +==== `max_retries` + +The number of times to retry sending the monitoring metrics after a failure. +After the specified number of retries, the metrics are typically dropped. The +default value is `3`. For more information, see <>. + +[float] +==== `parameters` + +Dictionary of HTTP parameters to pass within the url with index operations. + +[float] +==== `password` + +The password that {beatname_uc} uses to authenticate with the {es} instances for +shipping monitoring data. + +[float] +==== `metrics.period` + +The time interval (in seconds) when metrics are sent to the {es} cluster. A new +snapshot of {beatname_uc} metrics is generated and scheduled for publishing each +period. The default value is 10 * time.Second. + +[float] +==== `state.period` + +The time interval (in seconds) when state information are sent to the {es} cluster. A new +snapshot of {beatname_uc} state is generated and scheduled for publishing each +period. The default value is 60 * time.Second. + +[float] +==== `protocol` + +The name of the protocol to use when connecting to the {es} cluster. The options +are: `http` or `https`. The default is `http`. If you specify a URL for `hosts`, +however, the value of protocol is overridden by the scheme you specify in the URL. + +[float] +==== `proxy_url` + +The URL of the proxy to use when connecting to the {es} cluster. For more +information, see <>. + +[float] +==== `timeout` + +The HTTP request timeout in seconds for the {es} request. The default is `90`. + +[float] +==== `ssl` + +Configuration options for Transport Layer Security (TLS) or Secure Sockets Layer +(SSL) parameters like the certificate authority (CA) to use for HTTPS-based +connections. If the `ssl` section is missing, the host CAs are used for +HTTPS connections to {es}. For more information, see <>. + +[float] +==== `username` + +The user ID that {beatname_uc} uses to authenticate with the {es} instances for +shipping monitoring data. + diff --git a/libbeat/docs/monitoring/shared-monitor-config.asciidoc b/libbeat/docs/monitoring/shared-monitor-config.asciidoc index f7dabd8df4e..557465d29ae 100644 --- a/libbeat/docs/monitoring/shared-monitor-config.asciidoc +++ b/libbeat/docs/monitoring/shared-monitor-config.asciidoc @@ -6,16 +6,15 @@ //// Use the appropriate variables defined in the index.asciidoc file to //// resolve Beat names: beatname_uc and beatname_lc. //// Use the following include to pull this content into a doc file: -//// include::../../libbeat/docs/monitoring/configuring.asciidoc[] +//// include::../../libbeat/docs/monitoring/shared-monitor-config.asciidoc[] //// Make sure this content appears below a level 2 heading. ////////////////////////////////////////////////////////////////////////// [role="xpack"] [[configuration-monitor]] -=== Settings for internal monitoring collection +=== Settings for internal collection -Use the following settings to configure internal collection when you are not -using {metricbeat} to collect monitoring data. +Use the following settings to configure internal collection. You specify these settings in the `monitoring` section of the +{beatname_lc}.yml+ config file: @@ -37,7 +36,7 @@ configuration option contains the following fields: The maximum number of metrics to bulk in a single {es} bulk API index request. The default is `50`. For more information, see <>. -==== `backoff.init` +===== `backoff.init` The number of seconds to wait before trying to reconnect to Elasticsearch after a network error. After waiting `backoff.init` seconds, {beatname_uc} tries to @@ -45,79 +44,79 @@ reconnect. If the attempt fails, the backoff timer is increased exponentially up to `backoff.max`. After a successful connection, the backoff timer is reset. The default is 1s. -==== `backoff.max` +===== `backoff.max` The maximum number of seconds to wait before attempting to connect to Elasticsearch after a network error. The default is 60s. -==== `compression_level` +===== `compression_level` The gzip compression level. Setting this value to `0` disables compression. The compression level must be in the range of `1` (best speed) to `9` (best compression). The default value is `0`. Increasing the compression level reduces the network usage but increases the CPU usage. -==== `headers` +===== `headers` Custom HTTP headers to add to each request. For more information, see <>. -==== `hosts` +===== `hosts` The list of {es} nodes to connect to. Monitoring metrics are distributed to these nodes in round robin order. For more information, see <>. -==== `max_retries` +===== `max_retries` The number of times to retry sending the monitoring metrics after a failure. After the specified number of retries, the metrics are typically dropped. The default value is `3`. For more information, see <>. -==== `parameters` +===== `parameters` Dictionary of HTTP parameters to pass within the url with index operations. -==== `password` +===== `password` The password that {beatname_uc} uses to authenticate with the {es} instances for shipping monitoring data. -==== `metrics.period` +===== `metrics.period` The time interval (in seconds) when metrics are sent to the {es} cluster. A new snapshot of {beatname_uc} metrics is generated and scheduled for publishing each period. The default value is 10 * time.Second. -==== `state.period` +===== `state.period` The time interval (in seconds) when state information are sent to the {es} cluster. A new snapshot of {beatname_uc} state is generated and scheduled for publishing each period. The default value is 60 * time.Second. -==== `protocol` +===== `protocol` The name of the protocol to use when connecting to the {es} cluster. The options are: `http` or `https`. The default is `http`. If you specify a URL for `hosts`, however, the value of protocol is overridden by the scheme you specify in the URL. -==== `proxy_url` +===== `proxy_url` The URL of the proxy to use when connecting to the {es} cluster. For more information, see <>. -==== `timeout` +===== `timeout` The HTTP request timeout in seconds for the {es} request. The default is `90`. -==== `ssl` +===== `ssl` Configuration options for Transport Layer Security (TLS) or Secure Sockets Layer (SSL) parameters like the certificate authority (CA) to use for HTTPS-based connections. If the `ssl` section is missing, the host CAs are used for HTTPS connections to {es}. For more information, see <>. -==== `username` +===== `username` The user ID that {beatname_uc} uses to authenticate with the {es} instances for shipping monitoring data. diff --git a/libbeat/docs/security/users.asciidoc b/libbeat/docs/security/users.asciidoc index 1864f90aab3..7d12922db35 100644 --- a/libbeat/docs/security/users.asciidoc +++ b/libbeat/docs/security/users.asciidoc @@ -76,13 +76,7 @@ endif::has_central_config[] [[privileges-to-publish-monitoring]] ==== Privileges needed to publish and view monitoring information -{security} provides built-in users and roles for monitoring. The privileges and -roles needed depend on the method used to collect monitoring data. - -===== Internal collection - -For <>, {security} -provides the +{beat_default_index_prefix}_system+ +{security} provides the +{beat_default_index_prefix}_system+ {stack-ov}/built-in-users.html[built-in user] and +{beat_default_index_prefix}_system+ {stack-ov}/built-in-roles.html[built-in role] for sending monitoring information. You can use the built-in user, or @@ -101,47 +95,6 @@ If you use the +{beat_default_index_prefix}_system+ user, make sure you |`monitoring_user` and `kibana_user` roles |==== -ifndef::serverless[] -===== {metricbeat} collection - -For <>, {security} -provides the `remote_monitoring_user` {stack-ov}/built-in-users.html[built-in -user], and the `remote_monitoring_collector` and `remote_monitoring_agent` -{stack-ov}/built-in-roles.html[built-in roles] for collecting and sending -monitoring information. You can use the built-in user, or -create a user who has the privileges needed to collect and send monitoring -information. - -If you use the `remote_monitoring_user` user, make sure you -<>. - -If you don't use the `remote_monitoring_user` user: - -. Create a user on the production cluster who will collect and send monitoring -information. - -. Assign the following roles to the user: -+ -[options="header"] -|==== -|Role | Why needed? -|`remote_monitoring_collector` -|Collect monitoring metrics from {beatname_uc} -|`remote_monitoring_agent` -|Send monitoring data to the monitoring cluster -|==== - -. Assign the following role to users who will view the monitoring data in -{kib}: - -[options="header"] -|==== -|Role | Why needed? -|`monitoring_user` -|Use *Stack Monitoring* in {kib} to monitor {beatname_uc} -|==== -endif::serverless[] - [[privileges-to-publish-events]] ==== Privileges needed to publish events