Add guidelines about scaling Beats in Cloud Foundry (#22802)

Move common Cloud Foundry docs to a shared document that is imported
from Filebeat and Metricbeat docs, and add basic guidelines about scaling Beats
in Cloud Foundry.

Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>
Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

filebeat/docs/running-on-cloudfoundry.asciidoc

@@ -1,73 +1 @@
-[[running-on-cloudfoundry]]
-=== Run {beatname_uc} on Cloud Foundry
-
-You can use {beatname_uc} on Cloud Foundry to retrieve and ship logs.
-
-ifeval::["{release-state}"=="unreleased"]
-
-However, version {version} of {beatname_uc} has not yet been
-released, no build is currently available for this version.
-
-endif::[]
-
-
-[float]
-==== Cloud Foundry credentials
-
-{beatname_uc} needs credentials created with UAA so it can connect to loggregator to receive the logs. The `uaac`
-command will create the required credentials for connecting to loggregator.
-
-["source","sh",subs="attributes"]
-------------------------------------------------
-uaac client add {beatname_lc} --name {beatname_lc} --secret changeme --authorized_grant_types client_credentials,refresh_token --authorities doppler.firehose,cloud_controller.admin_read_only
-------------------------------------------------
-
-[WARNING]
-=======================================
-*Use a unique secret:* The `uaac` command shown here is an example. Remember to
-replace `changeme` with your secret, and update the +{beatname_lc}.yml+ file to
-use your chosen secret.
-=======================================
-
-
-[float]
-==== Cloud Foundry deploy manifests
-
-You deploy {beatname_uc} as an application with no route.
-
-Cloud Foundry requires that 3 files exist inside of a directory to allow {beatname_uc} to be pushed. The commands
-below provide the basic steps for getting it up and running.
-
-["source", "sh", subs="attributes"]
-------------------------------------------------
-curl -L -O https://artifacts.elastic.co/downloads/beats/{beatname_lc}/{beatname_lc}-{version}-linux-x86_64.tar.gz
-tar xzvf {beatname_lc}-{version}-linux-x86_64.tar.gz
-cd {beatname_lc}-{version}-linux-x86_64
-curl -L -O https://raw.githubusercontent.com/elastic/beats/{branch}/deploy/cloudfoundry/{beatname_lc}/{beatname_lc}.yml
-# Update api_address, client_id, client_secret in downloaded {beatname_lc}.
-curl -L -O https://raw.githubusercontent.com/elastic/beats/{branch}/deploy/cloudfoundry/{beatname_lc}/manifest.yml
-------------------------------------------------
-
-
-[float]
-==== Deploy
-
-To deploy {beatname_uc} to Cloud Foundry, run:
-
-["source", "sh", subs="attributes"]
-------------------------------------------------
-cf push
-------------------------------------------------
-
-To check the status, run:
-
-["source", "sh", subs="attributes"]
-------------------------------------------------
-$ cf apps
-
-name       requested state   instances   memory   disk   urls
-filebeat   started           1/1         512M     1G
-------------------------------------------------
-
-Log events should start flowing to Elasticsearch. The events are annotated with
-metadata added by the <<add-cloudfoundry-metadata>> processor.
+include::{libbeat-dir}/shared-cloudfoundry.asciidoc[]

libbeat/docs/shared-cloudfoundry.asciidoc

@@ -0,0 +1,128 @@
+[[running-on-cloudfoundry]]
+=== Run {beatname_uc} on Cloud Foundry
+
+ifeval::["{beatname_lc}"=="filebeat"]
+You can use {beatname_uc} on Cloud Foundry to retrieve and ship logs.
+endif::[]
+ifeval::["{beatname_lc}"=="metricbeat"]
+You can use {beatname_uc} on Cloud Foundry to retrieve and ship metrics.
+endif::[]
+
+ifeval::["{release-state}"=="unreleased"]
+
+However, version {version} of {beatname_uc} has not yet been
+released, no build is currently available for this version.
+
+endif::[]
+
+
+==== Create Cloud Foundry credentials
+
+To connect to loggregator and receive the logs, {beatname_uc} requires credentials created with UAA. The `uaac`
+command creates the required credentials for connecting to loggregator.
+
+["source","sh",subs="attributes"]
+------------------------------------------------
+uaac client add {beatname_lc} --name {beatname_lc} --secret changeme --authorized_grant_types client_credentials,refresh_token --authorities doppler.firehose,cloud_controller.admin_read_only
+------------------------------------------------
+
+[WARNING]
+=======================================
+*Use a unique secret:* The `uaac` command shown here is an example. Remember to
+replace `changeme` with your secret, and update the +{beatname_lc}.yml+ file to
+use your chosen secret.
+=======================================
+
+
+==== Download Cloud Foundry deploy manifests
+
+You deploy {beatname_uc} as an application with no route.
+
+Cloud Foundry requires that 3 files exist inside of a directory to allow {beatname_uc} to be pushed. The commands
+below provide the basic steps for getting it up and running.
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+curl -L -O https://artifacts.elastic.co/downloads/beats/{beatname_lc}/{beatname_lc}-{version}-linux-x86_64.tar.gz
+tar xzvf {beatname_lc}-{version}-linux-x86_64.tar.gz
+cd {beatname_lc}-{version}-linux-x86_64
+curl -L -O https://raw.githubusercontent.com/elastic/beats/{branch}/deploy/cloudfoundry/{beatname_lc}/{beatname_lc}.yml
+curl -L -O https://raw.githubusercontent.com/elastic/beats/{branch}/deploy/cloudfoundry/{beatname_lc}/manifest.yml
+------------------------------------------------
+
+You need to modify the +{beatname_lc}.yml+ file to set the `api_address`,
+`client_id` and `client_secret`.
+
+
+==== Deploy {beatname_uc}
+
+To deploy {beatname_uc} to Cloud Foundry, run:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+cf push
+------------------------------------------------
+
+To check the status, run:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+$ cf apps
+
+name       requested state   instances   memory   disk   urls
+{beatname_lc}   started           1/1         512M     1G
+------------------------------------------------
+
+ifeval::["{beatname_lc}"=="filebeat"]
+Log events should start flowing to Elasticsearch.
+endif::[]
+ifeval::["{beatname_lc}"=="metricbeat"]
+Metric events should start flowing to Elasticsearch.
+endif::[]
+The events are annotated with metadata added by the <<add-cloudfoundry-metadata>> processor.
+
+==== Scale {beatname_uc}
+
+A single instance of {beatname_uc} can ship more than a hundred thousand events
+per minute. If your Cloud Foundry deployment is producing more events than
+{beatname_uc} can collect and ship, the Firehose will start dropping events, and it
+will mark {beatname_uc} as a slow consumer. If the problems persist, {beatname_uc} may
+be disconnected from the Firehose.
+In such cases, you will need to scale {beatname_uc} to avoid losing events.
+
+The main settings you need to take into account are:
+
+ifeval::["{beatname_lc}"=="filebeat"]
+* The `shard_id` specified in the
+  <<filebeat-input-cloudfoundry,`cloudfoundry` input configuration>>. The
+  Firehose will divide the events amongst all the {beatname_uc} instances with
+  the same value for this setting. All the instances with the same `shard_id`
+  should have the same configuration.
+endif::[]
+ifeval::["{beatname_lc}"=="metricbeat"]
+* The `shard_id` specified in the
+  <<metricbeat-module-cloudfoundry,`cloudfoundry` module>>. The
+  Firehose will divide the events amongst all the {beatname_uc} instances with
+  the same value for this setting. All instances with the same `shard_id`
+  should have the same configuration.
+endif::[]
+* Number of {beatname_uc} instances. When {beatname_uc} is deployed as a Cloud
+  Foundry application, it can be scaled up and down like any other application,
+  with `cf scale` or by specifying the number of instances in the manifest.
+* <<configuring-output,Output configuration>>. In some cases, you can fine-tune
+  the output configuration to improve the events throughput. Some outputs
+  support multiple workers. The number of workers can be changed to take better
+  advantage of the available resources.
+
+Some basic recommendations to adjust these settings when {beatname_uc} is not
+able to collect all events:
+
+* If {beatname_uc} is hitting its CPU limits, you will need to increase the
+  number of {beatname_uc} instances deployed with the same `shard_id`.
+* If {beatname_uc} has some spare CPU, there may be some backpressure from the
+  output. Try to increase the number of workers in the output. If this doesn't
+  help, the bottleneck may be in the network or in the service receiving the
+  events sent by {beatname_uc}.
+* If you need to modify the memory limit of {beatname_uc}, remember that CPU
+  shares assigned to Cloud Foundry applications depend on the configured memory
+  limit. You may need to check the other recommendations after that.

metricbeat/docs/running-on-cloudfoundry.asciidoc

@@ -1,73 +1 @@
-[[running-on-cloudfoundry]]
-=== Run {beatname_uc} on Cloud Foundry
-
-You can use {beatname_uc} on Cloud Foundry to retrieve and ship metrics.
-
-ifeval::["{release-state}"=="unreleased"]
-
-However, version {version} of {beatname_uc} has not yet been
-released, no build is currently available for this version.
-
-endif::[]
-
-
-[float]
-==== Cloud Foundry credentials
-
-{beatname_uc} needs credentials created with UAA so it can connect to loggregator to receive the logs. The `uaac`
-command will create the required credentials for connecting to loggregator.
-
-["source","sh",subs="attributes"]
-------------------------------------------------
-uaac client add {beatname_lc} --name {beatname_lc} --secret changeme --authorized_grant_types client_credentials,refresh_token --authorities doppler.firehose,cloud_controller.admin_read_only
-------------------------------------------------
-
-[WARNING]
-=======================================
-*Use a unique secret:* The `uaac` command shown here is an example. Remember to
-replace `changeme` with your secret, and update the +{beatname_lc}.yml+ file to
-use your chosen secret.
-=======================================
-
-
-[float]
-==== Cloud Foundry deploy manifests
-
-You deploy {beatname_uc} as an application with no route.
-
-Cloud Foundry requires that 3 files exist inside of a directory to allow {beatname_uc} to be pushed. The commands
-below provide the basic steps for getting it up and running.
-
-["source", "sh", subs="attributes"]
-------------------------------------------------
-curl -L -O https://artifacts.elastic.co/downloads/beats/{beatname_lc}/{beatname_lc}-{version}-linux-x86_64.tar.gz
-tar xzvf {beatname_lc}-{version}-linux-x86_64.tar.gz
-cd {beatname_lc}-{version}-linux-x86_64
-curl -L -O https://raw.githubusercontent.com/elastic/beats/{branch}/deploy/cloudfoundry/{beatname_lc}/{beatname_lc}.yml
-# Update api_address, client_id, client_secret in downloaded {beatname_lc}.
-curl -L -O https://raw.githubusercontent.com/elastic/beats/{branch}/deploy/cloudfoundry/{beatname_lc}/manifest.yml
-------------------------------------------------
-
-
-[float]
-==== Deploy
-
-To deploy {beatname_uc} to Cloud Foundry, run:
-
-["source", "sh", subs="attributes"]
-------------------------------------------------
-cf push
-------------------------------------------------
-
-To check the status, run:
-
-["source", "sh", subs="attributes"]
-------------------------------------------------
-$ cf apps
-
-name         requested state   instances   memory   disk   urls
-metricbeat   started           1/1         512M     1G
-------------------------------------------------
-
-Metrics should start flowing to Elasticsearch. The events are annotated with
-metadata added by the <<add-cloudfoundry-metadata>> processor.
+include::{libbeat-dir}/shared-cloudfoundry.asciidoc[]