diff --git a/filebeat/docs/load-balancing.asciidoc b/filebeat/docs/load-balancing.asciidoc index b73059d78de..d890562190c 100644 --- a/filebeat/docs/load-balancing.asciidoc +++ b/filebeat/docs/load-balancing.asciidoc @@ -2,7 +2,7 @@ == Load Balancing Filebeat provides configuration options that you can use to fine -tune load balancing when sending events to multiple hosts. +tune load balancing when sending events to multiple hosts. To enable load balancing, you specify `loadbalance: true` when you configure the output. For example: @@ -32,7 +32,7 @@ The load balancer also supports multiple workers per host. The default is connections will be used. The total number of workers participating in load balancing is `number of hosts * workers`. + -Example: +Example: + [source,yaml] ------------------------------------------------------------------------------- @@ -46,7 +46,7 @@ output.logstash: worker: 2 ------------------------------------------------------------------------------- + -In this example, there are 4 workers participating in load balancing. +In this example, there are 4 workers participating in load balancing. * **Send events to `N` hosts in lock-step:** + @@ -54,7 +54,7 @@ You can configure Filebeat to send events to `N` hosts in lock-step by setting `spool_size = N * bulk_max_size`. In lock-step mode, the batch collected by the spooler is split up into smaller batches of size `bulk_max_size`. These smaller batches are load balanced between available connections. Filebeat waits for all -sub-batches to be published before it retrieves another batch from the spooler. +sub-batches to be published before it retrieves another batch from the spooler. + This mode requires more memory and CPU usage than the previous mode. + @@ -72,26 +72,3 @@ output.logstash: loadbalance: true bulk_max_size: 2048 ------------------------------------------------------------------------------- - -* **Send events in parallel and asynchronously:** -+ -You can configure Filebeat to send events in parallel and asynchronously by -setting `publish_async: true`. With this setting, Filebeat pushes a batch of -lines and then prepares a new batch of lines while waiting for the output to -ACK. This mode can improve load-balancing throughput, but requires the most -memory and CPU usage. -+ -Example: -+ -[source,yaml] -------------------------------------------------------------------------------- -filebeat.prospectors: -- input_type: log - paths: - - /var/log/*.log -filebeat.publish_async: true -output.logstash: - hosts: ["localhost:5044", "localhost:5045"] - loadbalance: true -------------------------------------------------------------------------------- - diff --git a/filebeat/docs/reference/configuration/filebeat-options.asciidoc b/filebeat/docs/reference/configuration/filebeat-options.asciidoc index 4e30b8d133d..ac45ad8e710 100644 --- a/filebeat/docs/reference/configuration/filebeat-options.asciidoc +++ b/filebeat/docs/reference/configuration/filebeat-options.asciidoc @@ -481,6 +481,8 @@ See <> for more information about how this setting affects load ===== publish_async +experimental[] + If enabled, the publisher pipeline in Filebeat operates in async mode preparing a new batch of lines while waiting for ACK. This option can improve load-balancing throughput at the cost of increased memory usage. The default value is false. diff --git a/filebeat/publisher/publisher.go b/filebeat/publisher/publisher.go index eb1a692ec08..07bbf5fefb9 100644 --- a/filebeat/publisher/publisher.go +++ b/filebeat/publisher/publisher.go @@ -6,6 +6,7 @@ import ( "github.com/elastic/beats/filebeat/input" "github.com/elastic/beats/libbeat/common" + "github.com/elastic/beats/libbeat/logp" "github.com/elastic/beats/libbeat/publisher" ) @@ -33,6 +34,7 @@ func New( pub publisher.Publisher, ) LogPublisher { if async { + logp.Warn("Using publish_async is experimental!") return newAsyncLogPublisher(in, out, pub) } return newSyncLogPublisher(in, out, pub)