Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #16450 to 7.x: [Filebeat] Update fileset input merging to replaces paths and append processors. #16525

Merged
merged 1 commit into from
Feb 24, 2020

Conversation

blakerouse
Copy link
Contributor

Cherry-pick of PR #16450 to 7.x branch. Original message:

What does this PR do?

Updates vendored package of go-ucfg to 0.8.3, needed to use the new field merging adjustments. Updates fileset input merging to replaces paths and append processors when merging configurations.

Why is it important?

Allows a container to define modules that contain processors at the same time as adding extra processors during autodiscovery.

docker run -l 'co.elastic.logs/processors.add_fields={"fields":{"foo":"bar"}}' -l co.elastic.logs/module=elasticsearch busybox echo hello

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • [ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • [ ]

How to test this PR locally

Run filebeat with configuration:

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

output.console:
  pretty: true

Start container:

docker run -l 'co.elastic.logs/processors.add_fields={"fields":{"foo":"bar"}}' -l co.elastic.logs/module=elasticsearch busybox echo hello

Related issues

Use cases

Screenshots

Logs

2020-02-20T06:39:46.264-0500    DEBUG   [autodiscover]  autodiscover/autodiscover.go:190        Generated config: map[audit:map[enabled:true input:map[paths:[/var/lib/docker/containers/fe58462fbaec77346b02d431375b39508c386bc31ef930320a4e82643ca02e93/*-json.log] processors:[map[add_fields:map[fields:map[foo:bar]]]] stream:all type:container]] deprecation:map[enabled:true input:map[paths:[/var/lib/docker/containers/fe58462fbaec77346b02d431375b39508c386bc31ef930320a4e82643ca02e93/*-json.log] processors:[map[add_fields:map[fields:map[foo:bar]]]] stream:all type:container]] gc:map[enabled:true input:map[paths:[/var/lib/docker/containers/fe58462fbaec77346b02d431375b39508c386bc31ef930320a4e82643ca02e93/*-json.log] processors:[map[add_fields:map[fields:map[foo:bar]]]] stream:all type:container]] module:elasticsearch server:map[enabled:true input:map[paths:[/var/lib/docker/containers/fe58462fbaec77346b02d431375b39508c386bc31ef930320a4e82643ca02e93/*-json.log] processors:[map[add_fields:map[fields:map[foo:bar]]]] stream:all type:container]] slowlog:map[enabled:true input:map[paths:[/var/lib/docker/containers/fe58462fbaec77346b02d431375b39508c386bc31ef930320a4e82643ca02e93/*-json.log] processors:[map[add_fields:map[fields:map[foo:bar]]]] stream:all type:container]]]
2020-02-20T06:39:46.264-0500    DEBUG   [autodiscover]  autodiscover/autodiscover.go:252        Got a meta field in the event
2020-02-20T06:39:46.265-0500    DEBUG   [autodiscover]  cfgfile/list.go:62      Starting reload procedure, current runners: 0
2020-02-20T06:39:46.265-0500    DEBUG   [autodiscover]  cfgfile/list.go:80      Start list: 1, Stop list: 0
2020-02-20T06:39:46.271-0500    DEBUG   [processors]    processors/processor.go:101     Generated new processors: add_locale=[format=offset], add_fields={"fields":{"foo":"bar"}}
2020-02-20T06:39:46.271-0500    DEBUG   [input] log/config.go:204       recursive glob enabled
2020-02-20T06:39:46.271-0500    DEBUG   [input] log/input.go:164        exclude_files: [(?-s:.)gz(?-m:$)]. Number of stats: 0
2020-02-20T06:39:46.271-0500    DEBUG   [input] log/input.go:185        input with previous states loaded: 0
2020-02-20T06:39:46.271-0500    INFO    log/input.go:152        Configured paths: [/var/lib/docker/containers/fe58462fbaec77346b02d431375b39508c386bc31ef930320a4e82643ca02e93/*-json.log]

…processors. (elastic#16450)

* Update go-ucfg to 0.8.3 to use the new field merging adjustments. Update fileset input merging to replaces paths and append procesors.

* Add changelog entry.

(cherry picked from commit 9695092)
@blakerouse blakerouse merged commit dcaf7c2 into elastic:7.x Feb 24, 2020
@blakerouse blakerouse deleted the backport_16450_7.x branch May 6, 2020 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants