Convert Filebeat elasticsearch.* to ECS #9293

webmat · 2018-11-29T18:45:24Z

Very little appears to be required for this module.

Caveats

elasticsearch.deprecation, elasticsearch.gc and elasticsearch.server look good already, no need to modify.
Slowlog has two representations for event duration. One is an int in milliseconds, which is what I used to populate event.duration. But if we parse the textual slowlog.took instead, we can get much more precise durations. Outside of the scope of this ECS mapping, however.

Renames

elasticsearch.audit.event_type => event.type
elasticsearch.audit.origin_address => source.ip
elasticsearch.audit.uri => url.original
elasticsearch.audit.request_body => http.request.body
elasticsearch.audit.principal => user.name

Copies

Populate event.duration (ns) without removing slowlog.took and slowlog.took_millis

TODO

Generate a few more `-expected.json` files prior to making changes
Save slowlog.took_millis * 1000000 to event.duration
Make sure to address the http.*.body.content move in ecs-migration and the code as well
- Delete http.request.body from ecs-migration
Alias renamed fields to their ECS counterpart
Document field migrations in ecs-migration.yml
Changelog
Migrate principal to user.name

ruflin · 2018-12-04T07:59:19Z

+1 on the Caveats

webmat · 2018-12-18T21:40:09Z

I'll be able to start on this again once #9645 is merged.

Notes: - Can't be aliased since `body` is moving to `body.content`. - Currently only affects Packetbeat, so it's been listed only there, even if these are ECS field defs. - This will affect the ES Filebeat module logs as well. A note as been added to elastic#9293, so it doesn't get forgotten.

ruflin

@ycombinator You should probably also have a look at this one.

filebeat/module/elasticsearch/audit/test/test.log-expected.json

ycombinator · 2019-01-08T12:38:05Z

Changes LGTM but other related PRs got merged recently so conflicts will need to be resolved on this one. Let me know if you need help with any of those.

webmat · 2019-01-08T15:33:43Z

Yeah I want to get more module migrations going and get those discussions going first. I'll fix these conflicts in a few days

webmat · 2019-01-15T17:51:47Z

jenkins, test this

webmat · 2019-01-15T18:34:32Z

jenkins, test this

ruflin · 2019-01-16T08:54:57Z

jenkins, test this

…dule

- elasticsearch.audit.event_type => event.type - elasticsearch.audit.origin_address => source.ip - elasticsearch.audit.uri => url.original - elasticsearch.audit.request_body => http.request.body

Not removing original, as there's a 1000000x scale difference

webmat · 2019-01-16T15:16:01Z

@ruflin @ycombinator Ready for a final review. All elasticsearch-related tests are successful here, even in Jenkins.

The failing filebeat testsuite in there is actually puzzling to me: all tasks are [success], and it concludes that the test run has failed.

But all of this is obviously unrelated to the ES Filebeat module :-)

Only caveat I'd like confirmation on: right now it would be possible to populate event.duration with nanoseconds precision, but I'd have to parse the textual field slowlog.took. I say we can call that an improvement for later. The important part for FF is to move things to the right fields. Do you agree?

ruflin · 2019-01-16T15:35:25Z

jenkins, test this

ruflin

LGTM but I would really want to see CI pass at least for Filebeat.

webmat · 2019-01-16T16:58:03Z

This last run only had flakiness for libbeat and metricbeat, so we're good.

ruflin · 2019-01-17T12:20:14Z

@webmat Let's get it in. Super unhappy about the CI failures but filebeat is excluded as you mentioned.

Can you also remove the WIP from the PR title?

ruflin mentioned this pull request Nov 29, 2018

Transition Beats to ECS #8655

Closed

webmat self-assigned this Nov 29, 2018

webmat added in progress Pull request is currently in progress. module Filebeat Filebeat ecs labels Nov 29, 2018

webmat mentioned this pull request Dec 7, 2018

Add http.request.body. elastic/ecs#263

Closed

webmat mentioned this pull request Dec 17, 2018

Fix recently translated Filebeat modules where integer coercions were added #9598

Closed

14 tasks

webmat mentioned this pull request Jan 3, 2019

Add two migrated fields to ecs-migration.yml from #9645. #9878

Merged

ruflin reviewed Jan 8, 2019

View reviewed changes

filebeat/module/elasticsearch/audit/test/test.log-expected.json Outdated Show resolved Hide resolved

webmat force-pushed the ecs-fb-elasticsearch branch from 7d03d37 to 78f1b0f Compare January 15, 2019 14:45

webmat requested a review from a team as a code owner January 15, 2019 14:45

webmat force-pushed the ecs-fb-elasticsearch branch from 78f1b0f to 63d4141 Compare January 15, 2019 14:52

webmat requested a review from a team as a code owner January 15, 2019 15:38

webmat added review and removed in progress Pull request is currently in progress. labels Jan 15, 2019

Mathieu Martin added 6 commits January 16, 2019 09:15

Get a few more expected.json logs created prior to modifying the mo…

af4fd92

…dule

Map a few fields from audit log to ECS:

abc238d

- elasticsearch.audit.event_type => event.type - elasticsearch.audit.origin_address => source.ip - elasticsearch.audit.uri => url.original - elasticsearch.audit.request_body => http.request.body

Populate event.duration based on slowlog.took_millis...

645dc8f

Not removing original, as there's a 1000000x scale difference

Pass multiplier for event.duration as a param, to reduce compilations

b827bf0

Alias all fields that were moved

0ec2eea

Document the field transitions in ecs-migration.yml

4c24fe5

Mathieu Martin added 4 commits January 16, 2019 09:15

Changelog

c533310

Transition principal to user.name

ba466a0

Alias .audit.principal to user.name

370d098

Document the principal to user.name alias

1dc24da

webmat force-pushed the ecs-fb-elasticsearch branch from 8a25060 to 1dc24da Compare January 16, 2019 14:15

ruflin approved these changes Jan 16, 2019

View reviewed changes

webmat changed the title ~~WIP Convert Filebeat elasticsearch.* to ECS~~ Convert Filebeat elasticsearch.* to ECS Jan 17, 2019

Changelog part 2

2c7568c

webmat merged commit c4b2ad8 into elastic:master Jan 17, 2019

webmat deleted the ecs-fb-elasticsearch branch January 17, 2019 19:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert Filebeat elasticsearch.* to ECS #9293

Convert Filebeat elasticsearch.* to ECS #9293

webmat commented Nov 29, 2018 •

edited

Loading

ruflin commented Dec 4, 2018

webmat commented Dec 18, 2018

ruflin left a comment

ycombinator commented Jan 8, 2019

webmat commented Jan 8, 2019

webmat commented Jan 15, 2019

webmat commented Jan 15, 2019

ruflin commented Jan 16, 2019

webmat commented Jan 16, 2019

ruflin commented Jan 16, 2019

ruflin left a comment

webmat commented Jan 16, 2019

ruflin commented Jan 17, 2019

Convert Filebeat elasticsearch.* to ECS #9293

Convert Filebeat elasticsearch.* to ECS #9293

Conversation

webmat commented Nov 29, 2018 • edited Loading

ruflin commented Dec 4, 2018

webmat commented Dec 18, 2018

ruflin left a comment

Choose a reason for hiding this comment

ycombinator commented Jan 8, 2019

webmat commented Jan 8, 2019

webmat commented Jan 15, 2019

webmat commented Jan 15, 2019

ruflin commented Jan 16, 2019

webmat commented Jan 16, 2019

ruflin commented Jan 16, 2019

ruflin left a comment

Choose a reason for hiding this comment

webmat commented Jan 16, 2019

ruflin commented Jan 17, 2019

webmat commented Nov 29, 2018 •

edited

Loading