Migrate to ECS Filebeat modules populating http size and duration metrics

[webmat]

This PR finishes the migration of http size and time metrics for all relevant Filebeat modules. This had already started happening in the more recent module transitions. This PR finishes up the work all at once.

Questions

• In all module transitions so far, I've left the module-specific total request time fields alone. I think we can get rid of them, as event.duration should be the single place to record the overall duration. I would remove them, and mark then in ecs-migration.yml as migrated, with alias: false. Any disagreements?
  • Note that any additional timing metrics such as various wait times are unaffected by this.
• Take a look at how I'm reducing Painless compilations across access log modules, when calculating event.duration. I think it's worth doing a follow-up PR to spread this practice around, and reduce the impact of calculating event.duration across all Fb modules even further. Objections?

Modules/filesets affected

• apache.access (apache2.access.body_sent.bytes)
• iis.access (iis.access.body_sent.bytes, iis.access.body_received.bytes, iis.access.request_time_ms)
• kibana.log (http.response.content_length)
• nginx.access (nginx.access.body_sent.bytes)
• traefik (traefik.access.body_sent.bytes, traefik.access.duration)

List of aliased field migrations

response size

• http.response.content_length => http.response.body.bytes (Kibana module)
• apache2.access.body_sent.bytes => http.response.body.bytes
• iis.access.body_sent.bytes => http.response.body.bytes
• nginx.access.body_sent.bytes => http.response.body.bytes
• traefik.access.body_sent.bytes => http.response.body.bytes

request size

• iis.access.body_received.bytes => http.request.body.bytes

Event duration

No aliases, as scale is different. Original fields removed.

• http.response.elapsed_time (Kibana module)
• iis.access.request_time_ms
• traefik.access.duration

TODO

• Change all HTTP size metrics
• Compute all event durations
• Look for other usage of http.response.content_length
• Look for other usage of http.response.elapsed_time
• Alias the fields for the migration
• Document in ecs-migration.yml
  • Delete any other web access event durations, and document as alias: false in ecs-migration.yml as well
• Changelog

[webmat]

@ruflin I'd like your opinion on the two questions outlined in the PR body. For both questions, you can look at the code, because I went ahead and did what I'm suggesting. Open to discussion, however.

This is pretty much ready to go, except for the changelog.

[webmat]

Unrelated test failure (heartbeat)

[ruflin]

+1 on on the painless part. For the event.duration not 100% sure I can follow. So we map the existing fields to event.duration or not use it at all?

[webmat]

@ruflin Ok ready for another review, including for both caveats. Now that my painless is working, the caveats will make more sense :-)

[cachedout]

From the @elastic/stack-monitoring perspective this LGTM but let's see if @ycombinator can come by and just verify that as well.

[ycombinator]

We should mention the removal of this field these two fields in the breaking changes section of the CHANGELOG.

[webmat]

Yes, agreed. I've often glossed over the details, for the module-specific fields (e.g. "migrate apache.access.*").

But these two deserve a special mention indeed, since they could have been mistaken for ECS fields.

[ycombinator]

Just a minor comment about mentioning breaking changes in the CHANGELOG. Otherwise LGTM.

[webmat]

Changelog too chatty? Rest of the PR is approved and hasn't been touched.

[webmat]

Rebased over master.

I'm starting another PR on top of this one, to finish the event.duration migration in other modules the same way I'm doing here.

[ruflin]

For the Changelog: If we document every single field we changed, it definitively gets too chatty. What I have in mind is that we will have a doc page with all the fields that we changed so users can go there to look it up.

[webmat]

@ruflin Yeah we should definitely have a generated page that explains "the ECS migration" in plain terms, and is followed by the exhaustive list of fields migrated (aliases, durations that moved to event.duration with a different scale, etc).

Then the changelogs related to ECS could be simplified, and all link to that "ECS migration" documentation page.

I'll rewrite the changelog a bit, then we should be good.

[webmat]

Ok, pushed one changelog entry per field (duration and body.bytes). Merging right away. Hit me up for a follow-up PR, if these are still problematic.