[Filebeat] Changes to text fields in elasticsearch module

[ycombinator]

This PR is an offshoot of conversations and decisions made in #10372 w.r.t text fields, but scoped to the elasticsearch module.

[elasticmachine]

Pinging @elastic/stack-monitoring

[ycombinator]

jenkins, test this

[ruflin]

Changelog?

[ruflin]

One thing I spot now: Below the message fields seems to contain everything including timestamp. Is that expected?

For the took. The plan is to extract the unit later or just remove it?

[ycombinator]

I didn't follow your question about the message field?

For the took, I think we should just the unit as we already have the actual numeric value in ms, which seems more useful? BTW, this is something we're doing in the case of the elasticsearch/server fileset as well:

beats/filebeat/module/elasticsearch/server/ingest/pipeline.json

Line 52 in 9d8de3e

"elasticsearch.server.gc.collection_duration.unit",

[webmat]

The point about message is that once the header has been parsed and moved to other fields, only the rest of the message should be left in message.

Right now, the message fields looks like an event.original, which is meant to contain the pristine initial copy of the event (if the user wants this).

For the took field, I would keep it until we're ready to interpret it, no? Right now event.duration is based on the took_millis (removed by the pipeline), for simplicity, which is often rounded to 0ms. Elasticsearch is too fast. But I think it's worth keeping the custom field with the more precise timing until we can leverage it better (or until the ES log format gives us nanoseconds).

[webmat]

I'm good with the text field changes.

However I'd keep the took custom field.

[ycombinator]

Alrighty, brought back the took field, this time as a keyword. @webmat @ruflin this is ready for another round of review, when you get a chance. Thanks!

[webmat]

Code LGTM

You just need to update fields.go accordingly :-)

[ycombinator]

jenkins, test this

[ycombinator]

jenkins, test this