[Log] Message field is keyword

[ruflin]

Setting up the custom logs integration in 8.8 for the dataset logs-foo-default makes the message field a keyword instead of a text (match_only_text?) field. This was not the case previously. Is it possible this is related to the switch of an input package?

The default proposed is generic as dataset, so if the custom logs is setup for logs-generic-default, it means the default is now switched to generic.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[jsoriano]

Is it possible this is related to the switch of an input package?

We have been investigating this issue a little bit, installing logs 2.0.0 (input package) and logs 1.1.2 (integration package) in 8.8.0 and we have found some differences on behaviour.

• The integration package installs the logs-log.log index template, that matches logs-log.log-* data streams.
• The input package installs the logs-generic index template, that matches logs-generic-* data streams.

In both cases, logs are sent to logs-generic-default, that matches the mappings provided by the input package, but doesn't match the mappings provided by the integrations package.

Elasticsearch includes a logs index pattern, with lower priority, that matches logs-* and includes a dynamic template for the message field, as match_only_text. This is the index pattern that the integration package ends up using.

We haven't found the reason why the logs integration package doesn't use its own mapping or if this is intentional, this would require more investigation.
But it looks like the behaviour of the input package is correct, it is using the template included in the package.

None of these packages include a mapping for message, nor do any of them include tests, that would have helped detect this.

I guess that the quickest solution to this is to include the expected mappings in the logs package. Apart of that we would need to investigate too why the integration package is using the logs index template, and if this was intentional. And add tests to the package.

[gsantoro]

I can confirm what @jsoriano said about the behaviour of the input package (only that since I have only tested that part).

In the linked PR I have provided a fix using the dynamic ECS templates as suggested at #5055.

The message field mapping is now correct. The side effect is that now there are other ECS fields defined in the index template.

I assume this is ok for a temporary fix

[jsoriano]

We haven't found the reason why the logs integration package doesn't use its own mapping or if this is intentional, this would require more investigation.

Issue created for this, as it appears to cause problems in more situations elastic/kibana#160775