[Proposal] Consolidate use of ecs.version field

[jsoriano]

We have different sources of ECS fields, and different places where ecs.version can be set. This proposal attempts to define some explicit rules about the expectations of the ecs.version field.

The different sources of ECS fields definitions are, at least:

• Fields imported from ecs using export, where the version used is determined in _dev/build/build.yml.
• Fields imported using import_mappings, with no explicit versioning, but can be combined with export and also uses the definition in _dev/build/build.yml.
• Fields included by Fleet with ecs@mappings component template, since 8.13.

The proposal would be focused on making the final documents to have the greater known ECS version affecting the package, with two main tasks:

• [elastic-package] If a version is set in _dev/build/build.yml, elastic-package system tests must check that ecs.version is set to this version or greater. Package developers may need to add pipeline processors to satisfy this check.
  • Validate version match of ECS in build.yml and generated docs elastic-package#1066
• [Fleet UI] If Fleet includes the ecs@mappings component template, it must also add a processor that sets ecs.version. The value used must be the max version between the latest version of ECS supported by the the stack and the version found in the document, if any. Notice that the versions of ECS and the stack are not fully aligned.

cc @ishleenk17 @zmoog @elastic/ecosystem @kpollich

[mrodm]

Is this issue elastic/elastic-package#1066 similar to the first task mentioned in the description ? @jsoriano

[elastic-package] If a version is set in _dev/build/build.yml, elastic-package system tests must check that ecs.version is set to this version or greater. Package developers may need to add pipeline processors to satisfy this check.

[jsoriano]

Yes! Good finding, I will link this issue here. Thanks.

[ishleenk17]

Thank you for creating the issue. This would be a pre-requisite for having a complete solution for adopting ecs@mappings for Integrations.

[ishleenk17]

• [elastic-package] If a version is set in _dev/build/build.yml, elastic-package system tests must check that ecs.version is set to this version or greater. Package developers may need to add pipeline processors to satisfy this check.

Would this mean that we will not be getting rid of build.yml ?
In case we are using ecs@mappings, we will not need this.

But if we have to overwrite some of the ECS fields (say in case of TSDB), then we might need build.yml?

[jsoriano]

Would this mean that we will not be getting rid of build.yml ?
In case we are using ecs@mappings, we will not need this.

By now build.yml is going to stay in case we need a escape hatch to explicitly import some field. It will be also needed for packages supporting versions of the stack older than 8.13.

But if we have to overwrite some of the ECS fields (say in case of TSDB), then we might need build.yml?

Correct, build.yml is still helpful on this case.

[flash1293]

About the test in elastic-package - nice addition, not sure about the priority as it won't take affect on recent versions of the stack anyway (are we automatically running tests on older stack versions?)

About the second part - makes a lot of sense to me, we need to make sure the fleet-provided pipeline processor runs after the package-provided pipeline so we can "upgrade" an outdated ecs.version property.

[ishleenk17]

About the second part - makes a lot of sense to me, we need to make sure the fleet-provided pipeline processor runs after the package-provided pipeline so we can "upgrade" an outdated ecs.version property.

If we can set it in the Fleet final pipeline, this can be achieved.

• The value used must be the max version between the version of the stack and the version found in the document, if any.

@jsoriano : Say, stack version is 8.14, then the max would be taken as that. But our latest ecs.version is 8.11.
So I think we will not be able to directly take the max of stack or document. Since ecs mappings version and stack version is not the same.

[jsoriano]

our latest ecs.version is 8.11. So I think we will not be able to directly take the max of stack or document. Since ecs mappings version and stack version is not the same.

Interesting, I thought ECS version was more coupled to the stack version. Fleet or some other component of the stack will need to know then what is the version being supported by ecs@mappings.

[flash1293]

@felixbarny do you know how the fleet plugin in Kibana could detect which ecs version ecs@mappings corresponds to?

[felixbarny]

The ECS version of ecs@mappings is always the same as the stack version.

[flash1293]

@felixbarny As @ishleenk17 noted above, the last technical release of https://github.com/elastic/ecs is 8.11, but I guess what you are saying is that if there isn't an explicit ecs release that just means that the new ecs version is identical to the previous one?

Two examples:

• If you are on 8.14, then the version of ecs@mappings is 8.14 which is completely identical to 8.11
• If we would release a new ecs version today, then it would be 8.15 to be aligned with the first stack version that includes it