[Elastic Agent] Change structure of Elastic Agent across the Stack

[ruflin]

In #18758 the decision was made to change the config structure of the Elastic Agent. This issue is to track the changes needed across all parts of the Stack. It is split up in different phases so the changes can be done in small chunks and it can be tested everything still works as expected.

Phase 1 - Remove data source

In Phase 1, the datasources array is removed from the config and all the configs on the datasource level are moved to the input level.
Before:

datasources:
  - namespace: default
    use_output: default
    inputs:
      - type: system/metrics
        streams:
          - metricset: cpu
            dataset: system.cpu

After:

inputs:
 - namespace: default
   use_output: default
   type: system/metrics
   streams:
     - metricset: cpu
       dataset: system.cpu

Changes to be made:

• Update config processing in Elastic Agent [Ingest Manager] New structure of agent configuration #19128
• Update config generation in Ingest Manager (Kibana). Datasources on saved object level should not be removed, only what is generated for the Agent is changed. [Ingest Manager] Replace datasources with inputs when generating agent config kibana#69226

Phase 2 - Adjust dataset naming

Based on elastic/package-registry#482 configs related to datasets should be adjusted

Before

inputs:
 - namespace: default
   use_output: default
   type: system/metrics
   streams:
     - metricset: cpu
       dataset: system.cpu

After

inputs:
 - dataset.namespace: default
   use_output: default
   type: system/metrics
   streams:
     - metricset: cpu
       dataset.name: system.cpu

• Add support for renamed configs on the Elastic Agent side [Ingest Manager] New structure of agent configuration #19128
• Update Kibana generation of configs accordingly [Ingest Manager] Replace datasources with inputs when generating agent config kibana#69226
• Adjust packages in package-storage
• Adjust package-registry and manifest
• Update integrations package generation if needed

Phase 3 - Remove datasources

The naming of datasources should be renamed in Kibana, the package-registry and the packages.

• Rename datasources (SO) in Kibana [Ingest Manager] Rename data sources to package configs kibana#70259
• Rename datasources list in package manifest to config_templatesUpdate package output structure package-registry#514
• Update packages in package-storage to new format
• Update integrations repositories to the new structure Update package output structure package-registry#514
• Update Kibana to support package registry changes [Ingest Manager] Support updated package output structure kibana#69864

Phase 4 - Add support for dataset.type

Currently the dataset.type is inherited from the input.type but this might not always correct. In the future it should be possible to set the dataset.type independently. Below is an example on how this could look (example does not make much sense):

inputs:
 - dataset.namespace: default
   use_output: default
   type: system/metrics
   streams:
     - metricset: cpu
       dataset.name: system.cpu
       dataset.type: logs

• Add support for it on the Elastic Agent side [Ingest Manager] New structure of agent configuration #19128
• Adjust package-registry and manifest as needed Update package output structure package-registry#514
• Add dataset.type to Kibana config generation [Ingest Manager] Support updated package output structure kibana#69864

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[michalpristas]

i think it will be easier to create one big PR which will include all the phases at once.
handling configuration transformation might be sometimes tricky and from my experience it is less work to do the change as a whole. (cc @ph)

splitting this issue into phases is a good thing though and provides a clear implementation plan, thank you for that @ruflin

[ruflin]

@jen-huang How do you see it on the Kibana side to do all changes at once? I think also on the registry and package side it is possible to do the changes in one go, but we will have to temporarily disable some tests to not have to merge red builds.

[ph]

If possible let's reduce the number of PR this might help for the testing too?

cc @EricDavisX for awareness.

[jen-huang]

I think Kibana changes are fine to do in one go as well, and agree with @ph that one PR per repo would make testing easier, and also make merge coordination easier too.

[EricDavisX]

Thanks for the ping. For my 2 cents, I will say that if it impacts teams outside of Ingest we may want to not put them thru the need to adjust their codebases multiple times. But also, It Depends, on if it will take us many weeks (just get it done once?) or 3 or more months (maybe better to do it in more digestable stages) for the phases.

Most helpful is the clear plan we can point teams to and the advanced notice of the pending merges, and good tests on our side to back up that we're making good changes.

[ruflin]

As for KB and Agent it seems doing all at once is easier, lets try to merge all the phases into 1. We should make sure that we coordinates the merges into master but also the backports to 7.x and do them quickly. Because as soon as the changes in the registry are made, it will affect master and 7.x

[jen-huang]

@ruflin I'd like propose moving Phase 3 - Remove datasources out of the initial set of PRs for this. The reason being is that Phase 3 1) has no changes needed on the agent side, 2) has breaking changes in package registry which will affect CI/tests, 3) requires deep changes in Kibana.

We can get the Kibana, Agent, and package registry changes from the other phases in first without worrying about breaking CI. This will get us to the new agent config structure right away for testing.

Then we can spend a bit more time on Phase 3 revisiting the structure of the package manifest and Kibana saved objects. (I recall that we discussed even removing datasources from manifest entirely and generating it on Kibana side?) I started going down the rabbit hole on Kibana side and it seems to me that with data sources going away, our SO mappings can be simplified. Not to mention the sheer amount of variable and component names containing datasource(s)... 😅

The SO mapping changes will also affect how Endpoint creates their config.

I'll have a Kibana PR up shortly with all changes except for Phase 3.

[jen-huang]

Here is the Kibana PR excluding Phase 3: elastic/kibana#69226

[ph]

@jen-huang Yup, I think its a really good changes after all, your proposal sound like a good plan. :)

[ruflin]

This looks great. I also realised if we remove Phase 3 for now, we don't need any changes on the registry and package side yet as all the dataset info is added by Kibana to the config. Or do I miss something here?

Will now test both Agent + Kibana PR together to see if things keep working.

[jen-huang]

After discussion with @ruflin and @michalpristas, we pulled Phase 4 (adding dataset.type) out of the Kibana PR (elastic/kibana#69226). I updated the items in this issue description for Phase 4 accordingly.

[jen-huang]

Hi all, the Kibana changes to support package registry output changes in elastic/package-registry#514 is ready for testing and review: elastic/kibana#69864. I have included an example generated agent config if anyone wants to test with agent standalone.

@michalpristas After discussing with @ruflin, we realized that the dataset.type field makes more sense on the stream level, rather than input level. Does agent support it at the stream level already? You can refer to the config example in my PR above to see the config structure. The package information was also moved under meta.

There is one more Kibana task remaining for this issue: Rename datasources (SO) in Kibana. As this is a deep change to remove datasources from everywhere in Kibana and also includes revising the saved object mappings, I will work on this next separate from the above PR. I created elastic/kibana#70018 to track the work that needs to happen here.

[ruflin]

Tested and works. @michalpristas I assume no "direct" changes are needed to get the Kibana PR in?

[jen-huang]

The renaming of data sources -> package configs Kibana PR has been merged: elastic/kibana#70259

The rest of the unchecked items on this list relates to package registry, but I think some (all?) of them are not really needed or have already been done. @ruflin can you review?

[ruflin]

Closing as all work on this is done.