The component run directory must be copied on upgrade

[cmacknz]

This is a follow up from #2050 where we discovered that the agent never properly migrated component state (like the Filebeat registry) on upgrade. This results in unnecessary log duplication on upgrade.

To properly fix this in the new V2 directory structure the contents of the component run directory must be copied from data/elastic-agent-$old_version_hash to data/ealstic-agent-$new_version_hash.

For example here the run directory structure for a Filebeat component started to run the log input type in 8.6.0:

/Library/Elastic/Agent/data/elastic-agent-123730/run/log-default/
├── meta.json
└── registry
    └── filebeat
        ├── log.json
        └── meta.json

The entire contents of /run/log-default need to be copied to the data directory of the new version on upgrade. Copying the contents ensures that the previous version can resume from the point prior to the upgrade if a downgrade is necessary without backwards compatibility issues.

Acceptance Criteria:

• The contents of the data/elastic-agent-$hash/run/ directory are copied on upgrade
• Log duplication is not observed after the upgrade using a test like the one described in Upgrade to 8.6 is not migrating persisted agent application state from pre-8.6 #2050 (comment)
• Downgrades can be performed successfully and can resume log ingestion from the point before the upgrade (in this case data duplication of lines added after the upgrade is expected).

[lucabelluccini]

Hello @cmacknz

How one would perform a downgrade if the upgrade is successful?
I've noticed the old path is not wiped if the upgrade is successful, but we do not document until when it's kept and how one would perform a downgrade properly.
Also, already indexed duplicated documents would be tough to identify.

[cmacknz]

How one would perform a downgrade if the upgrade is successful?

A downgrade is just an upgrade to a lower version if that makes sense to you. The agent is just changing a symlink to an active elastic-agent-$hash directory where $hash is the Git SHA of the target agent version. Fleet doesn't support downgrades however, only upgrades.

I've noticed the old path is not wiped if the upgrade is successful, but we do not document until when it's kept and how one would perform a downgrade properly.

The old version is removed when the upgrade succeeds, which is sometime after the new version has been running without issue. This is considered an implementation detail so it isn't something we document.

Also, already indexed duplicated documents would be tough to identify.

Yes, I suspect this is why we don't have many active reports of this problem. The agent never properly copied the Filebeat registry in any previous version.