[Meta] disk queue journey to GA

[leehinman]

Checklist to achieve experimental status

• add disk queue configuration to shipper configuration #119
• Add option to supply disk queue encryption key #120
• enrich diskqueue configuration with encryption_password when disk queue is encryption is enabled elastic-agent#1656
• Add disk queue options and operation to documentation #122
• The disk queue must assign an auto-incrementing ID to each element #27

Checklist to achieve beta status

• [Fleet] Add option to enable disk queue for elastic-agent-shipper kibana#141508
• Verify keystore is maintained during elastic-agent upgrades #123
• [Meta] Implement shipper performance testing #57
• Add disk queue metrics to shipper metrics #125
• Add explicit memory usage cap for the disk queue beats#30069
• Correctly handle enqueued events affected by agent policy changes #49

Checklist to achieve ga status

• [Fleet] Add feature so integrations can require elastic-agent-shipper disk queue kibana#141510
• [Fleet] Update agent dashboard visualization with elastic-agent-shipper queue metrics kibana#141512

[mbudge]

Wouldn't it be more efficient to use in-memory queues which temporarily fall back to on-disk queues if there's a network issue?

When the connectivity issue is resolved, push the events from the on-disk queue back to the output integration. I've built software which did this using channels in Go to add redundancy by caching data to disk. This was done when there was a problem writing to elastic using the go-elasticsearch bulk indexer and it worked well. The software would periodically check connectivity to the cluster and send the events when there was no longer an issue.

This could be done by having an option to flush the in-memory queue to the on-disk queue instead of dropping the events. This would reduce the resources required to run on-disk queues on busy production servers 24/7. It sounds like on disk queues on busy domain controllers or other critical systems might use more resource which could trigger out IT teams to complain. It's too complicated to have separate policies in large multi-regional deployments to have in-memory queues on a subset of endpoints.

[cmacknz]

It's a good idea, but exploring that needs to come after making the disk queue available at all. Today there is definitely a performance penalty to write to disk as it requires serializing the event twice (once to disk, once to the output) but we haven't thoroughly quantified what it is. We may also be able to improve that cost.