[META] OpenSearch Events Correlation Engine

[sbcd90]

Is your feature request related to a problem? Please describe.
OpenSearch is a scalable, flexible, and extensible open-source software suite for search, analytics, and observability applications licensed under Apache 2.0.
OpenSearch includes a data store and search engine where customers can store their business, operational, and security data from a variety of sources & run search queries on them.

Since the various customer infrastructure events, such as security events, observability events etc, spans across multiple indices & data streams, a strong correlation across these indices (or data streams) helps customers to identify patterns and dive into the relationship of events occurring across different systems in their infrastructure.

Describe the solution you'd like
Correlation Engine is an Events Knowledge Graph which can be used to identify and store connected events data spanning across multiple indices or data streams. Also, it helps generate insights by correlating the recent/historical data based on time windows provided by the client .

The Events Correlation Engine provides an approach to help customers correlate events across log sources by allowing customers to define their own Correlation Rules exactly once, while then generating correlations between events from different log sources automatically.

Describe alternatives you've considered
There are no direct alternatives to Events Correlation Engine in OpenSearch today which allows correlations of events across indices based on time windows.

Additional context
More detailed design covered as part of the RFC : #6779

Breaking the changes further into more granular issues for P0 items as below

• 1. Skeleton - Define the skeleton for the core plugin including test setup.

Correlation Query Service

• 1. Correlation Rule Ingestion Layer - APIs to create/update/delete rules and persist in form of Correlation Rule Index[add correlation rule layer for events-correlation-engine #7132 ]

Correlation Service

• 1. Event Ingestion Layer - REST apis to ingest events from dashboard, REST client & Transport layer to ingest events from downstream plugins.
• 2. Join Handler - the Join task determines immediate neighbors of a particular event, given the correlation rules defined by the user for the indices(or data streams) they wish to correlate.
• 4. Insertion Handler - In this layer, events are converted to k-dimensional vectors & are stored in the vector storage layer mentioned above along with their correlations.
• 5. Search Handler - this part of the Correlation Engine allows user to specify a particular event, & then converts it to a k-dimensional vector & then uses it to query its neighboring events which are actually its correlated events within a time window.
• 6. OS/Lucene/HNSW Storage/Query Layer - this is HNSW Graph based storage layer used to store all event vectors & query them at the vector level.
• 7. Index management of Correlation Engine indices