[Query Insights] Add user info in top queries results

[ansjcy]

Is your feature request related to a problem? Please describe

One of the top ask from the query insights users are the ability to identify users that submitted those heavy queries. We have made some experiments in #12529 to read from the thread environment context but after the discussion with the community, we learned that might not be the best solution for getting user related info.

We need to again carefully review what should be the best way to add user related information into top queries - Let's check the Identity service to see what are the gaps to get user info from there.

Describe the solution you'd like

Utilize identity service to add user related info into top queries results.

Related component

Search:Query Insights

Describe alternatives you've considered

No response

Additional context

No response

[ansjcy]

• IdentityService has a function getSubject to further get the application-wide uniquely identifying principal. (https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/identity/IdentityService.java#L50).
• When the feature flag (opensearch.experimental.feature.identity.enabled) is enabled, we load the identity plugins in node.java (https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/node/Node.java#L505).
• Security plugin already implements IdentityPlugin (https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java#L226), but the getSubject function just returns NoopSubject now (https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java#L2021).

So the gap here is, we need to correctly implement different Subject (and other related) classes for different auth methods in security plugin, and also the getSubject function in security plugin to be able to get identity info while using different auth methods, so that we can correctly retrieve the user related information in the query insights plugin. We might need some help from the security team about this.

@peternied Please let me know if this make sense, we can discuss further if we want to prioritize this later :)

[ansjcy]

Part of the work is done as part of #13374, in which we provided certain abilities to send customized labels in a search query as part of the X-Opaque-Id header, and top n queries service will surface that information.

We still need to continue the work on

• [Feature Request][RFC] Multi-tenancy as a construct in OpenSearch #13341
• [FEATURE] Default labeling rule for security based tenancy labeling security#4402

To introduce a separate header dedicated for labels.