Support for plugin dedicated node

[lezzago]

Is your feature request related to a problem? Please describe.
Currently plugins are utilizing a shared threadpool when it runs its operations. However, plugins are not able to scale up as well in comparison to OpenSearch core as the cluster grows. Plugins need access to more resources, so they can scale out well without affecting the performance of other operations.

Describe the solution you'd like
Similarly to the ML node, there needs to be a framework for users to configure a plugin dedicate node or have just a few plugins running on a single node, so customers can configure their cluster to optimize the performance for their use cases.

[owaiskazi19]

@lezzago did you get a chance to go through #2447? We are working on creating extensions to run independently of OpenSearch cluster.

[dblock]

@lezzago it sounds like you want to install plugins only on some (dedicated) nodes? But some plugins interact with shard data and therefore may need to be on a data node, while other plugins are stateless and will want to be on a search node. Creating a "plugin" node will run into all these exclusions and requirements, and will be very limiting, so I think we need to find a better idea.

Another way to look at the problem is to support installing plugins only on some nodes, and not on all. I propose the plugin framework leverages what we've built for dynamic node labels and creates a plugins.[name].present label at runtime, so requests can then be routed to any node with that label using the existing mechanism. Administrators can then decide how many nodes they want to dedicate to plugins and only install some plugins that support this feature there. WDYT?

(And yes, extensions are already being built with this in mind, maybe we can just use that for your scenarios?)

[lezzago]

Thanks @owaiskazi19 and @dblock, extensions should definitely be able to support the use cases I was thinking of for plugin dedicated nodes.

Another way to look at the problem is to support installing plugins only on some nodes, and not on all. I propose the plugin framework leverages what we've built for dynamic node labels and creates a plugins.[name].present label at runtime, so requests can then be routed to any node with that label using the existing mechanism. Administrators can then decide how many nodes they want to dedicate to plugins and only install some plugins that support this feature there. WDYT?

This way of thought was something closer to what I was initially thinking of and having this type of enhancement would be great since it would allow admins to scale up certain plugins that are known to utilize a lot of resources. For example, there have been users trying to setup 100,000+ monitors in the Alerting plugin and having these resource heavy plugins isolated from other parts of the cluster can really help with the overall performance of their cluster and not negatively impacts other plugins.

[dblock]

@lezzago sounds like we agree :) maybe you want to POC nodes advertising certain plugins and request routing sending requests to those?