Support dynamic node role

[ylwu-amzn]

OpenSearch doesn't support ML node role currently, check DiscoveryNodeRole.java, no ML role defined.

We need dedicated ML node for ML plugin

We have released a new ML plugin ml-commons in 1.3 and we are planning to adding more models.

ML model generally consuming more resources, especially for training process. We are going to support bigger ML models which might require more resources and special hardware like GPU. As OpenSearch doesn’t support ML node, we dispatch ML task to data node only. That means if user want to train some big model, they need to scale up all data nodes which seems costly and not reasonable. If we can support dedicated ML node, user don’t need to scale up their data node at all, just need to configure a new ML node(with different settings, more powerful instance type) and add it to cluster. And we can separate resource usage better by running ML task on dedicated node which can reduce impact to other tasks like search/ingestion.

What changes we need to make

OpenSearch

• Add new node type ml in DiscoveryNodeRole.java
• Support configuring ML node role in opensearch.yml
• Security: should be ok as ML node communicate with other nodes via transport action.

[Updated solution]: We plan to support user configuring any node role name in opensearch.yml file. Node will read the unknown roles as dynamic role rather than break node start process. Dynamic role abbreviations will be the same with full name. To avoid confusion, we enforce the node role full name and abbreviation case-insensitive(must be lower-case). We will return all role's full name including both built-in and dynamic roles in a new field node.roles when user call _cat/nodes API. Check more details in discussions below.

ML plugin

• If ML node exists, dispatch ML task to ML node; otherwise, dispatch to data node. Support dedicated ml node ml-commons#79

[dblock]

I like the problem that is being solved here (some tasks may compete/work better on specialized hardware and provisioning only some nodes to be GPU-enabled vs. others not be GPU enabled is cheaper) and understand what we're trying to accomplish (send certain tasks to certain nodes).

My suggestion is to generalize this to avoid solving this problem for each type of workload.

This can be treated as a request routing problem. For example, I'd like to be able to direct all search requests to search-only nodes (separate read/write). In #2859 I'd like to route requests in a weighted round robin. In #2447 I'd like not to have to install plugins on all nodes, and thus isolate execution between plugins across different hardware nodes (same as only some nodes are "ml" nodes because they have the "ml" plugin installed).

I think the generic version of this is to 1) make node roles dynamic (vs. hard-coded in Java), 2) make nodes advertise roles at runtime and be able to drop/add roles without restarts, 3) allow plugins to register and advertise roles at runtime, 4) enable routing to nodes that support a set of roles.

It doesn't have to be an existing "role" concept, it could be a new "capability" or "tag" concept.

If we implement this generically, a node that has the ML plugin installed would advertise a "plugin:ml" role and requests can be routed to it. A node that has a specific kind of GPU could advertise "os:gpu" and "os:gpu:NVIDIA T4". Round robin routing could then be policy-based with potential fallback (e.g. prefer nodes with "os:gpu" at 2x weight). Will need a way to say that an ML workload must be routed to an ML-capable node, but some other workload has a preference of ML-capable node and can fallback to a non-ML-capable data node.

[penghuo]

Two questions,

1. Does ML node contains index data?
2. Do we consider term compute node instead of ml node. In my option, ML algorithms is a type of computation task. but we could schedule any computation task on compute node, e.g. OpeSearch pipeline aggregation could execute distributively on compute node.

[jainankitk]

Can you add some more details on ML node interaction with storage layer? More specifically:

How is the ML node handled from shard allocation perspective? If ML nodes don't have any shards like master, do they fetch data from other nodes while processing ML task? Or we temporarily relocate the shard to ML nodes?

[reta]

It doesn't have to be an existing "role" concept, it could be a new "capability" or "tag" concept.

I like the idea of introducing the capabilities (fe search, ingest, data, ...). In this case the built-in roles could be assembled out of capabilities (to support existing features) but the routing / placement policies could rely on capabilities instead. For some capabilities we still need the support in core (fe could node store data or not), but others could be added dynamically (like gpu, ssd, ...) at any time.

[ylwu-amzn]

@penghuo Thanks for the question/suggestion

1. Does ML node contains index data?

Node can have 1 or multiple node roles. If a node has only "ml" node role, it won't store index data (no shard on it). If a node has both "data" and "ml" node roles, shard will be allocated to it. Basically only allocate shard to node which has "data" node role.

2. Do we consider term compute node instead of ml node. In my option, ML algorithms is a type of computation task. but we could schedule any computation task on compute node, e.g. OpeSearch pipeline aggregation could execute distributively on compute node.

Good point. I think it's good to use a more generic role name like "computation", so any computation-intensive tasks can run on "computation" nodes.

[ylwu-amzn]

Can you add some more details on ML node interaction with storage layer? More specifically:

How is the ML node handled from shard allocation perspective? If ML nodes don't have any shards like master, do they fetch data from other nodes while processing ML task? Or we temporarily relocate the shard to ML nodes?

@jainankitk thanks, this is a good question.
We are not going to change the shard allocation part, at least for the short-term view. It's possible that we meet data transfer bottleneck in future, then we may make some improvement like move ML model closer to data(like run model on shard level) or move data closer to ML model (transfer data to ML node). But this is not one-way-door. We can make continuous improvement and the improvement could be in ML plugin rather than OpenSearch core, unless that improvement is general and will benefit other use cases.

[ylwu-amzn]

@reta @dblock thanks for your suggestion. Like this

It doesn't have to be an existing "role" concept, it could be a new "capability" or "tag" concept.

After reading all comments and brainstorming possible solutions, I think we have multiple options to support dedicated node to run specific tasks, and ML task is just one example, maybe we can support other special tasks like image processing.

The main reasons why we need dedicated nodes for specific tasks like ML are

1. ML task uses shared resource of data nodes which may impact core functions. We can limit the memory usage and thread pool of ML tasks, but still possible to impact core function like searching and indexing.
2. Costly to scale. If some ML tasks needs special hardware like GPU, we need to scale up all data nodes which is costly and unreasonable.

We may have these options to solve the problems. Welcome any more suggestions/options.

Option1: add new node role

OpenSearch doesn't support ML node role currently, check DiscoveryNodeRole.java , no ML role defined.
Add a new role for ML/computation-intensive tasks like “ml” or more generally “computation”. Prototype code change link . I started a multi-node cluster (1ML node, 2 data nodes, 3 master nodes) and it works well, all ML tasks routes to ML node and no shard allocated to ML node.

Pros:

1. Less effort. Just leverage current node role framework.
2. ML node doesn't has shards. ML node failure has no impact on cluster state.

Cons:

1. Node roles are configured in opensearch.yml file. User can’t change node role on the fly.
2. If we have a new task type which needs to run on its own dedicated node in future, we need to add a new node role. But that depends on how possible it looks that we are going to add many new task types and if the new task types can use the general "computation" node role.

Option 2: enhance current node role framework

We keep some reserved/predefined roles like master/data/ingest. Enhance current node role framework to support any custom role and can assign/change node role on the fly.

Need to consider some challenges:

1. If change ML node role to data node role, shard will be allocated to it. If change data node role to ML, shard will be relocated to other data nodes. Shard moving between nodes may cause cluster performance down or unstable state. We can add some limitation like user can’t change role between data role with other roles, or user can’t change from/to reserved roles like master/data.
2. Assume one node role is dedicated for some kind of task, for example “ML” node role for ML tasks, “image_processing” role is for processing images tasks. If we change a node role from “ML” to “image_processing”, the ML tasks on it must stop immediately and maybe migrate to other ML node. This flexibility is good to let user repurpose node efficiently, but also risky to stop some running tasks especially if the task doesn’t support migration to other node. So user should be careful about changing node roles.
3. We need to persist the node role information somewhere (like OpenSearch index, or some local state file), so node will pick up correct latest node role after restarting. Compared with the static opensearch.yml file, we need to pay more attention on data consistency.

Pros:

1. Support assigning/changing node role on the fly. User can repurpose node easily.

Cons

1. There are some challenges to solve.
2. User needs to pay attention when assign/change node role on the fly. More flexibility, also more risk.

Option 3: keep current node role framework and add new node tag/capability

We build a brand new concept: "capability" or "tag". For easier discussion, here I just use "tag".

This option doesn’t change node role, so it’s safe for shard allocation and cluster management. But still have same second&third challenge of Option2.

Pros:

1. No impact to current node role
2. Support assigning/changing node tags on the fly.

Cons:

1. If a cluster only has master or data nodes. Adding new node tag can’t separate ML task from data node, it still uses shared resources of data nodes.

Option 4: get rid of current node role framework, build brand new node tag/capability framework

This options looks similar to option2, just name role as "tag"/"capability". We may think what other feature we can build in this new framework.

Pros:

1. Flexible. We can build any new features in this new framework.

Cons:

1. BWC: support old node roles in previous versions.
2. Same challenges of Option2

My conclusion by comparing these options

After comparing these options, I think Option1 is the easiest, and Option2 and Option3 will support more use case like assigning/changing node role/tag on the fly, but also bring more challenges. Option4 is something similar to Option2, but we may have more flexibility to build new features.

So, I wound suggest follow Option1 for short-term goal. And we can support Option2 later so user have flexibility to change node roles easily with some limitation (fe, can't change data roles). I think Option3 and Option4 are some long term solution, we can follow these options if we have some user cases which can't be solved by node role framework.

[dblock]

Option 1 solves a super narrow use-case, and its main advantage is "less effort". Is the prototype misleading? It doesn't say how calls will be routed to that node. I imagine you'll modify ml-commons/k-nn to say "route this to an ML node" - do you have that code? What happens if I don't have any nodes in my cluster with such role? Will that node sit idle when it's not busy? Does introducing a new node type have other impact, such as upgrades?

Is there a simple version of option 2 where there's no API to change node roles on the fly, but you can define custom node roles in the node's opensearch.yml and have the ML plugin prefer routing requests to a node that has the "ml" role?

node.roles: [ data, ingest, ml, image_processing, fizzbuzz ]

[reta]

Is there a simple version of option 2 where there's no API to change node roles on the fly, but you can define custom node roles in the node's opensearch.yml and have the ML plugin prefer routing requests to a node that has the "ml" role?

I think this would be the simplest starting point, we could also bake in the "dynamic" nature of roles (on the fly) for all custom ones but not for builtin ones (those deemed "static").

[ylwu-amzn]

Thanks @dblock .

I imagine you'll modify ml-commons/k-nn to say "route this to an ML node" - do you have that code?

Yes, in my prototype we need to filter out ml nodes, then route request. This is the code change ylwu-amzn/ml-commons@6883925.

What happens if I don't have any nodes in my cluster with such role?

That depends on plugin's logic. In my prototype, if ml node doesn't exist, will route request to data nodes. This prototype just demonstrate we can route request to ml node only if it exists. We may change this later, like only route request to ml nodes, and will throw exception if no ml node.

Will that node sit idle when it's not busy?

Yes, that's true. We should explain this in our documentation. If user don't run heavy ML tasks, maybe just run it on data node is enough. We need more experiments to find out the best configuration.

Does introducing a new node type have other impact, such as upgrades?

When upgrade, we need to consider BWC for the data layer and code logic. For data layer, the shard only allocated on data nodes, should be ok. For code logic, each plugin should consider BWC for their own case especially for mixed cluster. For ml-commons, ML tasks on old version routes to data nodes, so old tasks still consumes shared resource on data nodes, but new tasks only routes to ml nodes, both tasks can run independently as we don't have distributed model run on both data and ml nodes.

Is there a simple version of option 2 where there's no API to change node roles on the fly, but you can define custom node roles in the node's opensearch.yml and have the ML plugin prefer routing requests to a node that has the "ml" role?

Agree with @reta , this is the simplest and we can add "dynamic" nature for custom roles later (this is not one-way-door). Actually this is what I mean in Option1, In my prototype, I configured ml roles in opensearch.yml file, like this

node.name: ml-node-1
node.roles: [ ml ]

And add ml node role in OpenSearch core, check code change in my repo ylwu-amzn@b0915f4

[dblock]

Both @reta and I are saying that there's no public static final DiscoveryNodeRole ML_ROLE = in code, which is not option 1. Want to try to code the slightly more dynamic version @ylwu-amzn?

[ylwu-amzn]

there's no public static final DiscoveryNodeRole ML_ROLE = in code

Got it. If we don't add public static final DiscoveryNodeRole ML_ROLE = in code, we need to read node role from opensearch.yml dynamically and set it when node start. Looks more dynamic than option1 and less than option2. Sounds good to me. Thanks @dblock @reta . Let's keep this topic open for a while to check if others have any concern.

[xinlamzn]

Looks like we are aligned on the modified option #2. Could we summarize the proposal?

Also, what is the impact of the ML node failure to the cluster status? If a node is configured to be ML-only and failure, is there a signal on the cluster state? We need a generic way to detect any node failure.

[dblock]

Also, what is the impact of the ML node failure to the cluster status? If a node is configured to be ML-only and failure, is there a signal on the cluster state? We need a generic way to detect any node failure.

I think cluster state needs to get smarter. Naively, I would do this (bold would be new):

RED: some or all of (primary) shards are not ready, no available ml nodes
YELLOW: all of the primary shards allocated, but some/all of the replicas have not been allocated, some ml nodes down
GREEN: fully operational, all ml nodes up

There's a chicken and egg problem with this approach because the cluster doesn't know anything about ml nodes until a node dynamically publishes such a type. It doesn't know the total number of ml nodes, nor does it know whether the number is sufficient to execute tasks. So I think this needs to queue off actual tasks that require an ml node.

RED: some or all of (primary) shards are not ready, there are tasks that require ml nodes, but these cannot be executed at all because they cannot find an ml node and new tasks are being dropped/rejected
YELLOW: all of the primary shards allocated, but some/all of the replicas have not been allocated, tasks that prefer ml nodes are queued excessively, starving for more ml nodes
GREEN: fully operational, tasks that prefer ml nodes don't queue up

I feel like there must be more holes with this approach, so needs more thinking.

[ylwu-amzn]

Thanks @xinlamzn, good question! Thanks @dblock for the proposal, that's definitely an option.

I brainstormed possible options and listed as bellow, please help review:

Option1: user monitor state of custom role nodes by themselves

Keep current cluster state as is. ML node failure doesn’t impact cluster state. User should know how many ML nodes are expected to be running in cluster, and they can monitor ML nodes by calling GET _cat/nodes. For example, a cluster have 10 ML nodes in total, user may expect 10 ML nodes running is “GREEN”, 5 ML nodes running is “YELLOW” and no ML nodes running is “RED”.

Pros:

1. No impact to current cluster state.

Cons:

1. User needs to define state rules and do some calculation to tell the state

Option2: add new cluster state for custom role node

Keep current cluster state as is. Define new cluster state for different custom node roles. If the custom node role is defined in code like DiscoveryNodeRole.java , we can define corresponding state for each role. If we don’t define roles in code, node just pick up custom node roles from opensearch.yml dynamically, we should generate custom cluster state for the custom node role dynamically. And user can define rules for each custom role.

For example, node picks up ml role, then we can generate ml_nodes_status . User can define state rules for ml nodes like :

"green": 10, // >= 10
"yellow": 5, // <= 5 and >0
"red": 0, // <=0

If all data nodes running correctly and all primary/replica shards are good, but only 3 running ml nodes in cluster when call GET _cluster/health, then we will return "ml_nodes_status":"yelllow" together with the current cluster state "status":"green"

Pros:

1. No impact to current cluster state.
2. Compared with Option1, user don’t need to calculate cluster states

Cons:

1. User needs to define state rules
2. More cluster states.

UPDATE: : We can enhance this option to avoid the cons "More cluster states.", list this as Option2.1

Option2.1: add new cluster status for node, keep current cluster status for shard

As current cluster status from _cluster/health is only for shard health status (for example, if a data node doesn't have any shard on it, kill this data node won't impact cluster status). We can build a general node status for node failures. This node failure will cover all node roles. For example, user can configure min_nodes for ml role as 5, then if cluster running ml node >= 5, we set node_status as green; if running ml node less than 5 but greater than 0, then set node_status as yellow; if there is no running ml node, set node_status as red. If we have multiple data roles, we can calculate each node role status, then pick up the worst one. For example ml node is red and image_processing node is yellow, then the final node_status will be red. We don't return separate node status for each node role. User can deep dive into each node role by calling _cat/nodes.

Pros:

1. No impact to current cluster state, still means shard health status
2. Compared with Option2, user just need to monitor one node_status for all nodes. No need to monitor multiple node status for node roles.

Cons:

1. Compared with Option3 and Option4, user need to monitor 1 more status for node. But this option keeps the shard health status, will be easier for user if they need to monitor shard health status and node status separately.

Option3: enrich current cluster state to include failure of nodes with custom role, but user define rules for custom role node

User define the rule for each custom role like Option2. Then we can combine all status of custom role nodes and original cluster status and choose the worst one. For example, ml nodes status is yellow , image_processing nodes status is red and general cluster status is green (all data nodes running correctly). Then the final status will be red

Pros:

1. Just one cluster status returned

Cons:

1. This will break current cluster state semantic, will break the BWC if user only needs to monitor primary/replica shards.
2. User get a combined status, need to dive deep to know what’s the root cause. So maybe we should also add separate status for each custom role finally to avoid this.
3. User need to define cluster state rules

Option4: enrich current cluster state to include failure of nodes with custom role, user don’t need to define rules for custom role node

This is to improve Option3 to reduce the effort to define state rules for custom roles. This is what db mentioned :

RED: some or all of (primary) shards are not ready, no available ml nodes
YELLOW: all of the primary shards allocated, but some/all of the replicas have not been allocated, some ml nodes down
GREEN: fully operational, all ml nodes up

Like Db said, this is chicken and egg problem as we don’t know how many ml nodes are expected in a cluster.
or

RED: some or all of (primary) shards are not ready, there are tasks that require ml nodes, but these cannot be executed at all because they cannot find an ml node and new tasks are being dropped/rejected
YELLOW: all of the primary shards allocated, but some/all of the replicas have not been allocated, tasks that prefer ml nodes are queued excessively, starving for more ml nodes
GREEN: fully operational, tasks that prefer ml nodes don't queue up

This seems to be monitoring the ml node load, rather than node failure. If there are too many requests and ML tasks queue up or rejected, that may be by design, rather than caused by some ML node failure.

Pros:

1. Just one cluster status returned
2. No need to define rules for custom roles

Cons

1. This will break current cluster state semantic, will break the BWC if user only needs to monitor primary/replica shards.
2. User get a combined status, need to dive deep to know what’s the root cause. So maybe we should also add separate status for each custom role finally to avoid this.
3. There are some challenges like how do we know how many ml nodes are expected to be running in cluster.

My conclusion by comparing these options

I think we should not break the current cluster state semantic, which seems breaking the BWC if user only needs to monitor if primary/replica shards looks good or not. So I prefer not to choose option3 and 4.

For option1 it needs some manual effort from user side to calculate and monitor and less effort for development.
As option2 will add new status for each node role, seems not so scalable. So prefer to use Option2.1, just return one node_status for all nodes. For Option2.1, we needs more development effort than option1, but will reduce some calculation effort for user. Option1 is not a one way door. For agile development, I think it's ok to choose option1 to speed up the development and feature release. But not against option2.1 at all if we have enough bandwidth/time.

[dblock]

Two questions re:pros/cons of the last option.

1. This will break current cluster state semantic, will break the BWC if user only needs to monitor primary/replica shards.

Regarding backwards compatibility, those who don't have ml/custom nodes are 100% bcw, so is this valid?

2. User get a combined status, need to dive deep to know what’s the root cause.

Isn't this the case today where you have to dive deep under "yellow" to see, for example, that some master and some data nodes are down? So is this a new con?

[ylwu-amzn]

Thanks @dblock .

Regarding backwards compatibility, those who don't have ml/custom nodes are 100% bcw, so is this valid?

Agree, if cluster doesn't have ML node, there is no BWC issue.
I'm thinking this case: User may upgrade their cluster, then add one ML node to their existing cluster. Then the "red" cluster state doesn't mean bad primary shard any more. It may be caused by ML node failure. That means existing metrics monitoring shard health status will be impacted by ML node failure. But maybe it's ok or won't hurt too much if we document this clearly and users can have some other way to monitor shard status.

I added some updates under option2, just list as Option2.1. Also paste here

As current cluster status from _cluster/health is only for shard health status (for example, if a data node doesn't have any shard on it, kill this data node won't impact cluster status). We can build a general node status for node failures. This node failure will cover all node roles. For example, user can configure min_nodes for ml role as 5, then if cluster running ml node >= 5, we set node_status as green; if running ml node less than 5 but greater than 0, then set node_status as yellow; if there is no running ml node, set node_status as red. If we have multiple data roles, we can calculate each node role status, then pick up the worst one. For example ml node is red and image_processing node is yellow, then the final node_status will be red. We don't return separate node status for each node role. User can deep dive into each node role by calling _cat/nodes.

Isn't this the case today where you have to dive deep under "yellow" to see, for example, that some master and some data nodes are down? So is this a new con?

I think that depends on which level user are more interested to monitor. Current cluster status means the shard health status. So I think the shard level health info should be kept as this is important to current user.

[dblock]

Isn't this the case today where you have to dive deep under "yellow" to see, for example, that some master and some data nodes are down? So is this a new con?

I think that depends on which level user are more interested to monitor. Current cluster status means the shard health status. So I think the shard level health info should be kept as this is important to current user.

We do document this status as shard health in https://opensearch.org/docs/latest/opensearch/rest-api/cluster-health/. I do think a major release can absolutely expand on this definition if it makes sense.

[ylwu-amzn]

Isn't this the case today where you have to dive deep under "yellow" to see, for example, that some master and some data nodes are down? So is this a new con?

I think that depends on which level user are more interested to monitor. Current cluster status means the shard health status. So I think the shard level health info should be kept as this is important to current user.

We do document this status as shard health in https://opensearch.org/docs/latest/opensearch/rest-api/cluster-health/. I do think a major release can absolutely expand on this definition if it makes sense.

I'm ok to have some breaking part in a major release. I prefer to listen to community user's feedback to check if it's ok to

1. change the cluster status semantic to cover both shard health status and node status.
2. don't return shard health status in _cluster/health response

@reta do you have any suggestion?

[andrross]

Regarding the cluster health discussion, are these new ml nodes comparable to ingest nodes? Like ml nodes, I don't think the cluster knows how many ingest nodes should be present until they register themselves and it also doesn't know how many are actually needed to perform the ingest work. Is this an apt comparison? If so, how does a failure of a dedicated ingest node impact cluster health?

[reta]

@ylwu-amzn thanks for excellent summary, somewhat in the same vein as @andrross said, I think we should not alter the cluster health semantics from what it is right now (there are a lot of operational mitigation procedures which will be invalidated). However, I like the Option2 where we could extend the cluster heath response to include the nodes state by custom roles.

[ylwu-amzn]

Regarding the cluster health discussion, are these new ml nodes comparable to ingest nodes? Like ml nodes, I don't think the cluster knows how many ingest nodes should be present until they register themselves and it also doesn't know how many are actually needed to perform the ingest work. Is this an apt comparison? If so, how does a failure of a dedicated ingest node impact cluster health?

@andrross Thanks, agree that we actually don't know how many ingest or ml node should be present in a cluster due to the distributed nature (fe, 10 ml nodes configured, 5 failed to start, then we have 5 ml nodes running after cluster started, but later user can fix the failed ml node, then the running ml node will increase to 6, 7, to 10).
One options is to let user configure threshold like option2 or option2.1, so cluster can know which node status should be mapped to.

how does a failure of a dedicated ingest node impact cluster health?

If we keep current cluster health semantics (it means shard health status), the dedicated ingest node failure should not impact cluster health as it doesn't contain any shards. Like @reta said, we can keep cluster health semantics.

@reta , thanks for your opinion, for

I like the Option2 where we could extend the cluster heath response to include the nodes state by custom roles.

Do you mean we should add new node status for each custom role, like ml_node_status for ml node, and image_procesing_node_status for image_processing node? If user add more and more custom roles, the cluster health response will grow bigger. Do you think this will cause any scalability problem? If yes, maybe we should go with option2.1, just add one general node status for all nodes.

[reta]

Do you mean we should add new node status for each custom role, like ml_node_status for ml node, and image_procesing_node_status for image_processing node? If user add more and more custom roles, the cluster health response will grow bigger. Do you think this will cause any scalability problem? If yes, maybe we should go with option2.1, just add one general node status for all nodes.

This is a good point. My perspective was/is primarily on what we are looking from health API with respect to custom roles. Fe. it should be fairly trivial to spot there are no nodes with specific roles in the cluster, may be something like a new JSON element:

"nodes_by_roles": {
    "ml": 1,
    "ingest": 1,
    "...": ..
}

It should be fairly easy to troubleshoot, I am not sure it is going to be a scalability problem since cluster discovery collects this details anyway, having dozens of roles is realistic, but hundreds and more - does not look like it. @ylwu-amzn @dblock @xinlamzn what do you think guys?

[ylwu-amzn]

@andrross Thanks, good point. Built-in roles are mainly for OpenSearch core function, they are hard-coded and can't be changed. Wrong built-in roles will fail to start cluster or impact core function. For custom roles, they are mainly for plugin/extension function and wrong custom role only impact specific plugin, not the core function. But maybe the end user should consider the built-in and custom/extra roles the same as they treat the OpenSearch and plugin as a whole.

Is the difference between "built in" roles and these "extra" roles important for a user?

I think this is valid concern and may bring confusion/difficulty for user to understand. Missed this cons for adding new field.

creating a new node.roles field (without single-letter abbreviations) and deprecating node.role

I think this is an option. Like @dblock said, the current design to use one char as abbreviation seems not reasonable. Maybe good to deprecate node.role to avoid the confusion of two role columns.

[reta]

@ylwu-amzn I also agree, deprecation + node.roles is good tradeoff

[ylwu-amzn]

Thanks @reta for sharing your opinion. What do you think @dblock about "deprecation + node.roles"? I guess you are good for this option, just confirm.

[dblock]

Thanks @reta for sharing your opinion. What do you think @dblock about "deprecation + node.roles"? I guess you are good for this option, just confirm.

That would be my favorite and most looking forward option.

[ylwu-amzn]

So now we have conclusion for the open question: "how to extend _cat/nodes API?":

We will add a new filed node.roles and put full name of all roles (both built-in and custom roles) there. And we will deprecate node.role.

If anyone has new concerns, please don't hesitate to call out. We are open to tune or change direction.

[dtaivpp]

@reta or @dblock can we tag this for 2.1 release? I talked with @ylwu-amzn and it is ready to roll.

[reta]

@reta or @dblock can we tag this for 2.1 release? I talked with @ylwu-amzn and it is ready to roll.

Sorry for that @dtaivpp , lost this one somehow, looked and commented

[dtaivpp]

@reta all good I cant keep track of any of it 😅

[brijos]

Adding a meta issue to track the engineering work as well as future documentation and blog posts.

[dblock]

I CRed #3436, just waiting on another pair of eyes from @reta and it's good to go.

[ylwu-amzn]

PR merged. Close this issue. Thanks a lot for your help!