OOM errors caused by large queue of cluster state update tasks

[ywelsch]

We've had multiple reports of master nodes going OOM when dealing with large numbers of pending cluster state update tasks. I am mainly opening this issue to explain what the implementation issues are and to link to current work being done to fix them. Before talking about the actual issues, let me quickly recapitulate what these update tasks are. Cluster state update tasks are units of work that are processed on the master node to change the cluster state, for example when creating a new index, updating mappings or changing the routing table (allocating / starting / moving shards). Some tasks have timeouts, others do not. Most tasks that result from user action (e.g. index creation) have timeouts which are configurable on the request with the master_timeout parameter (default is 30 seconds). Tasks are submitted to a single-threaded thread pool executor that considers priorities as well as insertion order for choosing the tasks to execute.

1. The first issue is related to batching of cluster state updates. Even though a task might have timed out, the resources it will need to execute won't be cleaned up right away (in certain situations they won't be cleaned up at all which corresponds to a memory leak). The reason for this is that with the introduction of task batching a new map was added to cluster service that holds a reference to the UpdateTask. The task is only removed from this map once the task gets to execute (which can happen way later, or even not at all - see TieBreakingPrioritizedRunnable#scheduleTimeout and #runAndClean).

2. The second issue is that cluster state update tasks often reference large states. One of the worst offenders is the action listener in TransportMasterNodeAction.AsyncSingleAction which has a ClusterStateObserver to coordinate the retry mechanism if the action on the master node fails due to the node not being master anymore. The ClusterStateObserver in AsyncSingleAction keeps a reference to the full cluster state when the action was initiated. If the pending tasks queue grows quite large and has older items in it lots of cluster states can possibly be referenced. In ES 5.x this is less of an issue as in 2.x because 5.x has immutable ShardRoutings (Immutable ShardRouting #17821) which lets us reuse ShardRouting objects across cluster states. The solution here I think is to look at how ClusterStateObserver is used and see that keeping the full old cluster state around is not really needed (we only care about version / status and cluster state identity of the old state).

3. The third issue is that the pending task queue is unbounded. Here we could look at ways to constrain the number of tasks that can be submitted, possibly trading CPU time for memory.

Fixes:

• Remove cluster update task when task times out #21578 fixes the memory leak issue where the cluster service does not remove an update task from its internal data structures that are used for batching cluster state updates.
• Let ClusterStateObserver only hold onto state that's needed for change detection #21631 fixes the issue where ClusterStateObserver holds onto cluster states

[amazinganshul]

Will the changes be merged in 1.7.4 branch as well? We recently faced the same issue. Master nodes had heap configured at 8GB.

[jasontedor]

Will the changes be merged in 1.7.4 branch as well? We recently faced the same issue. Master nodes had heap configured at 8GB.

No. The 1.7 series has reached end-of-life and we not see any more changes.

[bleskes]

@ywelsch can you advice on the state of things here? It seems that the unbound nature of the queue hasn't been addressed (though it's tricky as to what we should do imo).

[ywelsch]

With the fixes done for 1) and 2), we have not gotten recent reports of this OOM kind anymore. As updates are not hogging memory anymore, remaining issues due to the unbounded nature of the pending cluster tasks queue have manifested in the form of the master node becoming slow/unavailable/CPU-bound when running into a high number of pending tasks. Other open issues capture work that helps with avoiding/better handling a large number of tasks, so we can close this one.