Task management

[clintongormley]

We need a task management API to allow management of long running tasks, like snapshot/restore, benchmarking, update-by-query etc.

The API should allow listing and aborting of ongoing tasks.

[1st]

+1. Waiting for this new feature!!!

[markharwood]

Part of killing long running tasks will be tied up in this new generic timeout handling mechanism: #4586

[shikhar]

Will this cover search requests? Since one can't know in advance very well if a search request will be long-running, I guess this task tracking would need to apply to all of them.

If this is not the plan, I think #7157 should be reopened.

[martijnvg]

@shikhar The task infrastructure will be generic enough that it can be adopted in the search api as well.

[efuquen]

👍 Not having this has been a pain.

[shadow000fire]

+1

[ashvins]

+1

[JDvorak]

👍

[sflint]

+1

[niemyjski]

+1

[raf64flo]

Looking at it carefully. 👍

[smramdani]

+1
Would be a great feature as "running queries" and "long queries" in the classical RDBMSs.

[kennycason]

+1

[tostasqb]

+1

[kiryam]

+1

[chenryn]

+1

[kjelle]

+1

[amontalenti]

+1

[BrickXu]

+1

[imotov]

I created a new meta issue #15117 where work on the task management will be tracked. Closing this one.

[garyelephant]

+1

[rfyiamcool]

thanks

[garyelephant]

Will this cover resource limit(similar with spark job executor's cpu,mem resource limit) using cgroups or something ? For Users who want to aggregate more than 1TB of index, it's likely to cause overall elasticsearch cluster overload, no response.

[imotov]

@garyelephant there are no immediate plans to do that as a part of task management effort. However, we are working on some improvements in this area. Please see the discussion in #11511.