-
Notifications
You must be signed in to change notification settings - Fork 24.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Show force merge/optimize progress #15975
Comments
If it can be done, it'll be via the task management API. Closing in favour of #15117 |
I think we dropped this issue on the floor at the end of the task management task. Now that the task management framework is in the place, should we revisit this issue? |
@imotov I'm not an expert in elasticsearch internal things, but here are my though as consumers:
|
Pinging @elastic/es-distributed |
@jpountz is there any way to track progress of a user driven forced merge? |
Not everything is exposed publicly today, but we could expose information about ongoing merges. Giving progress information is more complex due to the fact that Lucene doesn't merge document by document but data-structure by data-structure, eg. the inverted index of _id, then the inverted index of foo then doc values of seq_no, then doc values of foo, etc. |
I understand this is a bit more complex than tracking the progress of other tasks in the cluster, but even showing the number of indices which already finished (in a multi-index force-merge) and the number of data-structures (or doc_values) already processed would be nice. Is there any plan to add this functionality? Thanks |
using the tasks api to see that a forcemerge might still be running doesn't really tell us anything as far as % completion to give estimation... when this is combined with something like curator to forcemerge a bunch of indexes that meet criteria it would be useful to know when these will end so they don't step on each other or cause other contention issues. |
The ability to at least understand what index is currently merging would be very useful. We have had users accidentally start of a force merge for an index actively being written to and there was no apparent way to identify which index was being operated on by a given task. |
This is static information that is part of the force merge request. Relates to elastic#15975
Since some large indices with many segments may take for a long time to force merge, it would be beneficial to:
I feel it is essential to be able to know what is happening with forge merges that take extended amount of time. |
Hi, thats something we need too |
Even something as high level as whether the merge is running or complete would be huge. I don't need to have a play-by-play on what's going on in Lucene, I just need to know when I can start indexing documents again. Please address this soon. |
I am able to see merge: current by running below command:
Is this the one to monitor to check the current merge in progress? |
The task list contains a node-level task for each node holding shards that are to be force-merged. I think we could decorate this task with things like how many shards it targets, how many of those shard-level tasks are complete, the shard of the currently-running task, and how long that shard-level task has been running. We have all of that information available I think, it's more a question of exposing it in the task list. More detailed progress from Lucene would be nice too, but that's a much larger ask, so let's rule that out of scope for now. I'm marking this as |
But what exactly does
I really hope it is the former because the merge process been running for 7 days now and there is absolutely no way currently to check if anything is happening and how much time I still have to wait for the merge to complete.
I wish there was detailed explanation of |
+1 I have a similar issue with a large dataset that I'm trying to use in rally. |
Are we able to expose this somehow, maybe pending tasks?
At the moment it just runs until finished, but I'm hoping there may be a way to figure out some kind of progress percentage, even a rough one.
I realise this is all a little vague, but I don't really know if it's possible, just that it'd be nice to have :p
The text was updated successfully, but these errors were encountered: