[ML] Job in index: Get datafeed and job stats from index #34591
Conversation
|
Pinging @elastic/ml-core |
davidkyle
changed the title
| Set<String> excludeJobIds = stats.stream().map(GetJobsStatsAction.Response.JobStats::getJobId).collect(Collectors.toSet()); | ||
| return requestedJobIds.stream().filter(jobId -> !excludeJobIds.contains(jobId) && | ||
| !mlMetadata.isJobDeleted(jobId)).collect(Collectors.toList()); |
There was a problem hiding this comment.
This is the tricky bit as it is no longer easy to find if the job is being deleted without making an async call for each job. Before this change if a job was being deleted then the calls to gatherForecastStats and gatherDataCountsAndModelSizeStats would not be made, it is still safe to make those calls if those documents are deleted the response accepts null for forecast stats and model size stats and gather data counts returns a default constructed object if the document is not found. Some jobs will take a long time to delete and the job config document is the last thing to be removed so it is possible that forecast stats are in the process of deletion meaning that in a long running delete subsequent calls. It's a reasonable argument to say the API is doing what is says is it doing in this case. Compare with GET jobs with returns all jobs including deleting jobs.
One helpful change is to add an excludeDeleting parameter to jobConfigProvider.expandJobsIds when called in doExecute which would make the behaviour closer to the current albeit with race.
|
Closing due to git shenanigans. #34645 has been raised instead |
For both actions the datafeed and job Ids expressions must be expanded from the index documents.
Jindex feature branch