Skip to content

Commit

Permalink
[ML] handle broken setup with state alias being an index (#58999)
Browse files Browse the repository at this point in the history
.ml-state-write is supposed to be an index alias, however by accident it can become an index. If
.ml-state-write is a concrete index instead of an alias ML stops working. This change improves error
handling by setting the job to failed and properly log and audit the problem. The user still has to
manually fix the problem. This change should lead to a quicker resolution of the problem.

fixes #58482
  • Loading branch information
Hendrik Muhs committed Jul 3, 2020
1 parent 2c2486d commit ca3da7a
Showing 1 changed file with 14 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.core.internal.io.IOUtils;
import org.elasticsearch.index.analysis.AnalysisRegistry;
import org.elasticsearch.indices.InvalidAliasNameException;
import org.elasticsearch.persistent.PersistentTasksCustomMetadata.PersistentTask;
import org.elasticsearch.rest.RestStatus;
import org.elasticsearch.threadpool.ThreadPool;
Expand Down Expand Up @@ -426,7 +427,19 @@ protected void doRun() {
e -> closeHandler.accept(e, true)
));
},
e -> closeHandler.accept(e, true));
e -> {
if (ExceptionsHelper.unwrapCause(e) instanceof InvalidAliasNameException) {
String msg = "Detected a problem with your setup of machine learning, the state index alias ["
+ AnomalyDetectorsIndex.jobStateIndexWriteAlias()
+ "] exists as index but must be an alias.";
logger.error(new ParameterizedMessage("[{}] {}", jobId, msg), e);
auditor.error(jobId, msg);
setJobState(jobTask, JobState.FAILED, msg, e2 -> closeHandler.accept(e, true));
} else {
closeHandler.accept(e, true);
}
}
);

// Make sure the state index and alias exist
ActionListener<Boolean> resultsMappingUpdateHandler = ActionListener.wrap(
Expand Down

0 comments on commit ca3da7a

Please sign in to comment.