Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove multiple paths from ShardPath #72713

Merged
merged 3 commits into from
May 5, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ protected void findAndProcessShardPath(OptionSet options, Environment environmen
.resolve(Integer.toString(shId.id()));
if (Files.exists(shardPathLocation)) {
final ShardPath shardPath = ShardPath.loadShardPath(logger, shId, indexSettings.customDataPath(),
new Path[]{shardPathLocation}, dataPath);
shardPathLocation, dataPath);
if (shardPath != null) {
consumer.accept(shardPath);
}
Expand Down
49 changes: 17 additions & 32 deletions server/src/main/java/org/elasticsearch/index/shard/ShardPath.java
Original file line number Diff line number Diff line change
Expand Up @@ -99,59 +99,44 @@ public boolean isCustomDataPath() {
}

/**
* This method walks through the nodes shard paths to find the data and state path for the given shard. If multiple
* directories with a valid shard state exist the one with the highest version will be used.
* This method resolves the node's shard path using the given {@link NodeEnvironment}.
* <b>Note:</b> this method resolves custom data locations for the shard if such a custom data path is provided.
*/
public static ShardPath loadShardPath(Logger logger, NodeEnvironment env,
ShardId shardId, String customDataPath) throws IOException {
final Path shardPath = env.availableShardPath(shardId);
final Path sharedDataPath = env.sharedDataPath();
return loadShardPath(logger, shardId, customDataPath, new Path[] { shardPath }, sharedDataPath);
return loadShardPath(logger, shardId, customDataPath, shardPath, sharedDataPath);
}

/**
* This method walks through the nodes shard paths to find the data and state path for the given shard. If multiple
* directories with a valid shard state exist the one with the highest version will be used.
* This method resolves the node's shard path using the given data paths.
* <b>Note:</b> this method resolves custom data locations for the shard.
*/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment looks out of date now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I tweaked these in 16975ec

public static ShardPath loadShardPath(Logger logger, ShardId shardId, String customDataPath, Path[] availableShardPaths,
public static ShardPath loadShardPath(Logger logger, ShardId shardId, String customDataPath, Path shardPath,
Path sharedDataPath) throws IOException {
final String indexUUID = shardId.getIndex().getUUID();
Path loadedPath = null;
for (Path path : availableShardPaths) {
// EMPTY is safe here because we never call namedObject
ShardStateMetadata load = ShardStateMetadata.FORMAT.loadLatestState(logger, NamedXContentRegistry.EMPTY, path);
if (load != null) {
if (load.indexUUID.equals(indexUUID) == false && IndexMetadata.INDEX_UUID_NA_VALUE.equals(load.indexUUID) == false) {
logger.warn("{} found shard on path: [{}] with a different index UUID - this "
+ "shard seems to be leftover from a different index with the same name. "
+ "Remove the leftover shard in order to reuse the path with the current index", shardId, path);
throw new IllegalStateException(shardId + " index UUID in shard state was: " + load.indexUUID
+ " expected: " + indexUUID + " on shard path: " + path);
}
if (loadedPath == null) {
loadedPath = path;
} else{
throw new IllegalStateException(shardId + " more than one shard state found");
}
// EMPTY is safe here because we never call namedObject
ShardStateMetadata load = ShardStateMetadata.FORMAT.loadLatestState(logger, NamedXContentRegistry.EMPTY, shardPath);
if (load != null) {
if (load.indexUUID.equals(indexUUID) == false && IndexMetadata.INDEX_UUID_NA_VALUE.equals(load.indexUUID) == false) {
logger.warn("{} found shard on path: [{}] with a different index UUID - this "
+ "shard seems to be leftover from a different index with the same name. "
+ "Remove the leftover shard in order to reuse the path with the current index", shardId, shardPath);
throw new IllegalStateException(shardId + " index UUID in shard state was: " + load.indexUUID
+ " expected: " + indexUUID + " on shard path: " + shardPath);
}

}
if (loadedPath == null) {
return null;
} else {
final Path dataPath;
final Path statePath = loadedPath;
final boolean hasCustomDataPath = Strings.isNotEmpty(customDataPath);
if (hasCustomDataPath) {
dataPath = NodeEnvironment.resolveCustomLocation(customDataPath, shardId, sharedDataPath);
} else {
dataPath = statePath;
dataPath = shardPath;
}
logger.debug("{} loaded data path [{}], state path [{}]", shardId, dataPath, statePath);
return new ShardPath(hasCustomDataPath, dataPath, statePath, shardId);
logger.debug("{} loaded data path [{}], state path [{}]", shardId, dataPath, shardPath);
return new ShardPath(hasCustomDataPath, dataPath, shardPath, shardId);
}
return null;
}

/**
Expand Down