From 2e79246c1a9c754a28fb7f96bf572542e7163c3b Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Mon, 13 Jan 2014 21:53:44 +0100 Subject: [PATCH 01/34] [DOCS] Added docs for tribe node Related #4708 --- docs/reference/modules.asciidoc | 2 + docs/reference/modules/tribe.asciidoc | 60 +++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) create mode 100644 docs/reference/modules/tribe.asciidoc diff --git a/docs/reference/modules.asciidoc b/docs/reference/modules.asciidoc index a9490ec9101ce..4ca327d1da0c0 100644 --- a/docs/reference/modules.asciidoc +++ b/docs/reference/modules.asciidoc @@ -17,6 +17,8 @@ include::modules/network.asciidoc[] include::modules/node.asciidoc[] +include::modules/tribe_node.asciidoc[] + include::modules/plugins.asciidoc[] include::modules/scripting.asciidoc[] diff --git a/docs/reference/modules/tribe.asciidoc b/docs/reference/modules/tribe.asciidoc new file mode 100644 index 0000000000000..b7481aec37238 --- /dev/null +++ b/docs/reference/modules/tribe.asciidoc @@ -0,0 +1,60 @@ +[[modules-tribe]] +== Tribe node + +The _tribes_ feature allows a _tribe node_ to act as a federated client across +multiple clusters. + +WARNING: This feature is EXPERIMENTAL -- use at your own risk. + +The tribe node works by retrieving the cluster state from all connected +clusters and merging them into a global cluster state. With this information +at hand, it is able to perform read and write operations against the nodes in +all clusters as if they were local. + +The `elasticsearch.yml` config file for a tribe node just needs to list the +clusters that should be joined, for instance: + +[source,yaml] +-------------------------------- +tribe: + t1: <1> + cluster.name: cluster_one + t2: <1> + cluster.name: cluster_two +-------------------------------- +<1> `t1` and `t2` are aribitrary names representing the connection to each + cluster. + +The example above configures connections to two clusters, name `t1` and `t2` +respectively. The tribe node will create a <> to +connect each cluster using <> by default. Any +other settings for the connection can be configured under `tribe.{name}`, just +like the `cluster.name` in the example. + +The merged global cluster state means that almost all operations work in the +same way as a single cluster: distributed search, suggest, percolation, +indexing, etc. + +However, there are a few exceptions: + +* The merged view cannot handle indices with the same name in multiple + clusters. It will pick one of them and discard the other. + +* Master level read operations (eg <>, <>) + need to have the `local` flag set to `true` as the tribe node does not + have a single master node. + +* Master level write operations (eg <>) are not + allowed. These should be performed on a single cluster. + +The tribe node can be configured to block all write operations and all +metadata operations with: + +[source,yaml] +-------------------------------- +tribe: + blocks: + write: true + metadata: true +-------------------------------- + From 0751f0b7c66429382be01bbab9007f043f1adb3c Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Mon, 13 Jan 2014 22:01:12 +0100 Subject: [PATCH 02/34] [DOCS] Fixed link to tribe.asciidoc --- docs/reference/modules.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/modules.asciidoc b/docs/reference/modules.asciidoc index 4ca327d1da0c0..79150367c1de5 100644 --- a/docs/reference/modules.asciidoc +++ b/docs/reference/modules.asciidoc @@ -17,7 +17,7 @@ include::modules/network.asciidoc[] include::modules/node.asciidoc[] -include::modules/tribe_node.asciidoc[] +include::modules/tribe.asciidoc[] include::modules/plugins.asciidoc[] From a1efa1f7aa24631040af8f988135277ff20d2326 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Mon, 13 Jan 2014 21:45:42 +0100 Subject: [PATCH 03/34] Remove `ElasticsearchInterruptedException` and handle interrupt state correctly. InterruptedExceptions should be handled by either rethrowing or restoring the interrupt state (i.e. calling `Thread.currentThread().interrupt()`). This is important since the caller of the is method or subequent method calls might also be interested in this exception. If we ignore the interrupt state the caller might be left unaware of the exception and blocks again on a subsequent method. Closes #4712 --- .../ElasticsearchInterruptedException.java | 36 ------------------- .../elasticsearch/action/ActionFuture.java | 30 ++++++++-------- .../action/support/AdapterActionFuture.java | 8 +++-- .../common/util/concurrent/EsAbortPolicy.java | 4 +-- .../index/service/InternalIndexService.java | 4 +-- .../transport/PlainTransportFuture.java | 8 +++-- .../org/elasticsearch/tribe/TribeService.java | 12 ++++--- 7 files changed, 37 insertions(+), 65 deletions(-) delete mode 100644 src/main/java/org/elasticsearch/ElasticsearchInterruptedException.java diff --git a/src/main/java/org/elasticsearch/ElasticsearchInterruptedException.java b/src/main/java/org/elasticsearch/ElasticsearchInterruptedException.java deleted file mode 100644 index 57a7e20d4e8c9..0000000000000 --- a/src/main/java/org/elasticsearch/ElasticsearchInterruptedException.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Licensed to Elasticsearch under one or more contributor - * license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright - * ownership. Elasticsearch licenses this file to you under - * the Apache License, Version 2.0 (the "License"); you may - * not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.elasticsearch; - -/** - * The same as {@link InterruptedException} simply a runtime one. - * - * - */ -public class ElasticsearchInterruptedException extends ElasticsearchException { - - public ElasticsearchInterruptedException(String message) { - super(message); - } - - public ElasticsearchInterruptedException(String message, Throwable cause) { - super(message, cause); - } -} diff --git a/src/main/java/org/elasticsearch/action/ActionFuture.java b/src/main/java/org/elasticsearch/action/ActionFuture.java index 3454aaa8a8438..3848536b0d11b 100644 --- a/src/main/java/org/elasticsearch/action/ActionFuture.java +++ b/src/main/java/org/elasticsearch/action/ActionFuture.java @@ -34,9 +34,9 @@ public interface ActionFuture extends Future { /** - * Similar to {@link #get()}, just wrapping the {@link InterruptedException} with - * {@link org.elasticsearch.ElasticsearchInterruptedException}, and throwing the actual - * cause of the {@link java.util.concurrent.ExecutionException}. + * Similar to {@link #get()}, just catching the {@link InterruptedException} with + * restoring the interrupted state on the thread and throwing an {@link org.elasticsearch.ElasticsearchIllegalStateException}, + * and throwing the actual cause of the {@link java.util.concurrent.ExecutionException}. *

*

Note, the actual cause is unwrapped to the actual failure (for example, unwrapped * from {@link org.elasticsearch.transport.RemoteTransportException}. The root failure is @@ -45,9 +45,9 @@ public interface ActionFuture extends Future { T actionGet() throws ElasticsearchException; /** - * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just wrapping the {@link InterruptedException} with - * {@link org.elasticsearch.ElasticsearchInterruptedException}, and throwing the actual - * cause of the {@link java.util.concurrent.ExecutionException}. + * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just catching the {@link InterruptedException} with + * restoring the interrupted state on the thread and throwing an {@link org.elasticsearch.ElasticsearchIllegalStateException}, + * and throwing the actual cause of the {@link java.util.concurrent.ExecutionException}. *

*

Note, the actual cause is unwrapped to the actual failure (for example, unwrapped * from {@link org.elasticsearch.transport.RemoteTransportException}. The root failure is @@ -56,9 +56,9 @@ public interface ActionFuture extends Future { T actionGet(String timeout) throws ElasticsearchException; /** - * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just wrapping the {@link InterruptedException} with - * {@link org.elasticsearch.ElasticsearchInterruptedException}, and throwing the actual - * cause of the {@link java.util.concurrent.ExecutionException}. + * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just catching the {@link InterruptedException} with + * restoring the interrupted state on the thread and throwing an {@link org.elasticsearch.ElasticsearchIllegalStateException}, + * and throwing the actual cause of the {@link java.util.concurrent.ExecutionException}. *

*

Note, the actual cause is unwrapped to the actual failure (for example, unwrapped * from {@link org.elasticsearch.transport.RemoteTransportException}. The root failure is @@ -69,9 +69,9 @@ public interface ActionFuture extends Future { T actionGet(long timeoutMillis) throws ElasticsearchException; /** - * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just wrapping the {@link InterruptedException} with - * {@link org.elasticsearch.ElasticsearchInterruptedException}, and throwing the actual - * cause of the {@link java.util.concurrent.ExecutionException}. + * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just catching the {@link InterruptedException} with + * restoring the interrupted state on the thread and throwing an {@link org.elasticsearch.ElasticsearchIllegalStateException}, + * and throwing the actual cause of the {@link java.util.concurrent.ExecutionException}. *

*

Note, the actual cause is unwrapped to the actual failure (for example, unwrapped * from {@link org.elasticsearch.transport.RemoteTransportException}. The root failure is @@ -80,9 +80,9 @@ public interface ActionFuture extends Future { T actionGet(long timeout, TimeUnit unit) throws ElasticsearchException; /** - * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just wrapping the {@link InterruptedException} with - * {@link org.elasticsearch.ElasticsearchInterruptedException}, and throwing the actual - * cause of the {@link java.util.concurrent.ExecutionException}. + * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just catching the {@link InterruptedException} with + * restoring the interrupted state on the thread and throwing an {@link org.elasticsearch.ElasticsearchIllegalStateException}, + * and throwing the actual cause of the {@link java.util.concurrent.ExecutionException}. *

*

Note, the actual cause is unwrapped to the actual failure (for example, unwrapped * from {@link org.elasticsearch.transport.RemoteTransportException}. The root failure is diff --git a/src/main/java/org/elasticsearch/action/support/AdapterActionFuture.java b/src/main/java/org/elasticsearch/action/support/AdapterActionFuture.java index 4c8899705aae9..f5acc4f0427e4 100644 --- a/src/main/java/org/elasticsearch/action/support/AdapterActionFuture.java +++ b/src/main/java/org/elasticsearch/action/support/AdapterActionFuture.java @@ -20,7 +20,7 @@ package org.elasticsearch.action.support; import org.elasticsearch.ElasticsearchException; -import org.elasticsearch.ElasticsearchInterruptedException; +import org.elasticsearch.ElasticsearchIllegalStateException; import org.elasticsearch.ElasticsearchTimeoutException; import org.elasticsearch.action.ActionFuture; import org.elasticsearch.action.ActionListener; @@ -44,7 +44,8 @@ public T actionGet() throws ElasticsearchException { try { return get(); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException(e.getMessage()); + Thread.currentThread().interrupt(); + throw new ElasticsearchIllegalStateException("Future got interrupted", e); } catch (ExecutionException e) { throw rethrowExecutionException(e); } @@ -72,7 +73,8 @@ public T actionGet(long timeout, TimeUnit unit) throws ElasticsearchException { } catch (TimeoutException e) { throw new ElasticsearchTimeoutException(e.getMessage()); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException(e.getMessage()); + Thread.currentThread().interrupt(); + throw new ElasticsearchIllegalStateException("Future got interrupted", e); } catch (ExecutionException e) { throw rethrowExecutionException(e); } diff --git a/src/main/java/org/elasticsearch/common/util/concurrent/EsAbortPolicy.java b/src/main/java/org/elasticsearch/common/util/concurrent/EsAbortPolicy.java index 8c5ee7150fcb7..cf6239445d8c1 100644 --- a/src/main/java/org/elasticsearch/common/util/concurrent/EsAbortPolicy.java +++ b/src/main/java/org/elasticsearch/common/util/concurrent/EsAbortPolicy.java @@ -20,7 +20,6 @@ package org.elasticsearch.common.util.concurrent; import org.elasticsearch.ElasticsearchIllegalStateException; -import org.elasticsearch.ElasticsearchInterruptedException; import org.elasticsearch.common.metrics.CounterMetric; import java.util.concurrent.BlockingQueue; @@ -44,7 +43,8 @@ public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) { try { ((SizeBlockingQueue) queue).forcePut(r); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException(e.getMessage(), e); + Thread.currentThread().interrupt(); + throw new ElasticsearchIllegalStateException("forced execution, but got interrupted", e); } return; } diff --git a/src/main/java/org/elasticsearch/index/service/InternalIndexService.java b/src/main/java/org/elasticsearch/index/service/InternalIndexService.java index 88a752fd6e2d2..130dc91253744 100644 --- a/src/main/java/org/elasticsearch/index/service/InternalIndexService.java +++ b/src/main/java/org/elasticsearch/index/service/InternalIndexService.java @@ -24,7 +24,6 @@ import com.google.common.collect.UnmodifiableIterator; import org.elasticsearch.ElasticsearchException; import org.elasticsearch.ElasticsearchIllegalStateException; -import org.elasticsearch.ElasticsearchInterruptedException; import org.elasticsearch.cluster.metadata.IndexMetaData; import org.elasticsearch.common.Nullable; import org.elasticsearch.common.inject.*; @@ -276,7 +275,8 @@ public void run() { try { latch.await(); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException("interrupted closing index [ " + index().name() + "]", e); + logger.debug("Interrupted closing index [{}]", e, index().name()); + Thread.currentThread().interrupt(); } } diff --git a/src/main/java/org/elasticsearch/transport/PlainTransportFuture.java b/src/main/java/org/elasticsearch/transport/PlainTransportFuture.java index 14a1c2c64051a..812eec9f64b0b 100644 --- a/src/main/java/org/elasticsearch/transport/PlainTransportFuture.java +++ b/src/main/java/org/elasticsearch/transport/PlainTransportFuture.java @@ -20,7 +20,7 @@ package org.elasticsearch.transport; import org.elasticsearch.ElasticsearchException; -import org.elasticsearch.ElasticsearchInterruptedException; +import org.elasticsearch.ElasticsearchIllegalStateException; import org.elasticsearch.ElasticsearchTimeoutException; import org.elasticsearch.common.util.concurrent.BaseFuture; @@ -44,7 +44,8 @@ public V txGet() throws ElasticsearchException { try { return get(); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException(e.getMessage()); + Thread.currentThread().interrupt(); + throw new ElasticsearchIllegalStateException("Future got interrupted", e); } catch (ExecutionException e) { if (e.getCause() instanceof ElasticsearchException) { throw (ElasticsearchException) e.getCause(); @@ -61,7 +62,8 @@ public V txGet(long timeout, TimeUnit unit) throws ElasticsearchException { } catch (TimeoutException e) { throw new ElasticsearchTimeoutException(e.getMessage()); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException(e.getMessage()); + Thread.currentThread().interrupt(); + throw new ElasticsearchIllegalStateException("Future got interrupted", e); } catch (ExecutionException e) { if (e.getCause() instanceof ElasticsearchException) { throw (ElasticsearchException) e.getCause(); diff --git a/src/main/java/org/elasticsearch/tribe/TribeService.java b/src/main/java/org/elasticsearch/tribe/TribeService.java index f28224ef96ba1..e41700afdd9ff 100644 --- a/src/main/java/org/elasticsearch/tribe/TribeService.java +++ b/src/main/java/org/elasticsearch/tribe/TribeService.java @@ -22,7 +22,7 @@ import com.google.common.collect.ImmutableMap; import com.google.common.collect.Lists; import org.elasticsearch.ElasticsearchException; -import org.elasticsearch.ElasticsearchInterruptedException; +import org.elasticsearch.ElasticsearchIllegalStateException; import org.elasticsearch.cluster.*; import org.elasticsearch.cluster.block.ClusterBlock; import org.elasticsearch.cluster.block.ClusterBlockLevel; @@ -150,8 +150,11 @@ public ClusterState execute(ClusterState currentState) throws Exception { @Override public void onFailure(String source, Throwable t) { - logger.error("{}", t, source); - latch.countDown(); + try { + logger.error("{}", t, source); + } finally { + latch.countDown(); + } } @Override @@ -162,7 +165,8 @@ public void clusterStateProcessed(String source, ClusterState oldState, ClusterS try { latch.await(); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException(e.getMessage(), e); + Thread.currentThread().interrupt(); + throw new ElasticsearchIllegalStateException("Interrupted while starting [" + this.getClass().getSimpleName()+ "]", e); } for (InternalNode node : nodes) { try { From b379bf5668c5bf3a6edffbc2cd9ae97f29a8b7d6 Mon Sep 17 00:00:00 2001 From: Lee Hinman Date: Thu, 26 Dec 2013 11:25:25 -0700 Subject: [PATCH 04/34] Default to not accepting type wrapper in indexing requests Currently it is possible to index a document as: ``` POST /myindex/mytype/1 { "foo"...} ``` or as: ``` POST /myindex/mytype/1 { "mytype": { "foo"... } } ``` This makes indexing non-deterministic and fields can be misinterpreted as type names. This changes makes Elasticsearch accept only the first form by default, ie without the type wrapper. This can be changed by setting `index.mapping.allow_type_wrapper` to `true`` when creating the index. Closes #4484 --- docs/reference/docs/index_.asciidoc | 17 +++-- .../index/mapper/DocumentMapper.java | 21 +++--- .../aliases/IndexAliasesTests.java | 2 +- .../mapper/all/SimpleAllMapperTests.java | 4 +- .../elasticsearch/index/mapper/all/test1.json | 34 +++++---- .../mapper/simple/SimpleMapperTests.java | 16 +++++ .../index/mapper/simple/test1-withtype.json | 43 +++++++++++ .../index/mapper/simple/test1.json | 68 +++++++++--------- .../ParseDocumentTypeLevelsTests.java | 54 +++++++------- .../org/elasticsearch/index/query/data.json | 72 +++++++++---------- .../percolator/PercolatorTests.java | 22 ++---- 11 files changed, 200 insertions(+), 153 deletions(-) create mode 100644 src/test/java/org/elasticsearch/index/mapper/simple/test1-withtype.json diff --git a/docs/reference/docs/index_.asciidoc b/docs/reference/docs/index_.asciidoc index c193cbdeaaf9a..de062c80ff37c 100644 --- a/docs/reference/docs/index_.asciidoc +++ b/docs/reference/docs/index_.asciidoc @@ -33,7 +33,7 @@ The result of the above index operation is: The index operation automatically creates an index if it has not been created before (check out the -<> for manually +<> for manually creating an index), and also automatically creates a dynamic type mapping for the specific type if one has not yet been created (check out the <> @@ -44,12 +44,21 @@ objects will automatically be added to the mapping definition of the type specified. Check out the <> section for more information on mapping definitions. -Though explained on the <> section, -it's important to note that the format of the JSON document can also -include the type (very handy when using JSON mappers), for example: +Note that the format of the JSON document can also include the type (very handy +when using JSON mappers) if the `index.mapping.allow_type_wrapper` setting is +set to true, for example: [source,js] -------------------------------------------------- +$ curl -XPOST 'http://localhost:9200/twitter' -d '{ + "settings": { + "index": { + "mapping.allow_type_wrapper": true + } + } +}' +{"acknowledged":true} + $ curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{ "tweet" : { "user" : "kimchy", diff --git a/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java b/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java index d2745d5136374..98157f288704c 100644 --- a/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java +++ b/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java @@ -239,6 +239,8 @@ protected ParseContext initialValue() { } }; + public static final String ALLOW_TYPE_WRAPPER = "index.mapping.allow_type_wrapper"; + private final String index; private final Settings indexSettings; @@ -494,18 +496,15 @@ public ParsedDocument parse(SourceToParse source, @Nullable ParseListener listen } else if (token != XContentParser.Token.FIELD_NAME) { throw new MapperParsingException("Malformed content, after first object, either the type field or the actual properties should exist"); } - if (type.equals(parser.currentName())) { - // first field is the same as the type, this might be because the type is provided, and the object exists within it - // or because there is a valid field that by chance is named as the type - - // Note, in this case, we only handle plain value types, an object type will be analyzed as if it was the type itself - // and other same level fields will be ignored - token = parser.nextToken(); + // first field is the same as the type, this might be because the + // type is provided, and the object exists within it or because + // there is a valid field that by chance is named as the type. + // Because of this, by default wrapping a document in a type is + // disabled, but can be enabled by setting + // index.mapping.allow_type_wrapper to true + if (type.equals(parser.currentName()) && indexSettings.getAsBoolean(ALLOW_TYPE_WRAPPER, false)) { + parser.nextToken(); countDownTokens++; - // commented out, allow for same type with START_OBJECT, we do our best to handle it except for the above corner case -// if (token != XContentParser.Token.START_OBJECT) { -// throw new MapperException("Malformed content, a field with the same name as the type must be an object with the properties/fields within it"); -// } } for (RootMapper rootMapper : rootMappersOrdered) { diff --git a/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java b/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java index 34653c9c35ba4..5599857fd1849 100644 --- a/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java +++ b/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java @@ -853,6 +853,6 @@ private void assertHits(SearchHits hits, String... ids) { } private String source(String id, String nameValue) { - return "{ type1 : { \"id\" : \"" + id + "\", \"name\" : \"" + nameValue + "\" } }"; + return "{ \"id\" : \"" + id + "\", \"name\" : \"" + nameValue + "\" }"; } } diff --git a/src/test/java/org/elasticsearch/index/mapper/all/SimpleAllMapperTests.java b/src/test/java/org/elasticsearch/index/mapper/all/SimpleAllMapperTests.java index 0d8d8bebd7634..1f15caf861892 100644 --- a/src/test/java/org/elasticsearch/index/mapper/all/SimpleAllMapperTests.java +++ b/src/test/java/org/elasticsearch/index/mapper/all/SimpleAllMapperTests.java @@ -231,11 +231,11 @@ public void testRandom() throws Exception { // reparse it DocumentMapper builtDocMapper = MapperTestUtils.newParser().parse(builtMapping); - byte[] json = jsonBuilder().startObject().startObject("test") + byte[] json = jsonBuilder().startObject() .field("foo", "bar") .field("_id", 1) .field("foobar", "foobar") - .endObject().endObject().bytes().array(); + .endObject().bytes().array(); Document doc = builtDocMapper.parse(new BytesArray(json)).rootDoc(); AllField field = (AllField) doc.getField("_all"); if (enabled) { diff --git a/src/test/java/org/elasticsearch/index/mapper/all/test1.json b/src/test/java/org/elasticsearch/index/mapper/all/test1.json index c3e72dfa9a8d4..834400a48cbe8 100644 --- a/src/test/java/org/elasticsearch/index/mapper/all/test1.json +++ b/src/test/java/org/elasticsearch/index/mapper/all/test1.json @@ -1,20 +1,18 @@ { - "person":{ - "_boost":3.7, - "_id":"1", - "name":{ - "first":"shay", - "last":"banon" + "_boost":3.7, + "_id":"1", + "name":{ + "first":"shay", + "last":"banon" + }, + "address":{ + "first":{ + "location":"first location" }, - "address":{ - "first":{ - "location":"first location" - }, - "last":{ - "location":"last location" - } - }, - "simple1":1, - "simple2":2 - } -} \ No newline at end of file + "last":{ + "location":"last location" + } + }, + "simple1":1, + "simple2":2 +} diff --git a/src/test/java/org/elasticsearch/index/mapper/simple/SimpleMapperTests.java b/src/test/java/org/elasticsearch/index/mapper/simple/SimpleMapperTests.java index dd7291b22bb43..79951e46a9630 100644 --- a/src/test/java/org/elasticsearch/index/mapper/simple/SimpleMapperTests.java +++ b/src/test/java/org/elasticsearch/index/mapper/simple/SimpleMapperTests.java @@ -22,6 +22,8 @@ import com.google.common.base.Charsets; import org.elasticsearch.common.bytes.BytesArray; import org.elasticsearch.common.bytes.BytesReference; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; import org.elasticsearch.index.mapper.*; import org.elasticsearch.index.mapper.ParseContext.Document; import org.elasticsearch.test.ElasticsearchTestCase; @@ -128,4 +130,18 @@ public void testNoDocumentSent() throws Exception { assertThat(e.getMessage(), equalTo("failed to parse, document is empty")); } } + + @Test + public void testTypeWrapperWithSetting() throws Exception { + String mapping = copyToStringFromClasspath("/org/elasticsearch/index/mapper/simple/test-mapping.json"); + Settings settings = ImmutableSettings.settingsBuilder().put("index.mapping.allow_type_wrapper", true).build(); + DocumentMapper docMapper = MapperTestUtils.newParser(settings).parse(mapping); + + assertThat((String) docMapper.meta().get("param1"), equalTo("value1")); + + BytesReference json = new BytesArray(copyToBytesFromClasspath("/org/elasticsearch/index/mapper/simple/test1-withtype.json")); + Document doc = docMapper.parse(json).rootDoc(); + assertThat(doc.get(docMapper.uidMapper().names().indexName()), equalTo(Uid.createUid("person", "1"))); + assertThat(doc.get(docMapper.mappers().name("first").mapper().names().indexName()), equalTo("shay")); + } } diff --git a/src/test/java/org/elasticsearch/index/mapper/simple/test1-withtype.json b/src/test/java/org/elasticsearch/index/mapper/simple/test1-withtype.json new file mode 100644 index 0000000000000..096554abb201b --- /dev/null +++ b/src/test/java/org/elasticsearch/index/mapper/simple/test1-withtype.json @@ -0,0 +1,43 @@ +{ + person:{ + _boost:3.7, + _id:"1", + name:{ + first:"shay", + last:"banon" + }, + address:{ + first:{ + location:"first location" + }, + last:{ + location:"last location" + } + }, + age:32, + birthDate:"1977-11-15", + nerd:true, + dogs:["buck", "mia"], + complex:[ + { + value1:"value1" + }, + { + value2:"value2" + } + ], + complex2:[ + [ + { + value1:"value1" + } + ], + [ + { + value2:"value2" + } + ] + ], + nullValue:null + } +} diff --git a/src/test/java/org/elasticsearch/index/mapper/simple/test1.json b/src/test/java/org/elasticsearch/index/mapper/simple/test1.json index 0d6786929995f..93507daffefe8 100644 --- a/src/test/java/org/elasticsearch/index/mapper/simple/test1.json +++ b/src/test/java/org/elasticsearch/index/mapper/simple/test1.json @@ -1,43 +1,41 @@ { - person:{ - _boost:3.7, - _id:"1", - name:{ - first:"shay", - last:"banon" + _boost:3.7, + _id:"1", + name:{ + first:"shay", + last:"banon" + }, + address:{ + first:{ + location:"first location" }, - address:{ - first:{ - location:"first location" - }, - last:{ - location:"last location" - } + last:{ + location:"last location" + } + }, + age:32, + birthDate:"1977-11-15", + nerd:true, + dogs:["buck", "mia"], + complex:[ + { + value1:"value1" }, - age:32, - birthDate:"1977-11-15", - nerd:true, - dogs:["buck", "mia"], - complex:[ + { + value2:"value2" + } + ], + complex2:[ + [ { value1:"value1" - }, + } + ], + [ { value2:"value2" } - ], - complex2:[ - [ - { - value1:"value1" - } - ], - [ - { - value2:"value2" - } - ] - ], - nullValue:null - } -} \ No newline at end of file + ] + ], + nullValue:null +} diff --git a/src/test/java/org/elasticsearch/index/mapper/typelevels/ParseDocumentTypeLevelsTests.java b/src/test/java/org/elasticsearch/index/mapper/typelevels/ParseDocumentTypeLevelsTests.java index 2bde20e14f2f9..56d1a961c8d6a 100644 --- a/src/test/java/org/elasticsearch/index/mapper/typelevels/ParseDocumentTypeLevelsTests.java +++ b/src/test/java/org/elasticsearch/index/mapper/typelevels/ParseDocumentTypeLevelsTests.java @@ -26,9 +26,7 @@ import org.elasticsearch.test.ElasticsearchTestCase; import org.junit.Test; -import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.equalTo; -import static org.hamcrest.Matchers.nullValue; /** * @@ -68,9 +66,9 @@ public void testTypeLevel() throws Exception { .endObject().endObject() .bytes()); - assertThat(doc.rootDoc().get("test1"), equalTo("value1")); - assertThat(doc.rootDoc().get("test2"), equalTo("value2")); - assertThat(doc.rootDoc().get("inner.inner_field"), equalTo("inner_value")); + assertThat(doc.rootDoc().get("type.test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("type.test2"), equalTo("value2")); + assertThat(doc.rootDoc().get("type.inner.inner_field"), equalTo("inner_value")); } @Test @@ -109,10 +107,10 @@ public void testTypeLevelWithFieldTypeAsValue() throws Exception { .endObject().endObject() .bytes()); - assertThat(doc.rootDoc().get("type"), equalTo("value_type")); - assertThat(doc.rootDoc().get("test1"), equalTo("value1")); - assertThat(doc.rootDoc().get("test2"), equalTo("value2")); - assertThat(doc.rootDoc().get("inner.inner_field"), equalTo("inner_value")); + assertThat(doc.rootDoc().get("type.type"), equalTo("value_type")); + assertThat(doc.rootDoc().get("type.test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("type.test2"), equalTo("value2")); + assertThat(doc.rootDoc().get("type.inner.inner_field"), equalTo("inner_value")); } @Test @@ -131,9 +129,9 @@ public void testNoLevelWithFieldTypeAsObject() throws Exception { .bytes()); // in this case, we analyze the type object as the actual document, and ignore the other same level fields - assertThat(doc.rootDoc().get("type_field"), equalTo("type_value")); - assertThat(doc.rootDoc().get("test1"), nullValue()); - assertThat(doc.rootDoc().get("test2"), nullValue()); + assertThat(doc.rootDoc().get("type.type_field"), equalTo("type_value")); + assertThat(doc.rootDoc().get("test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("test2"), equalTo("value2")); } @Test @@ -151,10 +149,10 @@ public void testTypeLevelWithFieldTypeAsObject() throws Exception { .endObject().endObject() .bytes()); - assertThat(doc.rootDoc().get("type.type_field"), equalTo("type_value")); - assertThat(doc.rootDoc().get("test1"), equalTo("value1")); - assertThat(doc.rootDoc().get("test2"), equalTo("value2")); - assertThat(doc.rootDoc().get("inner.inner_field"), equalTo("inner_value")); + assertThat(doc.rootDoc().get("type.type.type_field"), equalTo("type_value")); + assertThat(doc.rootDoc().get("type.test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("type.test2"), equalTo("value2")); + assertThat(doc.rootDoc().get("type.inner.inner_field"), equalTo("inner_value")); } @Test @@ -172,10 +170,10 @@ public void testNoLevelWithFieldTypeAsValueNotFirst() throws Exception { .endObject().endObject() .bytes()); - assertThat(doc.rootDoc().get("type"), equalTo("value_type")); - assertThat(doc.rootDoc().get("test1"), equalTo("value1")); - assertThat(doc.rootDoc().get("test2"), equalTo("value2")); - assertThat(doc.rootDoc().get("inner.inner_field"), equalTo("inner_value")); + assertThat(doc.rootDoc().get("type.type"), equalTo("value_type")); + assertThat(doc.rootDoc().get("type.test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("type.test2"), equalTo("value2")); + assertThat(doc.rootDoc().get("type.inner.inner_field"), equalTo("inner_value")); } @Test @@ -193,10 +191,10 @@ public void testTypeLevelWithFieldTypeAsValueNotFirst() throws Exception { .endObject().endObject() .bytes()); - assertThat(doc.rootDoc().get("type"), equalTo("value_type")); - assertThat(doc.rootDoc().get("test1"), equalTo("value1")); - assertThat(doc.rootDoc().get("test2"), equalTo("value2")); - assertThat(doc.rootDoc().get("inner.inner_field"), equalTo("inner_value")); + assertThat(doc.rootDoc().get("type.type"), equalTo("value_type")); + assertThat(doc.rootDoc().get("type.test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("type.test2"), equalTo("value2")); + assertThat(doc.rootDoc().get("type.inner.inner_field"), equalTo("inner_value")); } @Test @@ -236,9 +234,9 @@ public void testTypeLevelWithFieldTypeAsObjectNotFirst() throws Exception { .endObject().endObject() .bytes()); - assertThat(doc.rootDoc().get("type.type_field"), equalTo("type_value")); - assertThat(doc.rootDoc().get("test1"), equalTo("value1")); - assertThat(doc.rootDoc().get("test2"), equalTo("value2")); - assertThat(doc.rootDoc().get("inner.inner_field"), equalTo("inner_value")); + assertThat(doc.rootDoc().get("type.type.type_field"), equalTo("type_value")); + assertThat(doc.rootDoc().get("type.test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("type.test2"), equalTo("value2")); + assertThat(doc.rootDoc().get("type.inner.inner_field"), equalTo("inner_value")); } } diff --git a/src/test/java/org/elasticsearch/index/query/data.json b/src/test/java/org/elasticsearch/index/query/data.json index ef942d9c55d2c..b3c6db83a07d2 100644 --- a/src/test/java/org/elasticsearch/index/query/data.json +++ b/src/test/java/org/elasticsearch/index/query/data.json @@ -1,47 +1,45 @@ { - person:{ - _boost:3.7, - _id:"1", - name:{ - first:"shay", - last:"banon" + _boost:3.7, + _id:"1", + name:{ + first:"shay", + last:"banon" + }, + address:{ + first:{ + location:"first location" }, - address:{ - first:{ - location:"first location" - }, - last:{ - location:"last location" - } + last:{ + location:"last location" + } + }, + age:32, + birthDate:"1977-11-15", + nerd:true, + dogs:["buck", "mia"], + complex:[ + { + value1:"value1" }, - age:32, - birthDate:"1977-11-15", - nerd:true, - dogs:["buck", "mia"], - complex:[ + { + value2:"value2" + } + ], + complex2:[ + [ { value1:"value1" - }, + } + ], + [ { value2:"value2" } - ], - complex2:[ - [ - { - value1:"value1" - } - ], - [ - { - value2:"value2" - } - ] - ], - nullValue:null, - "location":{ - "lat":1.1, - "lon":1.2 - } + ] + ], + nullValue:null, + "location":{ + "lat":1.1, + "lon":1.2 } } \ No newline at end of file diff --git a/src/test/java/org/elasticsearch/percolator/PercolatorTests.java b/src/test/java/org/elasticsearch/percolator/PercolatorTests.java index 6c20a50cfe2de..42859d7591321 100644 --- a/src/test/java/org/elasticsearch/percolator/PercolatorTests.java +++ b/src/test/java/org/elasticsearch/percolator/PercolatorTests.java @@ -164,11 +164,6 @@ public void testSimple2() throws Exception { .field("field2", "value") .endObject().endObject(); - XContentBuilder docWithType = XContentFactory.jsonBuilder().startObject().startObject("doc").startObject("type1") - .field("field1", 1) - .field("field2", "value") - .endObject().endObject().endObject(); - PercolateResponse response = client().preparePercolate().setSource(doc) .setIndices("test").setDocumentType("type1") .execute().actionGet(); @@ -187,13 +182,6 @@ public void testSimple2() throws Exception { assertThat(response.getMatches(), arrayWithSize(1)); assertThat(convertFromTextArray(response.getMatches(), "test"), arrayContaining("test1")); - response = client().preparePercolate() - .setIndices("test").setDocumentType("type1") - .setSource(docWithType).execute().actionGet(); - assertMatchCount(response, 1l); - assertThat(response.getMatches(), arrayWithSize(1)); - assertThat(convertFromTextArray(response.getMatches(), "test"), arrayContaining("test1")); - // add second query... client().prepareIndex("test", PercolatorService.TYPE_NAME, "test2") .setSource(XContentFactory.jsonBuilder().startObject().field("query", termQuery("field1", 1)).endObject()) @@ -430,7 +418,7 @@ public void multiplePercolators() throws Exception { percolate = client().preparePercolate() .setIndices("test").setDocumentType("type1") - .setSource(jsonBuilder().startObject().startObject("doc").startObject("type1").field("field1", "value2").endObject().endObject().endObject()) + .setSource(jsonBuilder().startObject().startObject("doc").field("field1", "value2").endObject().endObject()) .execute().actionGet(); assertMatchCount(percolate, 1l); assertThat(percolate.getMatches(), arrayWithSize(1)); @@ -471,7 +459,7 @@ public void dynamicAddingRemovingQueries() throws Exception { percolate = client().preparePercolate() .setIndices("test").setDocumentType("type1") - .setSource(jsonBuilder().startObject().startObject("doc").startObject("type1").field("field1", "value2").endObject().endObject().endObject()) + .setSource(jsonBuilder().startObject().startObject("doc").field("field1", "value2").endObject().endObject()) .execute().actionGet(); assertMatchCount(percolate, 1l); assertThat(percolate.getMatches(), arrayWithSize(1)); @@ -487,7 +475,7 @@ public void dynamicAddingRemovingQueries() throws Exception { .execute().actionGet(); PercolateSourceBuilder sourceBuilder = new PercolateSourceBuilder() - .setDoc(docBuilder().setDoc(jsonBuilder().startObject().startObject("type1").field("field1", "value2").endObject().endObject())) + .setDoc(docBuilder().setDoc(jsonBuilder().startObject().field("field1", "value2").endObject())) .setQueryBuilder(termQuery("color", "red")); percolate = client().preparePercolate() .setIndices("test").setDocumentType("type1") @@ -533,9 +521,9 @@ public void percolateWithSizeField() throws Exception { logger.info("--> percolate a document"); PercolateResponse percolate = client().preparePercolate().setIndices("test").setDocumentType("type1") .setSource(jsonBuilder().startObject() - .startObject("doc").startObject("type1") + .startObject("doc") .field("field1", "value1") - .endObject().endObject() + .endObject() .endObject()) .execute().actionGet(); assertMatchCount(percolate, 1l); From b987615f5e96cb6f568e025a449d1f4d7e5402df Mon Sep 17 00:00:00 2001 From: Igor Motov Date: Mon, 13 Jan 2014 11:22:13 -0500 Subject: [PATCH 05/34] Improve support for partial snapshots Fixes #4701. Changes behavior of the snapshot operation. The operation now fails if not all primary shards are available at the beginning of the snapshot operation. The restore operation no longer tries to restore indices with shards that failed or were missing during snapshot operation. --- docs/reference/modules/snapshots.asciidoc | 3 +- .../create/CreateSnapshotRequest.java | 43 +++++++-- .../create/CreateSnapshotRequestBuilder.java | 11 +++ .../create/TransportCreateSnapshotAction.java | 1 + .../restore/RestoreSnapshotRequest.java | 13 ++- .../cluster/metadata/SnapshotMetaData.java | 43 ++++++++- .../snapshots/RestoreService.java | 29 +++++- .../elasticsearch/snapshots/SnapshotInfo.java | 24 ++++- .../snapshots/SnapshotsService.java | 88 ++++++++++++++++--- .../snapshots/AbstractSnapshotTests.java | 30 ------- .../DedicatedClusterSnapshotRestoreTests.java | 76 +++++++++++++++- .../snapshots/RepositoriesTests.java | 2 + .../SharedClusterSnapshotRestoreTests.java | 9 +- .../test/ElasticsearchIntegrationTest.java | 22 +++++ 14 files changed, 320 insertions(+), 74 deletions(-) diff --git a/docs/reference/modules/snapshots.asciidoc b/docs/reference/modules/snapshots.asciidoc index 08aab370357f2..94bf40c4ab58c 100644 --- a/docs/reference/modules/snapshots.asciidoc +++ b/docs/reference/modules/snapshots.asciidoc @@ -101,7 +101,8 @@ supports <>. The snapshot request al `ignore_unavailable` option. Setting it to `true` will cause indices that do not exists to be ignored during snapshot creation. By default, when `ignore_unavailable` option is not set and an index is missing the snapshot request will fail. By setting `include_global_state` to false it's possible to prevent the cluster global state to be stored as part of -the snapshot. +the snapshot. By default, entire snapshot will fail if one or more indices participating in the snapshot don't have +all primary shards available. This behaviour can be changed by setting `partial` to `true`. The index snapshot process is incremental. In the process of making the index snapshot Elasticsearch analyses the list of the index files that are already stored in the repository and copies only files that were created or diff --git a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/CreateSnapshotRequest.java b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/CreateSnapshotRequest.java index 8100b6f272331..2030c3e2383d4 100644 --- a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/CreateSnapshotRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/CreateSnapshotRequest.java @@ -45,6 +45,7 @@ import static org.elasticsearch.common.settings.ImmutableSettings.Builder.EMPTY_SETTINGS; import static org.elasticsearch.common.settings.ImmutableSettings.readSettingsFromStream; import static org.elasticsearch.common.settings.ImmutableSettings.writeSettingsToStream; +import static org.elasticsearch.common.xcontent.support.XContentMapValues.nodeBooleanValue; /** * Create snapshot request @@ -70,6 +71,8 @@ public class CreateSnapshotRequest extends MasterNodeOperationRequest indices) { } /** - * Retuns a list of indices that should be included into the snapshot + * Returns a list of indices that should be included into the snapshot * * @return list of indices */ @@ -215,6 +218,27 @@ public CreateSnapshotRequest indicesOptions(IndicesOptions indicesOptions) { return this; } + + /** + * Returns true if indices with unavailable shards should be be partially snapshotted. + * + * @return the desired behaviour regarding indices options + */ + public boolean partial() { + return partial; + } + + /** + * Set to true to allow indices with unavailable shards to be partially snapshotted. + * + * @param partial true if indices with unavailable shards should be be partially snapshotted. + * @return this request + */ + public CreateSnapshotRequest partial(boolean partial) { + this.partial = partial; + return this; + } + /** * If set to true the request should wait for the snapshot completion before returning. * @@ -315,6 +339,7 @@ public CreateSnapshotRequest includeGlobalState(boolean includeGlobalState) { /** * Returns true if global state should be stored as part of the snapshot + * * @return true if global state should be stored as part of the snapshot */ public boolean includeGlobalState() { @@ -353,17 +378,15 @@ public CreateSnapshotRequest source(Map source) { throw new ElasticsearchIllegalArgumentException("malformed indices section, should be an array of strings"); } } else if (name.equals("ignore_unavailable") || name.equals("ignoreUnavailable")) { - assert entry.getValue() instanceof String; - ignoreUnavailable = Boolean.valueOf(entry.getValue().toString()); + ignoreUnavailable = nodeBooleanValue(entry.getValue()); } else if (name.equals("allow_no_indices") || name.equals("allowNoIndices")) { - assert entry.getValue() instanceof String; - allowNoIndices = Boolean.valueOf(entry.getValue().toString()); + allowNoIndices = nodeBooleanValue(entry.getValue()); } else if (name.equals("expand_wildcards_open") || name.equals("expandWildcardsOpen")) { - assert entry.getValue() instanceof String; - expandWildcardsOpen = Boolean.valueOf(entry.getValue().toString()); + expandWildcardsOpen = nodeBooleanValue(entry.getValue()); } else if (name.equals("expand_wildcards_closed") || name.equals("expandWildcardsClosed")) { - assert entry.getValue() instanceof String; - expandWildcardsClosed = Boolean.valueOf(entry.getValue().toString()); + expandWildcardsClosed = nodeBooleanValue(entry.getValue()); + } else if (name.equals("partial")) { + partial(nodeBooleanValue(entry.getValue())); } else if (name.equals("settings")) { if (!(entry.getValue() instanceof Map)) { throw new ElasticsearchIllegalArgumentException("malformed settings section, should indices an inner object"); @@ -450,6 +473,7 @@ public void readFrom(StreamInput in) throws IOException { settings = readSettingsFromStream(in); includeGlobalState = in.readBoolean(); waitForCompletion = in.readBoolean(); + partial = in.readBoolean(); } @Override @@ -462,5 +486,6 @@ public void writeTo(StreamOutput out) throws IOException { writeSettingsToStream(settings, out); out.writeBoolean(includeGlobalState); out.writeBoolean(waitForCompletion); + out.writeBoolean(partial); } } diff --git a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/CreateSnapshotRequestBuilder.java b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/CreateSnapshotRequestBuilder.java index 35aa98d998ce7..81dae3b44a55e 100644 --- a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/CreateSnapshotRequestBuilder.java +++ b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/CreateSnapshotRequestBuilder.java @@ -112,6 +112,17 @@ public CreateSnapshotRequestBuilder setWaitForCompletion(boolean waitForCompleti return this; } + /** + * If set to true the request should snapshot indices with unavailable shards + * + * @param partial true if request should snapshot indices with unavailable shards + * @return this builder + */ + public CreateSnapshotRequestBuilder setPartial(boolean partial) { + request.partial(partial); + return this; + } + /** * Sets repository-specific snapshot settings. *

diff --git a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/TransportCreateSnapshotAction.java b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/TransportCreateSnapshotAction.java index 61f0db8aef08e..5b54e414031fa 100644 --- a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/TransportCreateSnapshotAction.java +++ b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/TransportCreateSnapshotAction.java @@ -78,6 +78,7 @@ protected void masterOperation(final CreateSnapshotRequest request, ClusterState new SnapshotsService.SnapshotRequest("create_snapshot[" + request.snapshot() + "]", request.snapshot(), request.repository()) .indices(request.indices()) .indicesOptions(request.indicesOptions()) + .partial(request.partial()) .settings(request.settings()) .includeGlobalState(request.includeGlobalState()) .masterNodeTimeout(request.masterNodeTimeout()); diff --git a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/restore/RestoreSnapshotRequest.java b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/restore/RestoreSnapshotRequest.java index f5ba01cf650f3..b009e76a91b74 100644 --- a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/restore/RestoreSnapshotRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/restore/RestoreSnapshotRequest.java @@ -44,6 +44,7 @@ import static org.elasticsearch.common.settings.ImmutableSettings.Builder.EMPTY_SETTINGS; import static org.elasticsearch.common.settings.ImmutableSettings.readSettingsFromStream; import static org.elasticsearch.common.settings.ImmutableSettings.writeSettingsToStream; +import static org.elasticsearch.common.xcontent.support.XContentMapValues.nodeBooleanValue; /** * Restore snapshot request @@ -397,17 +398,13 @@ public RestoreSnapshotRequest source(Map source) { throw new ElasticsearchIllegalArgumentException("malformed indices section, should be an array of strings"); } } else if (name.equals("ignore_unavailable") || name.equals("ignoreUnavailable")) { - assert entry.getValue() instanceof String; - ignoreUnavailable = Boolean.valueOf(entry.getValue().toString()); + ignoreUnavailable = nodeBooleanValue(entry.getValue()); } else if (name.equals("allow_no_indices") || name.equals("allowNoIndices")) { - assert entry.getValue() instanceof String; - allowNoIndices = Boolean.valueOf(entry.getValue().toString()); + allowNoIndices = nodeBooleanValue(entry.getValue()); } else if (name.equals("expand_wildcards_open") || name.equals("expandWildcardsOpen")) { - assert entry.getValue() instanceof String; - expandWildcardsOpen = Boolean.valueOf(entry.getValue().toString()); + expandWildcardsOpen = nodeBooleanValue(entry.getValue()); } else if (name.equals("expand_wildcards_closed") || name.equals("expandWildcardsClosed")) { - assert entry.getValue() instanceof String; - expandWildcardsClosed = Boolean.valueOf(entry.getValue().toString()); + expandWildcardsClosed = nodeBooleanValue(entry.getValue()); } else if (name.equals("settings")) { if (!(entry.getValue() instanceof Map)) { throw new ElasticsearchIllegalArgumentException("malformed settings section, should indices an inner object"); diff --git a/src/main/java/org/elasticsearch/cluster/metadata/SnapshotMetaData.java b/src/main/java/org/elasticsearch/cluster/metadata/SnapshotMetaData.java index effa39eb1af4c..7235c855df40c 100644 --- a/src/main/java/org/elasticsearch/cluster/metadata/SnapshotMetaData.java +++ b/src/main/java/org/elasticsearch/cluster/metadata/SnapshotMetaData.java @@ -203,7 +203,8 @@ public static enum State { STARTED((byte) 1), SUCCESS((byte) 2), FAILED((byte) 3), - ABORTED((byte) 4); + ABORTED((byte) 4), + MISSING((byte) 5); private byte value; @@ -216,7 +217,43 @@ public byte value() { } public boolean completed() { - return this == SUCCESS || this == FAILED; + switch (this) { + case INIT: + return false; + case STARTED: + return false; + case SUCCESS: + return true; + case FAILED: + return true; + case ABORTED: + return false; + case MISSING: + return true; + default: + assert false; + return true; + } + } + + public boolean failed() { + switch (this) { + case INIT: + return false; + case STARTED: + return false; + case SUCCESS: + return false; + case FAILED: + return true; + case ABORTED: + return true; + case MISSING: + return true; + default: + assert false; + return false; + } } public static State fromValue(byte value) { @@ -231,6 +268,8 @@ public static State fromValue(byte value) { return FAILED; case 4: return ABORTED; + case 5: + return MISSING; default: throw new ElasticsearchIllegalArgumentException("No snapshot state for value [" + value + "]"); } diff --git a/src/main/java/org/elasticsearch/snapshots/RestoreService.java b/src/main/java/org/elasticsearch/snapshots/RestoreService.java index 22a5d5be79123..f08f5b25a5f19 100644 --- a/src/main/java/org/elasticsearch/snapshots/RestoreService.java +++ b/src/main/java/org/elasticsearch/snapshots/RestoreService.java @@ -162,6 +162,10 @@ public ClusterState execute(ClusterState currentState) { ImmutableMap.Builder shards = ImmutableMap.builder(); for (Map.Entry indexEntry : renamedIndices.entrySet()) { String index = indexEntry.getValue(); + // Make sure that index was fully snapshotted - don't restore + if (failed(snapshot, index)) { + throw new SnapshotRestoreException(snapshotId, "index [" + index + "] wasn't fully snapshotted - cannot restore"); + } RestoreSource restoreSource = new RestoreSource(snapshotId, index); String renamedIndex = indexEntry.getKey(); IndexMetaData snapshotIndexMetaData = metaData.index(index); @@ -391,6 +395,24 @@ private void processDeletedIndices(ClusterChangedEvent event) { } } + private boolean failed(Snapshot snapshot, String index) { + for (SnapshotShardFailure failure : snapshot.shardFailures()) { + if (index.equals(failure.index())) { + return true; + } + } + return false; + } + + private boolean failed(Snapshot snapshot, String index, int shard) { + for (SnapshotShardFailure failure : snapshot.shardFailures()) { + if (index.equals(failure.index()) && shard == failure.shardId()) { + return true; + } + } + return false; + } + /** * Adds restore completion listener *

@@ -427,16 +449,17 @@ public void clusterChanged(ClusterChangedEvent event) { /** * Checks if a repository is currently in use by one of the snapshots + * * @param clusterState cluster state - * @param repository repository id + * @param repository repository id * @return true if repository is currently in use by one of the running snapshots */ public static boolean isRepositoryInUse(ClusterState clusterState, String repository) { MetaData metaData = clusterState.metaData(); RestoreMetaData snapshots = metaData.custom(RestoreMetaData.TYPE); if (snapshots != null) { - for(RestoreMetaData.Entry snapshot : snapshots.entries()) { - if(repository.equals(snapshot.snapshotId().getRepository())) { + for (RestoreMetaData.Entry snapshot : snapshots.entries()) { + if (repository.equals(snapshot.snapshotId().getRepository())) { return true; } } diff --git a/src/main/java/org/elasticsearch/snapshots/SnapshotInfo.java b/src/main/java/org/elasticsearch/snapshots/SnapshotInfo.java index ebf05ab6d872f..768a0157945e6 100644 --- a/src/main/java/org/elasticsearch/snapshots/SnapshotInfo.java +++ b/src/main/java/org/elasticsearch/snapshots/SnapshotInfo.java @@ -24,7 +24,6 @@ import org.elasticsearch.common.io.stream.Streamable; import org.elasticsearch.common.joda.FormatDateTimeFormatter; import org.elasticsearch.common.joda.Joda; -import org.elasticsearch.common.unit.TimeValue; import org.elasticsearch.common.xcontent.ToXContent; import org.elasticsearch.common.xcontent.XContentBuilder; import org.elasticsearch.common.xcontent.XContentBuilderString; @@ -43,6 +42,8 @@ public class SnapshotInfo implements ToXContent, Streamable { private SnapshotState state; + private String reason; + private ImmutableList indices; private long startTime; @@ -67,6 +68,7 @@ public class SnapshotInfo implements ToXContent, Streamable { public SnapshotInfo(Snapshot snapshot) { name = snapshot.name(); state = snapshot.state(); + reason = snapshot.reason(); indices = snapshot.indices(); startTime = snapshot.startTime(); endTime = snapshot.endTime(); @@ -93,6 +95,15 @@ public SnapshotState state() { return state; } + /** + * Returns snapshot failure reason + * + * @return snapshot failure reason + */ + public String reason() { + return reason; + } + /** * Returns indices that were included into this snapshot * @@ -137,7 +148,7 @@ public int totalShards() { * @return number of failed shards */ public int failedShards() { - return totalShards - successfulShards; + return totalShards - successfulShards; } /** @@ -162,6 +173,9 @@ public ImmutableList shardFailures() { * Returns snapshot REST status */ public RestStatus status() { + if (state == SnapshotState.FAILED) { + return RestStatus.INTERNAL_SERVER_ERROR; + } if (shardFailures.size() == 0) { return RestStatus.OK; } @@ -179,6 +193,7 @@ public RestStatus status() { static final class Fields { static final XContentBuilderString INDICES = new XContentBuilderString("indices"); static final XContentBuilderString STATE = new XContentBuilderString("state"); + static final XContentBuilderString REASON = new XContentBuilderString("reason"); static final XContentBuilderString START_TIME = new XContentBuilderString("start_time"); static final XContentBuilderString START_TIME_IN_MILLIS = new XContentBuilderString("start_time_in_millis"); static final XContentBuilderString END_TIME = new XContentBuilderString("end_time"); @@ -202,6 +217,9 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws } builder.endArray(); builder.field(Fields.STATE, state); + if (reason != null) { + builder.field(Fields.REASON, reason); + } if (startTime != 0) { builder.field(Fields.START_TIME, DATE_TIME_FORMATTER.printer().print(startTime)); builder.field(Fields.START_TIME_IN_MILLIS, startTime); @@ -235,6 +253,7 @@ public void readFrom(StreamInput in) throws IOException { } indices = indicesListBuilder.build(); state = SnapshotState.fromValue(in.readByte()); + reason = in.readOptionalString(); startTime = in.readVLong(); endTime = in.readVLong(); totalShards = in.readVInt(); @@ -259,6 +278,7 @@ public void writeTo(StreamOutput out) throws IOException { out.writeString(index); } out.writeByte(state.value()); + out.writeOptionalString(reason); out.writeVLong(startTime); out.writeVLong(endTime); out.writeVInt(totalShards); diff --git a/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java b/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java index b5f398173c780..641d5c2a255d9 100644 --- a/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java +++ b/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java @@ -59,6 +59,7 @@ import static com.google.common.collect.Lists.newArrayList; import static com.google.common.collect.Maps.newHashMap; import static com.google.common.collect.Maps.newHashMapWithExpectedSize; +import static com.google.common.collect.Sets.newHashSet; /** * Service responsible for creating snapshots @@ -67,7 +68,7 @@ *

    *
  • On the master node the {@link #createSnapshot(SnapshotRequest, CreateSnapshotListener)} is called and makes sure that no snapshots is currently running * and registers the new snapshot in cluster state
    • - *
  • When cluster state is updated the {@link #beginSnapshot(ClusterState, SnapshotMetaData.Entry, CreateSnapshotListener)} method + *
  • When cluster state is updated the {@link #beginSnapshot(ClusterState, SnapshotMetaData.Entry, boolean, CreateSnapshotListener)} method * kicks in and initializes the snapshot in the repository and then populates list of shards that needs to be snapshotted in cluster state
    • *
  • Each data node is watching for these shards and when new shards scheduled for snapshotting appear in the cluster state, data nodes * start processing them through {@link #processIndexShardSnapshots(SnapshotMetaData)} method
    • @@ -187,7 +188,7 @@ public void clusterStateProcessed(String source, ClusterState oldState, final Cl threadPool.executor(ThreadPool.Names.SNAPSHOT).execute(new Runnable() { @Override public void run() { - beginSnapshot(newState, newSnapshot, listener); + beginSnapshot(newState, newSnapshot, request.partial, listener); } }); } @@ -243,9 +244,10 @@ private void validate(SnapshotRequest request, ClusterState state) throws Elasti * * @param clusterState cluster state * @param snapshot snapshot meta data + * @param partial allow partial snapshots * @param userCreateSnapshotListener listener */ - private void beginSnapshot(ClusterState clusterState, final SnapshotMetaData.Entry snapshot, final CreateSnapshotListener userCreateSnapshotListener) { + private void beginSnapshot(ClusterState clusterState, final SnapshotMetaData.Entry snapshot, final boolean partial, final CreateSnapshotListener userCreateSnapshotListener) { boolean snapshotCreated = false; try { Repository repository = repositoriesService.repository(snapshot.snapshotId().getRepository()); @@ -271,6 +273,7 @@ private void beginSnapshot(ClusterState clusterState, final SnapshotMetaData.Ent clusterService.submitStateUpdateTask("update_snapshot [" + snapshot + "]", new ProcessedClusterStateUpdateTask() { boolean accepted = false; SnapshotMetaData.Entry updatedSnapshot; + String failure = null; @Override public ClusterState execute(ClusterState currentState) { @@ -282,6 +285,15 @@ public ClusterState execute(ClusterState currentState) { if (entry.snapshotId().equals(snapshot.snapshotId())) { // Replace the snapshot that was just created ImmutableMap shards = shards(snapshot.snapshotId(), currentState, snapshot.indices()); + if (!partial) { + Set indicesWithMissingShards = indicesWithMissingShards(shards); + if (indicesWithMissingShards != null) { + updatedSnapshot = new SnapshotMetaData.Entry(snapshot.snapshotId(), snapshot.includeGlobalState(), State.FAILED, snapshot.indices(), shards); + entries.add(updatedSnapshot); + failure = "Indices don't have primary shards +[" + indicesWithMissingShards + "]"; + continue; + } + } updatedSnapshot = new SnapshotMetaData.Entry(snapshot.snapshotId(), snapshot.includeGlobalState(), State.STARTED, snapshot.indices(), shards); entries.add(updatedSnapshot); if (!completed(shards.values())) { @@ -310,8 +322,11 @@ public void clusterStateProcessed(String source, ClusterState oldState, ClusterS userCreateSnapshotListener.onResponse(); // Now that snapshot completion listener is registered we can end the snapshot if needed + // We should end snapshot only if 1) we didn't accept it for processing (which happens when there + // is nothing to do) and 2) there was a snapshot in metadata that we should end. Otherwise we should + // go ahead and continue working on this snapshot rather then end here. if (!accepted && updatedSnapshot != null) { - endSnapshot(updatedSnapshot); + endSnapshot(updatedSnapshot, failure); } } }); @@ -602,6 +617,25 @@ private boolean completed(Collection shard return true; } + /** + * Returns list of indices with missing shards + * + * @param shards list of shard statuses + * @return list of failed indices + */ + private Set indicesWithMissingShards(ImmutableMap shards) { + Set indices = null; + for (ImmutableMap.Entry entry : shards.entrySet()) { + if (entry.getValue().state() == State.MISSING) { + if (indices == null) { + indices = newHashSet(); + } + indices.add(entry.getKey().getIndex()); + } + } + return indices; + } + /** * Updates the shard status on master node * @@ -661,25 +695,38 @@ public void onFailure(String source, Throwable t) { * * @param entry snapshot */ - private void endSnapshot(final SnapshotMetaData.Entry entry) { + private void endSnapshot(SnapshotMetaData.Entry entry) { + endSnapshot(entry, null); + } + + + /** + * Finalizes the shard in repository and then removes it from cluster state + *

    + * This is non-blocking method that runs on a thread from SNAPSHOT thread pool + * + * @param entry snapshot + * @param failure failure reason or null if snapshot was successful + */ + private void endSnapshot(final SnapshotMetaData.Entry entry, final String failure) { threadPool.executor(ThreadPool.Names.SNAPSHOT).execute(new Runnable() { @Override public void run() { SnapshotId snapshotId = entry.snapshotId(); try { final Repository repository = repositoriesService.repository(snapshotId.getRepository()); - logger.trace("[{}] finalizing snapshot in repository", snapshotId); + logger.trace("[{}] finalizing snapshot in repository, state: [{}], failure[{}]", snapshotId, entry.state(), failure); ArrayList failures = newArrayList(); ArrayList shardFailures = newArrayList(); for (Map.Entry shardStatus : entry.shards().entrySet()) { ShardId shardId = shardStatus.getKey(); ShardSnapshotStatus status = shardStatus.getValue(); - if (status.state() == State.FAILED) { + if (status.state().failed()) { failures.add(new ShardSearchFailure(status.reason(), new SearchShardTarget(status.nodeId(), shardId.getIndex(), shardId.id()))); shardFailures.add(new SnapshotShardFailure(status.nodeId(), shardId.getIndex(), shardId.id(), status.reason())); } } - Snapshot snapshot = repository.finalizeSnapshot(snapshotId, null, entry.shards().size(), ImmutableList.copyOf(shardFailures)); + Snapshot snapshot = repository.finalizeSnapshot(snapshotId, failure, entry.shards().size(), ImmutableList.copyOf(shardFailures)); removeSnapshotFromClusterState(snapshotId, new SnapshotInfo(snapshot), null); } catch (Throwable t) { logger.warn("[{}] failed to finalize snapshot", t, snapshotId); @@ -841,16 +888,17 @@ public void onSnapshotFailure(SnapshotId snapshotId, Throwable t) { /** * Checks if a repository is currently in use by one of the snapshots + * * @param clusterState cluster state - * @param repository repository id + * @param repository repository id * @return true if repository is currently in use by one of the running snapshots */ public static boolean isRepositoryInUse(ClusterState clusterState, String repository) { MetaData metaData = clusterState.metaData(); SnapshotMetaData snapshots = metaData.custom(SnapshotMetaData.TYPE); if (snapshots != null) { - for(SnapshotMetaData.Entry snapshot : snapshots.entries()) { - if(repository.equals(snapshot.snapshotId().getRepository())) { + for (SnapshotMetaData.Entry snapshot : snapshots.entries()) { + if (repository.equals(snapshot.snapshotId().getRepository())) { return true; } } @@ -900,10 +948,9 @@ private ImmutableMap shards(Snaps ShardId shardId = new ShardId(index, i); ShardRouting primary = indexRoutingTable.shard(i).primaryShard(); if (primary == null || !primary.assignedToNode()) { - //TODO: Should we bailout completely or just mark this shard as failed? - builder.put(shardId, new SnapshotMetaData.ShardSnapshotStatus(null, State.FAILED, "primary shard is not allocated")); + builder.put(shardId, new SnapshotMetaData.ShardSnapshotStatus(null, State.MISSING, "primary shard is not allocated")); } else if (!primary.started()) { - builder.put(shardId, new SnapshotMetaData.ShardSnapshotStatus(primary.currentNodeId(), State.FAILED, "primary shard hasn't been started yet")); + builder.put(shardId, new SnapshotMetaData.ShardSnapshotStatus(primary.currentNodeId(), State.MISSING, "primary shard hasn't been started yet")); } else { builder.put(shardId, new SnapshotMetaData.ShardSnapshotStatus(primary.currentNodeId())); } @@ -985,6 +1032,8 @@ public static class SnapshotRequest { private IndicesOptions indicesOptions = IndicesOptions.strict(); + private boolean partial; + private Settings settings; private boolean includeGlobalState; @@ -1059,6 +1108,17 @@ public SnapshotRequest indicesOptions(IndicesOptions indicesOptions) { return this; } + /** + * Set to true if partial snapshot should be allowed + * + * @param partial true if partial snapshots should be allowed + * @return this request + */ + public SnapshotRequest partial(boolean partial) { + this.partial = partial; + return this; + } + /** * Returns cause for snapshot operation * diff --git a/src/test/java/org/elasticsearch/snapshots/AbstractSnapshotTests.java b/src/test/java/org/elasticsearch/snapshots/AbstractSnapshotTests.java index 9221a498d84f1..9114e8cc4e645 100644 --- a/src/test/java/org/elasticsearch/snapshots/AbstractSnapshotTests.java +++ b/src/test/java/org/elasticsearch/snapshots/AbstractSnapshotTests.java @@ -26,8 +26,6 @@ import org.elasticsearch.repositories.RepositoryMissingException; import org.elasticsearch.snapshots.mockstore.MockRepository; import org.elasticsearch.test.ElasticsearchIntegrationTest; -import org.junit.After; -import org.junit.Before; import org.junit.Ignore; import java.io.File; @@ -39,34 +37,6 @@ @Ignore public abstract class AbstractSnapshotTests extends ElasticsearchIntegrationTest { - - @After - public final void wipeAfter() { - wipeRepositories(); - } - - @Before - public final void wipeBefore() { - wipeRepositories(); - } - - /** - * Deletes repositories, supports wildcard notation. - */ - public static void wipeRepositories(String... repositories) { - // if nothing is provided, delete all - if (repositories.length == 0) { - repositories = new String[]{"*"}; - } - for (String repository : repositories) { - try { - client().admin().cluster().prepareDeleteRepository(repository).execute().actionGet(); - } catch (RepositoryMissingException ex) { - // ignore - } - } - } - public static long getFailureCount(String repository) { long failureCount = 0; for (RepositoriesService repositoriesService : cluster().getInstances(RepositoriesService.class)) { diff --git a/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java b/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java index 9e5cefa07bef3..c32bf948f431c 100644 --- a/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java +++ b/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java @@ -22,11 +22,13 @@ import com.carrotsearch.randomizedtesting.LifecycleScope; import org.elasticsearch.action.admin.cluster.repositories.put.PutRepositoryResponse; import org.elasticsearch.action.admin.cluster.snapshots.create.CreateSnapshotResponse; +import org.elasticsearch.action.admin.cluster.snapshots.restore.RestoreSnapshotResponse; import org.elasticsearch.client.Client; import org.elasticsearch.common.settings.ImmutableSettings; import org.elasticsearch.common.unit.TimeValue; import org.elasticsearch.snapshots.mockstore.MockRepositoryModule; import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.elasticsearch.test.junit.annotations.TestLogging; import org.elasticsearch.test.store.MockDirectoryHelper; import org.elasticsearch.threadpool.ThreadPool; import org.junit.Test; @@ -36,7 +38,8 @@ import static com.google.common.collect.Lists.newArrayList; import static org.elasticsearch.common.settings.ImmutableSettings.settingsBuilder; import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked; -import static org.hamcrest.Matchers.equalTo; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertThrows; +import static org.hamcrest.Matchers.*; /** */ @@ -126,4 +129,75 @@ public void snapshotDuringNodeShutdownTest() throws Exception { logger.info("Number of failed shards [{}]", snapshotInfo.shardFailures().size()); logger.info("--> done"); } + + @Test + @TestLogging("snapshots:TRACE") + public void restoreIndexWithMissingShards() throws Exception { + logger.info("--> start 2 nodes"); + cluster().startNode(settingsBuilder().put("gateway.type", "local")); + cluster().startNode(settingsBuilder().put("gateway.type", "local")); + wipeIndices("_all"); + + assertAcked(prepareCreate("test-idx-1", 2, settingsBuilder().put("number_of_shards", 6) + .put("number_of_replicas", 0) + .put(MockDirectoryHelper.RANDOM_NO_DELETE_OPEN_FILE, false))); + ensureGreen(); + + logger.info("--> indexing some data into test-idx-1"); + for (int i = 0; i < 100; i++) { + index("test-idx-1", "doc", Integer.toString(i), "foo", "bar" + i); + } + refresh(); + assertThat(client().prepareCount("test-idx-1").get().getCount(), equalTo(100L)); + + logger.info("--> shutdown one of the nodes"); + cluster().stopRandomNode(); + + assertAcked(prepareCreate("test-idx-2", 1, settingsBuilder().put("number_of_shards", 6) + .put("number_of_replicas", 0) + .put(MockDirectoryHelper.RANDOM_NO_DELETE_OPEN_FILE, false))); + ensureGreen("test-idx-2"); + + logger.info("--> indexing some data into test-idx-2"); + for (int i = 0; i < 100; i++) { + index("test-idx-2", "doc", Integer.toString(i), "foo", "bar" + i); + } + refresh(); + assertThat(client().prepareCount("test-idx-2").get().getCount(), equalTo(100L)); + + logger.info("--> create repository"); + logger.info("--> creating repository"); + PutRepositoryResponse putRepositoryResponse = client().admin().cluster().preparePutRepository("test-repo") + .setType("fs").setSettings(ImmutableSettings.settingsBuilder().put("location", newTempDir())).execute().actionGet(); + assertThat(putRepositoryResponse.isAcknowledged(), equalTo(true)); + + logger.info("--> start snapshot with default settings - should fail"); + CreateSnapshotResponse createSnapshotResponse = client().admin().cluster().prepareCreateSnapshot("test-repo", "test-snap-1").setWaitForCompletion(true).execute().actionGet(); + + assertThat(createSnapshotResponse.getSnapshotInfo().state(), equalTo(SnapshotState.FAILED)); + + createSnapshotResponse = client().admin().cluster().prepareCreateSnapshot("test-repo", "test-snap-2").setWaitForCompletion(true).setPartial(true).execute().actionGet(); + logger.info("State: [{}], Reason: [{}]", createSnapshotResponse.getSnapshotInfo().state(), createSnapshotResponse.getSnapshotInfo().reason()); + assertThat(createSnapshotResponse.getSnapshotInfo().totalShards(), equalTo(12)); + assertThat(createSnapshotResponse.getSnapshotInfo().successfulShards(), lessThan(12)); + assertThat(createSnapshotResponse.getSnapshotInfo().successfulShards(), greaterThan(6)); + assertThat(client().admin().cluster().prepareGetSnapshots("test-repo").setSnapshots("test-snap-2").execute().actionGet().getSnapshots().get(0).state(), equalTo(SnapshotState.SUCCESS)); + + assertAcked(client().admin().indices().prepareClose("test-idx-1", "test-idx-2").execute().actionGet()); + + logger.info("--> restore incomplete snapshot - should fail"); + assertThrows(client().admin().cluster().prepareRestoreSnapshot("test-repo", "test-snap-2").setRestoreGlobalState(false).setWaitForCompletion(true).execute(), SnapshotRestoreException.class); + + logger.info("--> restore snapshot for the index that was snapshotted completely"); + RestoreSnapshotResponse restoreSnapshotResponse = client().admin().cluster().prepareRestoreSnapshot("test-repo", "test-snap-2").setRestoreGlobalState(false).setIndices("test-idx-2").setWaitForCompletion(true).execute().actionGet(); + assertThat(restoreSnapshotResponse.getRestoreInfo(), notNullValue()); + assertThat(restoreSnapshotResponse.getRestoreInfo().totalShards(), equalTo(6)); + assertThat(restoreSnapshotResponse.getRestoreInfo().successfulShards(), equalTo(6)); + assertThat(restoreSnapshotResponse.getRestoreInfo().failedShards(), equalTo(0)); + + ensureGreen("test-idx-2"); + + assertThat(client().prepareCount("test-idx-2").get().getCount(), equalTo(100L)); + + } } diff --git a/src/test/java/org/elasticsearch/snapshots/RepositoriesTests.java b/src/test/java/org/elasticsearch/snapshots/RepositoriesTests.java index 049f130187f4a..7471f1472f2f1 100644 --- a/src/test/java/org/elasticsearch/snapshots/RepositoriesTests.java +++ b/src/test/java/org/elasticsearch/snapshots/RepositoriesTests.java @@ -30,6 +30,8 @@ import org.elasticsearch.cluster.metadata.RepositoryMetaData; import org.elasticsearch.common.settings.ImmutableSettings; import org.elasticsearch.repositories.RepositoryException; +import org.junit.After; +import org.junit.Before; import org.junit.Test; import static org.hamcrest.Matchers.equalTo; diff --git a/src/test/java/org/elasticsearch/snapshots/SharedClusterSnapshotRestoreTests.java b/src/test/java/org/elasticsearch/snapshots/SharedClusterSnapshotRestoreTests.java index 361ab50f4e511..b52d761081fd0 100644 --- a/src/test/java/org/elasticsearch/snapshots/SharedClusterSnapshotRestoreTests.java +++ b/src/test/java/org/elasticsearch/snapshots/SharedClusterSnapshotRestoreTests.java @@ -42,6 +42,8 @@ import org.elasticsearch.snapshots.mockstore.MockRepositoryModule; import org.elasticsearch.test.junit.annotations.TestLogging; import org.elasticsearch.test.store.MockDirectoryHelper; +import org.junit.After; +import org.junit.Before; import org.junit.Test; import java.io.File; @@ -53,7 +55,6 @@ */ public class SharedClusterSnapshotRestoreTests extends AbstractSnapshotTests { - @Override public Settings indexSettings() { // During restore we frequently restore index to exactly the same state it was before, that might cause the same @@ -496,10 +497,10 @@ public void unallocatedShardsTest() throws Exception { logger.info("--> snapshot"); CreateSnapshotResponse createSnapshotResponse = client.admin().cluster().prepareCreateSnapshot("test-repo", "test-snap").setWaitForCompletion(true).setIndices("test-idx").get(); + assertThat(createSnapshotResponse.getSnapshotInfo().state(), equalTo(SnapshotState.FAILED)); assertThat(createSnapshotResponse.getSnapshotInfo().successfulShards(), equalTo(0)); - assertThat(createSnapshotResponse.getSnapshotInfo().totalShards(), equalTo(3)); - assertThat(createSnapshotResponse.getSnapshotInfo().shardFailures().size(), equalTo(3)); - assertThat(createSnapshotResponse.getSnapshotInfo().shardFailures().get(0).reason(), startsWith("primary shard is not allocated")); + assertThat(createSnapshotResponse.getSnapshotInfo().totalShards(), equalTo(0)); + assertThat(createSnapshotResponse.getSnapshotInfo().reason(), startsWith("Indices don't have primary shards")); } @Test diff --git a/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java b/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java index f50345e1c734a..32dcfd890d7b3 100644 --- a/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java +++ b/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java @@ -57,6 +57,7 @@ import org.elasticsearch.index.merge.policy.*; import org.elasticsearch.indices.IndexMissingException; import org.elasticsearch.indices.IndexTemplateMissingException; +import org.elasticsearch.repositories.RepositoryMissingException; import org.elasticsearch.rest.RestStatus; import org.elasticsearch.search.SearchService; import org.junit.After; @@ -189,6 +190,7 @@ public final void before() throws IOException { wipeIndices("_all"); wipeTemplates(); randomIndexTemplate(); + wipeRepositories(); logger.info("[{}#{}]: before test", getTestClass().getSimpleName(), getTestName()); } catch (OutOfMemoryError e) { if (e.getMessage().contains("unable to create new native thread")) { @@ -237,6 +239,7 @@ public final void after() throws IOException { wipeIndices("_all"); // wipe after to make sure we fail in the test that // didn't ack the delete wipeTemplates(); + wipeRepositories(); ensureAllSearchersClosed(); ensureAllFilesClosed(); logger.info("[{}#{}]: cleaned up after test", getTestClass().getSimpleName(), getTestName()); @@ -368,6 +371,25 @@ public static void wipeTemplates(String... templates) { } } + /** + * Deletes repositories, supports wildcard notation. + */ + public static void wipeRepositories(String... repositories) { + if (cluster().size() > 0) { + // if nothing is provided, delete all + if (repositories.length == 0) { + repositories = new String[]{"*"}; + } + for (String repository : repositories) { + try { + client().admin().cluster().prepareDeleteRepository(repository).execute().actionGet(); + } catch (RepositoryMissingException ex) { + // ignore + } + } + } + } + /** * Creates one or more indices and asserts that the indices are acknowledged. If one of the indices * already exists this method will fail and wipe all the indices created so far. From 22a96e6a18819efc17076a0f0c6a79d9e2b2a902 Mon Sep 17 00:00:00 2001 From: Benjamin Vetter Date: Sun, 12 Jan 2014 11:26:39 +0100 Subject: [PATCH 06/34] Added stopwords: _none_ to the docs #329 --- docs/reference/analysis/analyzers.asciidoc | 8 ++++++++ .../analysis/analyzers/standard-analyzer.asciidoc | 2 +- docs/reference/analysis/analyzers/stop-analyzer.asciidoc | 2 +- 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/reference/analysis/analyzers.asciidoc b/docs/reference/analysis/analyzers.asciidoc index b97231b0ab366..22881a68904d2 100644 --- a/docs/reference/analysis/analyzers.asciidoc +++ b/docs/reference/analysis/analyzers.asciidoc @@ -49,6 +49,14 @@ index : stopwords : [test1, test2, test3] -------------------------------------------------- +[[analyzers-stopwords]] +=== Stopwords + +The `stopwords` parameter can be used to provide a custom set of stopwords. As +certain analyzers use a default list of stopwords while others don't, please +check out the individual analyzer sections. In case you want the analyzer to +use no stopwords at all, simply provide `stopwords: _none_` + Below is a list of the built in analyzers. include::analyzers/standard-analyzer.asciidoc[] diff --git a/docs/reference/analysis/analyzers/standard-analyzer.asciidoc b/docs/reference/analysis/analyzers/standard-analyzer.asciidoc index 14027ffde0bb3..0b6faed771212 100644 --- a/docs/reference/analysis/analyzers/standard-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/standard-analyzer.asciidoc @@ -17,7 +17,7 @@ type: [cols="<,<",options="header",] |======================================================================= |Setting |Description -|`stopwords` |A list of stopword to initialize the stop filter with. +|`stopwords` |A list of stopwords to initialize the stop filter with. Defaults to an 'empty' stopword list added[1.0.0.Beta1, Previously defaulted to the English stopwords list] |`max_token_length` |The maximum token length. If a token is seen that diff --git a/docs/reference/analysis/analyzers/stop-analyzer.asciidoc b/docs/reference/analysis/analyzers/stop-analyzer.asciidoc index 18599af0f9053..2a1bfd7311347 100644 --- a/docs/reference/analysis/analyzers/stop-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/stop-analyzer.asciidoc @@ -12,7 +12,7 @@ The following are settings that can be set for a `stop` analyzer type: [cols="<,<",options="header",] |======================================================================= |Setting |Description -|`stopwords` |A list of stopword to initialize the stop filter with. +|`stopwords` |A list of stopwords to initialize the stop filter with. Defaults to the english stop words. |`stopwords_path` |A path (either relative to `config` location, or From ba8e012be90c8f3847bb638a99a75a1b994cf8c5 Mon Sep 17 00:00:00 2001 From: Benjamin Vetter Date: Mon, 13 Jan 2014 20:04:30 +0100 Subject: [PATCH 07/34] Referring to stop analyzer for stopword docs #329 --- docs/reference/analysis/analyzers.asciidoc | 8 -------- docs/reference/analysis/analyzers/lang-analyzer.asciidoc | 3 ++- .../analysis/analyzers/pattern-analyzer.asciidoc | 3 ++- .../analysis/analyzers/snowball-analyzer.asciidoc | 5 +++-- .../analysis/analyzers/standard-analyzer.asciidoc | 3 ++- docs/reference/analysis/analyzers/stop-analyzer.asciidoc | 3 ++- 6 files changed, 11 insertions(+), 14 deletions(-) diff --git a/docs/reference/analysis/analyzers.asciidoc b/docs/reference/analysis/analyzers.asciidoc index 22881a68904d2..b97231b0ab366 100644 --- a/docs/reference/analysis/analyzers.asciidoc +++ b/docs/reference/analysis/analyzers.asciidoc @@ -49,14 +49,6 @@ index : stopwords : [test1, test2, test3] -------------------------------------------------- -[[analyzers-stopwords]] -=== Stopwords - -The `stopwords` parameter can be used to provide a custom set of stopwords. As -certain analyzers use a default list of stopwords while others don't, please -check out the individual analyzer sections. In case you want the analyzer to -use no stopwords at all, simply provide `stopwords: _none_` - Below is a list of the built in analyzers. include::analyzers/standard-analyzer.asciidoc[] diff --git a/docs/reference/analysis/analyzers/lang-analyzer.asciidoc b/docs/reference/analysis/analyzers/lang-analyzer.asciidoc index d3505dd1d370e..f963e4b0f1b47 100644 --- a/docs/reference/analysis/analyzers/lang-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/lang-analyzer.asciidoc @@ -11,7 +11,8 @@ following types are supported: `arabic`, `armenian`, `basque`, All analyzers support setting custom `stopwords` either internally in the config, or by using an external stopwords file by setting -`stopwords_path`. +`stopwords_path`. Check <> for +more details. The following analyzers support setting custom `stem_exclusion` list: `arabic`, `armenian`, `basque`, `brazilian`, `bulgarian`, `catalan`, diff --git a/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc b/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc index 04f71bfc77334..12f8799f8f962 100644 --- a/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc @@ -15,7 +15,8 @@ type: |`flags` |The regular expression flags. |`stopwords` |A list of stopwords to initialize the stop filter with. Defaults to an 'empty' stopword list coming[1.0.0.RC1, Previously -defaulted to the English stopwords list] +defaulted to the English stopwords list]. Check +<> for more details. |=================================================================== *IMPORTANT*: The regular expression should match the *token separators*, diff --git a/docs/reference/analysis/analyzers/snowball-analyzer.asciidoc b/docs/reference/analysis/analyzers/snowball-analyzer.asciidoc index 234f41db15517..64804fcb359f9 100644 --- a/docs/reference/analysis/analyzers/snowball-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/snowball-analyzer.asciidoc @@ -41,8 +41,9 @@ filter>> and defaults to `English`. Note that not all the language analyzers have a default set of stopwords provided. The `stopwords` parameter can be used to provide stopwords for the -languages that has no defaults, or to simply replace the default set -with your custom list. A default set of stopwords for many of these +languages that have no defaults, or to simply replace the default set +with your custom list. Check <> +for more details. A default set of stopwords for many of these languages is available from for instance https://github.com/apache/lucene-solr/tree/trunk/lucene/analysis/common/src/resources/org/apache/lucene/analysis/[here] and diff --git a/docs/reference/analysis/analyzers/standard-analyzer.asciidoc b/docs/reference/analysis/analyzers/standard-analyzer.asciidoc index 0b6faed771212..4aae94a69bb10 100644 --- a/docs/reference/analysis/analyzers/standard-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/standard-analyzer.asciidoc @@ -19,7 +19,8 @@ type: |Setting |Description |`stopwords` |A list of stopwords to initialize the stop filter with. Defaults to an 'empty' stopword list added[1.0.0.Beta1, Previously -defaulted to the English stopwords list] +defaulted to the English stopwords list]. Check +<> for more details. |`max_token_length` |The maximum token length. If a token is seen that exceeds this length then it is discarded. Defaults to `255`. |======================================================================= diff --git a/docs/reference/analysis/analyzers/stop-analyzer.asciidoc b/docs/reference/analysis/analyzers/stop-analyzer.asciidoc index 2a1bfd7311347..9a19772795f65 100644 --- a/docs/reference/analysis/analyzers/stop-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/stop-analyzer.asciidoc @@ -14,8 +14,9 @@ The following are settings that can be set for a `stop` analyzer type: |Setting |Description |`stopwords` |A list of stopwords to initialize the stop filter with. Defaults to the english stop words. - |`stopwords_path` |A path (either relative to `config` location, or absolute) to a stopwords file configuration. |======================================================================= +Use `stopwords: _none_` to explicitly specify an 'empty' stopword list. + From 6829b67ff02602f0250b56df4b0e65b4689d6368 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Tue, 14 Jan 2014 16:11:45 +0100 Subject: [PATCH 08/34] Fix Release tool to run smoke tests off tags this commit allows to run the release tool for smoke testing without being on the actually released branch. This commit also added a list of plugins that will be installed for smoke testing to see if the plugin startup mechanism works correctly. --- dev-tools/build_release.py | 78 +++++++++++++++++++++++++++++--------- 1 file changed, 60 insertions(+), 18 deletions(-) diff --git a/dev-tools/build_release.py b/dev-tools/build_release.py index ed6a5a0e6572e..a49b8d0e44511 100644 --- a/dev-tools/build_release.py +++ b/dev-tools/build_release.py @@ -58,6 +58,12 @@ """ env = os.environ +PLUGINS = [('bigdesk', 'lukas-vlcek/bigdesk'), + ('paramedic', 'karmi/elasticsearch-paramedic'), + ('segmentspy', 'polyfractal/elasticsearch-segmentspy'), + ('inquisitor', 'polyfractal/elasticsearch-inquisitor'), + ('head', 'mobz/elasticsearch-head')] + LOG = env.get('ES_RELEASE_LOG', '/tmp/elasticsearch_release.log') def log(msg): @@ -117,10 +123,11 @@ def verify_mvn_java_version(version, mvn): # Returns the hash of the current git HEAD revision def get_head_hash(): - return get_hash('HEAD') + return os.popen(' git rev-parse --verify HEAD 2>&1').read().strip() -def get_hash(version): - return os.popen('git rev-parse --verify %s 2>&1' % (version)).read().strip() +# Returns the hash of the given tag revision +def get_tag_hash(tag): + return os.popen('git show-ref --tags %s --hash 2>&1' % (tag)).read().strip() # Returns the name of the current branch def get_current_branch(): @@ -133,6 +140,10 @@ def get_current_branch(): def release_branch(version): return 'release_branch_%s' % version +# runs get fetch on the given remote +def fetch(remote): + run('git fetch %s' % remote) + # Creates a new release branch from the given source branch # and rebases the source branch from the remote before creating # the release branch. Note: This fails if the source branch @@ -309,7 +320,7 @@ def generate_checksums(files): res = res + [os.path.join(directory, checksum_file), release_file] return res -def download_and_verify(release, files, base_url='https://download.elasticsearch.org/elasticsearch/elasticsearch'): +def download_and_verify(release, files, plugins=None, base_url='https://download.elasticsearch.org/elasticsearch/elasticsearch'): print('Downloading and verifying release %s from %s' % (release, base_url)) tmp_dir = tempfile.mkdtemp() try: @@ -326,11 +337,11 @@ def download_and_verify(release, files, base_url='https://download.elasticsearch urllib.request.urlretrieve(url, checksum_file) print(' Verifying checksum %s' % (checksum_file)) run('cd %s && sha1sum -c %s' % (tmp_dir, os.path.basename(checksum_file))) - smoke_test_release(release, downloaded_files, get_hash('v%s' % release)) + smoke_test_release(release, downloaded_files, get_tag_hash('v%s' % release), plugins) finally: shutil.rmtree(tmp_dir) -def smoke_test_release(release, files, expected_hash): +def smoke_test_release(release, files, expected_hash, plugins): for release_file in files: if not os.path.isfile(release_file): raise RuntimeError('Smoketest failed missing file %s' % (release_file)) @@ -344,9 +355,20 @@ def smoke_test_release(release, files, expected_hash): continue # nothing to do here es_run_path = os.path.join(tmp_dir, 'elasticsearch-%s' % (release), 'bin/elasticsearch') print(' Smoke testing package [%s]' % release_file) + es_plugin_path = os.path.join(tmp_dir, 'elasticsearch-%s' % (release),'bin/plugin') + plugin_names = {} + for name, plugin in plugins: + print(' Install plugin [%s] from [%s]' % (name, plugin)) + run('%s %s %s' % (es_plugin_path, '-install', plugin)) + plugin_names[name] = True + + if release.startswith("0.90."): + background = '' # 0.90.x starts in background automatically + else: + background = '-d' print(' Starting elasticsearch deamon from [%s]' % os.path.join(tmp_dir, 'elasticsearch-%s' % release)) - run('%s; %s -Des.node.name=smoke_tester -Des.cluster.name=prepare_release -Des.discovery.zen.ping.multicast.enabled=false -d' - % (java_exe(), es_run_path)) + run('%s; %s -Des.node.name=smoke_tester -Des.cluster.name=prepare_release -Des.discovery.zen.ping.multicast.enabled=false %s' + % (java_exe(), es_run_path, background)) conn = HTTPConnection('127.0.0.1', 9200, 20); wait_for_node_startup() try: @@ -360,9 +382,25 @@ def smoke_test_release(release, files, expected_hash): if version['build_snapshot']: raise RuntimeError('Expected non snapshot version') if version['build_hash'].strip() != expected_hash: - raise RuntimeError('HEAD hash does not match expected [%s] but got [%s]' % (get_head_hash(), version['build_hash'])) + raise RuntimeError('HEAD hash does not match expected [%s] but got [%s]' % (expected_hash, version['build_hash'])) print(' Running REST Spec tests against package [%s]' % release_file) run_mvn('test -Dtests.rest=%s -Dtests.class=*.*RestTests' % ("127.0.0.1:9200")) + print(' Verify if plugins are listed in _nodes') + conn.request('GET', '/_nodes?plugin=true&pretty=true') + res = conn.getresponse() + if res.status == 200: + nodes = json.loads(res.read().decode("utf-8"))['nodes'] + for _, node in nodes.items(): + node_plugins = node['plugins'] + for node_plugin in node_plugins: + if not plugin_names.get(node_plugin['name'], False): + raise RuntimeError('Unexpeced plugin %s' % node_plugin['name']) + del plugin_names[node_plugin['name']] + if plugin_names: + raise RuntimeError('Plugins not loaded %s' % list(plugin_names.keys())) + + else: + raise RuntimeError('Expected HTTP 200 but got %s' % res.status) else: raise RuntimeError('Expected HTTP 200 but got %s' % res.status) finally: @@ -471,14 +509,11 @@ def check_s3_credentials(): print('Preparing Release from branch [%s] running tests: [%s] dryrun: [%s]' % (src_branch, run_tests, dry_run)) print(' JAVA_HOME is [%s]' % JAVA_HOME) print(' Running with maven command: [%s] ' % (MVN)) - release_version = find_release_version(src_branch) - - if not smoke_test_version and not dry_run: - smoke_test_version = release_version - elif smoke_test_version: - print("Skipping build - smoketest only against version %s" % smoke_test_version) if build: + release_version = find_release_version(src_branch) + if not dry_run: + smoke_test_version = release_version head_hash = get_head_hash() run_mvn('clean') # clean the env! print(' Release version: [%s]' % release_version) @@ -497,11 +532,14 @@ def check_s3_credentials(): print(''.join(['-' for _ in range(80)])) print('Building Release candidate') input('Press Enter to continue...') - print(' Running maven builds now and publish to sonartype- run-tests [%s]' % run_tests) + if not dry_run: + print(' Running maven builds now and publish to sonartype - run-tests [%s]' % run_tests) + else: + print(' Running maven builds now run-tests [%s]' % run_tests) build_release(run_tests=run_tests, dry_run=dry_run, cpus=cpus) artifacts = get_artifacts(release_version) artifacts_and_checksum = generate_checksums(artifacts) - smoke_test_release(release_version, artifacts, get_head_hash()) + smoke_test_release(release_version, artifacts, get_head_hash(), PLUGINS) print(''.join(['-' for _ in range(80)])) print('Finish Release -- dry_run: %s' % dry_run) input('Press Enter to continue...') @@ -530,5 +568,9 @@ def check_s3_credentials(): run('git tag -d v%s' % release_version) # we delete this one anyways run('git branch -D %s' % (release_branch(release_version))) + else: + print("Skipping build - smoketest only against version %s" % smoke_test_version) + if smoke_test_version: - download_and_verify(smoke_test_version, artifact_names(smoke_test_version)) + fetch(remote) + download_and_verify(smoke_test_version, artifact_names(smoke_test_version), plugins=PLUGINS) From 99eae50f9c23bfe73a7a52273dab06fb196b1e10 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Tue, 14 Jan 2014 17:00:32 +0100 Subject: [PATCH 09/34] run mvn clean before smoketesting --- dev-tools/build_release.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/dev-tools/build_release.py b/dev-tools/build_release.py index a49b8d0e44511..6566b34d57620 100644 --- a/dev-tools/build_release.py +++ b/dev-tools/build_release.py @@ -338,6 +338,7 @@ def download_and_verify(release, files, plugins=None, base_url='https://download print(' Verifying checksum %s' % (checksum_file)) run('cd %s && sha1sum -c %s' % (tmp_dir, os.path.basename(checksum_file))) smoke_test_release(release, downloaded_files, get_tag_hash('v%s' % release), plugins) + print(' SUCCESS') finally: shutil.rmtree(tmp_dir) @@ -570,6 +571,7 @@ def check_s3_credentials(): run('git branch -D %s' % (release_branch(release_version))) else: print("Skipping build - smoketest only against version %s" % smoke_test_version) + run_mvn('clean') # clean the env! if smoke_test_version: fetch(remote) From f2710c16ebd918f646be9d0ab64b4871c25be4c2 Mon Sep 17 00:00:00 2001 From: Rob Cherry Date: Mon, 4 Nov 2013 12:55:27 -0500 Subject: [PATCH 10/34] excluding all fields of an object should not remove parent. When excluding '*.f1' from `{ "obj": { "f1": 1, "f2": 2 } }` XContentMapValues.filter returns `{ "obj": { "f2": 2}}`. When run on `{ "obj": { "f1" : 1 }}` we should return `{ "obj": { }}` to maintain object structure. People currently need to always check whether `obj` is there or not. Closes #4715 Closes #4047 Related to #4491 --- .../xcontent/support/XContentMapValues.java | 29 ++--- .../support/XContentMapValuesTests.java | 117 +++++++++++++++--- .../search/fields/SearchFieldsTests.java | 2 +- 3 files changed, 114 insertions(+), 34 deletions(-) diff --git a/src/main/java/org/elasticsearch/common/xcontent/support/XContentMapValues.java b/src/main/java/org/elasticsearch/common/xcontent/support/XContentMapValues.java index 27b2c1ba67e4a..ab316e9f31a0d 100644 --- a/src/main/java/org/elasticsearch/common/xcontent/support/XContentMapValues.java +++ b/src/main/java/org/elasticsearch/common/xcontent/support/XContentMapValues.java @@ -154,24 +154,18 @@ private static void filter(Map map, Map into, St } sb.append(key); String path = sb.toString(); - boolean excluded = false; - for (String exclude : excludes) { - if (Regex.simpleMatch(exclude, path)) { - excluded = true; - break; - } - } - if (excluded) { + + if (Regex.simpleMatch(excludes, path)) { sb.setLength(mark); continue; } - boolean exactIncludeMatch; + + boolean exactIncludeMatch = false; // true if the current position was specifically mentioned + boolean pathIsPrefixOfAnInclude = false; // true if potentially a sub scope can be included if (includes.length == 0) { // implied match anything exactIncludeMatch = true; } else { - exactIncludeMatch = false; - boolean pathIsPrefixOfAnInclude = false; for (String include : includes) { // check for prefix matches as well to see if we need to zero in, something like: obj1.arr1.* or *.field // note, this does not work well with middle matches, like obj1.*.obj3 @@ -198,11 +192,12 @@ private static void filter(Map map, Map into, St break; } } - if (!pathIsPrefixOfAnInclude && !exactIncludeMatch) { - // skip subkeys, not interesting. - sb.setLength(mark); - continue; - } + } + + if (!(pathIsPrefixOfAnInclude || exactIncludeMatch)) { + // skip subkeys, not interesting. + sb.setLength(mark); + continue; } @@ -210,7 +205,7 @@ private static void filter(Map map, Map into, St Map innerInto = Maps.newHashMap(); // if we had an exact match, we want give deeper excludes their chance filter((Map) entry.getValue(), innerInto, exactIncludeMatch ? Strings.EMPTY_ARRAY : includes, excludes, sb); - if (!innerInto.isEmpty()) { + if (exactIncludeMatch || !innerInto.isEmpty()) { into.put(entry.getKey(), innerInto); } } else if (entry.getValue() instanceof List) { diff --git a/src/test/java/org/elasticsearch/common/xcontent/support/XContentMapValuesTests.java b/src/test/java/org/elasticsearch/common/xcontent/support/XContentMapValuesTests.java index eff2bdd450678..e2fba5ac17062 100644 --- a/src/test/java/org/elasticsearch/common/xcontent/support/XContentMapValuesTests.java +++ b/src/test/java/org/elasticsearch/common/xcontent/support/XContentMapValuesTests.java @@ -26,6 +26,7 @@ import org.elasticsearch.common.xcontent.XContentHelper; import org.elasticsearch.common.xcontent.XContentType; import org.elasticsearch.test.ElasticsearchTestCase; +import org.hamcrest.Matchers; import org.junit.Test; import java.util.Arrays; @@ -33,7 +34,6 @@ import java.util.List; import java.util.Map; -import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.*; import static org.hamcrest.core.IsEqual.equalTo; @@ -46,6 +46,7 @@ public void testFilter() throws Exception { XContentBuilder builder = XContentFactory.jsonBuilder().startObject() .field("test1", "value1") .field("test2", "value2") + .field("something_else", "value3") .endObject(); Map source = XContentFactory.xContent(XContentType.JSON).createParser(builder.string()).mapAndClose(); @@ -59,8 +60,9 @@ public void testFilter() throws Exception { assertThat(filter.get("test2").toString(), equalTo("value2")); filter = XContentMapValues.filter(source, Strings.EMPTY_ARRAY, new String[]{"test1"}); - assertThat(filter.size(), equalTo(1)); + assertThat(filter.size(), equalTo(2)); assertThat(filter.get("test2").toString(), equalTo("value2")); + assertThat(filter.get("something_else").toString(), equalTo("value3")); // more complex object... builder = XContentFactory.jsonBuilder().startObject() @@ -200,20 +202,6 @@ public void testExtractRawValue() throws Exception { assertThat(XContentMapValues.extractRawValues("path1.xxx.path2.yyy.test", map).get(0).toString(), equalTo("value")); } - @Test - public void testThatFilteringWithNestedArrayAndExclusionWorks() throws Exception { - XContentBuilder builder = XContentFactory.jsonBuilder().startObject() - .startArray("coordinates") - .startArray().value("foo").endArray() - .endArray() - .endObject(); - - Tuple> mapTuple = XContentHelper.convertToMap(builder.bytes(), true); - Map filteredSource = XContentMapValues.filter(mapTuple.v2(), Strings.EMPTY_ARRAY, new String[]{"nonExistingField"}); - - assertThat(mapTuple.v2(), equalTo(filteredSource)); - } - @Test public void prefixedNamesFilteringTest() { Map map = new HashMap(); @@ -368,4 +356,101 @@ public void filterWithEmptyIncludesExcludes() { assertThat(filteredMap.get("field").toString(), equalTo("value")); } + + @SuppressWarnings({"unchecked"}) + @Test + public void testThatFilterIncludesEmptyObjectWhenUsingIncludes() throws Exception { + XContentBuilder builder = XContentFactory.jsonBuilder().startObject() + .startObject("obj") + .endObject() + .endObject(); + + Tuple> mapTuple = XContentHelper.convertToMap(builder.bytes(), true); + Map filteredSource = XContentMapValues.filter(mapTuple.v2(), new String[]{"obj"}, Strings.EMPTY_ARRAY); + + assertThat(mapTuple.v2(), equalTo(filteredSource)); + } + + @Test + public void testThatFilterIncludesEmptyObjectWhenUsingExcludes() throws Exception { + XContentBuilder builder = XContentFactory.jsonBuilder().startObject() + .startObject("obj") + .endObject() + .endObject(); + + Tuple> mapTuple = XContentHelper.convertToMap(builder.bytes(), true); + Map filteredSource = XContentMapValues.filter(mapTuple.v2(), Strings.EMPTY_ARRAY, new String[]{"nonExistingField"}); + + assertThat(mapTuple.v2(), equalTo(filteredSource)); + } + + @Test + public void testNotOmittingObjectsWithExcludedProperties() throws Exception { + XContentBuilder builder = XContentFactory.jsonBuilder().startObject() + .startObject("obj") + .field("f1", "v1") + .endObject() + .endObject(); + + Tuple> mapTuple = XContentHelper.convertToMap(builder.bytes(), true); + Map filteredSource = XContentMapValues.filter(mapTuple.v2(), Strings.EMPTY_ARRAY, new String[]{"obj.f1"}); + + assertThat(filteredSource.size(), equalTo(1)); + assertThat(filteredSource, hasKey("obj")); + assertThat(((Map) filteredSource.get("obj")).size(), equalTo(0)); + } + + @SuppressWarnings({"unchecked"}) + @Test + public void testNotOmittingObjectWithNestedExcludedObject() throws Exception { + XContentBuilder builder = XContentFactory.jsonBuilder().startObject() + .startObject("obj1") + .startObject("obj2") + .startObject("obj3") + .endObject() + .endObject() + .endObject() + .endObject(); + + // implicit include + Tuple> mapTuple = XContentHelper.convertToMap(builder.bytes(), true); + Map filteredSource = XContentMapValues.filter(mapTuple.v2(), Strings.EMPTY_ARRAY, new String[]{"*.obj2"}); + + assertThat(filteredSource.size(), equalTo(1)); + assertThat(filteredSource, hasKey("obj1")); + assertThat(((Map) filteredSource.get("obj1")).size(), Matchers.equalTo(0)); + + // explicit include + filteredSource = XContentMapValues.filter(mapTuple.v2(), new String[]{"obj1"}, new String[]{"*.obj2"}); + assertThat(filteredSource.size(), equalTo(1)); + assertThat(filteredSource, hasKey("obj1")); + assertThat(((Map) filteredSource.get("obj1")).size(), Matchers.equalTo(0)); + + // wild card include + filteredSource = XContentMapValues.filter(mapTuple.v2(), new String[]{"*.obj2"}, new String[]{"*.obj3"}); + assertThat(filteredSource.size(), equalTo(1)); + assertThat(filteredSource, hasKey("obj1")); + assertThat(((Map) filteredSource.get("obj1")), hasKey("obj2")); + assertThat(((Map) ((Map) filteredSource.get("obj1")).get("obj2")).size(), Matchers.equalTo(0)); + } + + @SuppressWarnings({"unchecked"}) + @Test + public void testIncludingObjectWithNestedIncludedObject() throws Exception { + XContentBuilder builder = XContentFactory.jsonBuilder().startObject() + .startObject("obj1") + .startObject("obj2") + .endObject() + .endObject() + .endObject(); + + Tuple> mapTuple = XContentHelper.convertToMap(builder.bytes(), true); + Map filteredSource = XContentMapValues.filter(mapTuple.v2(), new String[]{"*.obj2"}, Strings.EMPTY_ARRAY); + + assertThat(filteredSource.size(), equalTo(1)); + assertThat(filteredSource, hasKey("obj1")); + assertThat(((Map) filteredSource.get("obj1")).size(), equalTo(1)); + assertThat(((Map) filteredSource.get("obj1")), hasKey("obj2")); + assertThat(((Map) ((Map) filteredSource.get("obj1")).get("obj2")).size(), equalTo(0)); + } } diff --git a/src/test/java/org/elasticsearch/search/fields/SearchFieldsTests.java b/src/test/java/org/elasticsearch/search/fields/SearchFieldsTests.java index 4abed6dbb7289..314c235cf38ed 100644 --- a/src/test/java/org/elasticsearch/search/fields/SearchFieldsTests.java +++ b/src/test/java/org/elasticsearch/search/fields/SearchFieldsTests.java @@ -253,7 +253,7 @@ public void testPartialFields() throws Exception { SearchResponse response = client().prepareSearch("test") .addPartialField("partial1", "obj1.arr1.*", null) - .addPartialField("partial2", null, "obj1.*") + .addPartialField("partial2", null, "obj1") .execute().actionGet(); assertThat("Failures " + Arrays.toString(response.getShardFailures()), response.getShardFailures().length, equalTo(0)); From 9440655ab9091cefe33fe1853f91a0897670fc55 Mon Sep 17 00:00:00 2001 From: Igor Motov Date: Tue, 14 Jan 2014 10:43:15 -0500 Subject: [PATCH 11/34] Fix possible race condition in the restoreIndexWithMissingShards test Due to a race condition the index creation operation might still try to create an index on the closing node. --- .../snapshots/DedicatedClusterSnapshotRestoreTests.java | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java b/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java index c32bf948f431c..7728007d67bdd 100644 --- a/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java +++ b/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java @@ -24,6 +24,7 @@ import org.elasticsearch.action.admin.cluster.snapshots.create.CreateSnapshotResponse; import org.elasticsearch.action.admin.cluster.snapshots.restore.RestoreSnapshotResponse; import org.elasticsearch.client.Client; +import org.elasticsearch.common.Priority; import org.elasticsearch.common.settings.ImmutableSettings; import org.elasticsearch.common.unit.TimeValue; import org.elasticsearch.snapshots.mockstore.MockRepositoryModule; @@ -152,6 +153,7 @@ public void restoreIndexWithMissingShards() throws Exception { logger.info("--> shutdown one of the nodes"); cluster().stopRandomNode(); + assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("1m").setWaitForNodes("<2").execute().actionGet().isTimedOut(), equalTo(false)); assertAcked(prepareCreate("test-idx-2", 1, settingsBuilder().put("number_of_shards", 6) .put("number_of_replicas", 0) From d1a4f889aeabe1a99cd0ba62766b585715f1623b Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Tue, 14 Jan 2014 19:38:39 +0100 Subject: [PATCH 12/34] Fix DestructiveOperationsIntegrationTests to wait for index to be allocated before closing --- ...DestructiveOperationsIntegrationTests.java | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java b/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java index b7a26dd185afc..cd0ab6efec846 100644 --- a/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java +++ b/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java @@ -53,12 +53,12 @@ public void testDestructiveOperations() throws Exception { try { // should fail since index1 is the only index. client().admin().indices().prepareDelete("i*").get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} try { client().admin().indices().prepareDelete("_all").get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} settings = ImmutableSettings.builder() @@ -78,26 +78,26 @@ public void testDestructiveOperations() throws Exception { assertAcked(client().admin().indices().prepareCreate("index1").get()); assertAcked(client().admin().indices().prepareCreate("1index").get()); - + ensureYellow();// wait for primaries to be allocated // Should succeed, since no wildcards assertAcked(client().admin().indices().prepareClose("1index").get()); try { client().admin().indices().prepareClose("_all").get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} try { assertAcked(client().admin().indices().prepareOpen("_all").get()); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) { } try { client().admin().indices().prepareClose("*").get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} try { assertAcked(client().admin().indices().prepareOpen("*").get()); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) { } @@ -124,12 +124,12 @@ public void testDestructiveOperations() throws Exception { try { client().prepareDeleteByQuery("_all").setQuery(QueryBuilders.matchAllQuery()).get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} try { client().prepareDeleteByQuery().setQuery(QueryBuilders.matchAllQuery()).get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} settings = ImmutableSettings.builder() @@ -157,12 +157,12 @@ public void testDestructiveOperations() throws Exception { client().admin().indices().prepareDeleteMapping("1index").setType("1").get(); try { client().admin().indices().prepareDeleteMapping("_all").setType("1").get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} try { client().admin().indices().prepareDeleteMapping().setType("1").get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} settings = ImmutableSettings.builder() @@ -170,10 +170,10 @@ public void testDestructiveOperations() throws Exception { .build(); assertAcked(client().admin().cluster().prepareUpdateSettings().setTransientSettings(settings)); - client().admin().indices().preparePutMapping("1index").setType("1").setSource("field1", "type=string").get(); - client().admin().indices().prepareDeleteMapping().setType("1").get(); - client().admin().indices().preparePutMapping("1index").setType("1").setSource("field1", "type=string").get(); - client().admin().indices().prepareDeleteMapping("_all").setType("1").get(); + assertAcked(client().admin().indices().preparePutMapping("1index").setType("1").setSource("field1", "type=string")); + assertAcked(client().admin().indices().prepareDeleteMapping().setType("1")); + assertAcked(client().admin().indices().preparePutMapping("1index").setType("1").setSource("field1", "type=string")); + assertAcked(client().admin().indices().prepareDeleteMapping("_all").setType("1")); } } From 411739fe3b5b2410fa9594edf27087718162225f Mon Sep 17 00:00:00 2001 From: Britta Weber Date: Wed, 8 Jan 2014 10:34:48 +0100 Subject: [PATCH 13/34] Make PUT and DELETE consistent for _mapping, _alias and _warmer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit See issue #4071 PUT options for _mapping: Single type can now be added with `[PUT|POST] {index|_all|*|regex|blank}/[_mapping|_mappings]/type` and `[PUT|POST] {index|_all|*|regex|blank}/type/[_mapping|_mappings]` PUT options for _warmer: PUT with a single warmer can now be done with `[PUT|POST] {index|_all|*|prefix*|blank}/{type|_all|*|prefix*|blank}/[_warmer|_warmers]/warmer_name` PUT options for _alias: Single alias can now be PUT with `[PUT|POST] {index|_all|*|prefix*|blank}/[_alias|_aliases]/alias` DELETE options _mapping: Several mappings can be deleted at once by defining several indices and types with `[DELETE] /{index}/{type}` `[DELETE] /{index}/{type}/_mapping` `[DELETE] /{index}/_mapping/{type}` where `index= * | _all | glob pattern | name1, name2, …` `type= * | _all | glob pattern | name1, name2, …` Alternatively, the keyword `_mapings` can be used. DELETE options for _warmer: Several warmers can be deleted at once by defining several indices and names with `[DELETE] /{index}/_warmer/{type}` where `index= * | _all | glob pattern | name1, name2, …` `type= * | _all | glob pattern | name1, name2, …` Alternatively, the keyword `_warmers` can be used. DELETE options for _alias: Several aliases can be deleted at once by defining several indices and names with `[DELETE] /{index}/_alias/{type}` where `index= * | _all | glob pattern | name1, name2, …` `type= * | _all | glob pattern | name1, name2, …` Alternatively, the keyword `_aliases` can be used. --- docs/reference/indices/aliases.asciidoc | 29 +- .../reference/indices/delete-mapping.asciidoc | 21 +- docs/reference/indices/put-mapping.asciidoc | 22 ++ docs/reference/indices/warmers.asciidoc | 52 ++- rest-api-spec/api/indices.delete_alias.json | 2 +- rest-api-spec/api/indices.delete_mapping.json | 2 +- rest-api-spec/api/indices.delete_warmer.json | 4 +- rest-api-spec/api/indices.put_alias.json | 4 +- rest-api-spec/api/indices.put_mapping.json | 4 +- rest-api-spec/api/indices.put_warmer.json | 4 +- .../all_path_options.yaml | 225 +++++++++++++ .../all_path_options.yaml | 286 +++++++++++++++++ .../all_path_options.yaml | 214 +++++++++++++ .../indices.put_alias/all_path_options.yaml | 127 ++++++++ .../indices.put_mapping/all_path_options.yaml | 178 +++++++++++ .../all_path_options.yaml | 113 +++++++ .../test/indices.put_warmer/10_basic.yaml | 1 + .../indices.put_warmer/all_path_options.yaml | 128 ++++++++ .../indices/alias/IndicesAliasesRequest.java | 299 ++++++++++++++---- .../alias/IndicesAliasesRequestBuilder.java | 121 +++++-- .../alias/TransportIndicesAliasesAction.java | 40 ++- ...eleteMappingClusterStateUpdateRequest.java | 10 +- .../mapping/delete/DeleteMappingRequest.java | 62 ++-- .../delete/DeleteMappingRequestBuilder.java | 4 +- .../delete/TransportDeleteMappingAction.java | 28 +- .../warmer/delete/DeleteWarmerRequest.java | 49 ++- .../delete/DeleteWarmerRequestBuilder.java | 4 +- .../delete/TransportDeleteWarmerAction.java | 31 +- .../cluster/metadata/AliasAction.java | 66 ++-- .../cluster/metadata/MetaData.java | 34 +- .../metadata/MetaDataMappingService.java | 23 +- .../common/util/CollectionUtils.java | 11 + .../indices/TypeMissingException.java | 10 +- .../alias/delete/AliasesMissingException.java | 46 +++ .../delete/RestIndexDeleteAliasesAction.java | 8 +- .../alias/put/RestIndexPutAliasAction.java | 20 +- .../delete/RestDeleteMappingAction.java | 7 +- .../mapping/put/RestPutMappingAction.java | 20 +- .../warmer/delete/RestDeleteWarmerAction.java | 5 +- .../warmer/put/RestPutWarmerAction.java | 14 + .../warmer/IndexWarmerMissingException.java | 14 +- .../delete/DeleteWarmerRequestTests.java | 4 +- .../aliases/IndexAliasesTests.java | 61 +++- .../elasticsearch/cluster/ack/AckTests.java | 2 +- .../cluster/metadata/MetaDataTests.java | 12 +- .../indices/IndicesOptionsTests.java | 120 ++++++- .../mapping/SimpleDeleteMappingTests.java | 40 ++- .../LocalGatewayIndicesWarmerTests.java | 2 +- .../warmer/SimpleIndicesWarmerTests.java | 6 +- .../mlt/MoreLikeThisActionTests.java | 4 +- ...DestructiveOperationsIntegrationTests.java | 18 +- 51 files changed, 2314 insertions(+), 297 deletions(-) create mode 100644 rest-api-spec/test/indices.delete_alias/all_path_options.yaml create mode 100644 rest-api-spec/test/indices.delete_mapping/all_path_options.yaml create mode 100644 rest-api-spec/test/indices.delete_warmer/all_path_options.yaml create mode 100644 rest-api-spec/test/indices.put_alias/all_path_options.yaml create mode 100644 rest-api-spec/test/indices.put_mapping/all_path_options.yaml create mode 100644 rest-api-spec/test/indices.put_settings/all_path_options.yaml create mode 100644 rest-api-spec/test/indices.put_warmer/all_path_options.yaml create mode 100644 src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/AliasesMissingException.java diff --git a/docs/reference/indices/aliases.asciidoc b/docs/reference/indices/aliases.asciidoc index 5451ac530b3ee..53f484cadcf50 100644 --- a/docs/reference/indices/aliases.asciidoc +++ b/docs/reference/indices/aliases.asciidoc @@ -153,17 +153,22 @@ curl -XGET 'http://localhost:9200/alias2/_search?q=user:kimchy&routing=2,3' [float] [[alias-adding]] -=== Add a single index alias +=== Add a single alias -There is also an api to add a single index alias, with options: +An alias can also be added with the endpoint + +`PUT /{index}/_alias/{name}` + + +where [horizontal] -`index`:: The index to alias refers to. This is a required option. -`alias`:: The name of the alias. This is a required option. +`index`:: The index to alias refers to. Can be any of `blank | * | _all | glob pattern | name1, name2, …` +`name`:: The name of the alias. This is a required option. `routing`:: An optional routing that can be associated with an alias. `filter`:: An optional filter that can be associated with an alias. -The rest endpoint is: `/{index}/_alias/{alias}`. +You can also use the plural `_aliases`. [float] ==== Examples: @@ -191,16 +196,18 @@ curl -XPUT 'localhost:9200/users/_alias/user_12' -d '{ [float] [[deleting]] -=== Delete a single index alias +=== Delete aliases + + +The rest endpoint is: `/{index}/_alias/{name}` -Th API to delete a single index alias, has options: +where [horizontal] -`index`:: The index the alias is in, the needs to be deleted. This is - a required option. -`alias`:: The name of the alias to delete. This is a required option. +`index`:: `* | _all | glob pattern | name1, name2, …` +`name`:: `* | _all | glob pattern | name1, name2, …` -The rest endpoint is: `/{index}/_alias/{alias}`. Example: +Alternatively you can use the plural `_aliases`. Example: [source,js] -------------------------------------------------- diff --git a/docs/reference/indices/delete-mapping.asciidoc b/docs/reference/indices/delete-mapping.asciidoc index 6f066dbfdcefd..9b72d9cb02f19 100644 --- a/docs/reference/indices/delete-mapping.asciidoc +++ b/docs/reference/indices/delete-mapping.asciidoc @@ -1,8 +1,25 @@ [[indices-delete-mapping]] == Delete Mapping -Allow to delete a mapping (type) along with its data. The REST endpoint -is `/{index}/{type}` with `DELETE` method. +Allow to delete a mapping (type) along with its data. The REST endpoints are + +[source,js] +-------------------------------------------------- + +[DELETE] /{index}/{type} + +[DELETE] /{index}/{type}/_mapping + +[DELETE] /{index}/_mapping/{type} + +-------------------------------------------------- + +where + +[horizontal] + +`index`:: `* | _all | glob pattern | name1, name2, …` +`type`:: `* | _all | glob pattern | name1, name2, …` Note, most times, it make more sense to reindex the data into a fresh index compared to delete large chunks of it. diff --git a/docs/reference/indices/put-mapping.asciidoc b/docs/reference/indices/put-mapping.asciidoc index b55c19096d42f..77486259001c4 100644 --- a/docs/reference/indices/put-mapping.asciidoc +++ b/docs/reference/indices/put-mapping.asciidoc @@ -59,3 +59,25 @@ $ curl -XPUT 'http://localhost:9200/kimchy,elasticsearch/tweet/_mapping' -d ' } ' -------------------------------------------------- + +All options: + +[source,js] +-------------------------------------------------- + +PUT /{index}/_mapping/{type} + + +-------------------------------------------------- + + +where + +[horizontal] +`{index}`:: `blank | * | _all | glob pattern | name1, name2, …` + +`{type}`:: Name of the type to add. Must be the name of the type defined in the body. + + +Instead of `_mapping` you can also use the plural `_mappings`. +The uri `PUT /{index}/{type}/_mapping` is still supported for backwardscompatibility. diff --git a/docs/reference/indices/warmers.asciidoc b/docs/reference/indices/warmers.asciidoc index abe16cb794ceb..bc7109465124b 100644 --- a/docs/reference/indices/warmers.asciidoc +++ b/docs/reference/indices/warmers.asciidoc @@ -112,25 +112,55 @@ curl -XPUT localhost:9200/test/type1/_warmer/warmer_1 -d '{ }' -------------------------------------------------- +All options: + +[source,js] +-------------------------------------------------- + +PUT _warmer/{warmer_name} + +PUT /{index}/_warmer/{warmer_name} + +PUT /{index}/{type}/_warmer/{warmer_name} + +-------------------------------------------------- + + +where + +[horizontal] +`{index}`:: `* | _all | glob pattern | name1, name2, …` + +`{type}`:: `* | _all | glob pattern | name1, name2, …` + +Instead of `_warmer` you can also use the plural `_warmers`. + + + [float] [[removing]] -=== Delete Warmer +=== Delete Warmers + +Warmers can be deleted using the following endpoint: + -Removing a warmer can be done against an index (or alias / indices) -based on its name. The provided name can be a simple wildcard expression -or omitted to remove all warmers. Some samples: [source,js] -------------------------------------------------- -# delete warmer named warmer_1 on test index -curl -XDELETE localhost:9200/test/_warmer/warmer_1 -# delete all warmers that start with warm on test index -curl -XDELETE localhost:9200/test/_warmer/warm* - -# delete all warmers for test index -curl -XDELETE localhost:9200/test/_warmer/ +[DELETE] /{index}/_warmer/{name} + -------------------------------------------------- + + +where + +[horizontal] +`{index}`:: `* | _all | glob pattern | name1, name2, …` + +`{name}`:: `* | _all | glob pattern | name1, name2, …` + +Instead of `_warmer` you can also use the plural `_warmers`. [float] [[warmer-retrieving]] diff --git a/rest-api-spec/api/indices.delete_alias.json b/rest-api-spec/api/indices.delete_alias.json index f930e64e380dd..03686046820d8 100644 --- a/rest-api-spec/api/indices.delete_alias.json +++ b/rest-api-spec/api/indices.delete_alias.json @@ -4,7 +4,7 @@ "methods": ["DELETE"], "url": { "path": "/{index}/_alias/{name}", - "paths": ["/{index}/_alias/{name}"], + "paths": ["/{index}/_alias/{name}", "/{index}/_aliases/{name}"], "parts": { "index": { "type" : "string", diff --git a/rest-api-spec/api/indices.delete_mapping.json b/rest-api-spec/api/indices.delete_mapping.json index d6a159e876174..6494b3231d974 100644 --- a/rest-api-spec/api/indices.delete_mapping.json +++ b/rest-api-spec/api/indices.delete_mapping.json @@ -4,7 +4,7 @@ "methods": ["DELETE"], "url": { "path": "/{index}/{type}/_mapping", - "paths": ["/{index}/{type}/_mapping", "/{index}/{type}"], + "paths": ["/{index}/{type}/_mapping", "/{index}/{type}", "/{index}/_mapping/{type}", "/{index}/{type}/_mappings", "/{index}/_mappings/{type}"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/api/indices.delete_warmer.json b/rest-api-spec/api/indices.delete_warmer.json index f8bbc5d29f792..be747cbeeec2b 100644 --- a/rest-api-spec/api/indices.delete_warmer.json +++ b/rest-api-spec/api/indices.delete_warmer.json @@ -3,8 +3,8 @@ "documentation": "http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-warmers.html", "methods": ["DELETE"], "url": { - "path": "/{index}/_warmer", - "paths": ["/{index}/_warmer", "/{index}/_warmer/{name}", "/{index}/{type}/_warmer/{name}"], + "path": "/{index}/_warmer/{name}", + "paths": ["/{index}/_warmer", "/{index}/_warmer/{name}", "/{index}/_warmers", "/{index}/_warmers/{name}"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/api/indices.put_alias.json b/rest-api-spec/api/indices.put_alias.json index cbb4c1a198181..db1072e4720a1 100644 --- a/rest-api-spec/api/indices.put_alias.json +++ b/rest-api-spec/api/indices.put_alias.json @@ -1,10 +1,10 @@ { "indices.put_alias": { "documentation": "http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html", - "methods": ["PUT"], + "methods": ["PUT", "POST"], "url": { "path": "/{index}/_alias/{name}", - "paths": ["/{index}/_alias/{name}", "/_alias/{name}", "/{index}/_alias", "/_alias"], + "paths": ["/{index}/_alias/{name}", "/_alias/{name}", "/{index}/_aliases/{name}", "/_aliases/{name}"], "parts": { "index": { "type" : "string", diff --git a/rest-api-spec/api/indices.put_mapping.json b/rest-api-spec/api/indices.put_mapping.json index 5056feaacc83d..eadba64f3ce4b 100644 --- a/rest-api-spec/api/indices.put_mapping.json +++ b/rest-api-spec/api/indices.put_mapping.json @@ -4,11 +4,11 @@ "methods": ["PUT", "POST"], "url": { "path": "/{index}/{type}/_mapping", - "paths": ["/{index}/{type}/_mapping"], + "paths": ["/{index}/{type}/_mapping", "/{index}/_mapping/{type}", "/_mapping/{type}", "/{index}/{type}/_mappings", "/{index}/_mappings/{type}", "/_mappings/{type}"], "parts": { "index": { "type" : "list", - "required" : true, + "required" : false, "description" : "A comma-separated list of index names; use `_all` to perform the operation on all indices" }, "type": { diff --git a/rest-api-spec/api/indices.put_warmer.json b/rest-api-spec/api/indices.put_warmer.json index 7b9a8ab857ea0..dedb425c585cb 100644 --- a/rest-api-spec/api/indices.put_warmer.json +++ b/rest-api-spec/api/indices.put_warmer.json @@ -1,10 +1,10 @@ { "indices.put_warmer": { "documentation": "http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-warmers.html", - "methods": ["PUT"], + "methods": ["PUT", "POST"], "url": { "path": "/{index}/_warmer/{name}", - "paths": ["/{index}/_warmer/{name}", "/{index}/{type}/_warmer/{name}"], + "paths": ["/_warmer/{name}", "/{index}/_warmer/{name}", "/{index}/{type}/_warmer/{name}", "/_warmers/{name}", "/{index}/_warmers/{name}", "/{index}/{type}/_warmers/{name}"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/test/indices.delete_alias/all_path_options.yaml b/rest-api-spec/test/indices.delete_alias/all_path_options.yaml new file mode 100644 index 0000000000000..bd158c39271bd --- /dev/null +++ b/rest-api-spec/test/indices.delete_alias/all_path_options.yaml @@ -0,0 +1,225 @@ +--- +setup: + + - do: + indices.create: + index: test_index1 + + - do: + indices.create: + index: test_index2 + + - do: + indices.create: + index: foo + + - do: + indices.put_alias: + name: alias1 + body: + routing: "routing value" + - do: + indices.put_alias: + name: alias2 + body: + routing: "routing value" + +--- +"check setup": + - do: + indices.get_alias: + name: alias1 + + - match: {test_index1.aliases.alias1.search_routing: "routing value"} + - match: {test_index2.aliases.alias1.search_routing: "routing value"} + - match: {foo.aliases.alias1.search_routing: "routing value"} + + - do: + indices.get_alias: + name: alias2 + + - match: {test_index1.aliases.alias2.search_routing: "routing value"} + - match: {test_index2.aliases.alias2.search_routing: "routing value"} + - match: {foo.aliases.alias2.search_routing: "routing value"} + +--- +"check delete with _all index": + - do: + indices.delete_alias: + index: _all + name: alias1 + + - do: + catch: missing + indices.get_alias: + name: alias1 + - do: + indices.get_alias: + name: alias2 + + - match: {test_index1.aliases.alias2.search_routing: "routing value"} + - match: {test_index2.aliases.alias2.search_routing: "routing value"} + - match: {foo.aliases.alias2.search_routing: "routing value"} + +--- +"check delete with * index": + - do: + indices.delete_alias: + index: "*" + name: alias1 + + - do: + catch: missing + indices.get_alias: + name: alias1 + - do: + indices.get_alias: + name: alias2 + + - match: {test_index1.aliases.alias2.search_routing: "routing value"} + - match: {test_index2.aliases.alias2.search_routing: "routing value"} + - match: {foo.aliases.alias2.search_routing: "routing value"} + +--- +"check delete with index list": + - do: + indices.delete_alias: + index: "test_index1,test_index2" + name: alias1 + + - do: + indices.get_alias: + name: alias1 + + - match: {foo.aliases.alias1.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_alias: + name: alias2 + + - match: {test_index1.aliases.alias2.search_routing: "routing value"} + - match: {test_index2.aliases.alias2.search_routing: "routing value"} + - match: {foo.aliases.alias2.search_routing: "routing value"} + +--- +"check delete with prefix* index": + - do: + indices.delete_alias: + index: "test_*" + name: alias1 + + - do: + indices.get_alias: + name: alias1 + + - match: {foo.aliases.alias1.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_alias: + name: alias2 + + - match: {test_index1.aliases.alias2.search_routing: "routing value"} + - match: {test_index2.aliases.alias2.search_routing: "routing value"} + - match: {foo.aliases.alias2.search_routing: "routing value"} + + +--- +"check delete with index list and * aliases": + - do: + indices.delete_alias: + index: "test_index1,test_index2" + name: "*" + + - do: + indices.get_alias: + name: alias1 + + - match: {foo.aliases.alias1.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_alias: + name: alias2 + + - match: {foo.aliases.alias2.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + +--- +"check delete with index list and _all aliases": + - do: + indices.delete_alias: + index: "test_index1,test_index2" + name: _all + + - do: + indices.get_alias: + name: alias1 + + - match: {foo.aliases.alias1.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_alias: + name: alias2 + + - match: {foo.aliases.alias2.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + +--- +"check delete with index list and wildcard aliases": + - do: + indices.delete_alias: + index: "test_index1,test_index2" + name: "*1" + + - do: + indices.get_alias: + name: alias1 + + - match: {foo.aliases.alias1.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_alias: + name: alias2 + + - match: {test_index1.aliases.alias2.search_routing: "routing value"} + - match: {test_index2.aliases.alias2.search_routing: "routing value"} + - match: {foo.aliases.alias2.search_routing: "routing value"} + +--- +"check 404 on no matching alias": + - do: + catch: missing + indices.delete_alias: + index: "*" + name: "non_existent" + + - do: + catch: missing + indices.delete_alias: + index: "non_existent" + name: "alias1" + + +--- +"check delete with blank index and blank alias": + - do: + catch: param + indices.delete_alias: + name: "alias1" + + - do: + catch: param + indices.delete_alias: + index: "test_index1" + diff --git a/rest-api-spec/test/indices.delete_mapping/all_path_options.yaml b/rest-api-spec/test/indices.delete_mapping/all_path_options.yaml new file mode 100644 index 0000000000000..9fda1db407283 --- /dev/null +++ b/rest-api-spec/test/indices.delete_mapping/all_path_options.yaml @@ -0,0 +1,286 @@ +setup: + + - do: + indices.create: + index: test_index1 + body: + mappings: { test_type1: { }} + + - do: + indices.create: + index: test_index2 + body: + mappings: { test_type2: { }} + - do: + indices.create: + index: foo + body: + mappings: { test_type2: { }} + +--- +"delete with _all index": + - do: + indices.delete_mapping: + index: _all + type: test_type2 + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_true: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_false: '' + +--- +"delete with * index": + - do: + indices.delete_mapping: + index: '*' + type: test_type2 + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_true: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_false: '' + +--- +"delete with prefix* index": + - do: + indices.delete_mapping: + index: test* + type: test_type2 + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_true: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_true: '' + +--- +"delete with list of indices": + + - do: + indices.delete_mapping: + index: test_index1,test_index2 + type: test_type2 + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_true: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_true: '' + +--- +"delete with index list and _all type": + - do: + indices.delete_mapping: + index: test_index1,test_index2 + type: _all + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_false: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_true: '' + + +--- +"delete with index list and * type": + - do: + indices.delete_mapping: + index: test_index1,test_index2 + type: '*' + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_false: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_true: '' + + +--- +"delete with index list and prefix* type": + - do: + indices.delete_mapping: + index: test_index1,test_index2 + type: '*2' + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_true: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_true: '' + +--- +"delete with index list and list of types": + - do: + indices.delete_mapping: + index: test_index1,test_index2 + type: test_type1,test_type2 + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_false: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_true: '' + +--- +"check 404 on no matching type": + - do: + catch: missing + indices.delete_mapping: + index: "*" + type: "non_existent" + + - do: + catch: missing + indices.delete_mapping: + index: "non_existent" + type: "test_type1" + + +--- +"check delete with blank index and blank alias": + - do: + catch: param + indices.delete_alias: + name: "alias1" + + - do: + catch: param + indices.delete_alias: + index: "test_index1" + + +--- +"check delete with blank index and blank type": + - do: + catch: param + indices.delete_mapping: + name: "test_type1" + + - do: + catch: param + indices.delete_mapping: + index: "test_index1" + diff --git a/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml b/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml new file mode 100644 index 0000000000000..2953a09643560 --- /dev/null +++ b/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml @@ -0,0 +1,214 @@ +setup: + - do: + indices.create: + index: test_index1 + + - do: + indices.create: + index: test_index2 + + - do: + indices.create: + index: foo + + - do: + indices.put_warmer: + index: "test_index1,test_index2,foo" + name: test_warmer1 + body: + query: + match_all: {} + + - do: + indices.put_warmer: + index: "test_index1,test_index2,foo" + name: test_warmer2 + body: + query: + match_all: {} + +--- +"Check setup": + + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.test_warmer1.source.query.match_all: {}} + - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer1.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} + - match: {foo.warmers.test_warmer1.source.query.match_all: {}} + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + + +--- +"check delete with _all index": + - do: + indices.delete_warmer: + index: _all + name: test_warmer1 + + - do: + catch: missing + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + + - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + +--- +"check delete with * index": + - do: + indices.delete_warmer: + index: "*" + name: test_warmer1 + + - do: + catch: missing + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + + - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + +--- +"check delete with index list": + - do: + indices.delete_warmer: + index: "test_index1,test_index2" + name: test_warmer1 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - match: {foo.warmers.test_warmer1.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + +--- +"check delete with prefix* index": + - do: + indices.delete_warmer: + index: "test_*" + name: test_warmer1 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - match: {foo.warmers.test_warmer1.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + + +--- +"check delete with index list and * warmers": + - do: + indices.delete_warmer: + index: "test_index1,test_index2" + name: "*" + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - match: {foo.warmers.test_warmer1.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + +--- +"check delete with index list and _all warmers": + - do: + indices.delete_warmer: + index: "test_index1,test_index2" + name: _all + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - match: {foo.warmers.test_warmer1.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + +--- +"check delete with index list and wildcard warmers": + - do: + indices.delete_warmer: + index: "test_index1,test_index2" + name: "*1" + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - match: {foo.warmers.test_warmer1.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + +--- +"check 404 on no matching test_warmer": + - do: + catch: missing + indices.delete_warmer: + index: "*" + name: "non_existent" + + - do: + catch: missing + indices.delete_warmer: + index: "non_existent" + name: "test_warmer1" + + +--- +"check delete with blank index and blank test_warmer": + - do: + catch: param + indices.delete_warmer: + name: "test_warmer1" + + - do: + catch: param + indices.delete_warmer: + index: "test_index1" + diff --git a/rest-api-spec/test/indices.put_alias/all_path_options.yaml b/rest-api-spec/test/indices.put_alias/all_path_options.yaml new file mode 100644 index 0000000000000..024f6b6445118 --- /dev/null +++ b/rest-api-spec/test/indices.put_alias/all_path_options.yaml @@ -0,0 +1,127 @@ +--- +setup: +# create three indices + + - do: + indices.create: + index: test_index1 + - do: + indices.create: + index: test_index2 + - do: + indices.create: + index: foo + +--- +"put alias per index": + + - do: + indices.put_alias: + index: test_index1 + name: alias + - do: + indices.put_alias: + index: test_index2 + name: alias + + - do: + indices.get_alias: + name: alias + + - match: {test_index1.aliases.alias: {}} + + - match: {test_index2.aliases.alias: {}} + + - is_false: foo + +--- +"put alias in _all index": + + - do: + indices.put_alias: + index: _all + name: alias + + - do: + indices.get_alias: + name: alias + + - match: {test_index1.aliases.alias: {}} + - match: {test_index2.aliases.alias: {}} + - match: {foo.aliases.alias: {}} + +--- +"put alias in * index": + + + - do: + indices.put_alias: + index: '*' + name: alias + + - do: + indices.get_alias: + name: alias + + - match: {test_index1.aliases.alias: {}} + - match: {test_index2.aliases.alias: {}} + - match: {foo.aliases.alias: {}} + +--- +"put alias prefix* index": + + + - do: + indices.put_alias: + index: "test_*" + name: alias + + - do: + indices.get_alias: + name: alias + + - match: {test_index1.aliases.alias: {}} + - match: {test_index2.aliases.alias: {}} + - is_false: foo + +--- +"put alias in list of indices": + + + - do: + indices.put_alias: + index: "test_index1,test_index2" + name: alias + + - do: + indices.get_alias: + name: alias + + - match: {test_index1.aliases.alias: {}} + - match: {test_index2.aliases.alias: {}} + - is_false: foo + +--- +"put alias with blank index": + + + - do: + indices.put_alias: + name: alias + + - do: + indices.get_alias: + name: alias + + - match: {test_index1.aliases.alias: {}} + - match: {test_index2.aliases.alias: {}} + - match: {foo.aliases.alias: {}} + +--- +"put alias with mising name": + + + - do: + catch: param + indices.put_alias: {} + diff --git a/rest-api-spec/test/indices.put_mapping/all_path_options.yaml b/rest-api-spec/test/indices.put_mapping/all_path_options.yaml new file mode 100644 index 0000000000000..db0df345c5db2 --- /dev/null +++ b/rest-api-spec/test/indices.put_mapping/all_path_options.yaml @@ -0,0 +1,178 @@ +setup: + - do: + indices.create: + index: test_index1 + - do: + indices.create: + index: test_index2 + - do: + indices.create: + index: foo + + +--- +"put one mapping per index": + - do: + indices.put_mapping: + index: test_index1 + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + - do: + indices.put_mapping: + index: test_index2 + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + + + - do: + indices.get_mapping: {} + + - match: {test_index1.test_type.properties.text.type: string} + - match: {test_index1.test_type.properties.text.analyzer: whitespace} + + - match: {test_index2.test_type.properties.text.type: string} + - match: {test_index2.test_type.properties.text.analyzer: whitespace} + + - match: {foo: {}} + +--- +"put mapping in _all index": + + - do: + indices.put_mapping: + index: _all + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + + - do: + indices.get_mapping: {} + + - match: {test_index1.test_type.properties.text.type: string} + - match: {test_index1.test_type.properties.text.analyzer: whitespace} + + - match: {test_index2.test_type.properties.text.type: string} + - match: {test_index2.test_type.properties.text.analyzer: whitespace} + + - match: {foo.test_type.properties.text.type: string} + - match: {foo.test_type.properties.text.analyzer: whitespace} + +--- +"put mapping in * index": + - do: + indices.put_mapping: + index: "*" + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + + - do: + indices.get_mapping: {} + + - match: {test_index1.test_type.properties.text.type: string} + - match: {test_index1.test_type.properties.text.analyzer: whitespace} + + - match: {test_index2.test_type.properties.text.type: string} + - match: {test_index2.test_type.properties.text.analyzer: whitespace} + + - match: {foo.test_type.properties.text.type: string} + - match: {foo.test_type.properties.text.analyzer: whitespace} + +--- +"put mapping in prefix* index": + - do: + indices.put_mapping: + index: "test_index*" + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + + - do: + indices.get_mapping: {} + + - match: {test_index1.test_type.properties.text.type: string} + - match: {test_index1.test_type.properties.text.analyzer: whitespace} + + - match: {test_index2.test_type.properties.text.type: string} + - match: {test_index2.test_type.properties.text.analyzer: whitespace} + + - match: {foo: {}} + +--- +"put mapping in list of indices": + - do: + indices.put_mapping: + index: [test_index1, test_index2] + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + + - do: + indices.get_mapping: {} + + - match: {test_index1.test_type.properties.text.type: string} + - match: {test_index1.test_type.properties.text.analyzer: whitespace} + + - match: {test_index2.test_type.properties.text.type: string} + - match: {test_index2.test_type.properties.text.analyzer: whitespace} + + - match: {foo: {}} + +--- +"put mapping with blank index": + - do: + indices.put_mapping: + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + + - do: + indices.get_mapping: {} + + - match: {test_index1.test_type.properties.text.type: string} + - match: {test_index1.test_type.properties.text.analyzer: whitespace} + + - match: {test_index2.test_type.properties.text.type: string} + - match: {test_index2.test_type.properties.text.analyzer: whitespace} + + - match: {foo.test_type.properties.text.type: string} + - match: {foo.test_type.properties.text.analyzer: whitespace} + +--- +"put mapping with mising type": + + + - do: + catch: param + indices.put_mapping: {} + diff --git a/rest-api-spec/test/indices.put_settings/all_path_options.yaml b/rest-api-spec/test/indices.put_settings/all_path_options.yaml new file mode 100644 index 0000000000000..b9ae712221675 --- /dev/null +++ b/rest-api-spec/test/indices.put_settings/all_path_options.yaml @@ -0,0 +1,113 @@ +setup: + - do: + indices.create: + index: test_index1 + - do: + indices.create: + index: test_index2 + - do: + indices.create: + index: foo + + +--- +"put settings per index": + - do: + indices.put_settings: + index: test_index1 + body: + refresh_interval: 1s + + - do: + indices.put_settings: + index: test_index2 + body: + refresh_interval: 1s + + + - do: + indices.get_settings: {} + + - match: {test_index1.settings.index.refresh_interval: 1s} + - match: {test_index2.settings.index.refresh_interval: 1s} + - is_false: foo.settings.index.refresh_interval + +--- +"put settings in _all index": + - do: + indices.put_settings: + index: _all + body: + refresh_interval: 1s + + - do: + indices.get_settings: {} + + - match: {test_index1.settings.index.refresh_interval: 1s} + - match: {test_index2.settings.index.refresh_interval: 1s} + - match: {foo.settings.index.refresh_interval: 1s} + +--- +"put settings in * index": + - do: + indices.put_settings: + index: '*' + body: + refresh_interval: 1s + + - do: + indices.get_settings: {} + + - match: {test_index1.settings.index.refresh_interval: 1s} + - match: {test_index2.settings.index.refresh_interval: 1s} + - match: {foo.settings.index.refresh_interval: 1s} + + +--- +"put settings in prefix* index": + - do: + indices.put_settings: + index: 'test*' + body: + refresh_interval: 1s + + - do: + indices.get_settings: {} + + - match: {test_index1.settings.index.refresh_interval: 1s} + - match: {test_index2.settings.index.refresh_interval: 1s} + - is_false: foo.settings.index.refresh_interval + +--- +"put settings in list of indices": + - skip: + version: 1 - 999 + reason: list of indices not implemented yet + - do: + indices.put_settings: + index: test_index1, test_index2 + body: + refresh_interval: 1s + + - do: + indices.get_settings: {} + + - match: {test_index1.settings.index.refresh_interval: 1s} + - match: {test_index2.settings.index.refresh_interval: 1s} + - is_false: foo.settings.index.refresh_interval + + +--- +"put settings in blank index": + - do: + indices.put_settings: + body: + refresh_interval: 1s + + - do: + indices.get_settings: {} + + - match: {test_index1.settings.index.refresh_interval: 1s} + - match: {test_index2.settings.index.refresh_interval: 1s} + - match: {foo.settings.index.refresh_interval: 1s} + diff --git a/rest-api-spec/test/indices.put_warmer/10_basic.yaml b/rest-api-spec/test/indices.put_warmer/10_basic.yaml index 5bcac146b68f7..0e2a622360590 100644 --- a/rest-api-spec/test/indices.put_warmer/10_basic.yaml +++ b/rest-api-spec/test/indices.put_warmer/10_basic.yaml @@ -32,6 +32,7 @@ - do: indices.delete_warmer: index: test_index + name: test_warmer - do: catch: missing diff --git a/rest-api-spec/test/indices.put_warmer/all_path_options.yaml b/rest-api-spec/test/indices.put_warmer/all_path_options.yaml new file mode 100644 index 0000000000000..1c884e38948b6 --- /dev/null +++ b/rest-api-spec/test/indices.put_warmer/all_path_options.yaml @@ -0,0 +1,128 @@ +--- +setup: + + - do: + indices.create: + index: test_index1 + - do: + indices.create: + index: test_index2 + - do: + indices.create: + index: foo + +--- +"put warmer per index": + + - do: + indices.put_warmer: + index: test_index1 + name: warmer + body: + query: + match_all: {} + - do: + indices.put_warmer: + index: test_index2 + name: warmer + body: + query: + match_all: {} + + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.warmer.source.query.match_all: {}} + - match: {test_index2.warmers.warmer.source.query.match_all: {}} + - is_false: foo + +--- +"put warmer in _all index": + - do: + indices.put_warmer: + index: _all + name: warmer + body: + query: + match_all: {} + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.warmer.source.query.match_all: {}} + - match: {test_index2.warmers.warmer.source.query.match_all: {}} + - match: {foo.warmers.warmer.source.query.match_all: {}} + +--- +"put warmer in * index": + - do: + indices.put_warmer: + index: "*" + name: warmer + body: + query: + match_all: {} + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.warmer.source.query.match_all: {}} + - match: {test_index2.warmers.warmer.source.query.match_all: {}} + - match: {foo.warmers.warmer.source.query.match_all: {}} + +--- +"put warmer prefix* index": + - do: + indices.put_warmer: + index: "test_index*" + name: warmer + body: + query: + match_all: {} + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.warmer.source.query.match_all: {}} + - match: {test_index2.warmers.warmer.source.query.match_all: {}} + - is_false: foo + +--- +"put warmer in list of indices": + - do: + indices.put_warmer: + index: [test_index1, test_index2] + name: warmer + body: + query: + match_all: {} + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.warmer.source.query.match_all: {}} + - match: {test_index2.warmers.warmer.source.query.match_all: {}} + - is_false: foo + +--- +"put warmer with blank index": + - do: + indices.put_warmer: + name: warmer + body: + query: + match_all: {} + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.warmer.source.query.match_all: {}} + - match: {test_index2.warmers.warmer.source.query.match_all: {}} + - match: {foo.warmers.warmer.source.query.match_all: {}} + +--- +"put warmer with mising name": + + + - do: + catch: param + indices.put_warmer: {} + + + + diff --git a/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequest.java b/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequest.java index 11f8a9bde3697..af6d19343bf39 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequest.java @@ -19,21 +19,25 @@ package org.elasticsearch.action.admin.indices.alias; +import com.carrotsearch.hppc.cursors.ObjectCursor; +import com.google.common.collect.ImmutableList; import com.google.common.collect.Lists; -import org.elasticsearch.ElasticsearchGenerationException; import org.elasticsearch.action.ActionRequestValidationException; +import org.elasticsearch.action.support.IndicesOptions; import org.elasticsearch.action.support.master.AcknowledgedRequest; import org.elasticsearch.cluster.metadata.AliasAction; +import org.elasticsearch.cluster.metadata.AliasAction.Type; +import org.elasticsearch.cluster.metadata.AliasMetaData; +import org.elasticsearch.cluster.metadata.MetaData; import org.elasticsearch.common.Strings; +import org.elasticsearch.common.collect.ImmutableOpenMap; import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; -import org.elasticsearch.common.xcontent.ToXContent; -import org.elasticsearch.common.xcontent.XContentBuilder; -import org.elasticsearch.common.xcontent.XContentFactory; -import org.elasticsearch.common.xcontent.XContentType; +import org.elasticsearch.common.util.CollectionUtils; import org.elasticsearch.index.query.FilterBuilder; import java.io.IOException; +import java.util.ArrayList; import java.util.List; import java.util.Locale; import java.util.Map; @@ -46,115 +50,260 @@ */ public class IndicesAliasesRequest extends AcknowledgedRequest { - private List aliasActions = Lists.newArrayList(); + private List allAliasActions = Lists.newArrayList(); + + private IndicesOptions indicesOptions = IndicesOptions.fromOptions(false, false, true, false); public IndicesAliasesRequest() { } - /** - * Adds an alias to the index. - * - * @param index The index - * @param alias The alias + /* + * Aliases can be added by passing multiple indices to the Request and + * deleted by passing multiple indices and aliases. They are expanded into + * distinct AliasAction instances when the request is processed. This class + * holds the AliasAction and in addition the arrays or alias names and + * indices that is later used to create the final AliasAction instances. */ - public IndicesAliasesRequest addAlias(String index, String alias) { - aliasActions.add(new AliasAction(AliasAction.Type.ADD, index, alias)); - return this; + public static class AliasActions { + private String[] indices = Strings.EMPTY_ARRAY; + private String[] aliases = Strings.EMPTY_ARRAY; + private AliasAction aliasAction; + + public AliasActions(AliasAction.Type type, String[] indices, String[] aliases) { + aliasAction = new AliasAction(type); + indices(indices); + aliases(aliases); + } + + public AliasActions(AliasAction.Type type, String index, String alias) { + aliasAction = new AliasAction(type); + indices(index); + aliases(alias); + } + + AliasActions(AliasAction.Type type, String[] index, String alias) { + aliasAction = new AliasAction(type); + indices(index); + aliases(alias); + } + + public AliasActions(AliasAction action) { + this.aliasAction = action; + indices(action.index()); + aliases(action.alias()); + } + + public AliasActions(Type type, String index, String[] aliases) { + aliasAction = new AliasAction(type); + indices(index); + aliases(aliases); + } + + public AliasActions() { + } + + public AliasActions filter(Map filter) { + aliasAction.filter(filter); + return this; + } + + public AliasActions filter(FilterBuilder filter) { + aliasAction.filter(filter); + return this; + } + + public Type actionType() { + return aliasAction.actionType(); + } + + public void routing(String routing) { + aliasAction.routing(routing); + } + + public void searchRouting(String searchRouting) { + aliasAction.searchRouting(searchRouting); + } + + public void indexRouting(String indexRouting) { + aliasAction.indexRouting(indexRouting); + } + + public AliasActions filter(String filter) { + aliasAction.filter(filter); + return this; + } + + public void indices(String... indices) { + List finalIndices = new ArrayList(); + for (String index : indices) { + if (index != null) { + finalIndices.add(index); + } + } + this.indices = finalIndices.toArray(new String[finalIndices.size()]); + } + + public void aliases(String... aliases) { + this.aliases = aliases; + } + + public String[] aliases() { + return aliases; + } + + public String[] indices() { + return indices; + } + + public AliasAction aliasAction() { + return aliasAction; + } + + public String[] concreteAliases(MetaData metaData, String concreteIndex) { + if (aliasAction.actionType() == Type.REMOVE) { + //for DELETE we expand the aliases + String[] indexAsArray = {concreteIndex}; + ImmutableOpenMap> aliasMetaData = metaData.findAliases(aliases, indexAsArray); + List finalAliases = new ArrayList (); + for (ObjectCursor> curAliases : aliasMetaData.values()) { + for (AliasMetaData aliasMeta: curAliases.value) { + finalAliases.add(aliasMeta.alias()); + } + } + return finalAliases.toArray(new String[finalAliases.size()]); + } else { + //for add we just return the current aliases + return aliases; + } + } + public AliasActions readFrom(StreamInput in) throws IOException { + indices = in.readStringArray(); + aliases = in.readStringArray(); + aliasAction = readAliasAction(in); + return this; + } + + public void writeTo(StreamOutput out) throws IOException { + out.writeStringArray(indices); + out.writeStringArray(aliases); + this.aliasAction.writeTo(out); + } } /** * Adds an alias to the index. - * - * @param index The index - * @param alias The alias - * @param filter The filter + * @param alias The alias + * @param indices The indices */ - public IndicesAliasesRequest addAlias(String index, String alias, String filter) { - aliasActions.add(new AliasAction(AliasAction.Type.ADD, index, alias, filter)); + public IndicesAliasesRequest addAlias(String alias, String... indices) { + addAliasAction(new AliasActions(AliasAction.Type.ADD, indices, alias)); return this; } + + public void addAliasAction(AliasActions aliasAction) { + allAliasActions.add(aliasAction); + } + + + public IndicesAliasesRequest addAliasAction(AliasAction action) { + addAliasAction(new AliasActions(action)); + return this; + } + /** * Adds an alias to the index. - * - * @param index The index * @param alias The alias * @param filter The filter + * @param indices The indices */ - public IndicesAliasesRequest addAlias(String index, String alias, Map filter) { - if (filter == null || filter.isEmpty()) { - aliasActions.add(new AliasAction(AliasAction.Type.ADD, index, alias)); - return this; - } - try { - XContentBuilder builder = XContentFactory.contentBuilder(XContentType.JSON); - builder.map(filter); - aliasActions.add(new AliasAction(AliasAction.Type.ADD, index, alias, builder.string())); - return this; - } catch (IOException e) { - throw new ElasticsearchGenerationException("Failed to generate [" + filter + "]", e); - } + public IndicesAliasesRequest addAlias(String alias, Map filter, String... indices) { + addAliasAction(new AliasActions(AliasAction.Type.ADD, indices, alias).filter(filter)); + return this; } /** * Adds an alias to the index. - * - * @param index The index * @param alias The alias * @param filterBuilder The filter + * @param indices The indices */ - public IndicesAliasesRequest addAlias(String index, String alias, FilterBuilder filterBuilder) { - if (filterBuilder == null) { - aliasActions.add(new AliasAction(AliasAction.Type.ADD, index, alias)); - return this; - } - try { - XContentBuilder builder = XContentFactory.jsonBuilder(); - filterBuilder.toXContent(builder, ToXContent.EMPTY_PARAMS); - builder.close(); - return addAlias(index, alias, builder.string()); - } catch (IOException e) { - throw new ElasticsearchGenerationException("Failed to build json for alias request", e); - } + public IndicesAliasesRequest addAlias(String alias, FilterBuilder filterBuilder, String... indices) { + addAliasAction(new AliasActions(AliasAction.Type.ADD, indices, alias).filter(filterBuilder)); + return this; } - + + /** * Removes an alias to the index. * - * @param index The index - * @param alias The alias + * @param indices The indices + * @param aliases The aliases */ - public IndicesAliasesRequest removeAlias(String index, String alias) { - aliasActions.add(new AliasAction(AliasAction.Type.REMOVE, index, alias)); + public IndicesAliasesRequest removeAlias(String[] indices, String... aliases) { + addAliasAction(new AliasActions(AliasAction.Type.REMOVE, indices, aliases)); return this; } - - public IndicesAliasesRequest addAliasAction(AliasAction action) { - aliasActions.add(action); + + /** + * Removes an alias to the index. + * + * @param index The index + * @param aliases The aliases + */ + public IndicesAliasesRequest removeAlias(String index, String... aliases) { + addAliasAction(new AliasActions(AliasAction.Type.REMOVE, index, aliases)); return this; } - List aliasActions() { - return this.aliasActions; + List aliasActions() { + return this.allAliasActions; } - public List getAliasActions() { + public List getAliasActions() { return aliasActions(); } @Override public ActionRequestValidationException validate() { ActionRequestValidationException validationException = null; - if (aliasActions.isEmpty()) { + if (allAliasActions.isEmpty()) { return addValidationError("Must specify at least one alias action", validationException); } - for (AliasAction aliasAction : aliasActions) { - if (!Strings.hasText(aliasAction.alias())) { - validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + "] requires an [alias] to be set", validationException); + for (AliasActions aliasAction : allAliasActions) { + if (aliasAction.actionType() == AliasAction.Type.ADD) { + if (aliasAction.aliases.length != 1) { + validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + + "] requires exactly one [alias] to be set", validationException); + } + if (!Strings.hasText(aliasAction.aliases[0])) { + validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + + "] requires an [alias] to be set", validationException); + } + } else { + if (aliasAction.aliases.length == 0) { + validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + + "]: aliases may not be empty", validationException); + } + for (String alias : aliasAction.aliases) { + if (!Strings.hasText(alias)) { + validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + + "]: [alias] may not be empty string", validationException); + } + } + if (CollectionUtils.isEmpty(aliasAction.indices)) { + validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + + "]: indices may not be empty", validationException); + } } - if (!Strings.hasText(aliasAction.index())) { - validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + "] requires an [index] to be set", validationException); + if (!CollectionUtils.isEmpty(aliasAction.indices)) { + for (String index : aliasAction.indices) { + if (!Strings.hasText(index)) { + validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + + "]: [index] may not be empty string", validationException); + } + } } } return validationException; @@ -165,7 +314,7 @@ public void readFrom(StreamInput in) throws IOException { super.readFrom(in); int size = in.readVInt(); for (int i = 0; i < size; i++) { - aliasActions.add(readAliasAction(in)); + allAliasActions.add(readAliasActions(in)); } readTimeout(in); } @@ -173,10 +322,20 @@ public void readFrom(StreamInput in) throws IOException { @Override public void writeTo(StreamOutput out) throws IOException { super.writeTo(out); - out.writeVInt(aliasActions.size()); - for (AliasAction aliasAction : aliasActions) { + out.writeVInt(allAliasActions.size()); + for (AliasActions aliasAction : allAliasActions) { aliasAction.writeTo(out); } writeTimeout(out); } + + public IndicesOptions indicesOptions() { + return indicesOptions; + } + + private AliasActions readAliasActions(StreamInput in) throws IOException { + AliasActions actions = new AliasActions(); + return actions.readFrom(in); + } + } diff --git a/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequestBuilder.java b/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequestBuilder.java index 3724dfdce4359..90413723af171 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequestBuilder.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequestBuilder.java @@ -20,6 +20,7 @@ package org.elasticsearch.action.admin.indices.alias; import org.elasticsearch.action.ActionListener; +import org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest.AliasActions; import org.elasticsearch.action.support.master.AcknowledgedRequestBuilder; import org.elasticsearch.client.IndicesAdminClient; import org.elasticsearch.client.internal.InternalIndicesAdminClient; @@ -36,15 +37,26 @@ public class IndicesAliasesRequestBuilder extends AcknowledgedRequestBuilder filter) { + request.addAlias(alias, filter, indices); + return this; + } + + /** + * Adds an alias to the index. + * + * @param index The indices * @param alias The alias * @param filter The filter */ public IndicesAliasesRequestBuilder addAlias(String index, String alias, Map filter) { - request.addAlias(index, alias, filter); + request.addAlias(alias, filter, index); return this; } /** * Adds an alias to the index. * - * @param index The index + * @param indices The indices * @param alias The alias * @param filterBuilder The filter */ - public IndicesAliasesRequestBuilder addAlias(String index, String alias, FilterBuilder filterBuilder) { - request.addAlias(index, alias, filterBuilder); + public IndicesAliasesRequestBuilder addAlias(String indices[], String alias, FilterBuilder filterBuilder) { + request.addAlias(alias, filterBuilder, indices); return this; } - + /** - * Adds an alias action to the request. + * Adds an alias to the index. * - * @param aliasAction The alias Action + * @param index The index + * @param alias The alias + * @param filterBuilder The filter */ - public IndicesAliasesRequestBuilder addAliasAction(AliasAction aliasAction) { - request.addAliasAction(aliasAction); + public IndicesAliasesRequestBuilder addAlias(String index, String alias, FilterBuilder filterBuilder) { + request.addAlias(alias, filterBuilder, index); return this; } /** - * Removes an alias to the index. + * Removes an alias from the index. * * @param index The index * @param alias The alias @@ -104,9 +144,54 @@ public IndicesAliasesRequestBuilder removeAlias(String index, String alias) { request.removeAlias(index, alias); return this; } - + + /** + * Removes aliases from the index. + * + * @param indices The indices + * @param aliases The aliases + */ + public IndicesAliasesRequestBuilder removeAlias(String[] indices, String... aliases) { + request.removeAlias(indices, aliases); + return this; + } + + /** + * Removes aliases from the index. + * + * @param index The index + * @param aliases The aliases + */ + public IndicesAliasesRequestBuilder removeAlias(String index, String[] aliases) { + request.removeAlias(index, aliases); + return this; + } + @Override protected void doExecute(ActionListener listener) { ((IndicesAdminClient) client).aliases(request, listener); } + + /** + * Adds an alias action to the request. + * + * @param aliasAction The alias action + */ + public IndicesAliasesRequestBuilder addAliasAction(AliasAction aliasAction) { + request.addAliasAction(aliasAction); + return this; + } + + /** + * Adds an alias action to the request. + * + * @param aliasAction The alias action + */ + public IndicesAliasesRequestBuilder addAliasAction( + AliasActions action) { + request.addAliasAction(action); + return this; + } + + } diff --git a/src/main/java/org/elasticsearch/action/admin/indices/alias/TransportIndicesAliasesAction.java b/src/main/java/org/elasticsearch/action/admin/indices/alias/TransportIndicesAliasesAction.java index 5fab726c33aed..64da64e5cfd6b 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/alias/TransportIndicesAliasesAction.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/alias/TransportIndicesAliasesAction.java @@ -22,6 +22,7 @@ import com.google.common.collect.Sets; import org.elasticsearch.ElasticsearchException; import org.elasticsearch.action.ActionListener; +import org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest.AliasActions; import org.elasticsearch.action.support.master.TransportMasterNodeOperationAction; import org.elasticsearch.cluster.ClusterService; import org.elasticsearch.cluster.ClusterState; @@ -33,9 +34,13 @@ import org.elasticsearch.cluster.metadata.MetaDataIndexAliasesService; import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.rest.action.admin.indices.alias.delete.AliasesMissingException; import org.elasticsearch.threadpool.ThreadPool; import org.elasticsearch.transport.TransportService; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; import java.util.Set; /** @@ -76,8 +81,10 @@ protected IndicesAliasesResponse newResponse() { @Override protected ClusterBlockException checkBlock(IndicesAliasesRequest request, ClusterState state) { Set indices = Sets.newHashSet(); - for (AliasAction aliasAction : request.aliasActions()) { - indices.add(aliasAction.index()); + for (AliasActions aliasAction : request.aliasActions()) { + for (String index : aliasAction.indices()) { + indices.add(index); + } } return state.blocks().indicesBlockedException(ClusterBlockLevel.METADATA, indices.toArray(new String[indices.size()])); } @@ -85,9 +92,36 @@ protected ClusterBlockException checkBlock(IndicesAliasesRequest request, Cluste @Override protected void masterOperation(final IndicesAliasesRequest request, final ClusterState state, final ActionListener listener) throws ElasticsearchException { + //Expand the indices names + List actions = request.aliasActions(); + List finalActions = new ArrayList(); + boolean hasOnlyDeletesButNoneCanBeDone = true; + Set aliases = new HashSet(); + for (AliasActions action : actions) { + //expand indices + String[] concreteIndices = state.metaData().concreteIndices(action.indices(), request.indicesOptions()); + //collect the aliases + for (String alias : action.aliases()) { + aliases.add(alias); + } + for (String index : concreteIndices) { + for (String alias : action.concreteAliases(state.metaData(), index)) { + AliasAction finalAction = new AliasAction(action.aliasAction()); + finalAction.index(index); + finalAction.alias(alias); + finalActions.add(finalAction); + //if there is only delete requests, none will be added if the types do not map to any existing type + hasOnlyDeletesButNoneCanBeDone = false; + } + } + } + if (hasOnlyDeletesButNoneCanBeDone && actions.size() != 0) { + throw new AliasesMissingException(aliases.toArray(new String[aliases.size()])); + } + request.aliasActions().clear(); IndicesAliasesClusterStateUpdateRequest updateRequest = new IndicesAliasesClusterStateUpdateRequest() .ackTimeout(request.timeout()).masterNodeTimeout(request.masterNodeTimeout()) - .actions(request.aliasActions().toArray(new AliasAction[request.aliasActions().size()])); + .actions(finalActions.toArray(new AliasAction[finalActions.size()])); indexAliasesService.indicesAliases(updateRequest, new ClusterStateUpdateListener() { @Override diff --git a/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingClusterStateUpdateRequest.java b/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingClusterStateUpdateRequest.java index 49ed091a19b95..5a2ef82eba17e 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingClusterStateUpdateRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingClusterStateUpdateRequest.java @@ -26,7 +26,7 @@ */ public class DeleteMappingClusterStateUpdateRequest extends IndicesClusterStateUpdateRequest { - private String type; + private String[] types; DeleteMappingClusterStateUpdateRequest() { @@ -35,15 +35,15 @@ public class DeleteMappingClusterStateUpdateRequest extends IndicesClusterStateU /** * Returns the type to be removed */ - public String type() { - return type; + public String[] types() { + return types; } /** * Sets the type to be removed */ - public DeleteMappingClusterStateUpdateRequest type(String type) { - this.type = type; + public DeleteMappingClusterStateUpdateRequest types(String[] types) { + this.types = types; return this; } } diff --git a/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingRequest.java b/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingRequest.java index 3d1f0fc450930..e342a5d767323 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingRequest.java @@ -23,8 +23,10 @@ import org.elasticsearch.action.ActionRequestValidationException; import org.elasticsearch.action.support.IndicesOptions; import org.elasticsearch.action.support.master.AcknowledgedRequest; +import org.elasticsearch.common.Strings; import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.util.CollectionUtils; import java.io.IOException; @@ -37,7 +39,7 @@ public class DeleteMappingRequest extends AcknowledgedRequest() { @Override public void onResponse(FlushResponse flushResponse) { + + // get all types that need to be deleted. + ImmutableOpenMap> result = state.metaData().findMappings( + request.indices(), request.types() + ); + // create OrFilter with type filters within to account for different types + OrFilterBuilder filterBuilder = new OrFilterBuilder(); + Set types = new HashSet(); + for (ObjectObjectCursor> typesMeta : result) { + for (ObjectObjectCursor type : typesMeta.value) { + filterBuilder.add(new TypeFilterBuilder(type.key)); + types.add(type.key); + } + } + request.types(types.toArray(new String[types.size()])); QuerySourceBuilder querySourceBuilder = new QuerySourceBuilder() - .setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(), FilterBuilders.typeFilter(request.type()))); + .setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(), filterBuilder)); deleteByQueryAction.execute(Requests.deleteByQueryRequest(request.indices()).source(querySourceBuilder), new ActionListener() { @Override public void onResponse(DeleteByQueryResponse deleteByQueryResponse) { @@ -126,7 +148,7 @@ public void onFailure(Throwable e) { protected void removeMapping() { DeleteMappingClusterStateUpdateRequest clusterStateUpdateRequest = new DeleteMappingClusterStateUpdateRequest() - .indices(request.indices()).type(request.type()) + .indices(request.indices()).types(request.types()) .ackTimeout(request.timeout()) .masterNodeTimeout(request.masterNodeTimeout()); diff --git a/src/main/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequest.java b/src/main/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequest.java index 90057b8fa6d9a..3c79b104517e4 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequest.java @@ -26,15 +26,18 @@ import org.elasticsearch.common.Strings; import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.util.CollectionUtils; import java.io.IOException; +import static org.elasticsearch.action.ValidateActions.addValidationError; + /** * A request to delete an index warmer. */ public class DeleteWarmerRequest extends AcknowledgedRequest { - private String name; + private String[] names = Strings.EMPTY_ARRAY; private IndicesOptions indicesOptions = IndicesOptions.fromOptions(false, false, true, false); private String[] indices = Strings.EMPTY_ARRAY; @@ -46,36 +49,60 @@ public class DeleteWarmerRequest extends AcknowledgedRequest listener) throws ElasticsearchException { - clusterService.submitStateUpdateTask("delete_warmer [" + request.name() + "]", new AckedClusterStateUpdateTask() { + clusterService.submitStateUpdateTask("delete_warmer [" + Arrays.toString(request.names()) + "]", new AckedClusterStateUpdateTask() { @Override public boolean mustAck(DiscoveryNode discoveryNode) { @@ -118,7 +119,7 @@ public TimeValue timeout() { @Override public void onFailure(String source, Throwable t) { - logger.debug("failed to delete warmer [{}] on indices [{}]", t, request.name(), request.indices()); + logger.debug("failed to delete warmer [{}] on indices [{}]", t, Arrays.toString(request.names()), request.indices()); listener.onFailure(t); } @@ -136,10 +137,16 @@ public ClusterState execute(ClusterState currentState) { if (warmers != null) { List entries = Lists.newArrayList(); for (IndexWarmersMetaData.Entry entry : warmers.entries()) { - if (request.name() == null || Regex.simpleMatch(request.name(), entry.name())) { - globalFoundAtLeastOne = true; - // don't add it... - } else { + boolean keepWarmer = true; + for (String warmer : request.names()) { + if (Regex.simpleMatch(warmer, entry.name()) || warmer.equals("_all")) { + globalFoundAtLeastOne = true; + keepWarmer = false; + // don't add it... + break; + } + } + if (keepWarmer) { entries.add(entry); } } @@ -153,11 +160,7 @@ public ClusterState execute(ClusterState currentState) { } if (!globalFoundAtLeastOne) { - if (request.name() == null) { - // full match, just return with no failure - return currentState; - } - throw new IndexWarmerMissingException(request.name()); + throw new IndexWarmerMissingException(request.names()); } if (logger.isInfoEnabled()) { @@ -169,8 +172,10 @@ public ClusterState execute(ClusterState currentState) { IndexWarmersMetaData warmers = indexMetaData.custom(IndexWarmersMetaData.TYPE); if (warmers != null) { for (IndexWarmersMetaData.Entry entry : warmers.entries()) { - if (Regex.simpleMatch(request.name(), entry.name())) { - logger.info("[{}] delete warmer [{}]", index, entry.name()); + for (String warmer : request.names()) { + if (Regex.simpleMatch(warmer, entry.name()) || warmer.equals("_all")) { + logger.info("[{}] delete warmer [{}]", index, entry.name()); + } } } } diff --git a/src/main/java/org/elasticsearch/cluster/metadata/AliasAction.java b/src/main/java/org/elasticsearch/cluster/metadata/AliasAction.java index d0834760f4dbd..29bcba360bbc2 100644 --- a/src/main/java/org/elasticsearch/cluster/metadata/AliasAction.java +++ b/src/main/java/org/elasticsearch/cluster/metadata/AliasAction.java @@ -82,7 +82,20 @@ public static Type fromValue(byte value) { private AliasAction() { } - + + public AliasAction(AliasAction other) { + this.actionType = other.actionType; + this.index = other.index; + this.alias = other.alias; + this.filter = other.filter; + this.indexRouting = other.indexRouting; + this.searchRouting = other.searchRouting; + } + + public AliasAction(Type actionType) { + this.actionType = actionType; + } + public AliasAction(Type actionType, String index, String alias) { this.actionType = actionType; this.index = index; @@ -99,10 +112,20 @@ public AliasAction(Type actionType, String index, String alias, String filter) { public Type actionType() { return actionType; } + + public AliasAction index(String index) { + this.index = index; + return this; + } public String index() { return index; } + + public AliasAction alias(String alias) { + this.alias = alias; + return this; + } public String alias() { return alias; @@ -181,42 +204,21 @@ public static AliasAction readAliasAction(StreamInput in) throws IOException { @Override public void readFrom(StreamInput in) throws IOException { actionType = Type.fromValue(in.readByte()); - index = in.readString(); - alias = in.readString(); - if (in.readBoolean()) { - filter = in.readString(); - } - if (in.readBoolean()) { - indexRouting = in.readString(); - } - if (in.readBoolean()) { - searchRouting = in.readString(); - } + index = in.readOptionalString(); + alias = in.readOptionalString(); + filter = in.readOptionalString(); + indexRouting = in.readOptionalString(); + searchRouting = in.readOptionalString(); } @Override public void writeTo(StreamOutput out) throws IOException { out.writeByte(actionType.value()); - out.writeString(index); - out.writeString(alias); - if (filter == null) { - out.writeBoolean(false); - } else { - out.writeBoolean(true); - out.writeString(filter); - } - if (indexRouting == null) { - out.writeBoolean(false); - } else { - out.writeBoolean(true); - out.writeString(indexRouting); - } - if (searchRouting == null) { - out.writeBoolean(false); - } else { - out.writeBoolean(true); - out.writeString(searchRouting); - } + out.writeOptionalString(index); + out.writeOptionalString(alias); + out.writeOptionalString(filter); + out.writeOptionalString(indexRouting); + out.writeOptionalString(searchRouting); } public static AliasAction newAddAliasAction(String index, String alias) { diff --git a/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java b/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java index 2ac352f3643fb..b2310d2532cd8 100644 --- a/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java +++ b/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java @@ -279,7 +279,7 @@ public ImmutableOpenMap> findAliases(final return ImmutableOpenMap.of(); } - boolean matchAllAliases = aliases.length == 0; + boolean matchAllAliases = matchAllAliases(aliases); ImmutableOpenMap.Builder> mapBuilder = ImmutableOpenMap.builder(); Iterable intersection = HppcMaps.intersection(ObjectOpenHashSet.from(concreteIndices), indices.keys()); for (String index : intersection) { @@ -298,6 +298,15 @@ public ImmutableOpenMap> findAliases(final } return mapBuilder.build(); } + + private boolean matchAllAliases(final String[] aliases) { + for (String alias : aliases) { + if (alias.equals("_all")) { + return true; + } + } + return aliases.length == 0; + } /** * Checks if at least one of the specified aliases exists in the specified concrete indices. Wildcards are supported in the @@ -331,6 +340,12 @@ public boolean hasAliases(final String[] aliases, String[] concreteIndices) { return false; } + /* + * Finds all mappings for types and concrete indices. Types are expanded to + * include all types that match the glob patterns in the types array. Empty + * types array, null or {"_all"} will be expanded to all types available for + * the given indices. + */ public ImmutableOpenMap> findMappings(String[] concreteIndices, final String[] types) { assert types != null; assert concreteIndices != null; @@ -343,7 +358,7 @@ public ImmutableOpenMap> findM for (String index : intersection) { IndexMetaData indexMetaData = indices.get(index); ImmutableOpenMap.Builder filteredMappings; - if (types.length == 0) { + if (isAllTypes(types)) { indexMapBuilder.put(index, indexMetaData.getMappings()); // No types specified means get it all } else { @@ -945,7 +960,18 @@ public String[] filteringAliases(String index, String... indicesOrAliases) { * @return true if the provided array maps to all indices, false otherwise */ public boolean isAllIndices(String[] aliasesOrIndices) { - return aliasesOrIndices == null || aliasesOrIndices.length == 0 || isExplicitAllIndices(aliasesOrIndices); + return aliasesOrIndices == null || aliasesOrIndices.length == 0 || isExplicitAllPattern(aliasesOrIndices); + } + + /** + * Identifies whether the array containing type names given as argument refers to all types + * The empty or null array identifies all types + * + * @param types the array containing index names + * @return true if the provided array maps to all indices, false otherwise + */ + public boolean isAllTypes(String[] types) { + return types == null || types.length == 0 || isExplicitAllPattern(types); } /** @@ -955,7 +981,7 @@ public boolean isAllIndices(String[] aliasesOrIndices) { * @param aliasesOrIndices the array containing index names * @return true if the provided array explicitly maps to all indices, false otherwise */ - public boolean isExplicitAllIndices(String[] aliasesOrIndices) { + public boolean isExplicitAllPattern(String[] aliasesOrIndices) { return aliasesOrIndices != null && aliasesOrIndices.length == 1 && "_all".equals(aliasesOrIndices[0]); } diff --git a/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java b/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java index 046dd44190f62..414bc40a55068 100644 --- a/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java +++ b/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java @@ -385,7 +385,7 @@ public ClusterState execute(final ClusterState currentState) throws Exception { } public void removeMapping(final DeleteMappingClusterStateUpdateRequest request, final ClusterStateUpdateListener listener) { - clusterService.submitStateUpdateTask("remove-mapping [" + request.type() + "]", Priority.HIGH, new AckedClusterStateUpdateTask() { + clusterService.submitStateUpdateTask("remove-mapping [" + Arrays.toString(request.types()) + "]", Priority.HIGH, new AckedClusterStateUpdateTask() { @Override public boolean mustAck(DiscoveryNode discoveryNode) { @@ -428,21 +428,30 @@ public ClusterState execute(ClusterState currentState) { String latestIndexWithout = null; for (String indexName : request.indices()) { IndexMetaData indexMetaData = currentState.metaData().index(indexName); + IndexMetaData.Builder indexBuilder = IndexMetaData.builder(indexMetaData); + if (indexMetaData != null) { - if (indexMetaData.mappings().containsKey(request.type())) { - builder.put(IndexMetaData.builder(indexMetaData).removeMapping(request.type())); - changed = true; - } else { + boolean isLatestIndexWithout = true; + for (String type : request.types()) { + if (indexMetaData.mappings().containsKey(type)) { + indexBuilder.removeMapping(type); + changed = true; + isLatestIndexWithout = false; + } + } + if (isLatestIndexWithout) { latestIndexWithout = indexMetaData.index(); } + } + builder.put(indexBuilder); } if (!changed) { - throw new TypeMissingException(new Index(latestIndexWithout), request.type()); + throw new TypeMissingException(new Index(latestIndexWithout), request.types()); } - logger.info("[{}] remove_mapping [{}]", request.indices(), request.type()); + logger.info("[{}] remove_mapping [{}]", request.indices(), request.types()); return ClusterState.builder(currentState).metaData(builder).build(); } diff --git a/src/main/java/org/elasticsearch/common/util/CollectionUtils.java b/src/main/java/org/elasticsearch/common/util/CollectionUtils.java index c24a6e67be0d9..9d6e2753a87db 100644 --- a/src/main/java/org/elasticsearch/common/util/CollectionUtils.java +++ b/src/main/java/org/elasticsearch/common/util/CollectionUtils.java @@ -187,5 +187,16 @@ public static int sortAndDedup(double[] array, int len) { } return uniqueCount; } + + /** + * Checks if the given array contains any elements. + * + * @param array The array to check + * + * @return false if the array contains an element, true if not or the array is null. + */ + public static boolean isEmpty(Object[] array) { + return array == null || array.length == 0; + } } diff --git a/src/main/java/org/elasticsearch/indices/TypeMissingException.java b/src/main/java/org/elasticsearch/indices/TypeMissingException.java index 494c2ada69353..ae2830b957595 100644 --- a/src/main/java/org/elasticsearch/indices/TypeMissingException.java +++ b/src/main/java/org/elasticsearch/indices/TypeMissingException.java @@ -23,17 +23,19 @@ import org.elasticsearch.index.IndexException; import org.elasticsearch.rest.RestStatus; +import java.util.Arrays; + /** * */ public class TypeMissingException extends IndexException { - public TypeMissingException(Index index, String type) { - super(index, "type[" + type + "] missing"); + public TypeMissingException(Index index, String... types) { + super(index, "type[" + Arrays.toString(types) + "] missing"); } - public TypeMissingException(Index index, String type, String message) { - super(index, "type[" + type + "] missing: " + message); + public TypeMissingException(Index index, String[] types, String message) { + super(index, "type[" + Arrays.toString(types) + "] missing: " + message); } diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/AliasesMissingException.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/AliasesMissingException.java new file mode 100644 index 0000000000000..dfc98b0cb823a --- /dev/null +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/AliasesMissingException.java @@ -0,0 +1,46 @@ +/* + * Licensed to Elasticsearch under one or more contributor + * license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright + * ownership. Elasticsearch licenses this file to you under + * the Apache License, Version 2.0 (the "License"); you may + * not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.elasticsearch.rest.action.admin.indices.alias.delete; + +import org.elasticsearch.ElasticsearchException; +import org.elasticsearch.rest.RestStatus; + +import java.util.Arrays; + +/** + * + */ +public class AliasesMissingException extends ElasticsearchException { + + private final String[] names; + + public AliasesMissingException(String... names) { + super("aliases [" + Arrays.toString(names) + "] missing"); + this.names = names; + } + + public String[] names() { + return this.names; + } + + @Override + public RestStatus status() { + return RestStatus.NOT_FOUND; + } +} diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/RestIndexDeleteAliasesAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/RestIndexDeleteAliasesAction.java index 4e03f603c685a..8cbd9d7709265 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/RestIndexDeleteAliasesAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/RestIndexDeleteAliasesAction.java @@ -21,6 +21,7 @@ import org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest; import org.elasticsearch.action.admin.indices.alias.IndicesAliasesResponse; import org.elasticsearch.client.Client; +import org.elasticsearch.common.Strings; import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.rest.*; @@ -35,15 +36,16 @@ public class RestIndexDeleteAliasesAction extends BaseRestHandler { public RestIndexDeleteAliasesAction(Settings settings, Client client, RestController controller) { super(settings, client); controller.registerHandler(DELETE, "/{index}/_alias/{name}", this); + controller.registerHandler(DELETE, "/{index}/_aliases/{name}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { - final String index = request.param("index"); - String alias = request.param("name"); + final String[] indices = Strings.splitStringByCommaToArray(request.param("index")); + final String[] aliases = Strings.splitStringByCommaToArray(request.param("name")); IndicesAliasesRequest indicesAliasesRequest = new IndicesAliasesRequest(); indicesAliasesRequest.timeout(request.paramAsTime("timeout", indicesAliasesRequest.timeout())); - indicesAliasesRequest.removeAlias(index, alias); + indicesAliasesRequest.removeAlias(indices, aliases); indicesAliasesRequest.masterNodeTimeout(request.paramAsTime("master_timeout", indicesAliasesRequest.masterNodeTimeout())); client.admin().indices().aliases(indicesAliasesRequest, new AcknowledgedRestResponseActionListener(request, channel, logger)); diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/put/RestIndexPutAliasAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/put/RestIndexPutAliasAction.java index 3b4fa4ee82815..c2e3b5d549262 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/put/RestIndexPutAliasAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/put/RestIndexPutAliasAction.java @@ -20,9 +20,11 @@ import org.elasticsearch.ElasticsearchIllegalArgumentException; import org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest; +import org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest.AliasActions; import org.elasticsearch.action.admin.indices.alias.IndicesAliasesResponse; import org.elasticsearch.client.Client; import org.elasticsearch.cluster.metadata.AliasAction; +import org.elasticsearch.common.Strings; import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.xcontent.XContentFactory; @@ -32,6 +34,7 @@ import java.io.IOException; import java.util.Map; +import static org.elasticsearch.rest.RestRequest.Method.POST; import static org.elasticsearch.rest.RestRequest.Method.PUT; /** @@ -43,13 +46,22 @@ public RestIndexPutAliasAction(Settings settings, Client client, RestController super(settings, client); controller.registerHandler(PUT, "/{index}/_alias/{name}", this); controller.registerHandler(PUT, "/_alias/{name}", this); + controller.registerHandler(PUT, "/{index}/_aliases/{name}", this); + controller.registerHandler(PUT, "/_aliases/{name}", this); controller.registerHandler(PUT, "/{index}/_alias", this); controller.registerHandler(PUT, "/_alias", this); + + controller.registerHandler(POST, "/{index}/_alias/{name}", this); + controller.registerHandler(POST, "/_alias/{name}", this); + controller.registerHandler(POST, "/{index}/_aliases/{name}", this); + controller.registerHandler(POST, "/_aliases/{name}", this); + controller.registerHandler(PUT, "/{index}/_aliases", this); + //we cannot add POST for "/_aliases" because this is the _aliases api already defined in RestIndicesAliasesAction } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { - String index = request.param("index"); + String[] indices = Strings.splitStringByCommaToArray(request.param("index")); String alias = request.param("name"); Map filter = null; String routing = null; @@ -70,7 +82,7 @@ public void handleRequest(final RestRequest request, final RestChannel channel) currentFieldName = parser.currentName(); } else if (token.isValue()) { if ("index".equals(currentFieldName)) { - index = parser.text(); + indices = Strings.splitStringByCommaToArray(parser.text()); } else if ("alias".equals(currentFieldName)) { alias = parser.text(); } else if ("routing".equals(currentFieldName)) { @@ -102,9 +114,11 @@ public void handleRequest(final RestRequest request, final RestChannel channel) IndicesAliasesRequest indicesAliasesRequest = new IndicesAliasesRequest(); indicesAliasesRequest.timeout(request.paramAsTime("timeout", indicesAliasesRequest.timeout())); - AliasAction aliasAction = new AliasAction(AliasAction.Type.ADD, index, alias); + String[] aliases = new String[] {alias}; + IndicesAliasesRequest.AliasActions aliasAction = new AliasActions(AliasAction.Type.ADD, indices, aliases); indicesAliasesRequest.addAliasAction(aliasAction); indicesAliasesRequest.masterNodeTimeout(request.paramAsTime("master_timeout", indicesAliasesRequest.masterNodeTimeout())); + if (routing != null) { aliasAction.routing(routing); diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/delete/RestDeleteMappingAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/delete/RestDeleteMappingAction.java index 28cff7e776350..d915609b243bf 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/delete/RestDeleteMappingAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/delete/RestDeleteMappingAction.java @@ -41,13 +41,18 @@ public RestDeleteMappingAction(Settings settings, Client client, RestController super(settings, client); controller.registerHandler(DELETE, "/{index}/{type}/_mapping", this); controller.registerHandler(DELETE, "/{index}/{type}", this); + controller.registerHandler(DELETE, "/{index}/_mapping/{type}", this); + + //support _mappings also + controller.registerHandler(DELETE, "/{index}/{type}/_mappings", this); + controller.registerHandler(DELETE, "/{index}/_mappings/{type}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { DeleteMappingRequest deleteMappingRequest = deleteMappingRequest(Strings.splitStringByCommaToArray(request.param("index"))); deleteMappingRequest.listenerThreaded(false); - deleteMappingRequest.type(request.param("type")); + deleteMappingRequest.types(Strings.splitStringByCommaToArray(request.param("type"))); deleteMappingRequest.timeout(request.paramAsTime("timeout", deleteMappingRequest.timeout())); deleteMappingRequest.masterNodeTimeout(request.paramAsTime("master_timeout", deleteMappingRequest.masterNodeTimeout())); deleteMappingRequest.indicesOptions(IndicesOptions.fromRequest(request, deleteMappingRequest.indicesOptions())); diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/put/RestPutMappingAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/put/RestPutMappingAction.java index e1f285d527daf..b6e9a5ebabcd0 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/put/RestPutMappingAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/put/RestPutMappingAction.java @@ -37,14 +37,30 @@ */ public class RestPutMappingAction extends BaseRestHandler { + @Inject public RestPutMappingAction(Settings settings, Client client, RestController controller) { super(settings, client); - controller.registerHandler(PUT, "/{index}/_mapping", this); + controller.registerHandler(PUT, "/{index}/_mapping/", this); controller.registerHandler(PUT, "/{index}/{type}/_mapping", this); + controller.registerHandler(PUT, "/{index}/_mapping/{type}", this); + controller.registerHandler(PUT, "/_mapping/{type}", this); - controller.registerHandler(POST, "/{index}/_mapping", this); + controller.registerHandler(POST, "/{index}/_mapping/", this); controller.registerHandler(POST, "/{index}/{type}/_mapping", this); + controller.registerHandler(POST, "/{index}/_mapping/{type}", this); + controller.registerHandler(POST, "/_mapping/{type}", this); + + //register the same paths, but with plural form _mappings + controller.registerHandler(PUT, "/{index}/_mappings/", this); + controller.registerHandler(PUT, "/{index}/{type}/_mappings", this); + controller.registerHandler(PUT, "/{index}/_mappings/{type}", this); + controller.registerHandler(PUT, "/_mappings/{type}", this); + + controller.registerHandler(POST, "/{index}/_mappings/", this); + controller.registerHandler(POST, "/{index}/{type}/_mappings", this); + controller.registerHandler(POST, "/{index}/_mappings/{type}", this); + controller.registerHandler(POST, "/_mappings/{type}", this); } @Override diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/delete/RestDeleteWarmerAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/delete/RestDeleteWarmerAction.java index 2ff34c8f28aa6..e54e4589921ad 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/delete/RestDeleteWarmerAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/delete/RestDeleteWarmerAction.java @@ -37,12 +37,13 @@ public RestDeleteWarmerAction(Settings settings, Client client, RestController c super(settings, client); controller.registerHandler(DELETE, "/{index}/_warmer", this); controller.registerHandler(DELETE, "/{index}/_warmer/{name}", this); - controller.registerHandler(DELETE, "/{index}/{type}/_warmer/{name}", this); + controller.registerHandler(DELETE, "/{index}/_warmers", this); + controller.registerHandler(DELETE, "/{index}/_warmers/{name}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { - DeleteWarmerRequest deleteWarmerRequest = new DeleteWarmerRequest(request.param("name")) + DeleteWarmerRequest deleteWarmerRequest = new DeleteWarmerRequest(Strings.splitStringByCommaToArray(request.param("name"))) .indices(Strings.splitStringByCommaToArray(request.param("index"))); deleteWarmerRequest.listenerThreaded(false); deleteWarmerRequest.timeout(request.paramAsTime("timeout", deleteWarmerRequest.timeout())); diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/put/RestPutWarmerAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/put/RestPutWarmerAction.java index 4a1d03435e09b..b0bc3b1f35eda 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/put/RestPutWarmerAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/put/RestPutWarmerAction.java @@ -27,6 +27,7 @@ import org.elasticsearch.common.settings.Settings; import org.elasticsearch.rest.*; +import static org.elasticsearch.rest.RestRequest.Method.POST; import static org.elasticsearch.rest.RestRequest.Method.PUT; /** @@ -36,8 +37,21 @@ public class RestPutWarmerAction extends BaseRestHandler { @Inject public RestPutWarmerAction(Settings settings, Client client, RestController controller) { super(settings, client); + controller.registerHandler(PUT, "/_warmer/{name}", this); controller.registerHandler(PUT, "/{index}/_warmer/{name}", this); controller.registerHandler(PUT, "/{index}/{type}/_warmer/{name}", this); + + controller.registerHandler(PUT, "/_warmers/{name}", this); + controller.registerHandler(PUT, "/{index}/_warmers/{name}", this); + controller.registerHandler(PUT, "/{index}/{type}/_warmers/{name}", this); + + controller.registerHandler(POST, "/_warmer/{name}", this); + controller.registerHandler(POST, "/{index}/_warmer/{name}", this); + controller.registerHandler(POST, "/{index}/{type}/_warmer/{name}", this); + + controller.registerHandler(POST, "/_warmers/{name}", this); + controller.registerHandler(POST, "/{index}/_warmers/{name}", this); + controller.registerHandler(POST, "/{index}/{type}/_warmers/{name}", this); } @Override diff --git a/src/main/java/org/elasticsearch/search/warmer/IndexWarmerMissingException.java b/src/main/java/org/elasticsearch/search/warmer/IndexWarmerMissingException.java index 7e8517e4b08fe..3892714bb0d67 100644 --- a/src/main/java/org/elasticsearch/search/warmer/IndexWarmerMissingException.java +++ b/src/main/java/org/elasticsearch/search/warmer/IndexWarmerMissingException.java @@ -21,20 +21,22 @@ import org.elasticsearch.ElasticsearchException; import org.elasticsearch.rest.RestStatus; +import java.util.Arrays; + /** * */ public class IndexWarmerMissingException extends ElasticsearchException { - private final String name; + private final String[] names; - public IndexWarmerMissingException(String name) { - super("index_warmer [" + name + "] missing"); - this.name = name; + public IndexWarmerMissingException(String... names) { + super("index_warmer [" + Arrays.toString(names) + "] missing"); + this.names = names; } - public String name() { - return this.name; + public String[] names() { + return this.names; } diff --git a/src/test/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequestTests.java b/src/test/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequestTests.java index 3cb7e1e579843..386abbd3efb16 100644 --- a/src/test/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequestTests.java +++ b/src/test/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequestTests.java @@ -48,7 +48,7 @@ public void testDeleteWarmerTimeoutBwComp_Pre0906Format() throws Exception { DeleteWarmerRequest inRequest = new DeleteWarmerRequest(); inRequest.readFrom(esBuffer); - assertThat(inRequest.name(), equalTo("warmer1")); + assertThat(inRequest.names()[0], equalTo("warmer1")); //timeout is default as we don't read it from the received buffer assertThat(inRequest.timeout().millis(), equalTo(new DeleteWarmerRequest().timeout().millis())); @@ -70,7 +70,7 @@ public void testDeleteWarmerTimeoutBwComp_Post0906Format() throws Exception { DeleteWarmerRequest inRequest = new DeleteWarmerRequest(); inRequest.readFrom(esBuffer); - assertThat(inRequest.name(), equalTo("warmer1")); + assertThat(inRequest.names()[0], equalTo("warmer1")); //timeout is default as we don't read it from the received buffer assertThat(inRequest.timeout().millis(), equalTo(outRequest.timeout().millis())); diff --git a/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java b/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java index 5599857fd1849..c01abde04fe51 100644 --- a/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java +++ b/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java @@ -35,6 +35,8 @@ import org.elasticsearch.common.unit.TimeValue; import org.elasticsearch.index.query.FilterBuilder; import org.elasticsearch.index.query.QueryBuilders; +import org.elasticsearch.indices.IndexMissingException; +import org.elasticsearch.rest.action.admin.indices.alias.delete.AliasesMissingException; import org.elasticsearch.search.SearchHit; import org.elasticsearch.search.SearchHits; import org.elasticsearch.search.facet.FacetBuilders; @@ -90,7 +92,7 @@ public void testAliases() throws Exception { ensureGreen(); logger.info("--> remove [alias1], Aliasing index [test_x] with [alias1]"); - admin().indices().aliases(indexAliasesRequest().removeAlias("test", "alias1").addAlias("test_x", "alias1")).actionGet(); + admin().indices().aliases(indexAliasesRequest().removeAlias("test", "alias1").addAlias("alias1", "test_x")).actionGet(); Thread.sleep(300); logger.info("--> indexing against [alias1], should work against [test_x]"); @@ -412,6 +414,43 @@ public void testDeletingByQueryFilteringAliases() throws Exception { assertHits(searchResponse.getHits(), "4"); } + + + @Test + public void testDeleteAliases() throws Exception { + + logger.info("--> creating index [test1]"); + admin().indices().create(createIndexRequest("test1")).actionGet(); + + logger.info("--> creating index [test2]"); + admin().indices().create(createIndexRequest("test2")).actionGet(); + + ensureGreen(); + + logger.info("--> adding filtering aliases to index [test1]"); + admin().indices().prepareAliases().addAlias("test1", "aliasToTest1").execute().actionGet(); + admin().indices().prepareAliases().addAlias("test1", "aliasToTests").execute().actionGet(); + admin().indices().prepareAliases().addAlias("test1", "foos", termFilter("name", "foo")).execute().actionGet(); + admin().indices().prepareAliases().addAlias("test1", "bars", termFilter("name", "bar")).execute().actionGet(); + admin().indices().prepareAliases().addAlias("test1", "tests", termFilter("name", "test")).execute().actionGet(); + + logger.info("--> adding filtering aliases to index [test2]"); + admin().indices().prepareAliases().addAlias("test2", "aliasToTest2").execute().actionGet(); + admin().indices().prepareAliases().addAlias("test2", "aliasToTests").execute().actionGet(); + admin().indices().prepareAliases().addAlias("test2", "foos", termFilter("name", "foo")).execute().actionGet(); + admin().indices().prepareAliases().addAlias("test2", "tests", termFilter("name", "test")).execute().actionGet(); + + String[] indices = {"test1", "test2"}; + String[] aliases = {"aliasToTest1", "foos", "bars", "tests", "aliasToTest2", "aliasToTests"}; + + admin().indices().prepareAliases().removeAlias(indices, aliases).execute().actionGet(); + + AliasesExistResponse response = admin().indices().prepareAliasesExist(aliases).execute().actionGet(); + assertThat(response.exists(), equalTo(false)); + + } + + @Test public void testWaitForAliasCreationMultipleShards() throws Exception { @@ -512,10 +551,16 @@ public void testSameAlias() throws Exception { assertThat(admin().indices().prepareAliases().removeAlias("test", "alias1").setTimeout(timeout).execute().actionGet().isAcknowledged(), equalTo(true)); assertThat(stopWatch.stop().lastTaskTime().millis(), lessThan(timeout.millis())); - logger.info("--> deleting alias1 one more time"); - stopWatch.start(); - assertThat(admin().indices().prepareAliases().removeAlias("test", "alias1").setTimeout(timeout).execute().actionGet().isAcknowledged(), equalTo(true)); - assertThat(stopWatch.stop().lastTaskTime().millis(), lessThan(timeout.millis())); + + } + + @Test(expected = AliasesMissingException.class) + public void testIndicesRemoveNonExistingAliasResponds404() throws Exception { + logger.info("--> creating index [test]"); + createIndex("test"); + ensureGreen(); + logger.info("--> deleting alias1 which does not exist"); + assertThat(admin().indices().prepareAliases().removeAlias("test", "alias1").execute().actionGet().isAcknowledged(), equalTo(true)); } @Test @@ -729,7 +774,7 @@ public void testIndicesGetAliases() throws Exception { assertThat(existsResponse.exists(), equalTo(false)); } - @Test(expected = ActionRequestValidationException.class) + @Test(expected = IndexMissingException.class) public void testAddAliasNullIndex() { admin().indices().prepareAliases().addAliasAction(AliasAction.newAddAliasAction(null, "alias1")) @@ -765,7 +810,7 @@ public void testAddAliasNullAliasNullIndex() { assertTrue("Should throw " + ActionRequestValidationException.class.getSimpleName(), false); } catch (ActionRequestValidationException e) { assertThat(e.validationErrors(), notNullValue()); - assertThat(e.validationErrors().size(), equalTo(2)); + assertThat(e.validationErrors().size(), equalTo(1)); } } @@ -808,7 +853,7 @@ public void tesRemoveAliasEmptyAlias() { @Test public void testRemoveAliasNullAliasNullIndex() { try { - admin().indices().prepareAliases().addAliasAction(AliasAction.newAddAliasAction(null, null)) + admin().indices().prepareAliases().addAliasAction(AliasAction.newRemoveAliasAction(null, null)) .execute().actionGet(); assertTrue("Should throw " + ActionRequestValidationException.class.getSimpleName(), false); } catch (ActionRequestValidationException e) { diff --git a/src/test/java/org/elasticsearch/cluster/ack/AckTests.java b/src/test/java/org/elasticsearch/cluster/ack/AckTests.java index c69964fe8cc56..15bb212977c6f 100644 --- a/src/test/java/org/elasticsearch/cluster/ack/AckTests.java +++ b/src/test/java/org/elasticsearch/cluster/ack/AckTests.java @@ -123,7 +123,7 @@ public void testDeleteWarmerAcknowledgement() { assertAcked(client().admin().indices().preparePutWarmer("custom_warmer") .setSearchRequest(client().prepareSearch("test").setTypes("test").setQuery(QueryBuilders.matchAllQuery()))); - assertAcked(client().admin().indices().prepareDeleteWarmer().setIndices("test").setName("custom_warmer")); + assertAcked(client().admin().indices().prepareDeleteWarmer().setIndices("test").setNames("custom_warmer")); for (Client client : clients()) { GetWarmersResponse getWarmersResponse = client.admin().indices().prepareGetWarmers().setLocal(true).get(); diff --git a/src/test/java/org/elasticsearch/cluster/metadata/MetaDataTests.java b/src/test/java/org/elasticsearch/cluster/metadata/MetaDataTests.java index 12b6a4fb3236d..8ff8c26a13fa4 100644 --- a/src/test/java/org/elasticsearch/cluster/metadata/MetaDataTests.java +++ b/src/test/java/org/elasticsearch/cluster/metadata/MetaDataTests.java @@ -369,37 +369,37 @@ public void testIsAllIndices_wildcard() throws Exception { @Test public void testIsExplicitAllIndices_null() throws Exception { MetaData metaData = MetaData.builder().build(); - assertThat(metaData.isExplicitAllIndices(null), equalTo(false)); + assertThat(metaData.isExplicitAllPattern(null), equalTo(false)); } @Test public void testIsExplicitAllIndices_empty() throws Exception { MetaData metaData = MetaData.builder().build(); - assertThat(metaData.isExplicitAllIndices(new String[0]), equalTo(false)); + assertThat(metaData.isExplicitAllPattern(new String[0]), equalTo(false)); } @Test public void testIsExplicitAllIndices_explicitAll() throws Exception { MetaData metaData = MetaData.builder().build(); - assertThat(metaData.isExplicitAllIndices(new String[]{"_all"}), equalTo(true)); + assertThat(metaData.isExplicitAllPattern(new String[]{"_all"}), equalTo(true)); } @Test public void testIsExplicitAllIndices_explicitAllPlusOther() throws Exception { MetaData metaData = MetaData.builder().build(); - assertThat(metaData.isExplicitAllIndices(new String[]{"_all", "other"}), equalTo(false)); + assertThat(metaData.isExplicitAllPattern(new String[]{"_all", "other"}), equalTo(false)); } @Test public void testIsExplicitAllIndices_normalIndexes() throws Exception { MetaData metaData = MetaData.builder().build(); - assertThat(metaData.isExplicitAllIndices(new String[]{"index1", "index2", "index3"}), equalTo(false)); + assertThat(metaData.isExplicitAllPattern(new String[]{"index1", "index2", "index3"}), equalTo(false)); } @Test public void testIsExplicitAllIndices_wildcard() throws Exception { MetaData metaData = MetaData.builder().build(); - assertThat(metaData.isExplicitAllIndices(new String[]{"*"}), equalTo(false)); + assertThat(metaData.isExplicitAllPattern(new String[]{"*"}), equalTo(false)); } @Test diff --git a/src/test/java/org/elasticsearch/indices/IndicesOptionsTests.java b/src/test/java/org/elasticsearch/indices/IndicesOptionsTests.java index e0e19a627830e..ff1b391657a90 100644 --- a/src/test/java/org/elasticsearch/indices/IndicesOptionsTests.java +++ b/src/test/java/org/elasticsearch/indices/IndicesOptionsTests.java @@ -55,6 +55,7 @@ import org.elasticsearch.common.Strings; import org.elasticsearch.common.bytes.BytesArray; import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.index.query.QueryBuilders; import org.elasticsearch.search.suggest.SuggestBuilder; import org.elasticsearch.search.warmer.IndexWarmersMetaData; import org.elasticsearch.test.ElasticsearchIntegrationTest; @@ -537,6 +538,103 @@ public void testDeleteMapping_wildcard() throws Exception { assertThat(client().admin().indices().prepareTypesExists("barbaz").setTypes("type1").get().isExists(), equalTo(false)); } + + @Test + public void testPutWarmer() throws Exception { + assertAcked(prepareCreate("foobar")); + ensureYellow(); + verify(client().admin().indices().preparePutWarmer("warmer1").setSearchRequest(client().prepareSearch().setIndices("foobar").setQuery(QueryBuilders.matchAllQuery())), false); + assertThat(client().admin().indices().prepareGetWarmers("foobar").setWarmers("warmer1").get().getWarmers().size(), equalTo(1)); + + } + + @Test + public void testPutWarmer_wildcard() throws Exception { + + assertAcked(prepareCreate("foo")); + assertAcked(prepareCreate("foobar")); + assertAcked(prepareCreate("bar")); + assertAcked(prepareCreate("barbaz")); + ensureYellow(); + + verify(client().admin().indices().preparePutWarmer("warmer1").setSearchRequest(client().prepareSearch().setIndices("foo*").setQuery(QueryBuilders.matchAllQuery())), false); + + assertThat(client().admin().indices().prepareGetWarmers("foo").setWarmers("warmer1").get().getWarmers().size(), equalTo(1)); + assertThat(client().admin().indices().prepareGetWarmers("foobar").setWarmers("warmer1").get().getWarmers().size(), equalTo(1)); + assertThat(client().admin().indices().prepareGetWarmers("bar").setWarmers("warmer1").get().getWarmers().size(), equalTo(0)); + assertThat(client().admin().indices().prepareGetWarmers("barbaz").setWarmers("warmer1").get().getWarmers().size(), equalTo(0)); + + verify(client().admin().indices().preparePutWarmer("warmer2").setSearchRequest(client().prepareSearch().setIndices().setQuery(QueryBuilders.matchAllQuery())), false); + + assertThat(client().admin().indices().prepareGetWarmers("foo").setWarmers("warmer2").get().getWarmers().size(), equalTo(1)); + assertThat(client().admin().indices().prepareGetWarmers("foobar").setWarmers("warmer2").get().getWarmers().size(), equalTo(1)); + assertThat(client().admin().indices().prepareGetWarmers("bar").setWarmers("warmer2").get().getWarmers().size(), equalTo(1)); + assertThat(client().admin().indices().prepareGetWarmers("barbaz").setWarmers("warmer2").get().getWarmers().size(), equalTo(1)); + + } + + @Test + public void testPutAlias() throws Exception { + assertAcked(prepareCreate("foobar")); + ensureYellow(); + verify(client().admin().indices().prepareAliases().addAlias("foobar", "foobar_alias"), false); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("foobar").get().exists(), equalTo(true)); + + } + + @Test + public void testPutAlias_wildcard() throws Exception { + + assertAcked(prepareCreate("foo")); + assertAcked(prepareCreate("foobar")); + assertAcked(prepareCreate("bar")); + assertAcked(prepareCreate("barbaz")); + ensureYellow(); + + verify(client().admin().indices().prepareAliases().addAlias("foo*", "foobar_alias"), false); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("foo").get().exists(), equalTo(true)); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("foobar").get().exists(), equalTo(true)); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("bar").get().exists(), equalTo(false)); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("barbaz").get().exists(), equalTo(false)); + + verify(client().admin().indices().prepareAliases().addAlias("*", "foobar_alias"), false); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("foo").get().exists(), equalTo(true)); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("foobar").get().exists(), equalTo(true)); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("bar").get().exists(), equalTo(true)); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("barbaz").get().exists(), equalTo(true)); + + } + @Test + public void testDeleteMapping_typeWildcard() throws Exception { + verify(client().admin().indices().prepareDeleteMapping("_all").setType("type1"), true); + + assertAcked(prepareCreate("foo").addMapping("type1", "field", "type=string")); + assertAcked(prepareCreate("foobar").addMapping("type2", "field", "type=string")); + assertAcked(prepareCreate("bar").addMapping("type3", "field", "type=string")); + assertAcked(prepareCreate("barbaz").addMapping("type4", "field", "type=string")); + + ensureYellow(); + + assertThat(client().admin().indices().prepareTypesExists("foo").setTypes("type1").get().isExists(), equalTo(true)); + assertThat(client().admin().indices().prepareTypesExists("foobar").setTypes("type2").get().isExists(), equalTo(true)); + assertThat(client().admin().indices().prepareTypesExists("bar").setTypes("type3").get().isExists(), equalTo(true)); + assertThat(client().admin().indices().prepareTypesExists("barbaz").setTypes("type4").get().isExists(), equalTo(true)); + + verify(client().admin().indices().prepareDeleteMapping("foo*").setType("type*"), false); + assertThat(client().admin().indices().prepareTypesExists("foo").setTypes("type1").get().isExists(), equalTo(false)); + assertThat(client().admin().indices().prepareTypesExists("foobar").setTypes("type2").get().isExists(), equalTo(false)); + assertThat(client().admin().indices().prepareTypesExists("bar").setTypes("type3").get().isExists(), equalTo(true)); + assertThat(client().admin().indices().prepareTypesExists("barbaz").setTypes("type4").get().isExists(), equalTo(true)); + + assertAcked(client().admin().indices().prepareDelete("foo*")); + + verify(client().admin().indices().prepareDeleteMapping("foo*").setType("type1"), true); + + verify(client().admin().indices().prepareDeleteMapping("_all").setType("type3","type4"), false); + assertThat(client().admin().indices().prepareTypesExists("bar").setTypes("type3").get().isExists(), equalTo(false)); + assertThat(client().admin().indices().prepareTypesExists("barbaz").setTypes("type4").get().isExists(), equalTo(false)); + } + @Test public void testDeleteWarmer() throws Exception { IndexWarmersMetaData.Entry entry = new IndexWarmersMetaData.Entry( @@ -545,15 +643,15 @@ public void testDeleteWarmer() throws Exception { assertAcked(prepareCreate("foobar").addCustom(new IndexWarmersMetaData(entry))); ensureYellow(); - verify(client().admin().indices().prepareDeleteWarmer().setIndices("foo").setName("test1"), true); + verify(client().admin().indices().prepareDeleteWarmer().setIndices("foo").setNames("test1"), true); assertThat(client().admin().indices().prepareGetWarmers("foobar").setWarmers("test1").get().getWarmers().size(), equalTo(1)); - verify(client().admin().indices().prepareDeleteWarmer().setIndices("foobar").setName("test1"), false); + verify(client().admin().indices().prepareDeleteWarmer().setIndices("foobar").setNames("test1"), false); assertThat(client().admin().indices().prepareGetWarmers("foobar").setWarmers("test1").get().getWarmers().size(), equalTo(0)); } @Test public void testDeleteWarmer_wildcard() throws Exception { - verify(client().admin().indices().prepareDeleteWarmer().setIndices("_all").setName("test1"), true); + verify(client().admin().indices().prepareDeleteWarmer().setIndices("_all").setNames("test1"), true); IndexWarmersMetaData.Entry entry = new IndexWarmersMetaData.Entry( "test1", new String[]{"type1"}, new BytesArray("{\"query\" : { \"match_all\" : {}}}") @@ -564,7 +662,7 @@ public void testDeleteWarmer_wildcard() throws Exception { assertAcked(prepareCreate("barbaz").addCustom(new IndexWarmersMetaData(entry))); ensureYellow(); - verify(client().admin().indices().prepareDeleteWarmer().setIndices("foo*").setName("test1"), false); + verify(client().admin().indices().prepareDeleteWarmer().setIndices("foo*").setNames("test1"), false); assertThat(client().admin().indices().prepareGetWarmers("foo").setWarmers("test1").get().getWarmers().size(), equalTo(0)); assertThat(client().admin().indices().prepareGetWarmers("foobar").setWarmers("test1").get().getWarmers().size(), equalTo(0)); assertThat(client().admin().indices().prepareGetWarmers("bar").setWarmers("test1").get().getWarmers().size(), equalTo(1)); @@ -572,9 +670,9 @@ public void testDeleteWarmer_wildcard() throws Exception { assertAcked(client().admin().indices().prepareDelete("foo*")); - verify(client().admin().indices().prepareDeleteWarmer().setIndices("foo*").setName("test1"), true); + verify(client().admin().indices().prepareDeleteWarmer().setIndices("foo*").setNames("test1"), true); - verify(client().admin().indices().prepareDeleteWarmer().setIndices("_all").setName("test1"), false); + verify(client().admin().indices().prepareDeleteWarmer().setIndices("_all").setNames("test1"), false); assertThat(client().admin().indices().prepareGetWarmers("bar").setWarmers("test1").get().getWarmers().size(), equalTo(0)); assertThat(client().admin().indices().prepareGetWarmers("barbaz").setWarmers("test1").get().getWarmers().size(), equalTo(0)); } @@ -622,12 +720,18 @@ public void testPutMapping() throws Exception { assertThat(client().admin().indices().prepareGetMappings("foobar").get().mappings().get("foobar").get("type2"), notNullValue()); assertThat(client().admin().indices().prepareGetMappings("bar").get().mappings().get("bar").get("type2"), notNullValue()); assertThat(client().admin().indices().prepareGetMappings("barbaz").get().mappings().get("barbaz").get("type2"), notNullValue()); + verify(client().admin().indices().preparePutMapping().setType("type3").setSource("field", "type=string"), false); + assertThat(client().admin().indices().prepareGetMappings("foo").get().mappings().get("foo").get("type3"), notNullValue()); + assertThat(client().admin().indices().prepareGetMappings("foobar").get().mappings().get("foobar").get("type3"), notNullValue()); + assertThat(client().admin().indices().prepareGetMappings("bar").get().mappings().get("bar").get("type3"), notNullValue()); + assertThat(client().admin().indices().prepareGetMappings("barbaz").get().mappings().get("barbaz").get("type3"), notNullValue()); + verify(client().admin().indices().preparePutMapping("c*").setType("type1").setSource("field", "type=string"), true); assertAcked(client().admin().indices().prepareClose("barbaz").get()); - verify(client().admin().indices().preparePutMapping("barbaz").setType("type3").setSource("field", "type=string"), false); - assertThat(client().admin().indices().prepareGetMappings("barbaz").get().mappings().get("barbaz").get("type3"), notNullValue()); + verify(client().admin().indices().preparePutMapping("barbaz").setType("type4").setSource("field", "type=string"), false); + assertThat(client().admin().indices().prepareGetMappings("barbaz").get().mappings().get("barbaz").get("type4"), notNullValue()); } @Test diff --git a/src/test/java/org/elasticsearch/indices/mapping/SimpleDeleteMappingTests.java b/src/test/java/org/elasticsearch/indices/mapping/SimpleDeleteMappingTests.java index a35f5139f99c3..3499b88238d76 100644 --- a/src/test/java/org/elasticsearch/indices/mapping/SimpleDeleteMappingTests.java +++ b/src/test/java/org/elasticsearch/indices/mapping/SimpleDeleteMappingTests.java @@ -19,14 +19,17 @@ package org.elasticsearch.indices.mapping; +import org.elasticsearch.action.ActionRequestValidationException; import org.elasticsearch.action.admin.indices.mapping.get.GetMappingsResponse; import org.elasticsearch.action.count.CountResponse; import org.elasticsearch.cluster.ClusterState; import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.elasticsearch.test.hamcrest.ElasticsearchAssertions; import org.junit.Test; import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked; import static org.hamcrest.Matchers.*; /** @@ -43,7 +46,7 @@ public void simpleDeleteMapping() throws Exception { } ensureGreen(); - client().admin().indices().prepareRefresh().execute().actionGet(); + refresh(); for (int i = 0; i < 10; i++) { CountResponse countResponse = client().prepareCount().setQuery(matchAllQuery()).execute().actionGet(); @@ -51,15 +54,13 @@ public void simpleDeleteMapping() throws Exception { } ClusterState clusterState = client().admin().cluster().prepareState().execute().actionGet().getState(); - for (int i = 0; i < 10 && !clusterState.metaData().index("test").mappings().containsKey("type1"); i++, Thread.sleep(100)) ; assertThat(clusterState.metaData().index("test").mappings().containsKey("type1"), equalTo(true)); GetMappingsResponse mappingsResponse = client().admin().indices().prepareGetMappings("test").setTypes("type1").execute().actionGet(); assertThat(mappingsResponse.getMappings().get("test").get("type1"), notNullValue()); - client().admin().indices().prepareDeleteMapping().setType("type1").execute().actionGet(); - Thread.sleep(500); // for now, we don't have ack logic, so just wait + ElasticsearchAssertions.assertAcked(client().admin().indices().prepareDeleteMapping().setIndices("test").setType("type1")); for (int i = 0; i < 10; i++) { CountResponse countResponse = client().prepareCount().setQuery(matchAllQuery()).execute().actionGet(); @@ -71,4 +72,35 @@ public void simpleDeleteMapping() throws Exception { mappingsResponse = client().admin().indices().prepareGetMappings("test").setTypes("type1").execute().actionGet(); assertThat(mappingsResponse.getMappings().get("test"), nullValue()); } + + + @Test + public void deleteMappingAllowNoBlankIndexAndNoEmptyStrings() throws Exception { + assertAcked(client().admin().indices().prepareCreate("index1").addMapping("1", "field1", "type=string").get()); + assertAcked(client().admin().indices().prepareCreate("1index").addMapping("1", "field1", "type=string").get()); + + // Should succeed, since no wildcards + client().admin().indices().prepareDeleteMapping("1index").setType("1").get(); + try { + client().admin().indices().prepareDeleteMapping("_all").get(); + fail(); + } catch (ActionRequestValidationException e) {} + + try { + client().admin().indices().prepareDeleteMapping("_all").setType("").get(); + fail(); + } catch (ActionRequestValidationException e) {} + + try { + client().admin().indices().prepareDeleteMapping().setType("1").get(); + fail(); + } catch (ActionRequestValidationException e) {} + + try { + client().admin().indices().prepareDeleteMapping("").setType("1").get(); + fail(); + } catch (ActionRequestValidationException e) {} + + } + } diff --git a/src/test/java/org/elasticsearch/indices/warmer/LocalGatewayIndicesWarmerTests.java b/src/test/java/org/elasticsearch/indices/warmer/LocalGatewayIndicesWarmerTests.java index 080d0f9bff42a..5a056c5e0ee46 100644 --- a/src/test/java/org/elasticsearch/indices/warmer/LocalGatewayIndicesWarmerTests.java +++ b/src/test/java/org/elasticsearch/indices/warmer/LocalGatewayIndicesWarmerTests.java @@ -127,7 +127,7 @@ public Settings onNodeStopped(String nodeName) throws Exception { logger.info("--> delete warmer warmer_1"); - DeleteWarmerResponse deleteWarmerResponse = client().admin().indices().prepareDeleteWarmer().setIndices("test").setName("warmer_1").execute().actionGet(); + DeleteWarmerResponse deleteWarmerResponse = client().admin().indices().prepareDeleteWarmer().setIndices("test").setNames("warmer_1").execute().actionGet(); assertThat(deleteWarmerResponse.isAcknowledged(), equalTo(true)); logger.info("--> verify warmers (delete) are registered in cluster state"); diff --git a/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java b/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java index 5e8d429767a4b..e1bcac1d2a9f8 100644 --- a/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java +++ b/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java @@ -175,10 +175,10 @@ public void deleteNonExistentIndexWarmerTest() { createIndex("test"); try { - client().admin().indices().prepareDeleteWarmer().setIndices("test").setName("foo").execute().actionGet(1000); + client().admin().indices().prepareDeleteWarmer().setIndices("test").setNames("foo").execute().actionGet(1000); assert false : "warmer foo should not exist"; } catch (IndexWarmerMissingException ex) { - assertThat(ex.name(), equalTo("foo")); + assertThat(ex.names()[0], equalTo("foo")); } } @@ -199,7 +199,7 @@ public void deleteIndexWarmerTest() { assertThat(entry.value.size(), equalTo(1)); assertThat(entry.value.iterator().next().name(), equalTo("custom_warmer")); - DeleteWarmerResponse deleteWarmerResponse = client().admin().indices().prepareDeleteWarmer().setIndices("test").setName("custom_warmer").get(); + DeleteWarmerResponse deleteWarmerResponse = client().admin().indices().prepareDeleteWarmer().setIndices("test").setNames("custom_warmer").get(); assertThat(deleteWarmerResponse.isAcknowledged(), equalTo(true)); getWarmersResponse = client().admin().indices().prepareGetWarmers("test").get(); diff --git a/src/test/java/org/elasticsearch/mlt/MoreLikeThisActionTests.java b/src/test/java/org/elasticsearch/mlt/MoreLikeThisActionTests.java index 5b0489851a172..40ad9ee42d7a0 100644 --- a/src/test/java/org/elasticsearch/mlt/MoreLikeThisActionTests.java +++ b/src/test/java/org/elasticsearch/mlt/MoreLikeThisActionTests.java @@ -93,8 +93,8 @@ public void testMoreLikeThisWithAliases() throws Exception { .startObject("text").field("type", "string").endObject() .endObject().endObject().endObject())); logger.info("Creating aliases alias release"); - client().admin().indices().aliases(indexAliasesRequest().addAlias("test", "release", termFilter("text", "release"))).actionGet(); - client().admin().indices().aliases(indexAliasesRequest().addAlias("test", "beta", termFilter("text", "beta"))).actionGet(); + client().admin().indices().aliases(indexAliasesRequest().addAlias("release", termFilter("text", "release"), "test")).actionGet(); + client().admin().indices().aliases(indexAliasesRequest().addAlias("beta", termFilter("text", "beta"), "test")).actionGet(); logger.info("Running Cluster Health"); assertThat(ensureGreen(), equalTo(ClusterHealthStatus.GREEN)); diff --git a/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java b/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java index cd0ab6efec846..d752bdea3e2df 100644 --- a/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java +++ b/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java @@ -149,7 +149,8 @@ public void testDestructiveOperations() throws Exception { .put(DestructiveOperations.REQUIRES_NAME, true) .build(); assertAcked(client().admin().cluster().prepareUpdateSettings().setTransientSettings(settings)); - + + assertAcked(client().admin().indices().prepareCreate("index1").addMapping("1", "field1", "type=string").get()); assertAcked(client().admin().indices().prepareCreate("1index").addMapping("1", "field1", "type=string").get()); @@ -158,20 +159,19 @@ public void testDestructiveOperations() throws Exception { try { client().admin().indices().prepareDeleteMapping("_all").setType("1").get(); fail(); - } catch (ElasticsearchIllegalArgumentException e) {} - + } catch (ElasticsearchIllegalArgumentException e) { + } try { - client().admin().indices().prepareDeleteMapping().setType("1").get(); + client().admin().indices().prepareDeleteMapping().setIndices("*").setType("1").get(); fail(); - } catch (ElasticsearchIllegalArgumentException e) {} + } catch (ElasticsearchIllegalArgumentException e) { + } - settings = ImmutableSettings.builder() - .put(DestructiveOperations.REQUIRES_NAME, false) - .build(); + settings = ImmutableSettings.builder().put(DestructiveOperations.REQUIRES_NAME, false).build(); assertAcked(client().admin().cluster().prepareUpdateSettings().setTransientSettings(settings)); assertAcked(client().admin().indices().preparePutMapping("1index").setType("1").setSource("field1", "type=string")); - assertAcked(client().admin().indices().prepareDeleteMapping().setType("1")); + assertAcked(client().admin().indices().prepareDeleteMapping().setIndices("*").setType("1")); assertAcked(client().admin().indices().preparePutMapping("1index").setType("1").setSource("field1", "type=string")); assertAcked(client().admin().indices().prepareDeleteMapping("_all").setType("1")); } From fb3ea1feb0951c0f18a9227826ce99327a443f17 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Tue, 14 Jan 2014 20:38:03 +0100 Subject: [PATCH 14/34] Use assertThat instead of plain asserts in MemoryCircuitBreakerTests --- .../common/breaker/MemoryCircuitBreakerTests.java | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/test/java/org/elasticsearch/common/breaker/MemoryCircuitBreakerTests.java b/src/test/java/org/elasticsearch/common/breaker/MemoryCircuitBreakerTests.java index 7a07d0b03b6a0..5565cd44c0d39 100644 --- a/src/test/java/org/elasticsearch/common/breaker/MemoryCircuitBreakerTests.java +++ b/src/test/java/org/elasticsearch/common/breaker/MemoryCircuitBreakerTests.java @@ -54,7 +54,7 @@ public void run() { if (tripped.get()) { assertThat("tripped too many times", true, equalTo(false)); } else { - assert tripped.compareAndSet(false, true); + assertThat(tripped.compareAndSet(false, true), equalTo(true)); } } catch (Throwable e2) { lastException.set(e2); @@ -86,7 +86,6 @@ public void testConstantFactor() throws Exception { breaker.addEstimateBytesAndMaybeBreak(3); fail("should never reach this"); } catch (CircuitBreakingException cbe) { - assert true; } // shouldn't throw an exception @@ -102,7 +101,6 @@ public void testConstantFactor() throws Exception { breaker.addEstimateBytesAndMaybeBreak(0); fail("should never reach this"); } catch (CircuitBreakingException cbe) { - assert true; } } } From ba7699a38b1bdaaccef20fb28888dd5fe5861ace Mon Sep 17 00:00:00 2001 From: Igor Motov Date: Tue, 14 Jan 2014 16:20:46 -0500 Subject: [PATCH 15/34] Add documentation for index.routing.allocation.*._name and index.routing.allocation.*._id options --- docs/reference/modules/cluster.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/modules/cluster.asciidoc b/docs/reference/modules/cluster.asciidoc index f089e2dad903f..9d0eefe3076c1 100644 --- a/docs/reference/modules/cluster.asciidoc +++ b/docs/reference/modules/cluster.asciidoc @@ -206,7 +206,8 @@ The `include`, `exclude` and `require` values can have generic simple matching wildcards, for example, `value1*`. A special attribute name called `_ip` can be used to match on node ip values. In addition `_host` attribute can be used to match on either the node's hostname or its ip -address. +address. Similarly `_name` and `_id` attributes can be used to match on +node name and node id accordingly. Obviously a node can have several attributes associated with it, and both the attribute name and value are controlled in the setting. For From 158483554dc2c11a0168ffbb728bece2619fec21 Mon Sep 17 00:00:00 2001 From: Andrew Raines Date: Mon, 13 Jan 2014 19:35:13 -0600 Subject: [PATCH 16/34] Shorten epoch to second precision. Closes #4696. --- .../org/elasticsearch/rest/action/cat/RestCountAction.java | 4 ++-- .../org/elasticsearch/rest/action/cat/RestHealthAction.java | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestCountAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestCountAction.java index 76181761e9eb8..7cb754bf57914 100644 --- a/src/main/java/org/elasticsearch/rest/action/cat/RestCountAction.java +++ b/src/main/java/org/elasticsearch/rest/action/cat/RestCountAction.java @@ -98,7 +98,7 @@ public void onFailure(Throwable t) { Table getTableWithHeader(final RestRequest request) { Table table = new Table(); table.startHeaders(); - table.addCell("time(ms)", "desc:time, in milliseconds since epoch UTC, that the count was executed"); + table.addCell("epoch", "desc:seconds since 1970-01-01 00:00:00, that the count was executed"); table.addCell("timestamp", "desc:time that the count was executed"); table.addCell("count", "desc:the document count"); table.endHeaders(); @@ -109,7 +109,7 @@ Table getTableWithHeader(final RestRequest request) { private Table buildTable(RestRequest request, CountResponse response) { Table table = getTableWithHeader(request); - long time = System.currentTimeMillis(); + long time = System.currentTimeMillis() / 1000; table.startRow(); table.addCell(time); table.addCell(dateFormat.print(time)); diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestHealthAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestHealthAction.java index 83a68f486c140..d3f0a96bd6867 100644 --- a/src/main/java/org/elasticsearch/rest/action/cat/RestHealthAction.java +++ b/src/main/java/org/elasticsearch/rest/action/cat/RestHealthAction.java @@ -81,7 +81,7 @@ public void onFailure(Throwable e) { Table getTableWithHeader(final RestRequest request) { Table t = new Table(); t.startHeaders(); - t.addCell("time(ms)", "desc:time, in milliseconds since epoch UTC, that the count was executed"); + t.addCell("epoch", "desc:seconds since 1970-01-01 00:00:00, that the count was executed"); t.addCell("timestamp", "desc:time that the count was executed"); t.addCell("cluster", "desc:cluster name"); t.addCell("status", "desc:health status"); @@ -100,7 +100,7 @@ Table getTableWithHeader(final RestRequest request) { private DateTimeFormatter dateFormat = DateTimeFormat.forPattern("HH:mm:ss"); private Table buildTable(final ClusterHealthResponse health, final RestRequest request) { - long time = System.currentTimeMillis(); + long time = System.currentTimeMillis() / 1000; Table t = getTableWithHeader(request); t.startRow(); t.addCell(time); From b35ca1aa751efc551f2e33b9d21be27b5e8dd324 Mon Sep 17 00:00:00 2001 From: Andrew Raines Date: Mon, 13 Jan 2014 21:00:56 -0600 Subject: [PATCH 17/34] Fix disk percent used calc in cat/allocation Closes #4670. --- .../rest/action/cat/RestAllocationAction.java | 27 +++++++------------ 1 file changed, 10 insertions(+), 17 deletions(-) diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestAllocationAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestAllocationAction.java index 29c88256d81ab..bd3f88d3891eb 100644 --- a/src/main/java/org/elasticsearch/rest/action/cat/RestAllocationAction.java +++ b/src/main/java/org/elasticsearch/rest/action/cat/RestAllocationAction.java @@ -117,7 +117,8 @@ Table getTableWithHeader(final RestRequest request) { table.addCell("shards", "text-align:right;desc:number of shards on node"); table.addCell("diskUsed", "text-align:right;desc:disk used (total, not just ES)"); table.addCell("diskAvail", "text-align:right;desc:disk available"); - table.addCell("diskRatio", "text-align:right;desc:percent disk used"); + table.addCell("diskTotal", "text-align:right;desc:total capacity of all volumes"); + table.addCell("diskPercent", "text-align:right;desc:percent disk used"); table.addCell("host", "desc:host of node"); table.addCell("ip", "desc:ip of node"); table.addCell("node", "desc:name of node"); @@ -143,34 +144,25 @@ private Table buildTable(RestRequest request, final ClusterStateResponse state, for (NodeStats nodeStats : stats.getNodes()) { DiscoveryNode node = nodeStats.getNode(); - long used = -1; - long avail = -1; - - Iterator diskIter = nodeStats.getFs().iterator(); - while (diskIter.hasNext()) { - FsStats.Info disk = diskIter.next(); - used += disk.getTotal().bytes() - disk.getAvailable().bytes(); - avail += disk.getAvailable().bytes(); - } - - String nodeId = node.id(); - int shardCount = -1; - if (allocs.containsKey(nodeId)) { + if (allocs.containsKey(node.id())) { shardCount = allocs.lget(); } - float ratio = -1; + long used = nodeStats.getFs().getTotal().getTotal().bytes() - nodeStats.getFs().getTotal().getAvailable().bytes(); + long avail = nodeStats.getFs().getTotal().getAvailable().bytes(); + short diskPercent = -1; if (used >= 0 && avail > 0) { - ratio = used / (float) avail; + diskPercent = (short) (used * 100 / (used + avail)); } table.startRow(); table.addCell(shardCount < 0 ? null : shardCount); table.addCell(used < 0 ? null : new ByteSizeValue(used)); table.addCell(avail < 0 ? null : new ByteSizeValue(avail)); - table.addCell(ratio < 0 ? null : String.format(Locale.ROOT, "%.1f%%", ratio * 100.0)); + table.addCell(nodeStats.getFs().getTotal().getTotal()); + table.addCell(diskPercent < 0 ? null : diskPercent); table.addCell(node == null ? null : node.getHostName()); table.addCell(node == null ? null : node.getHostAddress()); table.addCell(node == null ? "UNASSIGNED" : node.name()); @@ -185,6 +177,7 @@ private Table buildTable(RestRequest request, final ClusterStateResponse state, table.addCell(null); table.addCell(null); table.addCell(null); + table.addCell(null); table.addCell("UNASSIGNED"); table.endRow(); } From 349a8be4fd9be31b2e264a660ec38b24fc1795b7 Mon Sep 17 00:00:00 2001 From: Alexander Reelsen Date: Tue, 14 Jan 2014 22:31:42 +0100 Subject: [PATCH 18/34] Consistent REST API changes for GETting data * Made GET mappings consistent, supporting * /{index}/_mappings/{type} * /{index}/_mapping/{type} * /_mapping/{type} * Added "mappings" in the JSON response to align it with other responses * Made GET warmers consistent, support /{index}/_warmers/{type} and /_warmer, /_warner/{name} as well as wildcards and _all notation * Made GET aliases consistent, support /{index}/_aliases/{name} and /_alias, /_aliases/{name} as well as wildcards and _all notation * Made GET settings consistent, added /{index}/_setting/{name}, /_settings/{name} as well as supportings wildcards in settings name * Returning empty JSON instead of a 404, if a specific warmer/ setting/alias/type is missing * Added a ton of spec tests for all of the above * Added a couple of more integration tests for several features Relates #4071 --- docs/reference/indices/get-settings.asciidoc | 6 +- rest-api-spec/api/indices.get_alias.json | 4 +- rest-api-spec/api/indices.get_aliases.json | 6 +- rest-api-spec/api/indices.get_mapping.json | 2 +- rest-api-spec/api/indices.get_settings.json | 6 +- rest-api-spec/api/indices.get_warmer.json | 4 +- .../test/indices.delete_alias/10_basic.yaml | 3 +- .../all_path_options.yaml | 16 +- .../test/indices.get_alias/10_basic.yaml | 211 +++++++++++++++++ .../test/indices.get_aliases/10_basic.yaml | 214 +++++++++++++++++ .../test/indices.get_mapping/10_basic.yaml | 179 +++++++++++++-- .../indices.get_mapping/20_missing_type.yaml | 6 +- .../test/indices.get_mapping/40_aliases.yaml | 26 +++ .../test/indices.get_settings/10_basic.yaml | 185 +++++++++++++-- .../test/indices.get_settings/20_aliases.yaml | 26 +++ .../test/indices.get_warmer/10_basic.yaml | 215 ++++++++++++++++++ .../test/indices.put_mapping/10_basic.yaml | 16 +- .../indices.put_mapping/all_path_options.yaml | 66 +++--- .../test/indices.put_warmer/10_basic.yaml | 116 +++++++++- .../test/indices.put_warmer/20_aliases.yaml | 30 +++ .../settings/get/GetSettingsRequest.java | 21 +- .../get/GetSettingsRequestBuilder.java | 4 +- .../get/TransportGetSettingsAction.java | 6 +- .../cluster/metadata/MetaData.java | 6 +- .../org/elasticsearch/common/Strings.java | 11 + .../org/elasticsearch/rest/RestRequest.java | 9 + .../alias/get/RestGetAliasesAction.java | 9 +- .../get/RestGetIndicesAliasesAction.java | 22 +- .../mapping/get/RestGetMappingAction.java | 17 +- .../settings/RestGetSettingsAction.java | 15 +- .../warmer/get/RestGetWarmerAction.java | 6 +- .../action/support/RestXContentBuilder.java | 4 + .../warmer/SimpleIndicesWarmerTests.java | 30 +++ 33 files changed, 1345 insertions(+), 152 deletions(-) create mode 100644 rest-api-spec/test/indices.get_alias/10_basic.yaml create mode 100644 rest-api-spec/test/indices.get_aliases/10_basic.yaml create mode 100644 rest-api-spec/test/indices.get_mapping/40_aliases.yaml create mode 100644 rest-api-spec/test/indices.get_settings/20_aliases.yaml create mode 100644 rest-api-spec/test/indices.get_warmer/10_basic.yaml create mode 100644 rest-api-spec/test/indices.put_warmer/20_aliases.yaml diff --git a/docs/reference/indices/get-settings.asciidoc b/docs/reference/indices/get-settings.asciidoc index 0a0e10df861a4..a5950c2ee4b1a 100644 --- a/docs/reference/indices/get-settings.asciidoc +++ b/docs/reference/indices/get-settings.asciidoc @@ -39,12 +39,12 @@ curl -XGET 'http://localhost:9200/my-index/_settings?prefix=index.' curl -XGET 'http://localhost:9200/_all/_settings?prefix=index.routing.allocation.' -curl -XGET 'http://localhost:9200/2013-*/_settings?prefix=index.merge.' +curl -XGET 'http://localhost:9200/2013-*/_settings?name=index.merge.*' -curl -XGET 'http://localhost:9200/2013-*/index.merge./_settings' +curl -XGET 'http://localhost:9200/2013-*/_settings/index.merge.*' -------------------------------------------------- The first example returns all index settings the start with `index.` in the index `my-index`, the second example gets all index settings that start with `index.routing.allocation.` for all indices, lastly the third example returns all index settings that start with `index.merge.` -in indices that start with `2013-`. \ No newline at end of file +in indices that start with `2013-`. diff --git a/rest-api-spec/api/indices.get_alias.json b/rest-api-spec/api/indices.get_alias.json index 61126fa963e5e..e7fe6e4a5007c 100644 --- a/rest-api-spec/api/indices.get_alias.json +++ b/rest-api-spec/api/indices.get_alias.json @@ -3,8 +3,8 @@ "documentation": "http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html", "methods": ["GET"], "url": { - "path": "/_alias/{name}", - "paths": ["/_alias/{name}", "/{index}/_alias/{name}", "/{index}/_alias"], + "path": "/_alias/", + "paths": [ "/_alias", "/_alias/{name}", "/{index}/_alias/{name}", "/{index}/_alias"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/api/indices.get_aliases.json b/rest-api-spec/api/indices.get_aliases.json index 508c94e3c0cc8..f3abd7a560260 100644 --- a/rest-api-spec/api/indices.get_aliases.json +++ b/rest-api-spec/api/indices.get_aliases.json @@ -4,11 +4,15 @@ "methods": ["GET"], "url": { "path": "/_aliases", - "paths": ["/_aliases", "/{index}/_aliases"], + "paths": ["/_aliases", "/{index}/_aliases", "/{index}/_aliases/{name}", "/_aliases/{name}" ], "parts": { "index": { "type" : "list", "description" : "A comma-separated list of index names to filter aliases" + }, + "name": { + "type" : "list", + "description" : "A comma-separated list of alias names to filter" } }, "params": { diff --git a/rest-api-spec/api/indices.get_mapping.json b/rest-api-spec/api/indices.get_mapping.json index 75cf384ff3290..2438deec57a31 100644 --- a/rest-api-spec/api/indices.get_mapping.json +++ b/rest-api-spec/api/indices.get_mapping.json @@ -4,7 +4,7 @@ "methods": ["GET"], "url": { "path": "/_mapping", - "paths": ["/_mapping", "/{index}/_mapping", "/{index}/{type}/_mapping"], + "paths": ["/_mapping", "/{index}/_mapping", "/_mapping/{type}", "/{index}/_mapping/{type}"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/api/indices.get_settings.json b/rest-api-spec/api/indices.get_settings.json index ae6b6e33c38c2..b02434db13ef7 100644 --- a/rest-api-spec/api/indices.get_settings.json +++ b/rest-api-spec/api/indices.get_settings.json @@ -4,15 +4,15 @@ "methods": ["GET"], "url": { "path": "/_settings", - "paths": ["/_settings", "/{index}/_settings", "/{index}/{prefix}/_settings"], + "paths": ["/_settings", "/{index}/_settings", "/{index}/_settings/{name}", "/_settings/{name}"], "parts": { "index": { "type" : "list", "description" : "A comma-separated list of index names; use `_all` or empty string to perform the operation on all indices" }, - "prefix": { + "name": { "type" : "string", - "description" : "The prefix all settings must have in order to be included" + "description" : "The name of the settings that should be included" } }, "params": { diff --git a/rest-api-spec/api/indices.get_warmer.json b/rest-api-spec/api/indices.get_warmer.json index ae17b957ef6c4..40a1e834519d5 100644 --- a/rest-api-spec/api/indices.get_warmer.json +++ b/rest-api-spec/api/indices.get_warmer.json @@ -3,8 +3,8 @@ "documentation": "http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-warmers.html", "methods": ["GET"], "url": { - "path": "/{index}/_warmer", - "paths": ["/{index}/_warmer", "/{index}/_warmer/{name}", "/{index}/{type}/_warmer/{name}"], + "path": "/_warmer", + "paths": [ "/_warmer", "/{index}/_warmer", "/{index}/_warmer/{name}", "/_warmer/{name}", "/{index}/{type}/_warmer/{name}"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/test/indices.delete_alias/10_basic.yaml b/rest-api-spec/test/indices.delete_alias/10_basic.yaml index 9f3535f7ef628..87f61efc112bc 100644 --- a/rest-api-spec/test/indices.delete_alias/10_basic.yaml +++ b/rest-api-spec/test/indices.delete_alias/10_basic.yaml @@ -25,7 +25,8 @@ name: testali - do: - catch: missing indices.get_alias: index: testind name: testali + + - match: { '': {}} diff --git a/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml b/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml index 2953a09643560..a0ed0fdd419f7 100644 --- a/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml +++ b/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml @@ -49,12 +49,7 @@ setup: name: test_warmer1 - do: - catch: missing - indices.get_warmer: { index: _all, name: 'test_warmer1' } - - - do: - indices.get_warmer: { index: _all, name: 'test_warmer2' } - + indices.get_warmer: {} - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} @@ -68,12 +63,7 @@ setup: name: test_warmer1 - do: - catch: missing - indices.get_warmer: { index: _all, name: 'test_warmer1' } - - - do: - indices.get_warmer: { index: _all, name: 'test_warmer2' } - + indices.get_warmer: {} - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} @@ -146,7 +136,7 @@ setup: --- "check delete with index list and _all warmers": - do: - indices.delete_warmer: + indices.delete_warmer: index: "test_index1,test_index2" name: _all diff --git a/rest-api-spec/test/indices.get_alias/10_basic.yaml b/rest-api-spec/test/indices.get_alias/10_basic.yaml new file mode 100644 index 0000000000000..21ff08c4f8060 --- /dev/null +++ b/rest-api-spec/test/indices.get_alias/10_basic.yaml @@ -0,0 +1,211 @@ +--- +setup: + + - do: + indices.create: + index: test_index + + - do: + indices.create: + index: test_index_2 + + - do: + indices.put_alias: + index: test_index + name: test_alias + + - do: + indices.put_alias: + index: test_index + name: test_blias + + - do: + indices.put_alias: + index: test_index_2 + name: test_alias + + - do: + indices.put_alias: + index: test_index_2 + name: test_blias + +--- +"Get all aliases via /_alias": + + - do: + indices.get_alias: {} + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_blias: {}} + + +--- +"Get all aliases via /{index}/_alias/": + + - do: + indices.get_alias: + index: test_index + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get specific alias via /{index}/_alias/{name}": + + - do: + indices.get_alias: + index: test_index + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_alias/_all": + + - do: + indices.get_alias: + index: test_index + name: _all + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_alias/*": + + - do: + indices.get_alias: + index: test_index + name: '*' + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_alias/prefix*": + + - do: + indices.get_alias: + index: test_index + name: 'test_a*' + + - match: {test_index.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_alias/name,name": + + - do: + indices.get_alias: + index: test_index + name: 'test_alias,test_blias' + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get aliases via /_alias/{name}": + + - do: + indices.get_alias: + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /_all/_alias/{name}": + + - do: + indices.get_alias: + index: _all + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /*/_alias/{name}": + + - do: + indices.get_alias: + index: '*' + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /pref*/_alias/{name}": + + - do: + indices.get_alias: + index: '*2' + name: test_alias + + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_alias + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /name,name/_alias/{name}": + + - do: + indices.get_alias: + index: test_index,test_index_2 + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + + +--- +"Non-existent alias on an existing index returns an empty body": + + - do: + indices.get_alias: + index: test_index + name: non-existent + + - match: { '': {}} + +--- +"Existent and non-existent alias returns just the existing": + + - do: + indices.get_alias: + index: test_index + name: test_alias,non-existent + + - match: {test_index.aliases.test_alias: {}} + - is_false: test_index.aliases.non-existent + +--- +"Getting alias on an non-existent index should return 404": + + - do: + catch: missing + indices.get_alias: + index: non-existent + name: foo + + + diff --git a/rest-api-spec/test/indices.get_aliases/10_basic.yaml b/rest-api-spec/test/indices.get_aliases/10_basic.yaml new file mode 100644 index 0000000000000..0e235f223d519 --- /dev/null +++ b/rest-api-spec/test/indices.get_aliases/10_basic.yaml @@ -0,0 +1,214 @@ +--- +setup: + + - do: + indices.create: + index: test_index + + - do: + indices.create: + index: test_index_2 + + - do: + indices.put_alias: + index: test_index + name: test_alias + + - do: + indices.put_alias: + index: test_index + name: test_blias + + - do: + indices.put_alias: + index: test_index_2 + name: test_alias + + - do: + indices.put_alias: + index: test_index_2 + name: test_blias + +--- +"Get all aliases via /_aliases": + + - do: + indices.get_aliases: {} + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_blias: {}} + + +--- +"Get all aliases via /{index}/_aliases/": + + - do: + indices.get_aliases: + index: test_index + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get specific alias via /{index}/_aliases/{name}": + + - do: + indices.get_aliases: + index: test_index + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_aliases/_all": + + - do: + indices.get_aliases: + index: test_index + name: _all + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_aliases/*": + + - do: + indices.get_aliases: + index: test_index + name: '*' + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_aliases/prefix*": + + - do: + indices.get_aliases: + index: test_index + name: 'test_a*' + + - match: {test_index.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_aliases/name,name": + + - do: + indices.get_aliases: + index: test_index + name: 'test_alias,test_blias' + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get aliases via /_aliases/{name}": + + - do: + indices.get_aliases: + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /_all/_aliases/{name}": + + - do: + indices.get_aliases: + index: _all + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /*/_aliases/{name}": + + - do: + indices.get_aliases: + index: '*' + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /pref*/_aliases/{name}": + + - do: + indices.get_aliases: + index: '*2' + name: test_alias + + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_alias + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /name,name/_aliases/{name}": + + - do: + indices.get_aliases: + index: test_index,test_index_2 + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + + +--- +"Non-existent alias on an existing index returns matching indcies": + + - do: + indices.get_aliases: + index: test_index + name: non-existent + + - match: { test_index.aliases: {}} + +--- +"Existent and non-existent alias returns just the existing": + + - do: + indices.get_aliases: + index: test_index + name: test_alias,non-existent + + - match: {test_index.aliases.test_alias: {}} + - is_false: test_index.aliases.non-existent + +--- +"Getting alias on an non-existent index should return 404": + + - skip: + version: 1 - 999 + reason: not implemented yet + - do: + catch: missing + indices.get_aliases: + index: non-existent + name: foo + + + diff --git a/rest-api-spec/test/indices.get_mapping/10_basic.yaml b/rest-api-spec/test/indices.get_mapping/10_basic.yaml index 40416435bb96c..4ed9b761bb7b2 100644 --- a/rest-api-spec/test/indices.get_mapping/10_basic.yaml +++ b/rest-api-spec/test/indices.get_mapping/10_basic.yaml @@ -2,31 +2,172 @@ setup: - do: indices.create: - index: test_index + index: test_1 body: mappings: - test_type: - properties: - text: - type: string - analyzer: whitespace + type_1: {} + type_2: {} + - do: + indices.create: + index: test_2 + body: + mappings: + type_2: {} + type_3: {} --- -"Get index mapping": - - do: - indices.get_mapping: - index: test_index +"Get /_mapping": + + - do: + indices.get_mapping: {} - - match: {test_index.test_type.properties.text.type: string} - - match: {test_index.test_type.properties.text.analyzer: whitespace} + - match: { test_1.mappings.type_1.properties: {}} + - match: { test_1.mappings.type_2.properties: {}} + - match: { test_2.mappings.type_2.properties: {}} + - match: { test_2.mappings.type_3.properties: {}} --- -"Get type mapping": +"Get /{index}/_mapping": - - do: - indices.get_mapping: - index: test_index - type: test_type + - do: + indices.get_mapping: + index: test_1 + + - match: { test_1.mappings.type_1.properties: {}} + - match: { test_1.mappings.type_2.properties: {}} + - is_false: test_2 + + +--- +"Get /{index}/_mapping/_all": + + - do: + indices.get_mapping: + index: test_1 + type: _all + + - match: { test_1.mappings.type_1.properties: {}} + - match: { test_1.mappings.type_2.properties: {}} + - is_false: test_2 + +--- +"Get /{index}/_mapping/*": + + - do: + indices.get_mapping: + index: test_1 + type: '*' + + - match: { test_1.mappings.type_1.properties: {}} + - match: { test_1.mappings.type_2.properties: {}} + - is_false: test_2 + +--- +"Get /{index}/_mapping/{type}": + + - do: + indices.get_mapping: + index: test_1 + type: type_1 + + - match: { test_1.mappings.type_1.properties: {}} + - is_false: test_1.mappings.type_2 + - is_false: test_2 + +--- +"Get /{index}/_mapping/{type,type}": + + - do: + indices.get_mapping: + index: test_1 + type: type_1,type_2 + + - match: { test_1.mappings.type_1.properties: {}} + - match: { test_1.mappings.type_2.properties: {}} + - is_false: test_2 + +--- +"Get /{index}/_mapping/{type*}": + + - do: + indices.get_mapping: + index: test_1 + type: '*2' + + - match: { test_1.mappings.type_2.properties: {}} + - is_false: test_1.mappings.type_1 + - is_false: test_2 + +--- +"Get /_mapping/{type}": + + - do: + indices.get_mapping: + type: type_2 + + - match: { test_1.mappings.type_2.properties: {}} + - match: { test_2.mappings.type_2.properties: {}} + - is_false: test_1.mappings.type_1 + - is_false: test_2.mappings.type_3 + +--- +"Get /_all/_mapping/{type}": + + - do: + indices.get_mapping: + index: _all + type: type_2 + + - match: { test_1.mappings.type_2.properties: {}} + - match: { test_2.mappings.type_2.properties: {}} + - is_false: test_1.mappings.type_1 + - is_false: test_2.mappings.type_3 + +--- +"Get /*/_mapping/{type}": + + - do: + indices.get_mapping: + index: '*' + type: type_2 + + - match: { test_1.mappings.type_2.properties: {}} + - match: { test_2.mappings.type_2.properties: {}} + - is_false: test_1.mappings.type_1 + - is_false: test_2.mappings.type_3 + +--- +"Get /{index}/_mapping/{type}": + + - do: + indices.get_mapping: + index: test_2 + type: type_2 + + - match: { test_2.mappings.type_2.properties: {}} + - is_false: test_1 + - is_false: test_2.mappings.type_3 + +--- +"Get /index,index/_mapping/{type}": + + - do: + indices.get_mapping: + index: test_1,test_2 + type: type_2 + + - match: { test_1.mappings.type_2.properties: {}} + - match: { test_2.mappings.type_2.properties: {}} + - is_false: test_2.mappings.type_3 + +--- +"Get /index*/_mapping/{type}": + + - do: + indices.get_mapping: + index: '*2' + type: type_2 - - match: {test_index.test_type.properties.text.type: string} - - match: {test_index.test_type.properties.text.analyzer: whitespace} + - match: { test_2.mappings.type_2.properties: {}} + - is_false: test_1 + - is_false: test_2.mappings.type_3 diff --git a/rest-api-spec/test/indices.get_mapping/20_missing_type.yaml b/rest-api-spec/test/indices.get_mapping/20_missing_type.yaml index e7d748acf929f..91b1883197b0c 100644 --- a/rest-api-spec/test/indices.get_mapping/20_missing_type.yaml +++ b/rest-api-spec/test/indices.get_mapping/20_missing_type.yaml @@ -1,5 +1,5 @@ --- -"Raise 404 when type doesn't exist": +"Return empty response when type doesn't exist": - do: indices.create: index: test_index @@ -12,8 +12,8 @@ analyzer: whitespace - do: - catch: missing indices.get_mapping: index: test_index type: not_test_type - + + - match: { '': {}} diff --git a/rest-api-spec/test/indices.get_mapping/40_aliases.yaml b/rest-api-spec/test/indices.get_mapping/40_aliases.yaml new file mode 100644 index 0000000000000..9dff6edc9dc74 --- /dev/null +++ b/rest-api-spec/test/indices.get_mapping/40_aliases.yaml @@ -0,0 +1,26 @@ +--- +"Getting mapping for aliases should return the real index as key": + + - do: + indices.create: + index: test_index + body: + mappings: + test_type: + properties: + text: + type: string + analyzer: whitespace + + - do: + indices.put_alias: + index: test_index + name: test_alias + + - do: + indices.get_mapping: + index: test_alias + + - match: {test_index.mappings.test_type.properties.text.type: string} + - match: {test_index.mappings.test_type.properties.text.analyzer: whitespace} + diff --git a/rest-api-spec/test/indices.get_settings/10_basic.yaml b/rest-api-spec/test/indices.get_settings/10_basic.yaml index 50ff2cfc041d0..5e9ce8972985a 100644 --- a/rest-api-spec/test/indices.get_settings/10_basic.yaml +++ b/rest-api-spec/test/indices.get_settings/10_basic.yaml @@ -1,32 +1,167 @@ --- -"Test get indices settings": +setup: - do: - indices.create: - index: test-index - body: - settings: - index: - refresh_interval: -1 - number_of_shards: 2 - number_of_replicas: 3 - + indices.create: + index: test_1 - do: - indices.get_settings: - index: test-index + indices.create: + index: test_2 + +--- +"Get /_settings": - - match: - test-index.settings.index.number_of_replicas: "3" - - match: - test-index.settings.index.number_of_shards: "2" - - match: - test-index.settings.index.refresh_interval: "-1" + - do: + indices.get_settings: {} - - do: - indices.get_settings: + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_1.settings.index.number_of_replicas: "1"} + - match: { test_2.settings.index.number_of_shards: "5"} + - match: { test_2.settings.index.number_of_replicas: "1"} + +--- +"Get /{index}/_settings": + + - do: + indices.get_settings: + index: test_1 + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_1.settings.index.number_of_replicas: "1"} + - is_false: test_2 + + +--- +"Get /{index}/_settings/_all": + + - do: + indices.get_settings: + index: test_1 + name: _all + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_1.settings.index.number_of_replicas: "1"} + - is_false: test_2 + +--- +"Get /{index}/_settings/*": + + - do: + indices.get_settings: + index: test_1 + name: '*' + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_1.settings.index.number_of_replicas: "1"} + - is_false: test_2 + +--- +"Get /{index}/_settings/{name}": + + - do: + indices.get_settings: + index: test_1 + name: index.number_of_shards + + - match: { test_1.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2 + +--- +"Get /{index}/_settings/{name,name}": + + - do: + indices.get_settings: + index: test_1 + name: index.number_of_shards,index.number_of_replicas + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_1.settings.index.number_of_replicas: "1"} + - is_false: test_2 + +--- +"Get /{index}/_settings/{name*}": + + - do: + indices.get_settings: + index: test_1 + name: 'index.number_of_s*' + + - match: { test_1.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2 + +--- +"Get /_settings/{name}": + + - do: + indices.get_settings: + name: index.number_of_shards + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_2.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2.settings.index.number_of_replicas + +--- +"Get /_all/_settings/{name}": + + - do: + indices.get_settings: index: _all - prefix: index.number + name: index.number_of_shards + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_2.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2.settings.index.number_of_replicas + + +--- +"Get /*/_settings/{name}": + + - do: + indices.get_settings: + index: '*' + name: index.number_of_shards + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_2.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2.settings.index.number_of_replicas + +--- +"Get /{index}/_settings/{name}": + + - do: + indices.get_settings: + index: test_1 + name: index.number_of_shards + + - match: { test_1.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2 + +--- +"Get /index,index/_settings/{name}": + + - do: + indices.get_settings: + index: test_1,test_2 + name: index.number_of_shards + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_2.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2.settings.index.number_of_replicas + +--- +"Get /index*/_settings/{name}": + + - do: + indices.get_settings: + index: '*2' + name: index.number_of_shards - - match: - test-index.settings.index.number_of_replicas: "3" - - match: - test-index.settings.index.number_of_shards: "2" + - match: { test_2.settings.index.number_of_shards: "5"} + - is_false: test_1 + - is_false: test_2.settings.index.number_of_replicas diff --git a/rest-api-spec/test/indices.get_settings/20_aliases.yaml b/rest-api-spec/test/indices.get_settings/20_aliases.yaml new file mode 100644 index 0000000000000..da7678202ed34 --- /dev/null +++ b/rest-api-spec/test/indices.get_settings/20_aliases.yaml @@ -0,0 +1,26 @@ +--- +"Getting settings for aliases should return the real index as key": + + - do: + indices.create: + index: test-index + body: + settings: + index: + refresh_interval: -1 + number_of_shards: 2 + number_of_replicas: 3 + + - do: + indices.put_alias: + index: test-index + name: test-alias + + - do: + indices.get_settings: + index: test-alias + + - match: { test-index.settings.index.number_of_replicas: "3" } + - match: { test-index.settings.index.number_of_shards: "2" } + - match: { test-index.settings.index.refresh_interval: "-1" } + diff --git a/rest-api-spec/test/indices.get_warmer/10_basic.yaml b/rest-api-spec/test/indices.get_warmer/10_basic.yaml new file mode 100644 index 0000000000000..ed13f90c21bec --- /dev/null +++ b/rest-api-spec/test/indices.get_warmer/10_basic.yaml @@ -0,0 +1,215 @@ +--- +setup: + - do: + indices.create: + index: test_1 + - do: + indices.create: + index: test_2 + + - do: + cluster.health: + wait_for_status: yellow + - do: + indices.put_warmer: + index: test_1 + name: warmer_1 + body: { query: { match_all: { }}} + + - do: + indices.put_warmer: + index: test_1 + name: warmer_2 + body: { query: { match_all: { }}} + + - do: + indices.put_warmer: + index: test_2 + name: warmer_2 + body: { query: { match_all: { }}} + - do: + indices.put_warmer: + index: test_2 + name: warmer_3 + body: { query: { match_all: { }}} + + - do: + indices.refresh: {} + +--- +"Get /_warmer": + + - do: + indices.get_warmer: {} + + - match: { test_1.warmers.warmer_1.source.query.match_all: {}} + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - match: { test_2.warmers.warmer_3.source.query.match_all: {}} + +--- +"Get /{index}/_warmer": + + - do: + indices.get_warmer: + index: test_1 + + - match: { test_1.warmers.warmer_1.source.query.match_all: {}} + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_2 + + +--- +"Get /{index}/_warmer/_all": + + - do: + indices.get_warmer: + index: test_1 + name: _all + + - match: { test_1.warmers.warmer_1.source.query.match_all: {}} + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_2 + +--- +"Get /{index}/_warmer/*": + + - do: + indices.get_warmer: + index: test_1 + name: '*' + + - match: { test_1.warmers.warmer_1.source.query.match_all: {}} + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_2 + +--- +"Get /{index}/_warmer/{name}": + + - do: + indices.get_warmer: + index: test_1 + name: warmer_1 + + - match: { test_1.warmers.warmer_1.source.query.match_all: {}} + - is_false: test_1.warmers.warmer_2 + - is_false: test_2 + +--- +"Get /{index}/_warmer/{name,name}": + + - do: + indices.get_warmer: + index: test_1 + name: warmer_1,warmer_2 + + - match: { test_1.warmers.warmer_1.source.query.match_all: {}} + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_2 + +--- +"Get /{index}/_warmer/{name*}": + + - do: + indices.get_warmer: + index: test_1 + name: '*2' + + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_1.warmers.warmer_1 + - is_false: test_2 + +--- +"Get /_warmer/{name}": + + - do: + indices.get_warmer: + name: warmer_2 + + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_1.warmers.warmer_1 + - is_false: test_2.warmers.warmer_3 + +--- +"Get /_all/_warmer/{name}": + + - do: + indices.get_warmer: + index: _all + name: warmer_2 + + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_1.warmers.warmer_1 + - is_false: test_2.warmers.warmer_3 + +--- +"Get /*/_warmer/{name}": + + - do: + indices.get_warmer: + index: '*' + name: warmer_2 + + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_1.warmers.warmer_1 + - is_false: test_2.warmers.warmer_3 + +--- +"Get /{index}/_warmer/{name}": + + - do: + indices.get_warmer: + index: test_2 + name: warmer_2 + + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_1 + - is_false: test_2.warmers.warmer_3 + +--- +"Get /index,index/_warmer/{name}": + + - do: + indices.get_warmer: + index: test_1,test_2 + name: warmer_2 + + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_2.warmers.warmer_3 + +--- +"Get /index*/_warmer/{name}": + + - do: + indices.get_warmer: + index: '*2' + name: warmer_2 + + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_1 + - is_false: test_2.warmers.warmer_3 + +--- +"Empty response when no matching warmer": + + - do: + indices.get_warmer: + index: '*' + name: non_existent + + - match: { '': {}} + +--- +"Throw 404 on missing index": + + - do: + catch: missing + indices.get_warmer: + index: non_existent + name: '*' + + diff --git a/rest-api-spec/test/indices.put_mapping/10_basic.yaml b/rest-api-spec/test/indices.put_mapping/10_basic.yaml index c227da8673b5e..e02b948f9367a 100644 --- a/rest-api-spec/test/indices.put_mapping/10_basic.yaml +++ b/rest-api-spec/test/indices.put_mapping/10_basic.yaml @@ -22,10 +22,10 @@ indices.get_mapping: index: test_index - - match: {test_index.test_type.properties.text1.type: string} - - match: {test_index.test_type.properties.text1.analyzer: whitespace} - - match: {test_index.test_type.properties.text2.type: string} - - match: {test_index.test_type.properties.text2.analyzer: whitespace} + - match: {test_index.mappings.test_type.properties.text1.type: string} + - match: {test_index.mappings.test_type.properties.text1.analyzer: whitespace} + - match: {test_index.mappings.test_type.properties.text2.type: string} + - match: {test_index.mappings.test_type.properties.text2.analyzer: whitespace} - do: indices.put_mapping: @@ -56,7 +56,7 @@ indices.get_mapping: index: test_index - - match: {test_index.test_type.properties.text1.type: string} - - match: {test_index.test_type.properties.text1.fields.text_raw.index: not_analyzed} - - match: {test_index.test_type.properties.text2.type: string} - - match: {test_index.test_type.properties.text2.fields.text_raw.index: not_analyzed} + - match: {test_index.mappings.test_type.properties.text1.type: string} + - match: {test_index.mappings.test_type.properties.text1.fields.text_raw.index: not_analyzed} + - match: {test_index.mappings.test_type.properties.text2.type: string} + - match: {test_index.mappings.test_type.properties.text2.fields.text_raw.index: not_analyzed} diff --git a/rest-api-spec/test/indices.put_mapping/all_path_options.yaml b/rest-api-spec/test/indices.put_mapping/all_path_options.yaml index db0df345c5db2..1c26184513d72 100644 --- a/rest-api-spec/test/indices.put_mapping/all_path_options.yaml +++ b/rest-api-spec/test/indices.put_mapping/all_path_options.yaml @@ -37,13 +37,13 @@ setup: - do: indices.get_mapping: {} - - match: {test_index1.test_type.properties.text.type: string} - - match: {test_index1.test_type.properties.text.analyzer: whitespace} + - match: {test_index1.mappings.test_type.properties.text.type: string} + - match: {test_index1.mappings.test_type.properties.text.analyzer: whitespace} - - match: {test_index2.test_type.properties.text.type: string} - - match: {test_index2.test_type.properties.text.analyzer: whitespace} + - match: {test_index2.mappings.test_type.properties.text.type: string} + - match: {test_index2.mappings.test_type.properties.text.analyzer: whitespace} - - match: {foo: {}} + - is_false: foo --- "put mapping in _all index": @@ -62,14 +62,14 @@ setup: - do: indices.get_mapping: {} - - match: {test_index1.test_type.properties.text.type: string} - - match: {test_index1.test_type.properties.text.analyzer: whitespace} + - match: {test_index1.mappings.test_type.properties.text.type: string} + - match: {test_index1.mappings.test_type.properties.text.analyzer: whitespace} - - match: {test_index2.test_type.properties.text.type: string} - - match: {test_index2.test_type.properties.text.analyzer: whitespace} + - match: {test_index2.mappings.test_type.properties.text.type: string} + - match: {test_index2.mappings.test_type.properties.text.analyzer: whitespace} - - match: {foo.test_type.properties.text.type: string} - - match: {foo.test_type.properties.text.analyzer: whitespace} + - match: {foo.mappings.test_type.properties.text.type: string} + - match: {foo.mappings.test_type.properties.text.analyzer: whitespace} --- "put mapping in * index": @@ -87,14 +87,14 @@ setup: - do: indices.get_mapping: {} - - match: {test_index1.test_type.properties.text.type: string} - - match: {test_index1.test_type.properties.text.analyzer: whitespace} + - match: {test_index1.mappings.test_type.properties.text.type: string} + - match: {test_index1.mappings.test_type.properties.text.analyzer: whitespace} - - match: {test_index2.test_type.properties.text.type: string} - - match: {test_index2.test_type.properties.text.analyzer: whitespace} + - match: {test_index2.mappings.test_type.properties.text.type: string} + - match: {test_index2.mappings.test_type.properties.text.analyzer: whitespace} - - match: {foo.test_type.properties.text.type: string} - - match: {foo.test_type.properties.text.analyzer: whitespace} + - match: {foo.mappings.test_type.properties.text.type: string} + - match: {foo.mappings.test_type.properties.text.analyzer: whitespace} --- "put mapping in prefix* index": @@ -112,13 +112,13 @@ setup: - do: indices.get_mapping: {} - - match: {test_index1.test_type.properties.text.type: string} - - match: {test_index1.test_type.properties.text.analyzer: whitespace} + - match: {test_index1.mappings.test_type.properties.text.type: string} + - match: {test_index1.mappings.test_type.properties.text.analyzer: whitespace} - - match: {test_index2.test_type.properties.text.type: string} - - match: {test_index2.test_type.properties.text.analyzer: whitespace} + - match: {test_index2.mappings.test_type.properties.text.type: string} + - match: {test_index2.mappings.test_type.properties.text.analyzer: whitespace} - - match: {foo: {}} + - is_false: foo --- "put mapping in list of indices": @@ -136,13 +136,13 @@ setup: - do: indices.get_mapping: {} - - match: {test_index1.test_type.properties.text.type: string} - - match: {test_index1.test_type.properties.text.analyzer: whitespace} + - match: {test_index1.mappings.test_type.properties.text.type: string} + - match: {test_index1.mappings.test_type.properties.text.analyzer: whitespace} - - match: {test_index2.test_type.properties.text.type: string} - - match: {test_index2.test_type.properties.text.analyzer: whitespace} + - match: {test_index2.mappings.test_type.properties.text.type: string} + - match: {test_index2.mappings.test_type.properties.text.analyzer: whitespace} - - match: {foo: {}} + - is_false: foo --- "put mapping with blank index": @@ -159,14 +159,14 @@ setup: - do: indices.get_mapping: {} - - match: {test_index1.test_type.properties.text.type: string} - - match: {test_index1.test_type.properties.text.analyzer: whitespace} + - match: {test_index1.mappings.test_type.properties.text.type: string} + - match: {test_index1.mappings.test_type.properties.text.analyzer: whitespace} - - match: {test_index2.test_type.properties.text.type: string} - - match: {test_index2.test_type.properties.text.analyzer: whitespace} + - match: {test_index2.mappings.test_type.properties.text.type: string} + - match: {test_index2.mappings.test_type.properties.text.analyzer: whitespace} - - match: {foo.test_type.properties.text.type: string} - - match: {foo.test_type.properties.text.analyzer: whitespace} + - match: {foo.mappings.test_type.properties.text.type: string} + - match: {foo.mappings.test_type.properties.text.analyzer: whitespace} --- "put mapping with mising type": diff --git a/rest-api-spec/test/indices.put_warmer/10_basic.yaml b/rest-api-spec/test/indices.put_warmer/10_basic.yaml index 0e2a622360590..44313aa9bfa27 100644 --- a/rest-api-spec/test/indices.put_warmer/10_basic.yaml +++ b/rest-api-spec/test/indices.put_warmer/10_basic.yaml @@ -1,18 +1,24 @@ --- -"Basic test for warmers": +setup: - do: indices.create: index: test_index + - do: + indices.create: + index: test_idx + - do: cluster.health: wait_for_status: yellow - do: - catch: missing - indices.get_warmer: - index: test_index - name: test_warmer + indices.put_warmer: + index: test_idx + name: test_warmer2 + body: + query: + match_all: {} - do: indices.put_warmer: @@ -22,6 +28,8 @@ query: match_all: {} +--- +"Basic test for warmers": - do: indices.get_warmer: index: test_index @@ -35,7 +43,103 @@ name: test_warmer - do: - catch: missing indices.get_warmer: index: test_index name: test_warmer + + - match: { '': {}} + +--- +"Getting all warmers via /_warmer should work": + + - do: + indices.get_warmer: {} + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - match: {test_idx.warmers.test_warmer2.source.query.match_all: {}} + + +--- +"Getting warmers for several indices should work using *": + + - do: + indices.get_warmer: + index: '*' + name: '*' + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - match: {test_idx.warmers.test_warmer2.source.query.match_all: {}} + +--- +"Getting warmers for several indices should work using _all": + + - do: + indices.get_warmer: + index: _all + name: _all + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - match: {test_idx.warmers.test_warmer2.source.query.match_all: {}} + +--- +"Getting all warmers without specifying index should work": + + - do: + indices.get_warmer: + name: _all + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - match: {test_idx.warmers.test_warmer2.source.query.match_all: {}} + +--- +"Getting warmers for several indices should work using prefix*": + + - do: + indices.get_warmer: + index: test_i* + name: test_w* + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - match: {test_idx.warmers.test_warmer2.source.query.match_all: {}} + +--- +"Getting warmers for several indices should work using comma-separated lists": + + - do: + indices.get_warmer: + index: test_index,test_idx + name: test_warmer,test_warmer2 + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - match: {test_idx.warmers.test_warmer2.source.query.match_all: {}} + +--- +"Getting a non-existent warmer on an existing index should return an empty body": + + - do: + indices.get_warmer: + index: test_index + name: non-existent + + - match: { '': {}} + +--- +"Getting an existent and non-existent warmer should return the existent and no data about the non-existent warmer": + + - do: + indices.get_warmer: + index: test_index + name: test_warmer,non-existent + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - is_false: test_index.warmers.non-existent + +--- +"Getting warmer on an non-existent index should return 404": + + - do: + catch: missing + indices.get_warmer: + index: non-existent + name: foo + diff --git a/rest-api-spec/test/indices.put_warmer/20_aliases.yaml b/rest-api-spec/test/indices.put_warmer/20_aliases.yaml new file mode 100644 index 0000000000000..96d734475ac83 --- /dev/null +++ b/rest-api-spec/test/indices.put_warmer/20_aliases.yaml @@ -0,0 +1,30 @@ +--- +"Getting warmer for aliases should return the real index as key": + + - do: + indices.create: + index: test_index + + - do: + cluster.health: + wait_for_status: yellow + + - do: + indices.put_warmer: + index: test_index + name: test_warmer + body: + query: + match_all: {} + + - do: + indices.put_alias: + index: test_index + name: test_alias + + - do: + indices.get_warmer: + index: test_alias + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + diff --git a/src/main/java/org/elasticsearch/action/admin/indices/settings/get/GetSettingsRequest.java b/src/main/java/org/elasticsearch/action/admin/indices/settings/get/GetSettingsRequest.java index 3025caee581a7..311a60d262c84 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/settings/get/GetSettingsRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/settings/get/GetSettingsRequest.java @@ -20,6 +20,7 @@ package org.elasticsearch.action.admin.indices.settings.get; import org.elasticsearch.action.ActionRequestValidationException; +import org.elasticsearch.action.ValidateActions; import org.elasticsearch.action.support.IndicesOptions; import org.elasticsearch.action.support.master.MasterNodeOperationRequest; import org.elasticsearch.common.Strings; @@ -34,7 +35,7 @@ public class GetSettingsRequest extends MasterNodeOperationRequest entry : settings.getAsMap().entrySet()) { - if (entry.getKey().startsWith(request.prefix())) { + if (Regex.simpleMatch(request.names(), entry.getKey())) { settingsBuilder.put(entry.getKey(), entry.getValue()); } } diff --git a/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java b/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java index b2310d2532cd8..d20f0b2775cdd 100644 --- a/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java +++ b/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java @@ -376,12 +376,14 @@ public ImmutableOpenMap> findM return indexMapBuilder.build(); } - public ImmutableOpenMap> findWarmers(String[] concreteIndices, final String[] types, final String[] warmers) { - assert warmers != null; + public ImmutableOpenMap> findWarmers(String[] concreteIndices, final String[] types, final String[] uncheckedWarmers) { + assert uncheckedWarmers != null; assert concreteIndices != null; if (concreteIndices.length == 0) { return ImmutableOpenMap.of(); } + // special _all check to behave the same like not specifying anything for the warmers (not for the indices) + final String[] warmers = Strings.isAllOrWildcard(uncheckedWarmers) ? Strings.EMPTY_ARRAY : uncheckedWarmers; ImmutableOpenMap.Builder> mapBuilder = ImmutableOpenMap.builder(); Iterable intersection = HppcMaps.intersection(ObjectOpenHashSet.from(concreteIndices), indices.keys()); diff --git a/src/main/java/org/elasticsearch/common/Strings.java b/src/main/java/org/elasticsearch/common/Strings.java index aa85afa294d3b..45aa62f4246be 100644 --- a/src/main/java/org/elasticsearch/common/Strings.java +++ b/src/main/java/org/elasticsearch/common/Strings.java @@ -26,6 +26,7 @@ import org.apache.lucene.util.UnicodeUtil; import org.elasticsearch.ElasticsearchIllegalStateException; import org.elasticsearch.common.io.FastStringReader; +import org.elasticsearch.common.util.CollectionUtils; import java.io.BufferedReader; import java.io.IOException; @@ -1565,4 +1566,14 @@ public static String substring(String s, int beginIndex, int endIndex) { return s.substring(beginIndex, endIndex); } } + + /** + * If an array only consists of zero or one element, which is "*" or "_all" return an empty array + * which is usually used as everything + */ + public static boolean isAllOrWildcard(String[] data) { + return CollectionUtils.isEmpty(data) || + data.length == 1 && ("_all".equals(data[0]) || "*".equals(data[0])); + } + } diff --git a/src/main/java/org/elasticsearch/rest/RestRequest.java b/src/main/java/org/elasticsearch/rest/RestRequest.java index 781999bb31b4b..efd411e3742c4 100644 --- a/src/main/java/org/elasticsearch/rest/RestRequest.java +++ b/src/main/java/org/elasticsearch/rest/RestRequest.java @@ -158,4 +158,13 @@ public String[] paramAsStringArray(String key, String[] defaultValue) { } return Strings.splitStringByCommaToArray(value); } + + public String[] paramAsStringArrayOrEmptyIfAll(String key) { + String[] params = paramAsStringArray(key, Strings.EMPTY_ARRAY); + if (Strings.isAllOrWildcard(params)) { + return Strings.EMPTY_ARRAY; + } + return params; + } + } diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetAliasesAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetAliasesAction.java index 5d7f71635bbb3..6357395ef411a 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetAliasesAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetAliasesAction.java @@ -49,6 +49,7 @@ public class RestGetAliasesAction extends BaseRestHandler { @Inject public RestGetAliasesAction(Settings settings, Client client, RestController controller) { super(settings, client); + controller.registerHandler(GET, "/_alias/", this); controller.registerHandler(GET, "/_alias/{name}", this); controller.registerHandler(GET, "/{index}/_alias/{name}", this); controller.registerHandler(GET, "/{index}/_alias", this); @@ -56,7 +57,7 @@ public RestGetAliasesAction(Settings settings, Client client, RestController con @Override public void handleRequest(final RestRequest request, final RestChannel channel) { - String[] aliases = request.paramAsStringArray("name", Strings.EMPTY_ARRAY); + final String[] aliases = request.paramAsStringArrayOrEmptyIfAll("name"); final String[] indices = Strings.splitStringByCommaToArray(request.param("index")); final GetAliasesRequest getAliasesRequest = new GetAliasesRequest(aliases); getAliasesRequest.indices(indices); @@ -68,7 +69,11 @@ public void handleRequest(final RestRequest request, final RestChannel channel) public void onResponse(GetAliasesResponse response) { try { XContentBuilder builder = RestXContentBuilder.restContentBuilder(request); - if (response.getAliases().isEmpty()) { + // empty body, if indices were specified but no aliases were + if (indices.length > 0 && response.getAliases().isEmpty()) { + channel.sendResponse(new XContentRestResponse(request, OK, RestXContentBuilder.emptyBuilder(request))); + return; + } else if (response.getAliases().isEmpty()) { String message = String.format(Locale.ROOT, "alias [%s] missing", toNamesString(getAliasesRequest.aliases())); builder.startObject() .field("error", message) diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetIndicesAliasesAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetIndicesAliasesAction.java index c5122834a936d..91c17c8dad810 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetIndicesAliasesAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetIndicesAliasesAction.java @@ -29,6 +29,7 @@ import org.elasticsearch.cluster.metadata.MetaData; import org.elasticsearch.common.Strings; import org.elasticsearch.common.inject.Inject; +import org.elasticsearch.common.regex.Regex; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.xcontent.ToXContent; import org.elasticsearch.common.xcontent.XContentBuilder; @@ -37,6 +38,7 @@ import java.io.IOException; +import static org.elasticsearch.common.Strings.isAllOrWildcard; import static org.elasticsearch.rest.RestRequest.Method.GET; import static org.elasticsearch.rest.RestStatus.OK; @@ -50,16 +52,19 @@ public RestGetIndicesAliasesAction(Settings settings, Client client, RestControl super(settings, client); controller.registerHandler(GET, "/_aliases", this); controller.registerHandler(GET, "/{index}/_aliases", this); + controller.registerHandler(GET, "/{index}/_aliases/{name}", this); + controller.registerHandler(GET, "/_aliases/{name}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { final String[] indices = Strings.splitStringByCommaToArray(request.param("index")); + final String[] aliases = Strings.splitStringByCommaToArray(request.param("name")); ClusterStateRequest clusterStateRequest = Requests.clusterStateRequest() - .routingTable(false) - .nodes(false) - .indices(indices); + .routingTable(false) + .nodes(false) + .indices(indices); clusterStateRequest.listenerThreaded(false); @@ -71,20 +76,22 @@ public void onResponse(ClusterStateResponse response) { XContentBuilder builder = RestXContentBuilder.restContentBuilder(request); builder.startObject(); + final boolean isAllAliasesRequested = isAllOrWildcard(aliases); for (IndexMetaData indexMetaData : metaData) { builder.startObject(indexMetaData.index(), XContentBuilder.FieldCaseConversion.NONE); - builder.startObject("aliases"); + for (ObjectCursor cursor : indexMetaData.aliases().values()) { - AliasMetaData.Builder.toXContent(cursor.value, builder, ToXContent.EMPTY_PARAMS); + if (isAllAliasesRequested || Regex.simpleMatch(aliases, cursor.value.alias())) { + AliasMetaData.Builder.toXContent(cursor.value, builder, ToXContent.EMPTY_PARAMS); + } } - builder.endObject(); builder.endObject(); + builder.endObject(); } builder.endObject(); - channel.sendResponse(new XContentRestResponse(request, OK, builder)); } catch (Throwable e) { onFailure(e); @@ -101,4 +108,5 @@ public void onFailure(Throwable e) { } }); } + } \ No newline at end of file diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetMappingAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetMappingAction.java index 57909f5bfdea4..a5fef39bb0d37 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetMappingAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetMappingAction.java @@ -31,6 +31,7 @@ import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.common.xcontent.XContentBuilderString; import org.elasticsearch.index.Index; import org.elasticsearch.indices.IndexMissingException; import org.elasticsearch.indices.TypeMissingException; @@ -53,12 +54,15 @@ public RestGetMappingAction(Settings settings, Client client, RestController con controller.registerHandler(GET, "/_mapping", this); controller.registerHandler(GET, "/{index}/_mapping", this); controller.registerHandler(GET, "/{index}/{type}/_mapping", this); + controller.registerHandler(GET, "/{index}/_mappings/{type}", this); + controller.registerHandler(GET, "/{index}/_mapping/{type}", this); + controller.registerHandler(GET, "/_mapping/{type}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { final String[] indices = Strings.splitStringByCommaToArray(request.param("index")); - final String[] types = Strings.splitStringByCommaToArray(request.param("type")); + final String[] types = request.paramAsStringArrayOrEmptyIfAll("type"); boolean local = request.paramAsBooleanOptional("local", false); GetMappingsRequest getMappingsRequest = new GetMappingsRequest(); getMappingsRequest.indices(indices).types(types).local(local); @@ -74,7 +78,7 @@ public void onResponse(GetMappingsResponse response) { ImmutableOpenMap> mappingsByIndex = response.getMappings(); if (mappingsByIndex.isEmpty()) { if (indices.length != 0 && types.length != 0) { - channel.sendResponse(new XContentThrowableRestResponse(request, new TypeMissingException(new Index(indices[0]), types[0]))); + channel.sendResponse(new XContentRestResponse(request, OK, RestXContentBuilder.emptyBuilder(request))); } else if (indices.length != 0) { channel.sendResponse(new XContentThrowableRestResponse(request, new IndexMissingException(new Index(indices[0])))); } else if (types.length != 0) { @@ -87,12 +91,17 @@ public void onResponse(GetMappingsResponse response) { } for (ObjectObjectCursor> indexEntry : mappingsByIndex) { + if (indexEntry.value.isEmpty()) { + continue; + } builder.startObject(indexEntry.key, XContentBuilder.FieldCaseConversion.NONE); + builder.startObject(Fields.MAPPINGS); for (ObjectObjectCursor typeEntry : indexEntry.value) { builder.field(typeEntry.key); builder.map(typeEntry.value.sourceAsMap()); } builder.endObject(); + builder.endObject(); } builder.endObject(); @@ -112,4 +121,8 @@ public void onFailure(Throwable e) { } }); } + + static class Fields { + static final XContentBuilderString MAPPINGS = new XContentBuilderString("mappings"); + } } diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestGetSettingsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestGetSettingsAction.java index 506f2ffa9acaf..d87aa82148141 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestGetSettingsAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestGetSettingsAction.java @@ -46,34 +46,39 @@ public RestGetSettingsAction(Settings settings, Client client, RestController co super(settings, client); controller.registerHandler(GET, "/_settings", this); controller.registerHandler(GET, "/{index}/_settings", this); - controller.registerHandler(GET, "/{index}/{prefix}/_settings", this); + controller.registerHandler(GET, "/{index}/_settings/{name}", this); + controller.registerHandler(GET, "/_settings/{name}", this); + controller.registerHandler(GET, "/{index}/_setting/{name}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { + final String[] names = request.paramAsStringArrayOrEmptyIfAll("name"); GetSettingsRequest getSettingsRequest = new GetSettingsRequest() .indices(Strings.splitStringByCommaToArray(request.param("index"))) .indicesOptions(IndicesOptions.fromRequest(request, IndicesOptions.strict())) - .prefix(request.param("prefix")); + .names(names); client.admin().indices().getSettings(getSettingsRequest, new ActionListener() { @Override public void onResponse(GetSettingsResponse getSettingsResponse) { try { - boolean foundAny = false; XContentBuilder builder = RestXContentBuilder.restContentBuilder(request); builder.startObject(); for (ObjectObjectCursor cursor : getSettingsResponse.getIndexToSettings()) { + // no settings, jump over it to shorten the response data + if (cursor.value.getAsMap().isEmpty()) { + continue; + } builder.startObject(cursor.key, XContentBuilder.FieldCaseConversion.NONE); - foundAny = true; builder.startObject(Fields.SETTINGS); cursor.value.toXContent(builder, request); builder.endObject(); builder.endObject(); } builder.endObject(); - channel.sendResponse(new XContentRestResponse(request, foundAny ? OK : NOT_FOUND, builder)); + channel.sendResponse(new XContentRestResponse(request, OK, builder)); } catch (IOException e) { onFailure(e); } diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/get/RestGetWarmerAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/get/RestGetWarmerAction.java index ae2ea99a54132..0dbe4c528ddfb 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/get/RestGetWarmerAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/get/RestGetWarmerAction.java @@ -48,9 +48,11 @@ public class RestGetWarmerAction extends BaseRestHandler { @Inject public RestGetWarmerAction(Settings settings, Client client, RestController controller) { super(settings, client); - + controller.registerHandler(GET, "/_warmer", this); + controller.registerHandler(GET, "/_warmer/{name}", this); controller.registerHandler(GET, "/{index}/_warmer", this); controller.registerHandler(GET, "/{index}/_warmer/{name}", this); + controller.registerHandler(GET, "/{index}/_warmers/{name}", this); controller.registerHandler(GET, "/{index}/{type}/_warmer/{name}", this); } @@ -70,7 +72,7 @@ public void handleRequest(final RestRequest request, final RestChannel channel) public void onResponse(GetWarmersResponse response) { try { if (indices.length > 0 && response.warmers().isEmpty()) { - channel.sendResponse(new XContentThrowableRestResponse(request, new IndexMissingException(new Index(indices[0])))); + channel.sendResponse(new XContentRestResponse(request, OK, RestXContentBuilder.emptyBuilder(request))); return; } diff --git a/src/main/java/org/elasticsearch/rest/action/support/RestXContentBuilder.java b/src/main/java/org/elasticsearch/rest/action/support/RestXContentBuilder.java index fbde560fcf3e0..cff25557a669b 100644 --- a/src/main/java/org/elasticsearch/rest/action/support/RestXContentBuilder.java +++ b/src/main/java/org/elasticsearch/rest/action/support/RestXContentBuilder.java @@ -67,6 +67,10 @@ public static XContentBuilder restContentBuilder(RestRequest request, @Nullable return builder; } + public static XContentBuilder emptyBuilder(RestRequest request) throws IOException { + return restContentBuilder(request, request.hasContent() ? request.content() : null).startObject().endObject(); + } + /** * Directly writes the source to the output builder */ diff --git a/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java b/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java index e1bcac1d2a9f8..e24985b6ad9c6 100644 --- a/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java +++ b/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java @@ -231,6 +231,36 @@ public void ensureThatIndexWarmersCanBeChangedOnRuntime() throws Exception { assertThat(getWarmerRuns(), equalTo(warmerRunsAfterDisabling)); } + @Test + public void gettingAllWarmersUsingAllAndWildcardsShouldWork() throws Exception { + client().admin().indices().prepareCreate("test") + .setSettings(ImmutableSettings.settingsBuilder().put("index.number_of_shards", 1, "index.number_of_replicas", 0)) + .execute().actionGet(); + ensureGreen(); + + PutWarmerResponse putWarmerResponse = client().admin().indices().preparePutWarmer("custom_warmer") + .setSearchRequest(client().prepareSearch("test").setTypes("test").setQuery(QueryBuilders.matchAllQuery())) + .execute().actionGet(); + assertThat(putWarmerResponse.isAcknowledged(), equalTo(true)); + + PutWarmerResponse anotherPutWarmerResponse = client().admin().indices().preparePutWarmer("second_custom_warmer") + .setSearchRequest(client().prepareSearch("test").setTypes("test").setQuery(QueryBuilders.matchAllQuery())) + .execute().actionGet(); + assertThat(anotherPutWarmerResponse.isAcknowledged(), equalTo(true)); + + GetWarmersResponse getWarmersResponse = client().admin().indices().prepareGetWarmers("*").addWarmers("*").get(); + assertThat(getWarmersResponse.warmers().size(), is(1)); + + getWarmersResponse = client().admin().indices().prepareGetWarmers("_all").addWarmers("_all").get(); + assertThat(getWarmersResponse.warmers().size(), is(1)); + + getWarmersResponse = client().admin().indices().prepareGetWarmers("t*").addWarmers("c*").get(); + assertThat(getWarmersResponse.warmers().size(), is(1)); + + getWarmersResponse = client().admin().indices().prepareGetWarmers("test").addWarmers("custom_warmer", "second_custom_warmer").get(); + assertThat(getWarmersResponse.warmers().size(), is(1)); + } + private long getWarmerRuns() { IndicesStatsResponse indicesStatsResponse = client().admin().indices().prepareStats("test").clear().setWarmer(true).execute().actionGet(); return indicesStatsResponse.getIndex("test").getPrimaries().warmer.total(); From a3abcdc93a004bea53ba2d4cbab585f8820b660d Mon Sep 17 00:00:00 2001 From: Alexander Reelsen Date: Tue, 14 Jan 2014 22:39:28 +0100 Subject: [PATCH 19/34] Consistent APIs: Get field mapping API includes 'mappings' The get field mapping API now includes a mappings element after the index in its JSON Added more consistent endpoint /{index}/_mapping/{type}/field/{fields} and added endpoint /_mapping/{type}/field/{fields} which are also used in tests Added rest spec tests for wildcards and _all Relates #4071 NOTE: This is not yet complete for 1.0. We need to return an empty JSON document instead of a 404 if the field of an existing index and type is not found. However this is not possible with the current data structure being returned. Needs to be finished for 1.0. --- .../api/indices.get_field_mapping.json | 2 +- .../indices.get_field_mapping/10_basic.yaml | 25 +++-- .../50_field_wildcards.yaml | 104 +++++++++++++++--- .../mapping/get/GetFieldMappingsResponse.java | 2 + .../get/RestGetFieldMappingAction.java | 5 +- 5 files changed, 112 insertions(+), 26 deletions(-) diff --git a/rest-api-spec/api/indices.get_field_mapping.json b/rest-api-spec/api/indices.get_field_mapping.json index b683bf4ee0c88..1d823e23104a7 100644 --- a/rest-api-spec/api/indices.get_field_mapping.json +++ b/rest-api-spec/api/indices.get_field_mapping.json @@ -4,7 +4,7 @@ "methods": ["GET"], "url": { "path": "/_mapping/field/{field}", - "paths": ["/_mapping/field/{field}", "/{index}/_mapping/field/{field}", "/{index}/{type}/_mapping/field/{field}"], + "paths": ["/_mapping/field/{field}", "/{index}/_mapping/field/{field}", "/_mapping/{type}/field/{field}", "/{index}/_mapping/{type}/field/{field}"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/test/indices.get_field_mapping/10_basic.yaml b/rest-api-spec/test/indices.get_field_mapping/10_basic.yaml index 4d4614629c272..bac6fffc0a567 100644 --- a/rest-api-spec/test/indices.get_field_mapping/10_basic.yaml +++ b/rest-api-spec/test/indices.get_field_mapping/10_basic.yaml @@ -17,7 +17,7 @@ setup: indices.get_field_mapping: field: text - - match: {test_index.test_type.text.mapping.text.type: string} + - match: {test_index.mappings.test_type.text.mapping.text.type: string} --- "Get field mapping by index only": @@ -26,7 +26,7 @@ setup: index: test_index field: text - - match: {test_index.test_type.text.mapping.text.type: string} + - match: {test_index.mappings.test_type.text.mapping.text.type: string} --- "Get field mapping by type & field": @@ -37,7 +37,7 @@ setup: type: test_type field: text - - match: {test_index.test_type.text.mapping.text.type: string} + - match: {test_index.mappings.test_type.text.mapping.text.type: string} --- "Get field mapping by type & field, with another field that doesn't exist": @@ -48,8 +48,8 @@ setup: type: test_type field: [ text , text1 ] - - match: {test_index.test_type.text.mapping.text.type: string} - - is_false: test_index.test_type.text1 + - match: {test_index.mappings.test_type.text.mapping.text.type: string} + - is_false: test_index.mappings.test_type.text1 --- "Get field mapping with include_defaults": @@ -61,5 +61,16 @@ setup: field: text include_defaults: true - - match: {test_index.test_type.text.mapping.text.type: string} - - match: {test_index.test_type.text.mapping.text.analyzer: default} + - match: {test_index.mappings.test_type.text.mapping.text.type: string} + - match: {test_index.mappings.test_type.text.mapping.text.analyzer: default} + +--- +"Get field mapping should work without index specifying type and field": + + - do: + indices.get_field_mapping: + type: test_type + field: text + + - match: {test_index.mappings.test_type.text.mapping.text.type: string} + diff --git a/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml b/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml index 879d74988f5d7..fcf592632663e 100644 --- a/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml +++ b/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml @@ -23,6 +23,29 @@ setup: type: string index_name: t3 + - do: + indices.create: + index: test_index_2 + body: + mappings: + test_type_2: + properties: + t1: + type: string + t2: + type: string + obj: + path: just_name + properties: + t1: + type: string + i_t1: + type: string + index_name: t1 + i_t3: + type: string + index_name: t3 + --- "Get field mapping with * for fields": @@ -30,43 +53,92 @@ setup: indices.get_field_mapping: field: "*" - - match: {test_index.test_type.t1.full_name: t1 } - - match: {test_index.test_type.t2.full_name: t2 } - - match: {test_index.test_type.obj\.t1.full_name: obj.t1 } - - match: {test_index.test_type.obj\.i_t1.full_name: obj.i_t1 } - - match: {test_index.test_type.obj\.i_t3.full_name: obj.i_t3 } + - match: {test_index.mappings.test_type.t1.full_name: t1 } + - match: {test_index.mappings.test_type.t2.full_name: t2 } + - match: {test_index.mappings.test_type.obj\.t1.full_name: obj.t1 } + - match: {test_index.mappings.test_type.obj\.i_t1.full_name: obj.i_t1 } + - match: {test_index.mappings.test_type.obj\.i_t3.full_name: obj.i_t3 } --- "Get field mapping with t* for fields": - do: indices.get_field_mapping: + index: test_index field: "t*" # i_t1 matches the pattern using it's index name, but t1 already means a full name # of a field and thus takes precedence. - - match: {test_index.test_type.t1.full_name: t1 } - - match: {test_index.test_type.t2.full_name: t2 } - - match: {test_index.test_type.t3.full_name: obj.i_t3 } - - length: {test_index.test_type: 3} + - match: {test_index.mappings.test_type.t1.full_name: t1 } + - match: {test_index.mappings.test_type.t2.full_name: t2 } + - match: {test_index.mappings.test_type.t3.full_name: obj.i_t3 } + - length: {test_index.mappings.test_type: 3} --- "Get field mapping with *t1 for fields": - do: indices.get_field_mapping: + index: test_index field: "*t1" - - match: {test_index.test_type.t1.full_name: t1 } - - match: {test_index.test_type.obj\.t1.full_name: obj.t1 } - - match: {test_index.test_type.obj\.i_t1.full_name: obj.i_t1 } - - length: {test_index.test_type: 3} + - match: {test_index.mappings.test_type.t1.full_name: t1 } + - match: {test_index.mappings.test_type.obj\.t1.full_name: obj.t1 } + - match: {test_index.mappings.test_type.obj\.i_t1.full_name: obj.i_t1 } + - length: {test_index.mappings.test_type: 3} --- "Get field mapping with wildcarded relative names": - do: indices.get_field_mapping: + index: test_index + field: "i_*" + - match: {test_index.mappings.test_type.i_t1.full_name: obj.i_t1 } + - match: {test_index.mappings.test_type.i_t3.full_name: obj.i_t3 } + - length: {test_index.mappings.test_type: 2} + +--- +"Get field mapping should work using '_all' for indices and types": + + - do: + indices.get_field_mapping: + index: _all + type: _all + field: "i_*" + - match: {test_index.mappings.test_type.i_t1.full_name: obj.i_t1 } + - match: {test_index.mappings.test_type.i_t3.full_name: obj.i_t3 } + - length: {test_index.mappings.test_type: 2} + - match: {test_index_2.mappings.test_type_2.i_t1.full_name: obj.i_t1 } + - match: {test_index_2.mappings.test_type_2.i_t3.full_name: obj.i_t3 } + - length: {test_index_2.mappings.test_type_2: 2} + +--- +"Get field mapping should work using '*' for indices and types": + + - do: + indices.get_field_mapping: + index: '*' + type: '*' + field: "i_*" + - match: {test_index.mappings.test_type.i_t1.full_name: obj.i_t1 } + - match: {test_index.mappings.test_type.i_t3.full_name: obj.i_t3 } + - length: {test_index.mappings.test_type: 2} + - match: {test_index_2.mappings.test_type_2.i_t1.full_name: obj.i_t1 } + - match: {test_index_2.mappings.test_type_2.i_t3.full_name: obj.i_t3 } + - length: {test_index_2.mappings.test_type_2: 2} + +--- +"Get field mapping should work using comma_separated values for indices and types": + + - do: + indices.get_field_mapping: + index: 'test_index,test_index_2' + type: 'test_type,test_type_2' field: "i_*" - - match: {test_index.test_type.i_t1.full_name: obj.i_t1 } - - match: {test_index.test_type.i_t3.full_name: obj.i_t3 } - - length: {test_index.test_type: 2} + - match: {test_index.mappings.test_type.i_t1.full_name: obj.i_t1 } + - match: {test_index.mappings.test_type.i_t3.full_name: obj.i_t3 } + - length: {test_index.mappings.test_type: 2} + - match: {test_index_2.mappings.test_type_2.i_t1.full_name: obj.i_t1 } + - match: {test_index_2.mappings.test_type_2.i_t3.full_name: obj.i_t3 } + - length: {test_index_2.mappings.test_type_2: 2} + diff --git a/src/main/java/org/elasticsearch/action/admin/indices/mapping/get/GetFieldMappingsResponse.java b/src/main/java/org/elasticsearch/action/admin/indices/mapping/get/GetFieldMappingsResponse.java index da612f602d976..a62ea0d353bfc 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/mapping/get/GetFieldMappingsResponse.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/mapping/get/GetFieldMappingsResponse.java @@ -71,6 +71,7 @@ public FieldMappingMetaData fieldMappings(String index, String type, String fiel public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { for (Map.Entry>> indexEntry : mappings.entrySet()) { builder.startObject(indexEntry.getKey(), XContentBuilder.FieldCaseConversion.NONE); + builder.startObject("mappings"); for (Map.Entry> typeEntry : indexEntry.getValue().entrySet()) { builder.startObject(typeEntry.getKey(), XContentBuilder.FieldCaseConversion.NONE); for (Map.Entry fieldEntry : typeEntry.getValue().entrySet()) { @@ -81,6 +82,7 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws builder.endObject(); } builder.endObject(); + builder.endObject(); } return builder; } diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetFieldMappingAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetFieldMappingAction.java index 946f29a7e400c..269098d0a8796 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetFieldMappingAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetFieldMappingAction.java @@ -49,17 +49,18 @@ public class RestGetFieldMappingAction extends BaseRestHandler { public RestGetFieldMappingAction(Settings settings, Client client, RestController controller) { super(settings, client); controller.registerHandler(GET, "/_mapping/field/{fields}", this); + controller.registerHandler(GET, "/_mapping/{type}/field/{fields}", this); controller.registerHandler(GET, "/{index}/_mapping/field/{fields}", this); controller.registerHandler(GET, "/{index}/{type}/_mapping/field/{fields}", this); + controller.registerHandler(GET, "/{index}/_mapping/{type}/field/{fields}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { final String[] indices = Strings.splitStringByCommaToArray(request.param("index")); - final String[] types = Strings.splitStringByCommaToArray(request.param("type")); + final String[] types = request.paramAsStringArrayOrEmptyIfAll("type"); boolean local = request.paramAsBooleanOptional("local", false); final String[] fields = Strings.splitStringByCommaToArray(request.param("fields")); - GetFieldMappingsRequest getMappingsRequest = new GetFieldMappingsRequest(); getMappingsRequest.indices(indices).types(types).local(local).fields(fields).includeDefaults(request.paramAsBoolean("include_defaults", false)); getMappingsRequest.indicesOptions(IndicesOptions.fromRequest(request, getMappingsRequest.indicesOptions())); From 6389432b28817b82b0ae8d5c67f0f7a47c32cbaa Mon Sep 17 00:00:00 2001 From: Britta Weber Date: Tue, 14 Jan 2014 22:25:15 +0100 Subject: [PATCH 20/34] Use most recent cluster state to find types in delete mapping Previously, the cluster state before flushing was used to check which types map the given types pattern. However, this state might not be up to date. Instead use the more recent cluster state from clusterState.state() This fixes a test failure of PercolatorTests.testDeletePercolatorType Other changes: - use BoolFilter instead of OrFilter, because it is faster - throw exception immediately when no type matching the given patterns was found in cluster state --- .../delete/TransportDeleteMappingAction.java | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/TransportDeleteMappingAction.java b/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/TransportDeleteMappingAction.java index 113aba2a97aa8..f55e7149cb47a 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/TransportDeleteMappingAction.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/TransportDeleteMappingAction.java @@ -43,9 +43,11 @@ import org.elasticsearch.common.collect.ImmutableOpenMap; import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; -import org.elasticsearch.index.query.OrFilterBuilder; +import org.elasticsearch.index.Index; +import org.elasticsearch.index.query.BoolFilterBuilder; import org.elasticsearch.index.query.QueryBuilders; import org.elasticsearch.index.query.TypeFilterBuilder; +import org.elasticsearch.indices.TypeMissingException; import org.elasticsearch.node.settings.NodeSettingsService; import org.elasticsearch.threadpool.ThreadPool; import org.elasticsearch.transport.TransportService; @@ -117,18 +119,21 @@ protected void masterOperation(final DeleteMappingRequest request, final Cluster public void onResponse(FlushResponse flushResponse) { // get all types that need to be deleted. - ImmutableOpenMap> result = state.metaData().findMappings( + ImmutableOpenMap> result = clusterService.state().metaData().findMappings( request.indices(), request.types() ); // create OrFilter with type filters within to account for different types - OrFilterBuilder filterBuilder = new OrFilterBuilder(); + BoolFilterBuilder filterBuilder = new BoolFilterBuilder(); Set types = new HashSet(); for (ObjectObjectCursor> typesMeta : result) { for (ObjectObjectCursor type : typesMeta.value) { - filterBuilder.add(new TypeFilterBuilder(type.key)); + filterBuilder.should(new TypeFilterBuilder(type.key)); types.add(type.key); } } + if (types.size() == 0) { + throw new TypeMissingException(new Index("_all"), request.types(), "No index has the type."); + } request.types(types.toArray(new String[types.size()])); QuerySourceBuilder querySourceBuilder = new QuerySourceBuilder() .setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(), filterBuilder)); From 2f115b81035e76a928a4cf6629eb02fbc5175761 Mon Sep 17 00:00:00 2001 From: Britta Weber Date: Wed, 15 Jan 2014 00:22:44 +0100 Subject: [PATCH 21/34] update rest spec to be consistent with recent changes see issue #4071 --- rest-api-spec/api/indices.delete_alias.json | 8 ++++---- rest-api-spec/api/indices.delete_mapping.json | 6 +++--- rest-api-spec/api/indices.delete_warmer.json | 12 ++++++------ rest-api-spec/api/indices.put_alias.json | 5 ++--- rest-api-spec/api/indices.put_mapping.json | 3 +-- rest-api-spec/api/indices.put_warmer.json | 3 +-- 6 files changed, 17 insertions(+), 20 deletions(-) diff --git a/rest-api-spec/api/indices.delete_alias.json b/rest-api-spec/api/indices.delete_alias.json index 03686046820d8..532298866c044 100644 --- a/rest-api-spec/api/indices.delete_alias.json +++ b/rest-api-spec/api/indices.delete_alias.json @@ -7,14 +7,14 @@ "paths": ["/{index}/_alias/{name}", "/{index}/_aliases/{name}"], "parts": { "index": { - "type" : "string", + "type" : "list", "required" : true, - "description" : "The name of the index with an alias" + "description" : "A comma-separated list of index names (supports wildcards); use `_all` for all indices" }, "name": { - "type" : "string", + "type" : "list", "required" : true, - "description" : "The name of the alias to be deleted" + "description" : "A comma-separated list of aliases to delete (supports wildcards); use `_all` to delete all aliases for the specified indices." } }, "params": { diff --git a/rest-api-spec/api/indices.delete_mapping.json b/rest-api-spec/api/indices.delete_mapping.json index 6494b3231d974..3fbe66c0a65b6 100644 --- a/rest-api-spec/api/indices.delete_mapping.json +++ b/rest-api-spec/api/indices.delete_mapping.json @@ -9,12 +9,12 @@ "index": { "type" : "list", "required" : true, - "description" : "A comma-separated list of index names; use `_all` for all indices" + "description" : "A comma-separated list of index names (supports wildcards); use `_all` for all indices" }, "type": { - "type" : "string", + "type" : "list", "required" : true, - "description" : "The name of the document type to delete" + "description" : "A comma-separated list of document types to delete (supports wildcards); use `_all` to delete all document types in the specified indices." } }, "params": { diff --git a/rest-api-spec/api/indices.delete_warmer.json b/rest-api-spec/api/indices.delete_warmer.json index be747cbeeec2b..30357befe02e1 100644 --- a/rest-api-spec/api/indices.delete_warmer.json +++ b/rest-api-spec/api/indices.delete_warmer.json @@ -9,21 +9,21 @@ "index": { "type" : "list", "required" : true, - "description" : "A comma-separated list of index names to register warmer for; use `_all` or empty string to perform the operation on all indices" + "description" : "A comma-separated list of index names to delete warmers from (supports wildcards); use `_all` to perform the operation on all indices." }, "name" : { - "type" : "string", - "description" : "The name of the warmer (supports wildcards); leave empty to delete all warmers" - }, - "type": { "type" : "list", - "description" : "A comma-separated list of document types to register warmer for; use `_all` or empty string to perform the operation on all types" + "description" : "A comma-separated list of warmer names to delete (supports wildcards); use `_all` to delete all warmers in the specified indices. You must specify a name either in the uri or in the parameters." } }, "params": { "master_timeout": { "type" : "time", "description" : "Specify timeout for connection to master" + }, + "name" : { + "type" : "list", + "description" : "A comma-separated list of warmer names to delete (supports wildcards); use `_all` to delete all warmers in the specified indices. You must specify a name either in the uri or in the parameters." } } }, diff --git a/rest-api-spec/api/indices.put_alias.json b/rest-api-spec/api/indices.put_alias.json index db1072e4720a1..cb3172a49c6e2 100644 --- a/rest-api-spec/api/indices.put_alias.json +++ b/rest-api-spec/api/indices.put_alias.json @@ -7,9 +7,8 @@ "paths": ["/{index}/_alias/{name}", "/_alias/{name}", "/{index}/_aliases/{name}", "/_aliases/{name}"], "parts": { "index": { - "type" : "string", - "required" : true, - "description" : "The name of the index with an alias" + "type" : "list", + "description" : "A comma-separated list of index names the alias should point to (supports wildcards); use `_all` or omit to perform the operation on all indices." }, "name": { "type" : "string", diff --git a/rest-api-spec/api/indices.put_mapping.json b/rest-api-spec/api/indices.put_mapping.json index eadba64f3ce4b..09d298e076465 100644 --- a/rest-api-spec/api/indices.put_mapping.json +++ b/rest-api-spec/api/indices.put_mapping.json @@ -8,8 +8,7 @@ "parts": { "index": { "type" : "list", - "required" : false, - "description" : "A comma-separated list of index names; use `_all` to perform the operation on all indices" + "description" : "A comma-separated list of index names the mapping should be added to (supports wildcards); use `_all` or omit to add the mapping on all indices." }, "type": { "type" : "string", diff --git a/rest-api-spec/api/indices.put_warmer.json b/rest-api-spec/api/indices.put_warmer.json index dedb425c585cb..70ca77513e601 100644 --- a/rest-api-spec/api/indices.put_warmer.json +++ b/rest-api-spec/api/indices.put_warmer.json @@ -8,8 +8,7 @@ "parts": { "index": { "type" : "list", - "required" : true, - "description" : "A comma-separated list of index names to register the warmer for; use `_all` or empty string to perform the operation on all indices" + "description" : "A comma-separated list of index names to register the warmer for; use `_all` or omit to perform the operation on all indices" }, "name": { "type" : "string", From f8a427e2668094e3392994275d659cc8add72900 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 14:00:08 +0100 Subject: [PATCH 22/34] [DOCS] Moved fielddata circuit breaker higher up the page --- .../index-modules/fielddata.asciidoc | 64 ++++++++++--------- 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/docs/reference/index-modules/fielddata.asciidoc b/docs/reference/index-modules/fielddata.asciidoc index 3c962ea49b5b1..1c382c662c3da 100644 --- a/docs/reference/index-modules/fielddata.asciidoc +++ b/docs/reference/index-modules/fielddata.asciidoc @@ -24,7 +24,39 @@ field data after a certain time of inactivity. Defaults to `-1`. For example, can be set to `5m` for a 5 minute expiry. |======================================================================= -=== Field data formats +[float] +[[fielddata-circuit-breaker]] +=== Field data circuit breaker +The field data circuit breaker allows Elasticsearch to estimate the amount of +memory a field will required to be loaded into memory. It can then prevent the +field data loading by raising and exception. By default it is configured with +no limit (-1 bytes), but is automatically set to `indices.fielddata.cache.size` +if set. It can be configured with the following parameters: + +[cols="<,<",options="header",] +|======================================================================= +|Setting |Description +|`indices.fielddata.breaker.limit` |Maximum size of estimated field data +to allow loading. Defaults to 80% of the maximum JVM heap. +|`indices.fielddata.breaker.overhead` |A constant that all field data +estimations are multiplied with to determine a final estimation. Defaults to +1.03 +|======================================================================= + +Both the `indices.fielddata.breaker.limit` and +`indices.fielddata.breaker.overhead` can be changed dynamically using the +cluster update settings API. + +[float] +[[fielddata-monitoring]] +=== Monitoring field data + +You can monitor memory usage for field data as well as the field data circuit +breaker using +<> + +[[fielddata-formats]] +== Field data formats The field data format controls how field data should be stored. @@ -237,33 +269,3 @@ The `frequency` and `regex` filters can be combined: } -------------------------------------------------- -[float] -[[field-data-circuit-breaker]] -=== Field data circuit breaker -The field data circuit breaker allows Elasticsearch to estimate the amount of -memory a field will required to be loaded into memory. It can then prevent the -field data loading by raising and exception. By default it is configured with -no limit (-1 bytes), but is automatically set to `indices.fielddata.cache.size` -if set. It can be configured with the following parameters: - -[cols="<,<",options="header",] -|======================================================================= -|Setting |Description -|`indices.fielddata.breaker.limit` |Maximum size of estimated field data -to allow loading. Defaults to 80% of the maximum JVM heap. -|`indices.fielddata.breaker.overhead` |A constant that all field data -estimations are multiplied with to determine a final estimation. Defaults to -1.03 -|======================================================================= - -Both the `indices.fielddata.breaker.limit` and -`indices.fielddata.breaker.overhead` can be changed dynamically using the -cluster update settings API. - -[float] -[[field-data-monitoring]] -=== Monitoring field data - -You can monitor memory usage for field data as well as the field data circuit -breaker using -<> From a0b993e2dc197582668dc4dbbe8e64eb1f4ba53a Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 14:51:18 +0100 Subject: [PATCH 23/34] [DOCS] Tidied up cluster settings docs --- docs/reference/modules/cluster.asciidoc | 65 ++++++++++--------------- 1 file changed, 26 insertions(+), 39 deletions(-) diff --git a/docs/reference/modules/cluster.asciidoc b/docs/reference/modules/cluster.asciidoc index 9d0eefe3076c1..aaad74d62d864 100644 --- a/docs/reference/modules/cluster.asciidoc +++ b/docs/reference/modules/cluster.asciidoc @@ -12,70 +12,57 @@ handling nodes being added or removed. The following settings may be used: `cluster.routing.allocation.allow_rebalance`:: - Allow to control when rebalancing will happen based on the total - state of all the indices shards in the cluster. `always`, - `indices_primaries_active`, and `indices_all_active` are allowed, - defaulting to `indices_all_active` to reduce chatter during + Allow to control when rebalancing will happen based on the total + state of all the indices shards in the cluster. `always`, + `indices_primaries_active`, and `indices_all_active` are allowed, + defaulting to `indices_all_active` to reduce chatter during initial recovery. `cluster.routing.allocation.cluster_concurrent_rebalance`:: - Allow to control how many concurrent rebalancing of shards are + Allow to control how many concurrent rebalancing of shards are allowed cluster wide, and default it to `2`. `cluster.routing.allocation.node_initial_primaries_recoveries`:: - Allow to control specifically the number of initial recoveries - of primaries that are allowed per node. Since most times local - gateway is used, those should be fast and we can handle more of + Allow to control specifically the number of initial recoveries + of primaries that are allowed per node. Since most times local + gateway is used, those should be fast and we can handle more of those per node without creating load. `cluster.routing.allocation.node_concurrent_recoveries`:: - How many concurrent recoveries are allowed to happen on a node. + How many concurrent recoveries are allowed to happen on a node. Defaults to `2`. -added[1.0.0.RC1] - `cluster.routing.allocation.enable`:: Controls shard allocation for all indices, by allowing specific - kinds of shard to be allocated. Can be set to: + kinds of shard to be allocated. + added[1.0.0.RC1,Replaces `cluster.routing.allocation.disable*`] + Can be set to: * `all` (default) - Allows shard allocation for all kinds of shards. * `primaries` - Allows shard allocation only for primary shards. * `new_primaries` - Allows shard allocation only for primary shards for new indices. * `none` - No shard allocations of any kind are allowed for all indices. `cluster.routing.allocation.disable_new_allocation`:: - Allows to disable new primary allocations. Note, this will prevent - allocations for newly created indices. This setting really make - sense when dynamically updating it using the cluster update - settings API. This setting has been deprecated in favour - for `cluster.routing.allocation.enable`. - + deprecated[1.0.0.RC1,Replaced by `cluster.routing.allocation.enable`] `cluster.routing.allocation.disable_allocation`:: - Allows to disable either primary or replica allocation (does not - apply to newly created primaries, see `disable_new_allocation` - above). Note, a replica will still be promoted to primary if - one does not exist. This setting really make sense when - dynamically updating it using the cluster update settings API. - This setting has been deprecated in favour for `cluster.routing.allocation.enable`. + deprecated[1.0.0.RC1,Replaced by `cluster.routing.allocation.enable`] `cluster.routing.allocation.disable_replica_allocation`:: - Allows to disable only replica allocation. Similar to the previous - setting, mainly make sense when using it dynamically using the - cluster update settings API. This setting has been deprecated in - favour for `cluster.routing.allocation.enable`. + deprecated[1.0.0.RC1,Replaced by `cluster.routing.allocation.enable`] `cluster.routing.allocation.same_shard.host`:: Prevents that multiple instances of the same shard are allocated - on a single host. Defaults to `false`. This setting only applies + on a single host. Defaults to `false`. This setting only applies if multiple nodes are started on the same machine. `indices.recovery.concurrent_streams`:: - The number of streams to open (on a *node* level) to recover a - shard from a peer shard. Defaults to `3`. + The number of streams to open (on a *node* level) to recover a + shard from a peer shard. Defaults to `3`. [float] [[allocation-awareness]] @@ -182,8 +169,8 @@ set to `value1` and `value2` by setting [source,js] -------------------------------------------------- curl -XPUT localhost:9200/test/_settings -d '{ - "index.routing.allocation.include.tag" : "value1,value2" -}' + "index.routing.allocation.include.tag" : "value1,value2" +}' -------------------------------------------------- On the other hand, we can create an index that will be deployed on all @@ -193,11 +180,11 @@ nodes except for nodes with a `tag` of value `value3` by setting [source,js] -------------------------------------------------- curl -XPUT localhost:9200/test/_settings -d '{ - "index.routing.allocation.exclude.tag" : "value3" -}' + "index.routing.allocation.exclude.tag" : "value3" +}' -------------------------------------------------- -`index.routing.allocation.require.*` can be used to +`index.routing.allocation.require.*` can be used to specify a number of rules, all of which MUST match in order for a shard to be allocated to a node. This is in contrast to `include` which will include a node if ANY rule matches. @@ -229,7 +216,7 @@ curl -XPUT localhost:9200/test/_settings -d '{ "index.routing.allocation.include.group2" : "yyy", "index.routing.allocation.exclude.group3" : "zzz", "index.routing.allocation.require.group4" : "aaa" -}' +}' -------------------------------------------------- The provided settings can also be updated in real time using the update @@ -246,6 +233,6 @@ address: curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : { "cluster.routing.allocation.exclude._ip" : "10.0.0.1" - } -}' + } +}' -------------------------------------------------- From 353d599d05918dc847ec0753e8851fd80cc2e497 Mon Sep 17 00:00:00 2001 From: Alexander Reelsen Date: Wed, 15 Jan 2014 14:57:00 +0100 Subject: [PATCH 24/34] Increased max map pages for systemd to align with init.d settings in packages --- src/rpm/systemd/sysctl.d/elasticsearch.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/rpm/systemd/sysctl.d/elasticsearch.conf b/src/rpm/systemd/sysctl.d/elasticsearch.conf index daf621bd23ce2..62ea54d869751 100644 --- a/src/rpm/systemd/sysctl.d/elasticsearch.conf +++ b/src/rpm/systemd/sysctl.d/elasticsearch.conf @@ -1 +1 @@ -vm.max_map_count=65535 +vm.max_map_count=262144 From 3062e59f51701a14c82f0dffa35120232b3dd43a Mon Sep 17 00:00:00 2001 From: Lee Hinman Date: Wed, 15 Jan 2014 07:04:24 -0700 Subject: [PATCH 25/34] [DOCS] Fix default setting in circuit breaker documentation --- docs/reference/index-modules/fielddata.asciidoc | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/reference/index-modules/fielddata.asciidoc b/docs/reference/index-modules/fielddata.asciidoc index 1c382c662c3da..c958dcbc037e2 100644 --- a/docs/reference/index-modules/fielddata.asciidoc +++ b/docs/reference/index-modules/fielddata.asciidoc @@ -29,9 +29,9 @@ example, can be set to `5m` for a 5 minute expiry. === Field data circuit breaker The field data circuit breaker allows Elasticsearch to estimate the amount of memory a field will required to be loaded into memory. It can then prevent the -field data loading by raising and exception. By default it is configured with -no limit (-1 bytes), but is automatically set to `indices.fielddata.cache.size` -if set. It can be configured with the following parameters: +field data loading by raising and exception. By default the limit is configured +to 80% of the maximum JVM heap. It can be configured with the following +parameters: [cols="<,<",options="header",] |======================================================================= @@ -268,4 +268,3 @@ The `frequency` and `regex` filters can be combined: } } -------------------------------------------------- - From 93ba3b5e70a59e82dc79754fc84a54ab83dd8595 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 15:09:18 +0100 Subject: [PATCH 26/34] [DOCS] Tidied up layout of setup docs --- docs/reference/setup/as-a-service-win.asciidoc | 3 ++- docs/reference/setup/as-a-service.asciidoc | 8 +++++++- docs/reference/setup/repositories.asciidoc | 2 ++ 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/reference/setup/as-a-service-win.asciidoc b/docs/reference/setup/as-a-service-win.asciidoc index e786741b241ee..f77f194e6f91d 100644 --- a/docs/reference/setup/as-a-service-win.asciidoc +++ b/docs/reference/setup/as-a-service-win.asciidoc @@ -1,7 +1,7 @@ [[setup-service-win]] == Running as a Service on Windows -Windows users can configure Elasticsearch to run as a service to run in the background or start automatically +Windows users can configure Elasticsearch to run as a service to run in the background or start automatically at startup without any user interaction. This can be achieved through `service.bat` script under `bin/` folder which allows one to install, remove, manage or configure the service and potentially start and stop the service, all from the command-line. @@ -47,6 +47,7 @@ The service 'elasticsearch-service-x64' has been installed. NOTE: While a JRE can be used for the Elasticsearch service, due to its use of a client VM (as oppose to a server JVM which offers better performance for long-running applications) its usage is discouraged and a warning will be issued. +[float] === Customizing service settings There are two ways to customize the service settings: diff --git a/docs/reference/setup/as-a-service.asciidoc b/docs/reference/setup/as-a-service.asciidoc index f0e323385b81d..59f2163b92cca 100644 --- a/docs/reference/setup/as-a-service.asciidoc +++ b/docs/reference/setup/as-a-service.asciidoc @@ -3,6 +3,7 @@ In order to run elasticsearch as a service on your operating system, the provided packages try to make it as easy as possible for you to start and stop elasticsearch during reboot and upgrades. +[float] === Linux Currently our build automatically creates a debian package and an RPM package, which is available on the download page. The package itself does not have any dependencies, but you have to make sure that you installed a JDK. @@ -26,6 +27,7 @@ Each package features a configuration file, which allows you to set the followin `ES_JAVA_OPTS`:: Any additional java options you may want to apply. This may be useful, if you need to set the `node.name` property, but do not want to change the `elasticsearch.yml` configuration file, because it is distributed via a provisioning system like puppet or chef. Example: `ES_JAVA_OPTS="-Des.node.name=search-01"` `RESTART_ON_UPGRADE`:: Configure restart on package upgrade, defaults to `false`. This means you will have to restart your elasticsearch instance after installing a package manually. The reason for this is to ensure, that upgrades in a cluster do not result in a continouos shard reallocation resulting in high network traffic and reducing the response times of your cluster. +[float] ==== Debian/Ubuntu The debian package ships with everything you need as it uses standard debian tools like update `update-rc.d` to define the runlevels it runs on. The init script is placed at `/etc/init.d/elasticsearch` is you would expect it. The configuration file is placed at `/etc/default/elasticsearch`. @@ -38,6 +40,7 @@ sudo update-rc.d elasticsearch defaults 95 10 sudo /etc/init.d/elasticsearch start -------------------------------------------------- +[float] ===== Installing the oracle JDK The usual recommendation is to run the Oracle JDK with elasticsearch. However Ubuntu and Debian only ship the OpenJDK due to license issues. You can easily install the oracle installer package though. In case you are missing the `add-apt-repository` command under Debian GNU/Linux, make sure have at least Debian Wheezy and the package `python-software-properties` installed @@ -53,8 +56,10 @@ java -version The last command should verify a successful installation of the Oracle JDK. -==== RPM based distributions +[float] +==== RPM based distributions +[float] ===== Using chkconfig Some RPM based distributions are using `chkconfig` to enable and disable services. The init script is located at `/etc/init.d/elasticsearch`, where as the configuration file is placed at `/etc/sysconfig/elasticsearch`. Like the debian package the RPM package is not started by default after installation, you have to do this manually by entering the following commands @@ -66,6 +71,7 @@ sudo service elasticsearch start -------------------------------------------------- +[float] ===== Using systemd Distributions like SuSe do not use the `chkconfig` tool to register services, but rather `systemd` and its command `/bin/systemctl` to start and stop services (at least in newer versions, otherwise use the `chkconfig` commands above). The configuration file is also placed at `/etc/sysconfig/elasticsearch`. After installing the RPM, you have to change the systemd configuration and then start up elasticsearch diff --git a/docs/reference/setup/repositories.asciidoc b/docs/reference/setup/repositories.asciidoc index 838e3e33ed52d..aa0ae30e7b93d 100644 --- a/docs/reference/setup/repositories.asciidoc +++ b/docs/reference/setup/repositories.asciidoc @@ -6,6 +6,7 @@ We also have repositories available for APT and YUM based distributions. We have split the major versions in separate urls to avoid accidental upgrades across major version. For all 0.90.x releases use 0.90 as version number, for 1.0.x use 1.0, etc. +[float] === APT Download and install the Public Signing Key @@ -25,6 +26,7 @@ deb http://packages.elasticsearch.org/elasticsearch/0.90/debian stable main Run apt-get update and the repository is ready for use. +[float] === YUM Download and install the Public Signing Key From 12a095d797ce31584652671ea4393e1586331ac3 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 16:13:38 +0100 Subject: [PATCH 27/34] [DOCS] Tidied up the multi-indices docs --- docs/reference/api-conventions.asciidoc | 35 +++++++++++++++++-------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/docs/reference/api-conventions.asciidoc b/docs/reference/api-conventions.asciidoc index 091393ec1ae6d..b9e9516e514f3 100644 --- a/docs/reference/api-conventions.asciidoc +++ b/docs/reference/api-conventions.asciidoc @@ -26,17 +26,30 @@ support wildcards, for example: `test*`, and the ability to "add" (`+`) and "remove" (`-`), for example: `+test*,-test3`. All multi indices API support the following url query string parameters: -* `ignore_unavailable` - Controls whether to ignore if any specified indices are unavailable, this includes indices - that don't exist or closed indices. Either `true` or `false` can be specified. -* `allow_no_indices` - Controls whether to fail if a wildcard indices expressions results into no concrete indices. - Either `true` or `false` can be specified. For example if the wildcard expression `foo*` is specified and no indices - are available that start with `foo` then depending on this setting the request will fail. This setting is also applicable - when `_all`, `*` or no index has been specified. -* `expand_wildcards` - Controls to what kind of concrete indices wildcard indices expression expand to. If `open` is - specified then the wildcard expression if expanded to only open indices and if `closed` is specified then the wildcard - expression if expanded only to closed indices. Also both values (`open,closed`) can be specified to expand to all indices. - -The defaults settings for the above parameters dependent on the api being used. + +`ignore_unavailable`:: + +Controls whether to ignore if any specified indices are unavailable, this +includes indices that don't exist or closed indices. Either `true` or `false` +can be specified. + +`allow_no_indices`:: + +Controls whether to fail if a wildcard indices expressions results into no +concrete indices. Either `true` or `false` can be specified. For example if +the wildcard expression `foo*` is specified and no indices are available that +start with `foo` then depending on this setting the request will fail. This +setting is also applicable when `_all`, `*` or no index has been specified. + +`expand_wildcards`:: + +Controls to what kind of concrete indices wildcard indices expression expand +to. If `open` is specified then the wildcard expression if expanded to only +open indices and if `closed` is specified then the wildcard expression if +expanded only to closed indices. Also both values (`open,closed`) can be +specified to expand to all indices. + +The defaults settings for the above parameters depend on the api being used. NOTE: Single index APIs such as the <> and the <> do not support multiple indices. From 4d68d722a13e14b6c548939a36fdcc56f85e3d9b Mon Sep 17 00:00:00 2001 From: Alexander Reelsen Date: Wed, 15 Jan 2014 17:03:30 +0100 Subject: [PATCH 28/34] Added waiting for yellow cluster state in rest test In order to prevent rare timing issue, where the creation of an index happens in the cluster state too late and thus results in empty mappings. --- .../test/indices.get_field_mapping/50_field_wildcards.yaml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml b/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml index fcf592632663e..766be059d7831 100644 --- a/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml +++ b/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml @@ -46,6 +46,10 @@ setup: type: string index_name: t3 + - do: + cluster.health: + wait_for_status: yellow + --- "Get field mapping with * for fields": From baf3bd92240a05e6b8d37375b8be38719cd5716e Mon Sep 17 00:00:00 2001 From: David Pilato Date: Wed, 15 Jan 2014 17:10:11 +0100 Subject: [PATCH 29/34] Don't fail plugin manager test when downloading an actual plugin --- .../java/org/elasticsearch/plugin/PluginManagerTests.java | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/test/java/org/elasticsearch/plugin/PluginManagerTests.java b/src/test/java/org/elasticsearch/plugin/PluginManagerTests.java index 65f1032b18a57..1cb1dea67b1e8 100644 --- a/src/test/java/org/elasticsearch/plugin/PluginManagerTests.java +++ b/src/test/java/org/elasticsearch/plugin/PluginManagerTests.java @@ -227,11 +227,9 @@ private void singlePluginInstallAndRemove(String pluginShortName, String pluginC assertThat(plugins, notNullValue()); assertThat(plugins.length, is(0)); } catch (IOException e) { - logger.warn("--> IOException raised while downloading plugin [{}].", e, pluginShortName); - throw e; + logger.warn("--> IOException raised while downloading plugin [{}]. Skipping test.", e, pluginShortName); } catch (ElasticsearchTimeoutException e) { logger.warn("--> timeout exception raised while downloading plugin [{}]. Skipping test.", pluginShortName); - throw e; } } From faddd66e87d8cbd2466b9e4816a6ea04a728454d Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 17:50:24 +0100 Subject: [PATCH 30/34] [DOCS] Added breaking changes in 1.0 --- docs/reference/index-modules/store.asciidoc | 1 + docs/reference/index.asciidoc | 4 + docs/reference/migration/migrate_1_0.asciidoc | 345 ++++++++++++++++++ 3 files changed, 350 insertions(+) create mode 100644 docs/reference/migration/migrate_1_0.asciidoc diff --git a/docs/reference/index-modules/store.asciidoc b/docs/reference/index-modules/store.asciidoc index 247b36451ac83..8388ee2d5df78 100644 --- a/docs/reference/index-modules/store.asciidoc +++ b/docs/reference/index-modules/store.asciidoc @@ -81,6 +81,7 @@ Lucene `NIOFSDirectory`) using NIO. It allows multiple threads to read from the same file concurrently. It is not recommended on Windows because of a bug in the SUN Java implementation. +[[mmapfs]] [float] ==== MMap FS diff --git a/docs/reference/index.asciidoc b/docs/reference/index.asciidoc index fc2cb541813f8..1ae7a20dc4f4f 100644 --- a/docs/reference/index.asciidoc +++ b/docs/reference/index.asciidoc @@ -3,6 +3,8 @@ include::setup.asciidoc[] +include::migration/migrate_1_0.asciidoc[] + include::api-conventions.asciidoc[] include::docs.asciidoc[] @@ -29,3 +31,5 @@ include::testing.asciidoc[] include::glossary.asciidoc[] + + diff --git a/docs/reference/migration/migrate_1_0.asciidoc b/docs/reference/migration/migrate_1_0.asciidoc new file mode 100644 index 0000000000000..c74f7eafed0f1 --- /dev/null +++ b/docs/reference/migration/migrate_1_0.asciidoc @@ -0,0 +1,345 @@ +[[breaking-changes]] += Breaking changes in 1.0 + +This section discusses the changes that you need to be aware of when migrating +your application to Elasticsearch 1.0: + +== System and settings + +* Elasticsearch now runs in the foreground by default. There is no more `-f` + flag on the command line. Instead, to run elasticsearch as a daemon, use + the `-d` flag: + +[source,sh] +--------------- +./bin/elasticsearch -d +--------------- + +* Command line settings can now be passed without the `-Des.` prefix, for + instance: + +[source,sh] +--------------- +./bin/elasticsearch --node.name=search_1 --cluster.name=production +--------------- + +* Elasticsearch on 64 bit Linux now uses <> by default. Make + sure that you set <> to a sufficiently high + number. The RPM and Debian packages default this value to `262144`. + +* The RPM and Debian packages no longer start Elasticsearch by default. + +* The `cluster.routing.allocation` settings (`disable_allocation`, + `disable_new_allocation` and `disable_replica_location`) have been + <>: + +[source,yaml] +--------------- +cluster.routing.allocation.enable: all|primaries|new_primaries|none +--------------- + +== Stats and Info APIs + +The <>, <>, +<> and <> +APIs have all been changed to make their format more RESTful and less clumsy. + +For instance, if you just want the `nodes` section of the the `cluster_state`, +instead of: + +[source,sh] +--------------- +GET /_cluster/state?filter_metadata&filter_routing_table&filter_blocks +--------------- + +you now use: + +[source,sh] +--------------- +GET /_cluster/state/nodes +--------------- + +Simliarly for the `nodes_stats` API, if you want the `transport` and `http` +metrics only, instead of: + +[source,sh] +--------------- +GET /_nodes/stats?clear&transport&http +--------------- + +you now use: + +[source,sh] +--------------- +GET /_nodes/stats/transport,http +--------------- + +See the links above for full details. + + +== Indices APIs + +The `mapping`, `alias`, `settings`, and `warmer` index APIs are all similar +but there are subtle differences in the order of the URL and the response +body. For instance, adding a mapping and a warmer look slightly different: + +[source,sh] +--------------- +PUT /{index}/{type}/_mapping +PUT /{index}/_warmer/{name} +--------------- + +These URLs have been unified as: + +[source,sh] +--------------- +PUT /{indices}/_mapping/{type} +PUT /{indices}/_alias/{name} +PUT /{indices}/_warmer/{name} + +GET /{indices}/_mapping/{types} +GET /{indices}/_alias/{names} +GET /{indices}/_settings/{names} +GET /{indices}/_warmer/{names} + +DELETE /{indices}/_mapping/{types} +DELETE /{indices}/_alias/{names} +DELETE /{indices}/_warmer/{names} +--------------- + +All of the `{indices}`, `{types}` and `{names}` parameters can be replaced by: + + * `_all`, `*` or blank (ie left out altogether), all of which mean ``all'' + * wildcards like `test*` + * comma-separated lists: `index_1,test_*` + +The only exception is `DELETE` which doesn't accept blank (missing) +parameters. If you want to delete something, you should be specific. + +Similarly, the return values for `GET` have been unified with the following +rules: + +* Only return values that exist. If you try to `GET` a mapping which doesn't + exist, then the result will be an empty object: `{}`. We no longer throw a + `404` if the requested mapping/warmer/alias/setting doesn't exist. + +* The response format always has the index name, then the section, then the + element name, for instance: + +[source,json] +--------------- +{ + "my_index": { + "mappings": { + "my_type": {...} + } + } +} +--------------- ++ +This is a breaking change for the `get_mapping` API. + +In the future we will also provide plural versions to allow putting multiple mappings etc in a single request. + +See <>, <>, <>, +<>, +<>, <>, +<>, and <> for more details. + +== Index request + +Previously a document could be indexed as itself, or wrapped in an outer +object which specified the `type` name: + +[source,json] +--------------- +PUT /my_index/my_type/1 +{ + "my_type": { + ... doc fields ... + } +} +--------------- + +This led to some ambiguity when a document also included a field with the same +name as the `type`. We no longer accept the outer `type` wrapper, but this +behaviour can be reenabled on an index-by-index basis with the setting: +`index.mapping.allow_type_wrapper`. + +== Search requests + +While the `search` API takes a top-level `query` parameter, the +<>, <> and +<> requests expected the whole body to be a +query. These have been changed to all accept a top-level `query` parameter: + +[source,json] +--------------- +GET /_count +{ + "query": { + "match": { + "title": "Interesting stuff" + } + } +} +--------------- + +Also, the top-level `filter` parameter in search has been renamed to +<>, to indicate that it should not +be used as the primary way to filter search results (use a +<> instead), but only to filter +results AFTER facets/aggregations have been calculated. + +This example counts the top colors in all matching docs, but only returns docs +with color `red`: + +[source,json] +--------------- +GET /_search +{ + "query": { + "match_all": {} + }, + "aggs": { + "colors": { + "terms": { "field": "color" } + } + }, + "post_filter": { + "term": { + "color": "red" + } + } +} +--------------- + +== Multi-fields + +Multi-fields are dead! Long live multi-fields! Well, the field type +`multi_field` has been removed. Instead, any of the core field types +(excluding `object` and `nested`) now accept a `fields` parameter. It's the +same thing, but nicer. Instead of: + +[source,json] +--------------- +"title": { + "type": "multi_field", + "fields": { + "title": { "type": "string" }, + "raw": { "type": "string", "index": "not_analyzed" } + } +} +--------------- + +you can now write: + +[source,json] +--------------- +"title": { + "type": "string"; + "fields": { + "raw": { "type": "string", "index": "not_analyzed" } + } +} +--------------- + +Existing multi-fields will be upgraded to the new format automatically. + +== Stopwords + +Previously, the <> and +<> analyzers used the list of English stopwords +by default, which caused some hard to debug indexing issues. Now they are set to +use the empty stopwords list (ie `_none_`) instead. + +== Dates without years + +When dates are specified without a year, for example: `Dec 15 10:00:00` they +are treated as dates in 2000 during indexing and range searches... except for +the upper included bound `lte` where they were treated as dates in 1970! Now, +all https://github.com/elasticsearch/elasticsearch/issues/4451[dates without years] +use `1970` as the default. + +== Parameters + +* Geo queries used to use `miles` as the default unit. And we all know what + happened at NASA because of that decision. The new default unit is + https://github.com/elasticsearch/elasticsearch/issues/4515[`meters`]. + +* For all queries that support _fuzziness_, the `min_similarity`, `fuzziness` + and `edit_distance` parameters have been unified as the single parameter + `fuzziness`. See <> for details of accepted values. + +* The `ignore_missing` parameter has been replaced by the `expand_wildcards`, + `ignore_unavailable` and `allow_no_indices` parameters, all of which have + sensible defaults. See <> for more. + +* An index name (or pattern) is now required for destructive operations like + deleting indices: + +[source,sh] +--------------- +# v0.90 - delete all indices: +DELETE / + +# v1.0 - delete all indices: +DELETE /_all +DELETE /* +--------------- ++ +Setting `action.destructive_requires_name` to `true` provides further safety +by disabling wildcard expansion on destructive actions. + +== Return values + +* The `ok` return value has been removed from all response bodies as it added + no useful information. + +* The `found`, `not_found` and `exists` return values have been unified as + `found` on all relevant APIs. + +* Field values, in response to the <> + parameter, are now always returned as arrays. A field could have single or + multiple values, which meant that sometimes they were returned as scalars + and sometimes as arrays. By always returning arrays, this simplifies user + code. The only exception to this rule is when `fields` is used to retrieve + metadata like the `routing` value, which are always singular. Metadata + fields are always returned as scalars. + +* Settings, like `index.analysis.analyzer.default` are now returned as proper + nested JSON objects, which makes them easier to work with programatically: + +[source,json] +--------------- +{ + "index": { + "analysis": { + "analyzer": { + "default": xxx + } + } + } +} +--------------- ++ +You can choose to return them in flattened format by passing `?flat_settings` +in the query string. + +* The <> API no longer supports the text response + format, but does support JSON and YAML. + +== Deprecations + +* The `text` query has been removed. Use the + <> query instead. + +* The `field` query has been removed. Use the + <> query instead. + +* Per-document boosting with the <> field has + been removed. You can use the + <> instead. + + From 9e3f527721d7f3773b16315ef35e7cbb3e747ea4 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 18:00:13 +0100 Subject: [PATCH 31/34] [DOCS] Fixed asciidoc issue --- docs/reference/migration/migrate_1_0.asciidoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/reference/migration/migrate_1_0.asciidoc b/docs/reference/migration/migrate_1_0.asciidoc index c74f7eafed0f1..9972dfc1fae0b 100644 --- a/docs/reference/migration/migrate_1_0.asciidoc +++ b/docs/reference/migration/migrate_1_0.asciidoc @@ -1,8 +1,11 @@ [[breaking-changes]] = Breaking changes in 1.0 +[partintro] +-- This section discusses the changes that you need to be aware of when migrating your application to Elasticsearch 1.0: +-- == System and settings From c6155c5142f7995e054f9f3b7d82f923dd3620bc Mon Sep 17 00:00:00 2001 From: Alexander Reelsen Date: Wed, 15 Jan 2014 17:02:22 +0000 Subject: [PATCH 32/34] release [1.0.0.RC1] --- docs/reference/analysis/analyzers/pattern-analyzer.asciidoc | 2 +- pom.xml | 2 +- src/main/java/org/elasticsearch/Version.java | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc b/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc index 12f8799f8f962..424ecfa766a19 100644 --- a/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc @@ -14,7 +14,7 @@ type: |`pattern` |The regular expression pattern, defaults to `\W+`. |`flags` |The regular expression flags. |`stopwords` |A list of stopwords to initialize the stop filter with. -Defaults to an 'empty' stopword list coming[1.0.0.RC1, Previously +Defaults to an 'empty' stopword list added[1.0.0.RC1, Previously defaulted to the English stopwords list]. Check <> for more details. |=================================================================== diff --git a/pom.xml b/pom.xml index cab82e5d666ac..eab765b537eee 100644 --- a/pom.xml +++ b/pom.xml @@ -6,7 +6,7 @@ 4.0.0 org.elasticsearch elasticsearch - 1.0.0.RC1-SNAPSHOT + 1.0.0.RC1 jar Elasticsearch - Open Source, Distributed, RESTful Search Engine 2009 diff --git a/src/main/java/org/elasticsearch/Version.java b/src/main/java/org/elasticsearch/Version.java index b65b67e75f28f..d14a9f95a23e7 100644 --- a/src/main/java/org/elasticsearch/Version.java +++ b/src/main/java/org/elasticsearch/Version.java @@ -148,7 +148,7 @@ public class Version implements Serializable { public static final int V_1_0_0_Beta2_ID = /*00*/1000002; public static final Version V_1_0_0_Beta2 = new Version(V_1_0_0_Beta2_ID, false, org.apache.lucene.util.Version.LUCENE_46); public static final int V_1_0_0_RC1_ID = /*00*/1000051; - public static final Version V_1_0_0_RC1 = new Version(V_1_0_0_RC1_ID, true, org.apache.lucene.util.Version.LUCENE_46); + public static final Version V_1_0_0_RC1 = new Version(V_1_0_0_RC1_ID, false, org.apache.lucene.util.Version.LUCENE_46); public static final Version CURRENT = V_1_0_0_RC1; From 3d4891321b38d19d42924566fbfc12ae1ebb1788 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 18:22:56 +0100 Subject: [PATCH 33/34] [DOCS] Minor changes to the breaking changes doc --- docs/reference/migration/migrate_1_0.asciidoc | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/reference/migration/migrate_1_0.asciidoc b/docs/reference/migration/migrate_1_0.asciidoc index 9972dfc1fae0b..9ea28ea2abb42 100644 --- a/docs/reference/migration/migrate_1_0.asciidoc +++ b/docs/reference/migration/migrate_1_0.asciidoc @@ -4,7 +4,7 @@ [partintro] -- This section discusses the changes that you need to be aware of when migrating -your application to Elasticsearch 1.0: +your application to Elasticsearch 1.0. -- == System and settings @@ -267,8 +267,9 @@ use `1970` as the default. == Parameters -* Geo queries used to use `miles` as the default unit. And we all know what - happened at NASA because of that decision. The new default unit is +* Geo queries used to use `miles` as the default unit. And we + http://en.wikipedia.org/wiki/Mars_Climate_Orbiter[all know what + happened at NASA] because of that decision. The new default unit is https://github.com/elasticsearch/elasticsearch/issues/4515[`meters`]. * For all queries that support _fuzziness_, the `min_similarity`, `fuzziness` From 4643f7809872b86f0af033a4842658327cf6729d Mon Sep 17 00:00:00 2001 From: Igor Motov Date: Wed, 15 Jan 2014 12:57:05 -0500 Subject: [PATCH 34/34] [DOCS] Add documentation for URL repository --- docs/reference/modules/snapshots.asciidoc | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/reference/modules/snapshots.asciidoc b/docs/reference/modules/snapshots.asciidoc index 94bf40c4ab58c..38db497c20f06 100644 --- a/docs/reference/modules/snapshots.asciidoc +++ b/docs/reference/modules/snapshots.asciidoc @@ -71,6 +71,18 @@ on all data and master nodes. The following settings are supported: using size value notation, i.e. 1g, 10m, 5k. Defaults to `null` (unlimited chunk size). +[float] +===== Read-only URL Repository + +The URL repository (`"type": "url"`) can be used as an alternative read-only way to access data created by shared file +system repository is using shared file system to store snapshot. The URL specified in the `url` parameter should +point to the root of the shared filesystem repository. The following settings are supported: + +[horizontal] +`url`:: Location of the snapshots. Mandatory. +`concurrent_streams`:: Throttles the number of streams (per node) preforming snapshot operation. Defaults to `5` + + [float] === Snapshot @@ -170,7 +182,7 @@ http://docs.oracle.com/javase/6/docs/api/java/util/regex/Matcher.html#appendRepl ----------------------------------- $ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -d '{ "indices": "index_1,index_2", - "ignore_unavailable": "missing", + "ignore_unavailable": "true", "include_global_state": false, "rename_pattern": "index_(.)+", "rename_replacement": "restored_index_$1"