Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensures watch definitions are valid json #30692

Closed
wants to merge 5 commits into from
Closed

Ensures watch definitions are valid json #30692

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
36d0c12
Ensures watch definitions are valid json
May 17, 2018
6da9e8b
address reviewer feedback
May 17, 2018
7252b00
address reviewer feedback
May 17, 2018
9a4db9f
address reviewer feedback
tomcallahan May 23, 2018
dcd9b86
Merge remote-tracking branch 'origin/master' into ensure-watch-valid-…
tomcallahan May 24, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions x-pack/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/watch/WatchParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
*/
package org.elasticsearch.xpack.watcher.watch;

import com.fasterxml.jackson.core.JsonParseException;
import org.elasticsearch.ElasticsearchParseException;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.bytes.BytesReference;
Expand Down Expand Up @@ -174,6 +175,19 @@ public Watch parse(String id, boolean includeStatus, WatcherXContentParser parse
throw new ElasticsearchParseException("could not parse watch [{}]. unexpected field [{}]", id, currentFieldName);
}
}

try {
/* Make sure we are at the end of the available input data -- certain types of JSON errors will not manifest
until we try to consume additional tokens.
*/
if (parser.nextToken() != null) {
throw new ElasticsearchParseException("Received non-null token beyond end of watch definition!");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we set up this exception with all the information it needs, so we do not need to catch it and wrap below? It needlessly creates extra stack frames that must be read (by a human).

}
} catch (JsonParseException|ElasticsearchParseException e) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't catch JsonParseException in the loop condition when calling parser.nextToken(), do we really need to here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough -- I interpreted a desire to bubble up ElasticParseExceptions where possible, but if it is fine to just let that go up that makes sense.

throw new ElasticsearchParseException("could not parse watch [{}]. unexpected data beyond [line: {}, column: {}]",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor nit: maybe we can come up with a more explaining error message here, as unexpected data can mean anything - however I don't have a smarter idea as well to be honest. Is it unparseable/unknown or just too much data/unneeded additional data?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spinscale Actually it is this code that enables the JsonParseException for the original test case. Ryan asked that the code for the test case be changed, which resulted in a different error path. However, the key factor is that we now attempt to grab the next token beyond what we believe to be the last token of the watch definition. A previous revision of this code made that fact clear, at the expense of some more complex code.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea I think what @spinscale is saying is that he would like the error message changed. Maybe expected end of payload/data/json but received additional data

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hub-cap I was trying to respond to @spinscale's comment above in the LGTM -- I'll rework the error message

e, id, parser.getTokenLocation().lineNumber, parser.getTokenLocation().columnNumber);
}

if (trigger == null) {
throw new ElasticsearchParseException("could not parse watch [{}]. missing required field [{}]", id,
WatchField.TRIGGER.getPreferredName());
Expand Down
34 changes: 34 additions & 0 deletions x-pack/plugin/watcher/src/test/java/org/elasticsearch/xpack/watcher/watch/WatchTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import org.elasticsearch.action.support.WriteRequest;
import org.elasticsearch.client.Client;
import org.elasticsearch.common.ParseField;
import org.elasticsearch.common.bytes.BytesArray;
import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.logging.Loggers;
import org.elasticsearch.common.settings.Settings;
Expand All @@ -19,6 +20,7 @@
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.common.xcontent.json.JsonXContent;
import org.elasticsearch.index.query.MatchAllQueryBuilder;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.query.ScriptQueryBuilder;
Expand Down Expand Up @@ -296,6 +298,38 @@ public void testParserBadActions() throws Exception {
}
}

public void testParserConsumesEntireDefinition() throws Exception {
WatchParser wp = createWatchparser();
String watchDef="{\n" +
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typically we use XContentBuilder to build the json (see the test below for an example).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WatcherXContentParser is not different -- the real issue is that we do not consume enough tokens to encounter an actual error. The real crux of this PR is line 183 -- consuming an additional token beyond what we "think" is the end of the watch definition to ensure that we have looked at the whole thing.

In the example test, we have an extra closing curly brace. Up to and including that curly brace, the json is invalid -- we have to consume beyond the erroneous end brace to get the error.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re: XContentBuilder -- XContentBuilder won't let me build an invalid JSON.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use XContentBuilder to build valid json, and then append an extra token (eg comma as you have). This will ensure there isn't malformed json in the middle which tricks the test.

" \"trigger\": {\n" +
" \"schedule\": {\n" +
" \"interval\": \"10s\"\n" +
" }\n" +
" },\n" +
" \"input\": {\n" +
" \"simple\": {}\n" +
" }},\n" + /* NOTICE EXTRA CLOSING CURLY BRACE */
" \"condition\": {\n" +
" \"script\": {\n" +
" \"inline\": \"return false\"\n" +
" }\n" +
" },\n" +
" \"actions\": {\n" +
" \"logging\": {\n" +
" \"logging\": {\n" +
" \"text\": \"{{ctx.payload}}\"\n" +
" }\n" +
" }\n" +
" }\n" +
"}";
try {
wp.parseWithSecrets("failure", false, new BytesArray(watchDef), new DateTime(), XContentType.JSON, true);
fail("This watch should fail to parse as there is an extra closing curly brace in the middle of the watch");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use expectThrows instead of try/fail/catch

} catch (ElasticsearchParseException pe) {
assertThat(pe.getMessage().contains("unexpected data beyond"), is(true));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can use the containsString matcher instead of a boolean assert (the output on failure is much nicer).

}
}

public void testParserDefaults() throws Exception {
Schedule schedule = randomSchedule();
ScheduleRegistry scheduleRegistry = registry(schedule);
Expand Down