Improve include/exclude clause list speed and scalability
#11188
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…of clauses fail. The underlying automaton-backed implementation throws an error if there are too many states. In these circumstances this fix falls-back to using an implementation based on Set lookups for the excluded terms. If the global-ordinals execution mode is in effect this implementation also addresses the slowness identified in issue 11181 which is caused by traversing the TermsEnum - instead the excluded terms’ global ordinals are looked up individually and unset the bits of acceptable terms. This is significantly faster. Closes #11176
| // so we let it try and fall-back to Set based lookups if too | ||
| // complex. | ||
| return new TermListBackedStringFilter(includeValues, excludeValues); | ||
| } |
There was a problem hiding this comment.
I'm wondering that maybe we should directly use TermListBackedStringFilter in that case: I wouldn't expect automatons to speed things up a lot and having TermListBackedStringFilter only in this catch block makes it hard to test.
There was a problem hiding this comment.
Even it we remove that logic, it would be good though to keep the explanation that we don't use an automaton for the term list case in order not to blow up the maximum number of states.
|
Just left one question. Otherwise it looks good to me. |
…xclude clauses that include regexes. Otherwise the TermListBackedStringFilter is used for string-based terms
|
Thanks for review. Changed to only use AutomatonBackedStringFilter for regex cases. Performance still good. |
|
LGTM |
|
Pushed to master via caf7235 |
clintongormley
changed the title
include/exclude clause list speed and scalability
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the terms aggregations' IncludeExclude class the underlying automaton-backed implementation throws an error if there are too many states.
In these circumstances this fix falls-back to using an implementation based on Set lookups for the excluded terms.
If the global-ordinals execution mode is in effect this implementation also addresses the slowness identified in issue 11181 which is caused by traversing the TermsEnum - instead the excluded terms’ global ordinals are looked up individually and unset the bits of acceptable terms. This is significantly faster for high cardinality fields.
Closes #11176