[Backport 2.x] Supporting sparse semantic retrieval in neural search #343
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![opensearch-trigger-bot[bot]](https://avatars.githubusercontent.com/u/80134844?s=60&v=4){kind=small}
* sparse mapper field and query builder Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix typo Signed-off-by: zhichao-aws <zhichaog@amazon.com> * Add map result support in neural search for non text embedding models Signed-off-by: zane-neo <zaniu@amazon.com> * Fix compilation failure issue Signed-off-by: zane-neo <zaniu@amazon.com> * Add more UTs Signed-off-by: zane-neo <zaniu@amazon.com> * add sparse encoding processor Signed-off-by: xinyual <xinyual@amazon.com> * add sparse encoding processor Signed-off-by: xinyual <xinyual@amazon.com> * remove guava in gradle Signed-off-by: xinyual <xinyual@amazon.com> * modify access control Signed-off-by: xinyual <xinyual@amazon.com> * Add map result support in neural search for non text embedding models Signed-off-by: zane-neo <zaniu@amazon.com> * Fix compilation failure issue Signed-off-by: zane-neo <zaniu@amazon.com> * change output logic Signed-off-by: xinyual <xinyual@amazon.com> * create abstract Signed-off-by: xinyual <xinyual@amazon.com> * create abstract proccesor Signed-off-by: xinyual <xinyual@amazon.com> * add abstract class Signed-off-by: xinyual <xinyual@amazon.com> * remove duplicate code Signed-off-by: xinyual <xinyual@amazon.com> * remove duplicate code Signed-off-by: xinyual <xinyual@amazon.com> * remove dl process Signed-off-by: xinyual <xinyual@amazon.com> * move static to abstract class Signed-off-by: xinyual <xinyual@amazon.com> * update query rewrite logic Signed-off-by: zhichao-aws <zhichaog@amazon.com> * modify header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * merge conflict Signed-off-by: xinyual <xinyual@amazon.com> * delete index mapper, change to rank_features Signed-off-by: zhichao-aws <zhichaog@amazon.com> * remove unused import Signed-off-by: zhichao-aws <zhichaog@amazon.com> * list return result Signed-off-by: zhichao-aws <zhichaog@amazon.com> * refactor type and listTypeNestedMapKey, tidy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * forbid nested input. tidy. Signed-off-by: zhichao-aws <zhichaog@amazon.com> * tidy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * enable nested Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * Add ut it to sparse encoding processor (#6) * fix original UT problem Signed-off-by: xinyual <xinyual@amazon.com> * add UT IT Signed-off-by: xinyual <xinyual@amazon.com> * add more UT Signed-off-by: xinyual <xinyual@amazon.com> * add more ut Signed-off-by: xinyual <xinyual@amazon.com> * fix typo error Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> * utils, tidy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * rename to sparse_encoding query Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add validation and ut Signed-off-by: zhichao-aws <zhichaog@amazon.com> * sparse encoding query builder ut Signed-off-by: zhichao-aws <zhichaog@amazon.com> * rename Signed-off-by: zhichao-aws <zhichaog@amazon.com> * UT for utils Signed-off-by: zhichao-aws <zhichaog@amazon.com> * enrich sparse encoding IT mappings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add it Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add it Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add integ test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * rename resource file Signed-off-by: zhichao-aws <zhichaog@amazon.com> * tidy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * remove BoundedLinearQuery and TokenScoreUpperBound Signed-off-by: zhichao-aws <zhichaog@amazon.com> * tidy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add delta to loose the equal Signed-off-by: zhichao-aws <zhichaog@amazon.com> * move SparseEncodingQueryBuilder to upper level path Signed-off-by: zhichao-aws <zhichaog@amazon.com> * tidy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add it Signed-off-by: zhichao-aws <zhichaog@amazon.com> * Update src/main/java/org/opensearch/neuralsearch/ml/MLCommonsClientAccessor.java Co-authored-by: zane-neo <zaniu@amazon.com> Signed-off-by: zhichao-aws <zhichaog@amazon.com> * Update src/main/java/org/opensearch/neuralsearch/util/TokenWeightUtil.java Co-authored-by: zane-neo <zaniu@amazon.com> Signed-off-by: zhichao-aws <zhichaog@amazon.com> * restore gradle.propeties Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add release notes Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change field modifier to private for NLPProcessor Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add comments Signed-off-by: zhichao-aws <zhichaog@amazon.com> * use StringUtils to check Signed-off-by: zhichao-aws <zhichaog@amazon.com> * null check Signed-off-by: zhichao-aws <zhichaog@amazon.com> * modify changelog Signed-off-by: zhichao-aws <zhichaog@amazon.com> * nit Signed-off-by: zhichao-aws <zhichaog@amazon.com> * nit Signed-off-by: zhichao-aws <zhichaog@amazon.com> * remove query tokens from user interface Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * tidy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * update function name Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add javadoc Signed-off-by: zhichao-aws <zhichaog@amazon.com> * remove debug log including inference result Signed-off-by: zhichao-aws <zhichaog@amazon.com> * make query text and model id required Signed-off-by: zhichao-aws <zhichaog@amazon.com> * minor changes based on comments Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add locale to String.format Signed-off-by: zhichao-aws <zhichaog@amazon.com> * update mock model url Signed-off-by: zhichao-aws <zhichaog@amazon.com> --------- Signed-off-by: zhichao-aws <zhichaog@amazon.com> Signed-off-by: zane-neo <zaniu@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com> Co-authored-by: zane-neo <zaniu@amazon.com> Co-authored-by: xinyual <xinyual@amazon.com> (cherry picked from commit 7bef7a0)
opensearch-trigger-bot
bot
requested review from
heemin32,
navneet1v,
VijayanB,
vamshin,
jmazanec15,
naveentatikonda,
junqiu-lei,
martin-gaievski,
sean-zheng-amazon,
model-collapse,
wujunshen,
zane-neo,
ylwu-amzn and
jngz-es
as code owners
zane-neo
approved these changes
Sep 27, 2023
5 tasks
* fix apache http version Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add import Signed-off-by: zhichao-aws <zhichaog@amazon.com> --------- Signed-off-by: zhichao-aws <zhichaog@amazon.com>
|
waiting opensearch-project/ml-commons#1398 get merged to run sparse encoding integ test |
model-collapse
approved these changes
Sep 27, 2023
| listener.onResponse(vector); | ||
| }, e -> { | ||
| if (RetryUtil.shouldRetry(e, retryTime)) { | ||
| final int retryTimeAdd = retryTime + 1; | ||
| inferenceSentencesWithRetry(targetResponseFilters, modelId, inputText, retryTimeAdd, listener); | ||
| retryableInferenceSentencesWithVectorResult(targetResponseFilters, modelId, inputText, retryTimeAdd, listener); |
There was a problem hiding this comment.
Why you name function like that? An function should be a verb phrase.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport 7bef7a0 from #333