Support script score when doc value is disabled #1573

bugmakerrrrrr · 2024-03-21T16:37:13Z

Description

Today, script score is not supported if using lucene engine and doc value is disabled, because k-NN plugin uses doc value to execute exact query with script score. But actually, we can use ByteVectorValues/FloatVectorValues to execute this type of query.

In addition to , I think that we can totally disable doc value when using lucene engine, vector values can replace the function of doc value. Further, we can use FlatVectorsFormat to store vector values and implement our own native index functionality, so that we can avoid extra de/serialization and unify the script doc value interface. Any thoughts?

Btw, This PR has not added relevant tests yet.

Issues Resolved

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

bugmakerrrrrr · 2024-03-21T16:38:58Z

Currently, if we disable the doc value and use script score, the following exception will be thrown:

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_state_exception",
                "reason": "unexpected docvalues type NONE for field 'item_vector' (expected=BINARY). Re-index with correct docvalues type."
            }
        ],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
            {
                "shard": 0,
                "index": "products-shirts",
                "node": "M-rVnb_0SiqtyJw2-04XNw",
                "reason": {
                    "type": "illegal_state_exception",
                    "reason": "unexpected docvalues type NONE for field 'item_vector' (expected=BINARY). Re-index with correct docvalues type."
                }
            }
        ],
        "caused_by": {
            "type": "illegal_state_exception",
            "reason": "unexpected docvalues type NONE for field 'item_vector' (expected=BINARY). Re-index with correct docvalues type.",
            "caused_by": {
                "type": "illegal_state_exception",
                "reason": "unexpected docvalues type NONE for field 'item_vector' (expected=BINARY). Re-index with correct docvalues type."
            }
        }
    },
    "status": 500
}

navneet1v · 2024-03-21T23:38:37Z

Further, we can use FlatVectorsFormat to store vector values and implement our own native index functionality, so that we can avoid extra de/serialization and unify the script doc value interface.

This is a great idea. There is a github issue created for the same. There some challenges around the same. Ref: #1087 (comment)

navneet1v · 2024-03-21T23:39:00Z

@bugmakerrrrrr can you fix the gh actions.

navneet1v · 2024-03-21T23:41:02Z

I think that we can totally disable doc value when using lucene engine

Yes we should do this if possible. @jmazanec15 what your thought?

src/main/java/org/opensearch/knn/index/KNNVectorDVLeafFieldData.java

navneet1v · 2024-03-22T02:27:08Z

I did a very high level look on the PR, the logic seems to be pretty neat. But please remove all sysout statements and add docs on public functions and new interfaces which are created.

Along with fix the gh actions.

jmazanec15 · 2024-03-25T02:07:07Z

Further, we can use FlatVectorsFormat to store vector values and implement our own native index functionality, so that we can avoid extra de/serialization and unify the script doc value interface.

Right, #1087 calls this out. I think it makes sense to implement our own KnnVectorsFormat that uses the correct FlatVectorsFormat.

Edit: I meant at some point, not necessarily right now.

jmazanec15 · 2024-03-25T02:07:35Z

Yes we should do this if possible. @jmazanec15 what your thought?

Yes, I agree this can be disabled - no sense in storing both.

jmazanec15 · 2024-03-25T13:45:41Z

This looks pretty good to me. I think we will need to add a few test cases.

navneet1v · 2024-03-25T17:11:39Z

I think that we can totally disable doc value when using lucene engine

On this I would think we should do some BWC to see if for an index we suddenly stop sending doc values will it cause any issue.

@bugmakerrrrrr can we add a BWC tests for the same.

bugmakerrrrrr · 2024-03-26T03:28:46Z

I think that we can totally disable doc value when using lucene engine

On this I would think we should do some BWC to see if for an index we suddenly stop sending doc values will it cause any issue.

@bugmakerrrrrr can we add a BWC tests for the same.

@navneet1v Of course. But in this PR, I will focus on supporting script score, and open another PR to disable doc value.

codecov · 2024-03-27T05:59:07Z

Codecov Report

Attention: Patch coverage is 90.24390% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 84.97%. Comparing base (c861966) to head (46e9883).

Files	Patch %	Lines
...opensearch/knn/index/KNNVectorDVLeafFieldData.java	76.92%	2 Missing and 1 partial ⚠️
...opensearch/knn/index/KNNVectorScriptDocValues.java	96.42%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1573      +/-   ##
============================================
+ Coverage     84.96%   84.97%   +0.01%     
- Complexity     1366     1374       +8     
============================================
  Files           172      172              
  Lines          5566     5599      +33     
  Branches        546      553       +7     
============================================
+ Hits           4729     4758      +29     
- Misses          605      607       +2     
- Partials        232      234       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

navneet1v

Overall code looks good to me.

src/main/java/org/opensearch/knn/index/KNNVectorDVLeafFieldData.java

src/test/java/org/opensearch/knn/plugin/script/KNNScriptScoringIT.java

jmazanec15

LGTM

Signed-off-by: panguixin <panguixin@bytedance.com>

navneet1v

Looks good to me. Thanks for fixing the tests.

navneet1v · 2024-03-28T18:30:37Z

@jmazanec15 need approval from your side too on the PR.

opensearch-trigger-bot · 2024-03-28T20:01:10Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-1573-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 771c4b54a74b7c4406c71a8bf758378329cfe4d5
# Push it to GitHub
git push --set-upstream origin backport/backport-1573-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1573-to-2.x.

navneet1v · 2024-03-28T21:53:53Z

@bugmakerrrrrr can you manually raise the backport PR for this change.

bugmakerrrrrr · 2024-03-31T13:07:51Z

@bugmakerrrrrr can you manually raise the backport PR for this change.

@navneet1v Sure

) * support script score when doc value is disabled Signed-off-by: panguixin <panguixin@bytedance.com> * add test Signed-off-by: panguixin <panguixin@bytedance.com> * apply review comments Signed-off-by: panguixin <panguixin@bytedance.com> * fix test Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> (cherry picked from commit 771c4b5)

* support script score when doc value is disabled Signed-off-by: panguixin <panguixin@bytedance.com> * add test Signed-off-by: panguixin <panguixin@bytedance.com> * apply review comments Signed-off-by: panguixin <panguixin@bytedance.com> * fix test Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> (cherry picked from commit 771c4b5)

) (opensearch-project#1587) * support script score when doc value is disabled Signed-off-by: panguixin <panguixin@bytedance.com> * add test Signed-off-by: panguixin <panguixin@bytedance.com> * apply review comments Signed-off-by: panguixin <panguixin@bytedance.com> * fix test Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> (cherry picked from commit 771c4b5)

bugmakerrrrrr requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, martin-gaievski and ryanbogan as code owners March 21, 2024 16:37

navneet1v reviewed Mar 22, 2024

View reviewed changes

src/main/java/org/opensearch/knn/index/KNNVectorDVLeafFieldData.java Outdated Show resolved Hide resolved

navneet1v reviewed Mar 27, 2024

View reviewed changes

src/main/java/org/opensearch/knn/index/KNNVectorDVLeafFieldData.java Outdated Show resolved Hide resolved

src/test/java/org/opensearch/knn/plugin/script/KNNScriptScoringIT.java Outdated Show resolved Hide resolved

jmazanec15 previously approved these changes Mar 27, 2024

View reviewed changes

bugmakerrrrrr added 3 commits March 28, 2024 15:42

support script score when doc value is disabled

32762a5

Signed-off-by: panguixin <panguixin@bytedance.com>

add test

f30c093

Signed-off-by: panguixin <panguixin@bytedance.com>

apply review comments

b110b5f

Signed-off-by: panguixin <panguixin@bytedance.com>

bugmakerrrrrr dismissed jmazanec15’s stale review via b110b5f March 28, 2024 07:43

bugmakerrrrrr force-pushed the disable_docvalue branch from 543c456 to b110b5f Compare March 28, 2024 07:43

fix test

46e9883

Signed-off-by: panguixin <panguixin@bytedance.com>

navneet1v approved these changes Mar 28, 2024

View reviewed changes

jmazanec15 approved these changes Mar 28, 2024

View reviewed changes

navneet1v merged commit 771c4b5 into opensearch-project:main Mar 28, 2024
49 of 51 checks passed

jmazanec15 added the backport 2.x label Mar 28, 2024

jmazanec15 mentioned this pull request Mar 28, 2024

[Enhancement] Optimize the de-serialization of vector when reading from Doc Values #1050

Closed

bugmakerrrrrr mentioned this pull request Apr 2, 2024

[Backport 2.x] Support script score when doc value is disabled (#1573) #1587

Merged

5 tasks

This was referenced Apr 29, 2024

[BUG] Cluster crash when using script score #1647

Closed

Revert 'Support script score when doc value is disabled' #1662

Merged

bugmakerrrrrr mentioned this pull request May 9, 2024

Support script score when doc value is disabled and fix misusing DISI #1696

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support script score when doc value is disabled #1573

Support script score when doc value is disabled #1573

bugmakerrrrrr commented Mar 21, 2024

bugmakerrrrrr commented Mar 21, 2024

navneet1v commented Mar 21, 2024

navneet1v commented Mar 21, 2024

navneet1v commented Mar 21, 2024

navneet1v commented Mar 22, 2024

jmazanec15 commented Mar 25, 2024 •

edited

Loading

jmazanec15 commented Mar 25, 2024

jmazanec15 commented Mar 25, 2024

navneet1v commented Mar 25, 2024

bugmakerrrrrr commented Mar 26, 2024

codecov bot commented Mar 27, 2024 •

edited

Loading

navneet1v left a comment

jmazanec15 left a comment

navneet1v left a comment

navneet1v commented Mar 28, 2024

opensearch-trigger-bot bot commented Mar 28, 2024

navneet1v commented Mar 28, 2024

bugmakerrrrrr commented Mar 31, 2024

Support script score when doc value is disabled #1573

Support script score when doc value is disabled #1573

Conversation

bugmakerrrrrr commented Mar 21, 2024

Description

Issues Resolved

Check List

bugmakerrrrrr commented Mar 21, 2024

navneet1v commented Mar 21, 2024

navneet1v commented Mar 21, 2024

navneet1v commented Mar 21, 2024

navneet1v commented Mar 22, 2024

jmazanec15 commented Mar 25, 2024 • edited Loading

jmazanec15 commented Mar 25, 2024

jmazanec15 commented Mar 25, 2024

navneet1v commented Mar 25, 2024

bugmakerrrrrr commented Mar 26, 2024

codecov bot commented Mar 27, 2024 • edited Loading

Codecov Report

navneet1v left a comment

Choose a reason for hiding this comment

jmazanec15 left a comment

Choose a reason for hiding this comment

navneet1v left a comment

Choose a reason for hiding this comment

navneet1v commented Mar 28, 2024

opensearch-trigger-bot bot commented Mar 28, 2024

navneet1v commented Mar 28, 2024

bugmakerrrrrr commented Mar 31, 2024

jmazanec15 commented Mar 25, 2024 •

edited

Loading

codecov bot commented Mar 27, 2024 •

edited

Loading