Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce NativeEngineKNNVectorsFormat as a KNNVectorsFormat for Native engines #1855

Merged
merged 1 commit into from
Jul 24, 2024
Merged

Introduce NativeEngineKNNVectorsFormat as a KNNVectorsFormat for Native engines #1855

merged 1 commit into from
Jul 24, 2024

Conversation

navneet1v
Copy link
Collaborator

@navneet1v navneet1v commented Jul 18, 2024 •
edited
Loading

Description

Introduced NativeEngineKNNVectorsFormat as a KNNVectorsFormat for Native engines(faiss and nmslib). This PR doesn't change any code path or do the integration of the new format in the actual flow. This is more of a base work I am adding for next step of changes which are coming.

Changes include:

  1. Introduce the NativeEngineKNNVectorsFormat, readers and writers.
  2. Added the unit test for this new format.
  3. Currently the format is only writing the FlatVectors and not building the hnsw vector files. I will add this change in next few PRs.
  4. Not adding the backport label on this PR. I will add the backport label once 2.16 is released, as this change is not intended for 2.16 version of Opensearch.

Related Issues

#1853

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@navneet1v navneet1v added skip-changelog Enhancements Increases software capabilities beyond original client specifications labels Jul 18, 2024
@navneet1v navneet1v changed the title Introduced NativeEngineKNNVectorsFormat as a KNNVectorsFormat for Native engines Introduce NativeEngineKNNVectorsFormat as a KNNVectorsFormat for Native engines Jul 18, 2024
@navneet1v navneet1v force-pushed the codec branch 3 times, most recently from 143581f to 92cfae2 Compare July 18, 2024 23:47
@navneet1v
Introduce NativeEngineKNNVectorsFormat as a KNNVectorsFormat for Nati…
53a96c6 
…ve engines

Signed-off-by: Navneet Verma <navneev@amazon.com>
@navneet1v navneet1v requested a review from shatejas July 18, 2024 23:57
@navneet1v
Copy link
Collaborator Author

Not adding the backport label on this PR. I will add the backport label once 2.16 is released, as this change is not intended for 2.16 version of Opensearch.

@navneet1v navneet1v merged commit 1e03e59 into opensearch-project:main Jul 24, 2024
53 of 54 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 25, 2024
@navneet1v @github-actions
Introduce NativeEngineKNNVectorsFormat as a KNNVectorsFormat for Nati…
19c728c 
…ve engines (#1855)

Signed-off-by: Navneet Verma <navneev@amazon.com>
(cherry picked from commit 1e03e59)
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Jul 25, 2024
Merged
@jmazanec15 jmazanec15 mentioned this pull request Jul 26, 2024
Open
navneet1v added a commit that referenced this pull request Jul 27, 2024
@opensearch-trigger-bot @navneet1v
Introduce NativeEngineKNNVectorsFormat as a KNNVectorsFormat for Nati…
c1450cd 
…ve engines (#1855) (#1887)

Signed-off-by: Navneet Verma <navneev@amazon.com>
(cherry picked from commit 1e03e59)

Co-authored-by: Navneet Verma <navneev@amazon.com>
@navneet1v navneet1v mentioned this pull request Jul 30, 2024
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@heemin32 heemin32 heemin32 left review comments

@ryanbogan ryanbogan ryanbogan approved these changes

@jmazanec15 jmazanec15 jmazanec15 approved these changes

@VijayanB VijayanB Awaiting requested review from VijayanB VijayanB is a code owner

@vamshin vamshin Awaiting requested review from vamshin vamshin is a code owner

@naveentatikonda naveentatikonda Awaiting requested review from naveentatikonda naveentatikonda is a code owner

@junqiu-lei junqiu-lei Awaiting requested review from junqiu-lei junqiu-lei is a code owner

@martin-gaievski martin-gaievski Awaiting requested review from martin-gaievski martin-gaievski is a code owner

@luyuncheng luyuncheng Awaiting requested review from luyuncheng luyuncheng is a code owner

@shatejas shatejas Awaiting requested review from shatejas

Assignees
No one assigned
Labels
backport 2.x Enhancements Increases software capabilities beyond original client specifications skip-changelog
Projects
Vector Search RoadMap
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@navneet1v @heemin32 @ryanbogan @jmazanec15 @shatejas