Documentation for Binary Quantization Support with KNN Vector Search

[Vikasht34]

Description

This PR introduces detailed documentation for Binary Quantization support in the k-NN plugin, including how to configure, fine-tune, and optimize k-NN vector indices using Binary Quantization. Key highlights of this update are as follows:
Key Changes:

1. Introduction to Binary Quantization:

   • Binary Quantization allows vectors to be quantized into binary representations (1-bit, 2-bit, or 4-bit) to reduce memory footprint and speed up search performance.
   • It provides compression rates of 32x, 16x, and 8x respectively for 1-bit, 2-bit, and 4-bit quantization, drastically reducing memory usage.

2. Out-of-the-Box Experience:

   • Default 1-bit binary quantization is supported without any additional configuration, offering 32x compression and default parameters for ef_search and ef_construction (both set to 100).

3. Customization and Fine-tuning: Users can further optimize the indexing and search processes by specifying custom parameters like:

   • ef_construction: Adjusting the construction of HNSW graphs.
   • compression_level: Providing different compression levels such as 32x for 1-bit, 16x for 2-bit, and 8x for 4-bit.
   • oversample_factor: Fine-tuning search accuracy by oversampling neighbors during the search process.

4. Memory Estimation for Binary Quantization:

   • Provided memory estimation formulas for HNSW graph construction for 1-bit, 2-bit, and 4-bit quantization, helping users estimate the memory requirements based on their vector dimensions and index configurations.

5. Example Configurations:

   • Basic Binary Quantization Example: Demonstrates how to configure Binary Quantization with minimal setup and default 1-bit quantization.
   • Advanced Configurations: Shows how to customize parameters like compression level and ef_construction to achieve optimal performance for different use cases.

Issues Resolved

Closes #[insert issue number]

Version

2.17

Frontend features

If you're submitting documentation for an OpenSearch Dashboards feature, add a video that shows how a user will interact with the UI step by step. A voiceover is optional.

Checklist

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
  For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

[kolchfa-aws]

@Vikasht34 Please let me know when this PR is ready for doc review. Thanks!

[Vikasht34]

@Vikasht34 Please let me know when this PR is ready for doc review. Thanks!

Please start review!!

[kolchfa-aws]

Thanks, @Vikasht34! I made several rewrites. In the future, please work on a branch in your fork (not on main) because then maintainers can push the rewrites directly into your PR.

[natebower]

@kolchfa-aws @Vikasht34 Please see my changes and let me know if you have any questions. Thanks!