New solution to handle IVF_FLAT backward compatibility #76
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cydrain The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@cydrain 🔍 Important: PR Classification Needed! For efficient project management and a seamless review process, it's essential to classify your PR correctly. Here's how:
For any PR outside the kind/improvement category, ensure you link to the associated issue using the format: “issue: #”. Thanks for your efforts and contribution to the community!. |
|
/kind enhancement |
cydrain
force-pushed
the
caiyd_ivf_flat_nm_compatable_new_sol
branch
from
3799781
to
19e4384
Compare
|
/hold |
|
|
||
| uint8_t dummy8; | ||
| READ1(dummy8); | ||
| uint16_t dummy16; |
There was a problem hiding this comment.
8(used for cosine) + 8 + 16 + 32 (rest padding) = 64 ? Pls add some comments here or just use 64 bit to save the is_cosine
| @@ -81,9 +81,17 @@ namespace faiss { | |||
| static void write_index_header(const Index* idx, IOWriter* f) { | |||
| WRITE1(idx->d); | |||
| WRITE1(idx->ntotal); | |||
| Index::idx_t dummy = 1 << 20; | |||
| WRITE1(dummy); | |||
| WRITE1(idx->is_cosine); | |||
| { | ||
| std::vector<OnDiskInvertedLists::Slot> v( | ||
| od->slots.begin(), od->slots.end()); | ||
| WRITEVECTOR(v); |
There was a problem hiding this comment.
Has IVFFLAT used these inverted lists? May beyond my knowledge
There was a problem hiding this comment.
this is not a read code change
| // auto codes_size = ivfl->d * ivfl->ntotal * sizeof(float); | ||
|
|
||
| // IVF_FLAT_NM format, need convert to new format | ||
| if (remains == invlist_size + ids_size) { |
There was a problem hiding this comment.
little hack to decide the index type by binary size. Is there any other ways?
There was a problem hiding this comment.
I only think out 2 ways to detect IVF_FLAT_NM:
- use native IVF_FLAT format to parse a binary, if it throw an exception, then this binary should be IVF_FLAT_NM
- the way as this PR, read ivf_flat index header, then calculate the binary size
before this PR, I use method 1, but it's a little bit risky, so I change to method 2
|
Pls verify the compatibility with existing index binary. A byte difference may occur load fail. @elstic |
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
cydrain
force-pushed
the
caiyd_ivf_flat_nm_compatable_new_sol
branch
from
19e4384
to
b67f931
Compare
|
/lgtm |
|
/unhold |
Issue: #30