Commits on Mar 20, 2024

1. Performance optimization of IVF-flat / select_k (#2221) …

   This PR is a followup to #2169. To enable IVF-flat with k>256 we need an additional select_k invocation which was unexpectedly slow. There are two reasons for that:
   
   First problem is the data handed to select_k: The valid data length per row is much smaller than the conservative maximum that could be achieved by probing the N largest probes. Therefore each query row contains roughly ~50% dummy values. This is also the case for IVF-PQ, but did not show up as prominent due to the second reason.
   
   The second problem, and also a difference to the IVF-PQ algorithm - is that a 64bit payload data type is used for selectK. The performance of selectK with 64bit index type is significantly slower than with 32bit, especially when many elements are in the same range:
   ```
   Benchmark                                                           Time             CPU   Iterations
   -----------------------------------------------------------------------------------------------------
   SelectK/float/uint32_t/kRadix11bitsExtraPass/1/manual_time       1.68 ms         1.74 ms          413 1357#200000#512
   SelectK/float/uint32_t/kRadix11bitsExtraPass/3/manual_time       2.31 ms         2.37 ms          302 1357#200000#512#same-leading-bits
   SelectK/float/int64_t/kRadix11bitsExtraPass/1/manual_time        5.92 ms         5.98 ms          116 1357#200000#512
   SelectK/float/int64_t/kRadix11bitsExtraPass/3/manual_time        83.7 ms         83.8 ms            8 1357#200000#512#same-leading-bits
   -----------------------------------------------------------------------------------------------------
   ```
   The data distribution within a IVF-flat benchmark resulted in a select_k time of ~24ms. 
   
   ### scope:
   * additional parameter added to select_k to optionally pass individual row lengths for every batch entry. This parameter is utilized by both IVF-Flat and IVF-PQ and results in a ~2x speedup (50 nodes out of 5000) of the final `select_k`. 
   * refactor ivf-flat search to work with 32bit indices by storing positions instead of actual indices. This allows to utilize 32bit index type select_k for ~10x speedup in the final `select_k`.
   
   FYI @tfeher @achirkin 
   
   ### not in scope:
   * General optimization of select_k: In the current implementation there is no difference in the type of the payload and the actual index type. Especially the type of the histogram has a large effect on performance (due to the atomics).
   
   Authors:
     - Malte Förster (https://github.com/mfoerste4)
   
   Approvers:
     - Tamas Bela Feher (https://github.com/tfeher)
   
   URL: #2221

   

   mfoerste4 authored Mar 20, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 335236c
   • Browse repository at this point

   Copy the full SHA

   335236c View commit details

   Browse the repository at this point in the history