Skip to content

Commit

Permalink
Replace too long index file name with hash in ANN bench (#2280)
Browse files Browse the repository at this point in the history
Too long index file name would lead to a crash while calling the index serialization routines. Such long filenames can occur if we try to specialize many parameters for CAGRA ann index. This PR fixes the issue by replacing the long index file name with a hash. Drawback is the filename will not be descriptive.

Authors:
  - Tamas Bela Feher (https://github.com/tfeher)
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #2280
  • Loading branch information
tfeher authored May 21, 2024
1 parent ac6be9e commit a539c32
Showing 1 changed file with 7 additions and 3 deletions.
10 changes: 7 additions & 3 deletions python/raft-ann-bench/src/raft-ann-bench/run/__main__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Copyright (c) 2023, NVIDIA CORPORATION.
# Copyright (c) 2023-2024, NVIDIA CORPORATION.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -553,10 +553,14 @@ def add_algo_group(group_list):
index["build_param"], conf_file["dataset"]["dims"]
):
continue

index_filename = (
index_name
if len(index_name) < 128
else str(hash(index_name))
)
index["name"] = index_name
index["file"] = os.path.join(
args.dataset_path, args.dataset, "index", index_name
args.dataset_path, args.dataset, "index", index_filename
)
index["search_params"] = []
all_search_params = itertools.product(*search_param_lists)
Expand Down

0 comments on commit a539c32

Please sign in to comment.