bug: reactivate benchmarks with quick fixes (#2766)

* quick fix benchmark runs to make them work with current haystack version * fix minor typo * update readme. fix minor things to make benchmarks run again * Update Documentation & Code Style * fix typo in readme * update result files for reader and retriever querying * reduce batch size for update embeddings to prevent xlarge bulk_update requests that exceed elastic's limits (happening in dense 500k runs) * change default memory allocation back to normal. add note to readme * add first indexing results * add memory to docker cmd * full benchmarks results on commit c5a2651 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
deepset-ai · Sep 20, 2022 · 7e79a48 · 7e79a48
1 parent 9399ddf
commit 7e79a48
Show file tree

Hide file tree

Showing 25 changed files with 537 additions and 407 deletions.
diff --git a/docs/_src/api/api/file_converter.md b/docs/_src/api/api/file_converter.md
@@ -288,7 +288,7 @@ def convert(file_path: Path,
             id_hash_keys: Optional[List[str]] = None) -> List[Document]
 ```
 
-Reads text from a txt file and executes optional preprocessing steps.
+Reads text from a markdown file and executes optional preprocessing steps.
 
 **Arguments**:
 

diff --git a/docs/_src/benchmarks/farm_per_component.html b/docs/_src/benchmarks/farm_per_component.html
diff --git a/docs/_src/benchmarks/reader_performance.json b/docs/_src/benchmarks/reader_performance.json
@@ -2,7 +2,7 @@
     "chart_type": "BarChart",
     "title": "Reader Performance",
     "subtitle": "Time and Accuracy Benchmarks",
-    "description": "Performance benchmarks of different Readers that can be used off-the-shelf in Haystack. Some models are geared towards speed, while others are more performance-focused. Accuracy is measured as F1 score and speed as passages/sec (with passages of 384 tokens).  Each Reader is benchmarked using the SQuAD v2.0 development set, which contains 11866 question answer pairs. When tokenized using the BERT tokenizer and split using a sliding window approach, these become 12350 passages that are passed into the model. We set <i>max_seq_len=384</i> and <i>doc_stride=128</i>. These benchmarking tests are run using an AWS p3.2xlarge instance with a Nvidia V100 GPU with this <a href='https://github.com/deepset-ai/haystack/blob/main/test/benchmarks/reader.py'>script</a>. Please note that we are using the FARMReader class rather than the TransformersReader class. Also, the F1 measure that is reported here is in fact calculated on token level, rather than word level as is done in the official SQuAD script.",
+    "description": "Performance benchmarks of different Readers that can be used off-the-shelf in Haystack. Some models are geared towards speed, while others are more performance-focused. Accuracy is measured as F1 score and speed as passages/sec (with passages of 384 tokens).  Each Reader is benchmarked using the SQuAD v2.0 development set, which contains 11866 question answer pairs. When tokenized using the BERT tokenizer and split using a sliding window approach, these become 12350 passages that are passed into the model. We set <i>max_seq_len=384</i> and <i>doc_stride=128</i>. These benchmarking tests are run using an AWS p3.2xlarge instance with a Nvidia V100 GPU with this <a href='https://github.com/deepset-ai/haystack/blob/master/test/benchmarks/reader.py'>script</a>. Please note that we are using the FARMReader class rather than the TransformersReader class. Also, the F1 measure that is reported here is in fact calculated on token level, rather than word level as is done in the official SQuAD script.",
     "bars": "horizontal",
     "columns": [
         "Model",
@@ -11,34 +11,34 @@
     ],
     "data": [
         {
-            "F1": 82.58860575299658,
-            "Speed": 125.81040525892848,
+            "F1": 78.21542223321491,
+            "Speed": 118.41880460779026,
             "Model": "RoBERTa"
         },
         {
-            "F1": 78.87858491007042,
-            "Speed": 260.6443097981493,
+            "F1": 79.6479775678566,
+            "Speed": 235.0263836060577,
             "Model": "MiniLM"
         },
         {
-            "F1": 74.31182400443286,
-            "Speed": 121.08066567525722,
+            "F1": 73.49806294434092,
+            "Speed": 116.99833085428466,
             "Model": "BERT base"
         },
         {
-            "F1": 83.26306774734308,
-            "Speed": 42.21949937744112,
-            "Model": "BERT large"
+            "F1": 87.35038229973254,
+            "Speed": 43.45399164221017,
+            "Model": "RoBERTa large"
         },
         {
-            "F1": 84.50422699207468,
-            "Speed": 42.07400844838985,
-            "Model": "XLM-RoBERTa"
+            "F1": 82.29388851439239,
+            "Speed": 118.88131118898794,
+            "Model": "RoBERTa distilled"
         },
         {
-            "F1": 42.31925844723574,
-            "Speed": 222.91207128366702,
-            "Model": "DistilBERT"
+            "F1": 43.34028303155876,
+            "Speed": 198.4500591997352,
+            "Model": "distilBERT"
         }
     ]
 }
diff --git a/docs/_src/benchmarks/reader_results.csv b/docs/_src/benchmarks/reader_results.csv
@@ -0,0 +1,7 @@
+,EM,f1,top_n_accuracy,top_n,reader_time,seconds_per_query,EM_text_answer,f1_text_answer,top_n_accuracy_text_answer,top_n_EM_text_answer,top_n_f1_text_answer,Total_text_answer,EM_no_answer,f1_no_answer,top_n_accuracy_no_answer,Total_no_answer,passages_per_second,reader,error
+0,76.1672004045171,78.21542223321491,97.41277599865161,5,104.290868674987,0.0087890501158762,61.491731353358084,65.59301387433817,94.81943975700304,81.01586230172123,90.50667071721769,5926,90.8080808080808,90.8080808080808,100.0,5940,118.41880460779026,deepset/roberta-base-squad2,
+1,76.84982302376538,79.6479775678566,97.21894488454407,5,52.54729196999688,0.004428391367773207,65.89605129935876,71.49897094501964,94.4313196085049,79.96962537968275,89.70802565150441,5926,87.77777777777777,87.77777777777777,100.0,5940,235.02638360605766,deepset/minilm-uncased-squad2,
+2,70.0825889094893,73.49806294434092,95.53345693578291,5,105.55706145399017,0.008895757749367114,56.59804252446844,63.43705955071704,91.05636179547756,72.1228484643942,84.47853540832811,5926,83.53535353535354,83.53535353535354,100.0,5940,116.99833085428466,deepset/bert-base-cased-squad2,
+3,83.69290409573573,87.35038229973254,98.07854373841226,5,284.2086430560157,0.023951512140233923,75.19406007424907,82.51765716649112,96.15254809314884,82.33209584880188,91.89205932239825,5926,92.17171717171718,92.17171717171718,100.0,5940,43.45399164221017,deepset/roberta-large-squad2,
+4,79.50446654306421,82.29388851439239,97.52233271532108,5,103.88512606802396,0.008754856402159445,71.110361120486,76.69579499017553,95.03881201484981,81.72460344245697,90.89098996208568,5926,87.87878787878788,87.87878787878788,100.0,5940,118.88131118898794,deepset/roberta-base-squad2-distilled,
+5,38.53868194842407,43.340283031558755,95.43232765885725,5,62.23228176299017,0.005244588046771462,74.53594330070874,84.1504891077415,90.85386432669593,75.09281133985826,85.21259442024737,5926,2.6262626262626263,2.6262626262626263,100.0,5940,198.4500591997352,distilbert-base-uncased-distilled-squad,
diff --git a/docs/_src/benchmarks/reader_results.md b/docs/_src/benchmarks/reader_results.md
@@ -0,0 +1,8 @@
+|    |      EM |      f1 |   top_n_accuracy |   top_n |   reader_time |   seconds_per_query |   EM_text_answer |   f1_text_answer |   top_n_accuracy_text_answer |   top_n_EM_text_answer |   top_n_f1_text_answer |   Total_text_answer |   EM_no_answer |   f1_no_answer |   top_n_accuracy_no_answer |   Total_no_answer |   passages_per_second | reader                                  | error   |
+|---:|--------:|--------:|-----------------:|--------:|--------------:|--------------------:|-----------------:|-----------------:|-----------------------------:|-----------------------:|-----------------------:|--------------------:|---------------:|---------------:|---------------------------:|------------------:|----------------------:|:----------------------------------------|:--------|
+|  0 | 76.1672 | 78.2154 |          97.4128 |       5 |      104.291  |          0.00878905 |          61.4917 |          65.593  |                      94.8194 |                81.0159 |                90.5067 |                5926 |       90.8081  |       90.8081  |                        100 |              5940 |               118.419 | deepset/roberta-base-squad2             |         |
+|  1 | 76.8498 | 79.648  |          97.2189 |       5 |       52.5473 |          0.00442839 |          65.8961 |          71.499  |                      94.4313 |                79.9696 |                89.708  |                5926 |       87.7778  |       87.7778  |                        100 |              5940 |               235.026 | deepset/minilm-uncased-squad2           |         |
+|  2 | 70.0826 | 73.4981 |          95.5335 |       5 |      105.557  |          0.00889576 |          56.598  |          63.4371 |                      91.0564 |                72.1228 |                84.4785 |                5926 |       83.5354  |       83.5354  |                        100 |              5940 |               116.998 | deepset/bert-base-cased-squad2          |         |
+|  3 | 83.6929 | 87.3504 |          98.0785 |       5 |      284.209  |          0.0239515  |          75.1941 |          82.5177 |                      96.1525 |                82.3321 |                91.8921 |                5926 |       92.1717  |       92.1717  |                        100 |              5940 |                43.454 | deepset/roberta-large-squad2            |         |
+|  4 | 79.5045 | 82.2939 |          97.5223 |       5 |      103.885  |          0.00875486 |          71.1104 |          76.6958 |                      95.0388 |                81.7246 |                90.891  |                5926 |       87.8788  |       87.8788  |                        100 |              5940 |               118.881 | deepset/roberta-base-squad2-distilled   |         |
+|  5 | 38.5387 | 43.3403 |          95.4323 |       5 |       62.2323 |          0.00524459 |          74.5359 |          84.1505 |                      90.8539 |                75.0928 |                85.2126 |                5926 |        2.62626 |        2.62626 |                        100 |              5940 |               198.45  | distilbert-base-uncased-distilled-squad |         |
diff --git a/docs/_src/benchmarks/retriever_index_results.csv b/docs/_src/benchmarks/retriever_index_results.csv
@@ -0,0 +1,33 @@
+,retriever,doc_store,n_docs,indexing_time,docs_per_second,date_time,error
+31,sentence_transformers,elasticsearch,500000,4165.329931593005,120.03851032486591,2022-09-17 21:16:26.676611,
+28,dpr,elasticsearch,500000,6180.3500814459985,80.90156599721556,2022-09-17 16:45:36.329915,
+7,sentence_transformers,elasticsearch,1000,9.844315000998904,101.5814711230319,2022-09-17 08:26:44.750140,
+15,sentence_transformers,elasticsearch,10000,84.88162445399212,117.8111289024663,2022-09-17 08:43:51.615496,
+20,dpr,elasticsearch,100000,1230.761674939,81.2504988059173,2022-09-17 09:57:09.456572,
+0,elastic,elasticsearch,1000,1.7988810280075995,555.9011321096525,2022-09-17 08:21:11.001120,
+24,elastic,elasticsearch,500000,1031.7205082759901,484.6273733915616,2022-09-17 11:09:14.536450,
+16,elastic,elasticsearch,100000,206.83811076599522,483.4698964792531,2022-09-17 08:47:59.723168,
+8,elastic,elasticsearch,10000,20.81922616600059,480.3252493760203,2022-09-17 08:27:43.460016,
+4,dpr,elasticsearch,1000,12.114610354998149,82.54495775734364,2022-09-17 08:24:34.582262,
+12,dpr,elasticsearch,10000,123.21874477900565,81.15648327642944,2022-09-17 08:37:02.872786,
+23,sentence_transformers,elasticsearch,100000,831.661083732004,120.24128813537725,2022-09-17 10:50:54.888143,
+29,dpr,faiss_flat,500000,5485.974153257004,91.1415158059306,2022-09-17 18:19:50.024476,
+13,dpr,faiss_flat,10000,105.0845138200093,95.16150036272884,2022-09-17 08:39:25.198922,
+5,dpr,faiss_flat,1000,10.151226033005514,98.51026829159531,2022-09-17 08:25:18.819509,
+21,dpr,faiss_flat,100000,1068.483952116003,93.59054930302146,2022-09-17 10:15:58.178598,
+22,dpr,faiss_hnsw,100000,1195.8989856069966,83.61910261947709,2022-09-17 10:36:24.783587,
+14,dpr,faiss_hnsw,10000,114.6786555860017,87.20018515128682,2022-09-17 08:41:49.541725,
+6,dpr,faiss_hnsw,1000,10.710903742990922,93.36280336329109,2022-09-17 08:25:58.505307,
+30,dpr,faiss_hnsw,500000,6349.903588575005,78.7413530025274,2022-09-17 20:06:17.374842,
+9,elastic,opensearch_flat,10000,21.9185139649926,456.2353093814486,2022-09-17 08:28:40.051299,
+10,dpr,opensearch_flat,10000,123.36043137000524,81.06327036102982,2022-09-17 08:31:26.201876,
+17,elastic,opensearch_flat,100000,208.89336037798785,478.7131568904452,2022-09-17 08:52:13.178937,
+1,elastic,opensearch_flat,1000,2.2300678319879808,448.4168533602657,2022-09-17 08:21:45.610495,
+18,dpr,opensearch_flat,100000,1245.0073385680007,80.32081169498916,2022-09-17 09:13:46.979550,
+25,elastic,opensearch_flat,500000,1045.8889810429973,478.06221220667453,2022-09-17 11:28:08.906406,
+26,dpr,opensearch_flat,500000,6235.606822678994,80.18465791997863,2022-09-17 13:13:24.174105,
+2,dpr,opensearch_flat,1000,12.919973790005315,77.3995378205476,2022-09-17 08:22:43.256934,
+11,dpr,opensearch_hnsw,10000,121.49061479599914,82.3108848102505,2022-09-17 08:34:11.508717,
+3,dpr,opensearch_hnsw,1000,12.355953149002744,80.93264744053441,2022-09-17 08:23:36.852546,
+19,dpr,opensearch_hnsw,100000,1242.960266391994,80.45309468360983,2022-09-17 09:35:31.609351,
+27,dpr,opensearch_hnsw,500000,6237.192302857991,80.16427516125985,2022-09-17 14:59:57.763007,
diff --git a/docs/_src/benchmarks/retriever_index_results.md b/docs/_src/benchmarks/retriever_index_results.md
@@ -0,0 +1,34 @@
+|    | retriever             | doc_store       |   n_docs |   indexing_time |   docs_per_second | date_time                  | error   |
+|---:|:----------------------|:----------------|---------:|----------------:|------------------:|:---------------------------|:--------|
+| 31 | sentence_transformers | elasticsearch   |   500000 |      4165.33    |          120.039  | 2022-09-17 21:16:26.676611 |         |
+| 28 | dpr                   | elasticsearch   |   500000 |      6180.35    |           80.9016 | 2022-09-17 16:45:36.329915 |         |
+|  7 | sentence_transformers | elasticsearch   |     1000 |         9.84432 |          101.581  | 2022-09-17 08:26:44.750140 |         |
+| 15 | sentence_transformers | elasticsearch   |    10000 |        84.8816  |          117.811  | 2022-09-17 08:43:51.615496 |         |
+| 20 | dpr                   | elasticsearch   |   100000 |      1230.76    |           81.2505 | 2022-09-17 09:57:09.456572 |         |
+|  0 | elastic               | elasticsearch   |     1000 |         1.79888 |          555.901  | 2022-09-17 08:21:11.001120 |         |
+| 24 | elastic               | elasticsearch   |   500000 |      1031.72    |          484.627  | 2022-09-17 11:09:14.536450 |         |
+| 16 | elastic               | elasticsearch   |   100000 |       206.838   |          483.47   | 2022-09-17 08:47:59.723168 |         |
+|  8 | elastic               | elasticsearch   |    10000 |        20.8192  |          480.325  | 2022-09-17 08:27:43.460016 |         |
+|  4 | dpr                   | elasticsearch   |     1000 |        12.1146  |           82.545  | 2022-09-17 08:24:34.582262 |         |
+| 12 | dpr                   | elasticsearch   |    10000 |       123.219   |           81.1565 | 2022-09-17 08:37:02.872786 |         |
+| 23 | sentence_transformers | elasticsearch   |   100000 |       831.661   |          120.241  | 2022-09-17 10:50:54.888143 |         |
+| 29 | dpr                   | faiss_flat      |   500000 |      5485.97    |           91.1415 | 2022-09-17 18:19:50.024476 |         |
+| 13 | dpr                   | faiss_flat      |    10000 |       105.085   |           95.1615 | 2022-09-17 08:39:25.198922 |         |
+|  5 | dpr                   | faiss_flat      |     1000 |        10.1512  |           98.5103 | 2022-09-17 08:25:18.819509 |         |
+| 21 | dpr                   | faiss_flat      |   100000 |      1068.48    |           93.5905 | 2022-09-17 10:15:58.178598 |         |
+| 22 | dpr                   | faiss_hnsw      |   100000 |      1195.9     |           83.6191 | 2022-09-17 10:36:24.783587 |         |
+| 14 | dpr                   | faiss_hnsw      |    10000 |       114.679   |           87.2002 | 2022-09-17 08:41:49.541725 |         |
+|  6 | dpr                   | faiss_hnsw      |     1000 |        10.7109  |           93.3628 | 2022-09-17 08:25:58.505307 |         |
+| 30 | dpr                   | faiss_hnsw      |   500000 |      6349.9     |           78.7414 | 2022-09-17 20:06:17.374842 |         |
+|  9 | elastic               | opensearch_flat |    10000 |        21.9185  |          456.235  | 2022-09-17 08:28:40.051299 |         |
+| 10 | dpr                   | opensearch_flat |    10000 |       123.36    |           81.0633 | 2022-09-17 08:31:26.201876 |         |
+| 17 | elastic               | opensearch_flat |   100000 |       208.893   |          478.713  | 2022-09-17 08:52:13.178937 |         |
+|  1 | elastic               | opensearch_flat |     1000 |         2.23007 |          448.417  | 2022-09-17 08:21:45.610495 |         |
+| 18 | dpr                   | opensearch_flat |   100000 |      1245.01    |           80.3208 | 2022-09-17 09:13:46.979550 |         |
+| 25 | elastic               | opensearch_flat |   500000 |      1045.89    |          478.062  | 2022-09-17 11:28:08.906406 |         |
+| 26 | dpr                   | opensearch_flat |   500000 |      6235.61    |           80.1847 | 2022-09-17 13:13:24.174105 |         |
+|  2 | dpr                   | opensearch_flat |     1000 |        12.92    |           77.3995 | 2022-09-17 08:22:43.256934 |         |
+| 11 | dpr                   | opensearch_hnsw |    10000 |       121.491   |           82.3109 | 2022-09-17 08:34:11.508717 |         |
+|  3 | dpr                   | opensearch_hnsw |     1000 |        12.356   |           80.9326 | 2022-09-17 08:23:36.852546 |         |
+| 19 | dpr                   | opensearch_hnsw |   100000 |      1242.96    |           80.4531 | 2022-09-17 09:35:31.609351 |         |
+| 27 | dpr                   | opensearch_hnsw |   500000 |      6237.19    |           80.1643 | 2022-09-17 14:59:57.763007 |         |