docs: add latest benchmark results for v1.9.0

[tholor]

Proposed Changes

After (quick)fixing the benchmarks in #2766, I ran all of them now on tag v1.9.0 (i.e. commit ce36be8) to get some up-to-date results.

Few high-level observations from comparing the results to the last version of benchmarks we had (used the ones stored here for v1.8.0, but pretty sure we updated them properly the last time for v0.8.0 and then just carried over to newer versions).

Reader

• F1 performance shifted slightly (2x ~ 1-2% better, 2x 1-4% worse) => the only case standing out here a bit is the roberta model becoming somehow 4% worse
• Speed decreased slightly (biggest diff for MiniLM: from 260 passages/sec -> 239 passages/sec)

Retriever

• mAP Performance stayed the same. one exception is opensearch (hsnw), where we see better mAP values at larger scale (@>100k docs), which might be due to changes in our default knn params that we use here. 🎉
• indexing + querying speed: varying results depending on vectorDB + model combination (some are faster now, some slower, would need more detailed analysis incl some plots to understand if there's anything we should worry about) 🤔

Hardware
All benchmarks were run with:

• p3.2xlarge instance (v100 GPU)
• Python 3.7.4
• elasticsearch 7.9.2
• opensearch 2.2.1
• pytorch 1.9.1
• transformers 4.21.2

Notes for the reviewer

Not sure where to place the benchmarks results best in our new repo structure after the docs refactoring.
We used to have results in docs/v1.X.X/benchmarks but I guess we don't have these anymore after the migration of the docs?
Open for any suggestions here. Just important to keep it somehow tied to the v1.9.0 tag so that we which commit these results can be associated to.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added tests that demonstrate the correct behavior of the change
• I've used the conventional commit convention for my PR title
• I documented my code
• I ran pre-commit hooks and fixed any issue

[brandenchan]

Great to see some benchmarks again. I have a couple thoughts on this. Versioning is now handled differently in Haystack. v1.9.x now has its own branch and v1.9.0 has its own tag on this branch. If that's the case, we should perform the benchmarks on v1.9.0 and commit the results to the v1.9.x branch.

[tholor]

If that's the case, we should perform the benchmarks on v1.9.0

The benchmarks were performed on the tag v1.9.0, so not sure what you exactly request here?

... commit the results to the v1.9.x branch

Happy to merge this PR into the 1.9.0 branch and not main if this is the new style. However, I would expect that also for future releases benchmarks will only be added after the release (and the tag) as they run for 1-2 days (= we probably won't want to run them just for release candidates + it would slow down the release process quite a bit). Is this a problem? Where would you place them in the folder structure? Simply under docs/_src/benchmarks?
@masci: Is this in line with your thoughts?

[TuanaCelik]

Just adding this here:
The last discussion around Benchmarks we had during a Haystack Home meeting was that:

1. They start living in Haystack Home - this goes inline with Branden's experience with readme.
2. We decided to delay the decision of 'how' the sync would happen with the new repo
3. @masci and I had a brief conversation that an easy first release would involve having the last version up on Haystack Home to begin with and then add the other versions as a second iteration - this would just involve copying over the relevant files to HH - If this is delayed by a bit from the first release of HH we can have a page on HH that tells people 'Bare with us while we update our benchmarks page'

[brandenchan]

The benchmarks were performed on the tag v1.9.0, so not sure what you exactly request here?

That wasn't a request to benchmark differently, just a request that these benchmark results are merged into another branch.

If we do adopt this style where each version branch (e.g. v1.9.x) only has one set of benchmarking results, I would say that they could be saved in a docs/benchmarks folder.

[tholor]

Ok created a new PR #3355 that will merge the changes into the requested v1.9.x branch.
Closing this one.