Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: type of temperature param and adjust defaults for OpenAIAnswerGenerator #3073

Merged
merged 9 commits into from
Sep 16, 2022

Conversation

tholor
Copy link
Member

@tholor tholor commented Aug 19, 2022

Proposed Changes:

  • Small fix that changes the type of temperature param from int to float. Their API allows floats and this was causing problems in the YAML schema validation.
  • Also adjusting the default values as tests showed that with max_tokens = 7 quite many answers get cut off in the middle. GPT-3 tends to be a bit more verbose in it's answers
  • Increasing the default temperature slightly, as tests showed that we otherwise get often multiple answers generated that are exactly the same as the "top 1"

Checklist

@tholor tholor requested a review from a team as a code owner August 19, 2022 14:26
@tholor tholor requested review from vblagoje and removed request for a team August 19, 2022 14:26
@tholor tholor changed the title fix: type of temperature param and adjust defaults for OpenAIAnswerGenerator fix: type of temperature param and adjust defaults for OpenAIAnswerGenerator Aug 19, 2022
@Timoeller
Copy link
Contributor

Arr, I was playing with it as well today and have similar code changes.

About temperatur=0.2: when using it in our pipeline.eval function on AdversarialQA dataset with topk retriever=5 amd topk generator = 5 I mostly got 5 identical answers.

I will open a PR later as well with some more little tweaks.

@tholor tholor requested a review from a team as a code owner September 16, 2022 08:15
@vblagoje
Copy link
Member

LGTM @tholor

@tholor tholor merged commit 3134b0d into main Sep 16, 2022
@tholor tholor deleted the fix_type_gpt3 branch September 16, 2022 12:11
brandenchan pushed a commit that referenced this pull request Sep 21, 2022
…rGenerator` (#3073)

* fix: type of temperature param and adjust defaults

* update schema

* update api docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants