Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Fix prompt length computation #3448

Merged
merged 1 commit into from
Oct 24, 2022
Merged

Bug: Fix prompt length computation #3448

merged 1 commit into from
Oct 24, 2022

Conversation

Timoeller
Copy link
Contributor

Problem

The current prompt for the Answergenerator can exceed the allowed length. The prompt needs to account for the number of generated tokens as well since "The token count of your prompt plus max_tokens cannot exceed the model's context length.".

Proposed Changes:

Subtract max_tokens from leftover_token_len

Checklist

@Timoeller Timoeller requested a review from a team as a code owner October 21, 2022 10:05
@Timoeller Timoeller requested review from ZanSara and removed request for a team October 21, 2022 10:05
@ZanSara ZanSara added topic:retriever type:bug Something isn't working labels Oct 24, 2022
Copy link
Contributor

@ZanSara ZanSara left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@ZanSara ZanSara merged commit 9b931bb into main Oct 24, 2022
@ZanSara ZanSara deleted the openailength branch October 24, 2022 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic:retriever type:bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants