.Net: Depending on the code path used semantic functions will have different max tokens #2738
Assignees
Milestone
Comments
4 tasks
|
Related to #1362 |
github-merge-queue bot
pushed a commit
that referenced
this issue
Sep 16, 2023
### Motivation and Context ####⚠️ Breaking change. When `Kernel.InvokeSemanticFunctionAsync` is used, `max_tokens` is set to `256` by default. However if you call `Kernel.CreateSemanticFunction`, `max_tokens` is set to `null` by default so the model default value is used. The reason we are setting `max_tokens` to 256 is because OpenAI text completion models have a default of 16. However OpenAI chat completion models have a default of infinity. So the current default makes sense for text completions but not for chat completions. To fix this, the default value of `max_tokens` will always be set to `null`. #### Troubleshooting Enable informational logging and check for token usage logs e.g., ``` Action: GetCompletionsAsync. Azure OpenAI Deployment Name: text-davinci-003. Prompt tokens: 14. Completion tokens: 16. Total tokens: 30. ``` #### Here's the documentation for OpenAI: https://platform.openai.com/docs/api-reference/completions/create#max_tokens `max_tokens` `integer or null` `Optional` `Defaults to 16` The maximum number of [tokens](https://platform.openai.com/tokenizer) to generate in the completion. https://platform.openai.com/docs/api-reference/chat/create#max_tokens `max_tokens` `integer or null` `Optional` `Defaults to inf` The total length of input tokens and generated tokens is limited by the model's context length. Resolves: #2738 ### Description The property `max_tokens` is an optional so we should not set a default value in SK. This change makes SK consistent in this approach. ### Contribution Checklist - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄
SOE-YoungS
pushed a commit
to SOE-YoungS/semantic-kernel
that referenced
this issue
Nov 1, 2023
### Motivation and Context ####⚠️ Breaking change. When `Kernel.InvokeSemanticFunctionAsync` is used, `max_tokens` is set to `256` by default. However if you call `Kernel.CreateSemanticFunction`, `max_tokens` is set to `null` by default so the model default value is used. The reason we are setting `max_tokens` to 256 is because OpenAI text completion models have a default of 16. However OpenAI chat completion models have a default of infinity. So the current default makes sense for text completions but not for chat completions. To fix this, the default value of `max_tokens` will always be set to `null`. #### Troubleshooting Enable informational logging and check for token usage logs e.g., ``` Action: GetCompletionsAsync. Azure OpenAI Deployment Name: text-davinci-003. Prompt tokens: 14. Completion tokens: 16. Total tokens: 30. ``` #### Here's the documentation for OpenAI: https://platform.openai.com/docs/api-reference/completions/create#max_tokens `max_tokens` `integer or null` `Optional` `Defaults to 16` The maximum number of [tokens](https://platform.openai.com/tokenizer) to generate in the completion. https://platform.openai.com/docs/api-reference/chat/create#max_tokens `max_tokens` `integer or null` `Optional` `Defaults to inf` The total length of input tokens and generated tokens is limited by the model's context length. Resolves: microsoft#2738 ### Description The property `max_tokens` is an optional so we should not set a default value in SK. This change makes SK consistent in this approach. ### Contribution Checklist - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
Noticed this while create a new integration test
Calling
Kernel.InvokeSemanticFunctionAsync, the max tokens will default to 256. If you callKernel.CreateSemanticFunctionthe max tokens will not be set. Proposed fix is to use null as the default value as the property is optional.Here's the documentation for OpenAI:
https://platform.openai.com/docs/api-reference/completions/create#max_tokens
max_tokensinteger or nullOptionalDefaults to 16The maximum number of tokens to generate in the completion.
https://platform.openai.com/docs/api-reference/chat/create#max_tokens
max_tokensinteger or nullOptionalDefaults to infThe total length of input tokens and generated tokens is limited by the model's context length.
The text was updated successfully, but these errors were encountered: