Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: tools augmentation interrupted when include_images is true #2039

Open
1 of 3 tasks
corinagum opened this issue Sep 18, 2024 · 0 comments
Open
1 of 3 tasks

[Bug]: tools augmentation interrupted when include_images is true #2039

corinagum opened this issue Sep 18, 2024 · 0 comments
Labels
bug Something isn't working dotnet Change/fix applies to dotnet. If all three, use the 'JS & dotnet & Python' label Python Change/fix applies to Python. If all three, use the 'JS & dotnet & Python' label samples work wrt samples

Comments

@corinagum
Copy link
Collaborator

corinagum commented Sep 18, 2024 •
edited
Loading

Language

All 3 languages

Version

latest

Description

There are a number of issues

  • Sample does not work with Tools Augmentation. In the history, I can see an extraneous user message (empty) is being added after the tool_call is sent from the model, interrupting the flow where the model expects the next immediate message to be the tool_call response with tool_call_id.

Workaround

You can get this sample to work by changing the augmentation type in config.json to 'sequence'

Reproduction Steps

1. Set up CardGazer sample
4. Run sample by sending an image with text "Please turn this image into an Adaptive Card"
...

Tasks
Beta Give feedback
@corinagum corinagum added the bug Something isn't working label Sep 18, 2024
@corinagum corinagum self-assigned this Sep 18, 2024
@corinagum corinagum added the samples work wrt samples label Sep 18, 2024
corinagum pushed a commit that referenced this issue Sep 25, 2024
#2039 Fix CardGazer bug
e006305
@corinagum corinagum mentioned this issue Sep 25, 2024
Merged
1 task
@corinagum corinagum changed the title [Bug]: CardGazer sample does not work [Bug]: tools augmentation interrupted when include_images is true Sep 25, 2024
@corinagum corinagum added Python Change/fix applies to Python. If all three, use the 'JS & dotnet & Python' label dotnet Change/fix applies to dotnet. If all three, use the 'JS & dotnet & Python' label labels Sep 25, 2024
@corinagum corinagum mentioned this issue Sep 25, 2024
Open
corinagum added a commit that referenced this issue Sep 26, 2024
@corinagum
[JS] fix: CardGazer sample (#2059)
1e61623 
## Linked issues

#2039
#minor


<img width="935" alt="image"
src="https://github.com/user-attachments/assets/0018973c-9751-46d5-abd1-b6cbe889d9cc">

## Details

- `OpenAIModel.ts` docstring cleanup
- `PromptManager` - change order of tools and user input
- `UserInputMessage` - don't send messages if messages are empty.
- Update cardgazer to use 4o-mini and tools augmentation
- Update CardGazer sample to improve experience.
- Add note in README for Bot SSO sample


## Attestation Checklist

- [x] My code follows the style guidelines of this project

- I have checked for/fixed spelling, linting, and other errors
- I have commented my code for clarity
- I have made corresponding changes to the documentation (updating the
doc strings in the code is sufficient)
- My changes generate no new warnings
- I have added tests that validates my changes, and provides sufficient
test coverage. I have tested with:
  - Local testing
  - E2E testing in Teams
- New and existing unit tests pass locally with my changes

### Additional information

> Feel free to add other relevant information below

---------

Co-authored-by: Corina Gum <>
@corinagum corinagum removed their assignment Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working dotnet Change/fix applies to dotnet. If all three, use the 'JS & dotnet & Python' label Python Change/fix applies to Python. If all three, use the 'JS & dotnet & Python' label samples work wrt samples
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@corinagum