New Feature: Function calling w/ Context

[brandonh-msft]

Scenario

• Using auto-invoke tool calls
• User uploads files into a chat with a multi-agent system (implemented via Function Calling between Orchestrator and multiple, described, "agents" as Functions)
• Orchestrator determines Agent X is best suited to answer the user
• Agent X requires the files from the user to fully complete its job

Today, it seems the only things I can pass to a function call are JSON-serializable objects. However, in this scenario, I can send the LLM the image data via ImageContent message items but the LLM can only come back with, say, a filename that I surround this content with - not the content itself. This means that even if my Agent Function takes BinaryData as an input, the LLM doesn't provide it in its function call request. It can provide something like "Filename: X.png" which I could correlate back to the user's input, though, but I am unable to get back to that input to use it in the actual call to the agent.

So, enabling some way for me to get back to the original message which triggered the function call request is necessary for this scenario. I'm thinking something along the lines of an auto-injected parameter to the function, if the developer provides it on the signature (a la CancellationToken)