Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.Net: Is it possible to use binary image data with a handlebars template? #7150

Closed
sophialagerkranspandey opened this issue Jul 8, 2024 Discussed in #7121 · 1 comment · Fixed by #8401
Closed

.Net: Is it possible to use binary image data with a handlebars template? #7150

sophialagerkranspandey opened this issue Jul 8, 2024 Discussed in #7121 · 1 comment · Fixed by #8401
Assignees
Labels
bug Something isn't working Ignite .NET Issue or Pull requests regarding .NET code sk team issue A tag to denote issues that where created by the Semantic Kernel team (i.e., not the community)

Comments

@sophialagerkranspandey
Copy link
Contributor

Discussed in #7121

Originally posted by jmservera July 6, 2024
I'm trying to do something like this:

var getDescription= kernel.CreateFunctionFromPrompt(
    """
    <message role="system>You are an AI assistant that based on a picture or its description you transform it to a funny title
    in English, French and Spanish.
    The output should be a json file with "en", "fr" and "es" as keys for the translations.
    Here is the output schema:
        {
         "en":"English description",
         "fr": "French translation",
         "es": "Spanish translation"
        }
    </message>
    <message role="user">
    {{#if image}}
    <image>
    {{image}}
    </image>
    {{/if}}
    <text>
    {{#if caption}}
    Here are the captions for the picture: {{caption}}
    {{/if}}
    {{#if people}}
    Here's the people that appear in the picture: {{ people }}
    {{/if}}
    Please provide a funny sentence for this.
    </text>
    </message>
    """, templateFormat: "handlebars",promptTemplateFactory: new HandlebarsPromptTemplateFactory());

but the <image> tag seems to only take uri as valid value. I've tried with ImageContent's DataUri but then I get a token limit exception:

ImageContent image=new(new ReadOnlyMemory<byte>(bytes), "image/jpeg");
    parameters.Add("image",image.DataUri);

Thanks,
Juanma

@markwallace-microsoft markwallace-microsoft added .NET Issue or Pull requests regarding .NET code triage labels Jul 8, 2024
@github-actions github-actions bot changed the title Is it possible to use binary image data with a handlebars template? .Net: Is it possible to use binary image data with a handlebars template? Jul 8, 2024
@markwallace-microsoft markwallace-microsoft added bug Something isn't working and removed triage labels Jul 9, 2024
@MarceloAGG
Copy link
Contributor

MarceloAGG commented Aug 27, 2024

I didn't realize there was already an issue for this, just created a trivial PR which fixes the problem: #8401

@evchaki evchaki added the sk team issue A tag to denote issues that where created by the Semantic Kernel team (i.e., not the community) label Aug 28, 2024
github-merge-queue bot pushed a commit that referenced this issue Aug 30, 2024
…er so templates can use base64 encoded images. (#8401)

### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

At present, including images in prompt templates using base64 data
encoding is not possible. This limitation is due to
`ChatPromptParser.cs` exclusively calling the `ImageContent` constructor
that requires a URI, which leads to an `InvalidOperationException`. The
change required is straightforward and the limitation has been discussed
before, [for example
here](#7121).
Closes #7150.

### Description

The proposed trivial fix involves a simple check to determine if the
content starts with `data:`, and if it does the `ImageContent`
constructor that accepts a `dataUri` is utilized instead.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Marcelo Garcia 🛸 <marcgarc@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Ignite .NET Issue or Pull requests regarding .NET code sk team issue A tag to denote issues that where created by the Semantic Kernel team (i.e., not the community)
Projects
Archived in project
5 participants