Added a text completion connector to oobabooga text-generation-webui project #1357

jsboige · 2023-06-07T01:14:39Z

Motivation and Context

Oobabooga's text-generation-webui is the most popular open-source platform to host LLMs. It has a web API that supports blocking and streaming requests.
Having a connector to oobabooga brings semantic-kernel to the global "local Llamas" community.

Description

This PR adds to the solution a project similar to HuggingFace connectors project, and an additional integration test also similar to HuggingFace connector's
The code for the connector was based on the existing HuggingFace's, with a couple improvements (e.g. using web sockets for streaming API)

Contribution Checklist

The code builds clean without any errors or warnings
The PR follows SK Contribution Guidelines (https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
The code follows the .NET coding conventions (https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions) verified with dotnet format
All unit tests pass, and I have added new tests where possible
As for HuggingFace, the integration tests are meant to be tested manually together with oobabooga installed with a model selected and a running API. Both the blocking and streaming APIs were tested succesfully, though parameters mappings from sk's CompleteRequestSettings to oobabooga's TextCompletionRequest was done very roughly and would need adjusting since some units scales don't match, and oobabooga's API has a lot more parameters to feed, but I suppose it works well enough for an first version.
I didn't break anyone 😄

Merge last changes

jsboige · 2023-06-07T01:17:06Z

@microsoft-github-policy-service agree company="My Intelligence Agency"

dmytrostruk

@jsboige Awesome work, thank you! I left initial comments, when they will be resolved, I will test this functionality locally and if everything works as expected, we will merge it to main! Thanks again!

dotnet/src/Connectors/Connectors.AI.Oobabooga/TextCompletion/OoobaboogaTextCompletion.cs

dotnet/src/Connectors/Connectors.AI.Oobabooga/TextCompletion/TextCompletionRequest.cs

dotnet/src/Connectors/Connectors.AI.Oobabooga/TextCompletion/OoobaboogaTextCompletion.cs

dotnet/src/IntegrationTests/Connectors/Oobabooga/OobaboogaTextCompletionTests.cs

Merge main

… test websocket server

…ic-kernel

Merge upstream

…ic-kernel

dmytrostruk

Tested this connector locally using integration tests - everything works as expected.

Merge upstream

dmytrostruk · 2023-06-09T13:00:37Z

Please ignore Spell Check pipeline issue for now - we are going to fix that separately. PR still can be merged.

dotnet/src/Connectors/Connectors.AI.Oobabooga/TextCompletion/OobaboogaTextCompletion.cs

…stblocks

merging upstream

jsboige · 2023-07-03T13:25:12Z

@dmytrostruk , @RogerBarreto,
Hi guys, as you may see, I'm strill struggling trying to figure out what's going on with the TextCompletionStreamingResult alternatives.
After some more tinkering and refactoring, it would seem that Channels do actually support concurrent asynchronous broadcasts of completion streams, which does not fit my understanding of how they work.
Conversely, I'm now struggling to get BroadcastBlocks to work with a rare racing condition.
Accordingly, I added logging and many unit tests to investigate those issues, and there will be some cleanup whether we decide for a single All-use-cases best implementation or not.

In the mean time, I would really appreciate your feedback on those complicated issues.

Edit: It seems you've been busy, so I just reverted to a single Channel based Result class since it seems to do the job, and wrapped up the unit tests.
I removed the dependency to the Dataflow library and set aside the alternatives in a dedicated branch if you ever want to give it a try.

…it tests

merge upstream

dmytrostruk · 2023-07-04T13:57:22Z

@jsboige Thank you for the updates. I will review this PR and reply shortly.

update upstream

merge upstream

jsboige · 2023-07-07T08:08:33Z

@jsboige Thank you for the updates. I will review this PR and reply shortly.

Hi @dmytrostruk, do you have any news concerning this PR? I'll be happy to contribute some more, but since this has been open for quite some time now, and it does add a new project to the solution, it would be nice if we can wrap it up and move on.

Request answered

shawncal · 2023-07-08T04:00:57Z

@jsboige Thanks for all the hard work! We'll help take this home.
Note: Our PR process requires that all changes be synced up with main before merge, and that's obviously a moving target. Normally we'd merge the latest changes for you before merging your PR, but it looks like you have the "Allow edits by maintainers" box turned off, so we can't merge into your PR branch.

I've merged the whole PR into a feature branch in the sk repo, where we'll easily be able to address any conflicts and @dmytrostruk will bring it into main. If there are any further changes you'd like to push in the meantime, please add them to the "feature/oobabooga" branch.

Thanks again!

jsboige · 2023-07-08T14:54:37Z

Hi @shawncal , thanks for moving on with merging this PR.
About the "Allow edits by maintainers" feature, it appears that there is a bug preventing to enable that feature when the fork was made under an organization instead of a user, which is the case with this fork. Accordingly, I created a new Fork under my username and will start a new PR to that new branch from that new fork to help with finishing this up.

Hi, this is a new PR from a new user-level fork to account for a github bug preventing edits by maintainers on a PR made from an organization-level fork (see that [final comment](#1357 (comment)) on last PR) This PR is only an attempt at merging upstream's main for final integration of the oobabooga connector into main branch. Last set of commits made CompleteRequestSettings.MaxTokens Nullable ([#450f1d3a11eb95d6975da33f581d3997bed42906](#1367)), which broke this connector. Making TextCompletionRequest.MaxNewTokens also nullable fixed the issue. Note that Oobabooga [defaults max tokens to 200](https://github.com/oobabooga/text-generation-webui/blob/main/extensions/api/util.py#L24) Co-authored-by: Shawn Callegari <36091529+shawncal@users.noreply.github.com>

### Motivation and Context  Merge [feature/oobabooga](https://github.com/microsoft/semantic-kernel/tree/feature/oobabooga) branch to `main` with [Oobabooga](https://github.com/oobabooga/text-generation-webui) AI Connector functionality. Functionality verified with unit and integration testing. ### Description  From original PR (#1357): > This PR adds to the solution a project similar to HuggingFace connectors project, and an additional integration test also similar to HuggingFace connector's The code for the connector was based on the existing HuggingFace's, with a couple improvements (e.g. using web sockets for streaming API) ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#dev-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 Co-authored-by: Jean-Sylvain Boige <jsboige@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Jean-Sylvain Boige <jsboige@gmail.com> Co-authored-by: Shawn Callegari <36091529+shawncal@users.noreply.github.com> Co-authored-by: Gina Triolo <51341242+gitri-ms@users.noreply.github.com> Co-authored-by: Devis Lucato <dluc@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Craig Presti <146438+craigomatic@users.noreply.github.com> Co-authored-by: Craig Presti <craig.presti@microsoft.com> Co-authored-by: Mark Wallace <127216156+markwallace-microsoft@users.noreply.github.com> Co-authored-by: Teresa Hoang <125500434+teresaqhoang@users.noreply.github.com> Co-authored-by: Abby Harrison <54643756+awharrison-28@users.noreply.github.com> Co-authored-by: Tao Chen <TaoChenOSU@users.noreply.github.com> Co-authored-by: Aman Sachan <51973971+amsacha@users.noreply.github.com> Co-authored-by: cschadewitz <schadewitzcasey@gmail.com> Co-authored-by: Abby Harrison <abby.harrison@microsoft.com>

### Motivation and Context  Merge [feature/oobabooga](https://github.com/microsoft/semantic-kernel/tree/feature/oobabooga) branch to `main` with [Oobabooga](https://github.com/oobabooga/text-generation-webui) AI Connector functionality. Functionality verified with unit and integration testing. ### Description  From original PR (microsoft#1357): > This PR adds to the solution a project similar to HuggingFace connectors project, and an additional integration test also similar to HuggingFace connector's The code for the connector was based on the existing HuggingFace's, with a couple improvements (e.g. using web sockets for streaming API) ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#dev-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 Co-authored-by: Jean-Sylvain Boige <jsboige@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Jean-Sylvain Boige <jsboige@gmail.com> Co-authored-by: Shawn Callegari <36091529+shawncal@users.noreply.github.com> Co-authored-by: Gina Triolo <51341242+gitri-ms@users.noreply.github.com> Co-authored-by: Devis Lucato <dluc@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Craig Presti <146438+craigomatic@users.noreply.github.com> Co-authored-by: Craig Presti <craig.presti@microsoft.com> Co-authored-by: Mark Wallace <127216156+markwallace-microsoft@users.noreply.github.com> Co-authored-by: Teresa Hoang <125500434+teresaqhoang@users.noreply.github.com> Co-authored-by: Abby Harrison <54643756+awharrison-28@users.noreply.github.com> Co-authored-by: Tao Chen <TaoChenOSU@users.noreply.github.com> Co-authored-by: Aman Sachan <51973971+amsacha@users.noreply.github.com> Co-authored-by: cschadewitz <schadewitzcasey@gmail.com> Co-authored-by: Abby Harrison <abby.harrison@microsoft.com>

iamianM · 2023-08-01T01:56:58Z

Any chance this could be done for the python package please?

jsboige added 3 commits June 7, 2023 02:37

Added Text completion connector to oobabooga API

ea0b0c4

Merge pull request #1 from microsoft/main

c07af5f

Merge last changes

Moved file to the proper directory and addressed null warnings

aae5bfb

github-actions bot added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel labels Jun 7, 2023

dmytrostruk reviewed Jun 7, 2023

View reviewed changes

dmytrostruk added the PR: feedback to address Waiting for PR owner to address comments/questions label Jun 7, 2023

jsboige added 15 commits June 7, 2023 23:50

Merge pull request #2 from microsoft/main

e7c052e

Merge main

Fixed typo in file name

cf742c4

Removed HttpClientHandler parameter in favor of HttpClient

cafa7ce

Added websocket client for streaming as an optional ctor parameter

5e00589

introduced constants for streaming response object event types

33d3d70

Updated and explicited presence to repetition penalty conversion

1f54bcc

Added XML documentation to Oobabooga generation parameters

91f37f7

Added Mirostat to vocabulary

fef0a0a

Updated Oobabooga integration tests and added unit tests, introducing…

6ada20a

… test websocket server

Added Oobabooga integration tests instructions

6d1a4c5

Merge branch 'main' of https://github.com/MyIntelligenceAgency/semant…

79081de

…ic-kernel

Merge pull request #3 from microsoft/main

262f19c

Merge upstream

Merge branch 'main' of https://github.com/MyIntelligenceAgency/semant…

935fbdc

…ic-kernel

Updated Oobabooga connector to introduce the new ModelResult structure

7150d7d

Removed Async warning since System.Linq.Async is not available yet

7025f24

dmytrostruk previously approved these changes Jun 9, 2023

View reviewed changes

Merge pull request #4 from microsoft/main

7c0c830

Merge upstream

dmytrostruk added PR: ready to merge PR has been approved by all reviewers, and is ready to merge. and removed PR: feedback to address Waiting for PR owner to address comments/questions labels Jun 9, 2023

SergeyMenshykh reviewed Jun 9, 2023

View reviewed changes

dotnet/src/Connectors/Connectors.AI.Oobabooga/TextCompletion/OobaboogaTextCompletion.cs Outdated Show resolved Hide resolved

lemillermicrosoft removed the PR: ready to merge PR has been approved by all reviewers, and is ready to merge. label Jun 9, 2023

jsboige added 4 commits July 3, 2023 10:38

removing inheritence for Broadcastblock result

6d87abf

Trying to fix broadcastblock

65b7455

Adding some more comparison tests and some more trying to fix broadca…

f63ef76

…stblocks

Merge pull request #12 from microsoft/main

c9bbfc7

merging upstream

jsboige requested a review from dmytrostruk July 3, 2023 13:35

jsboige added 3 commits July 3, 2023 15:53

Fixed pooling stress tests without semaphore

76e2046

Added some more logging information

12ad62d

reverting to single Channel based streaming result and wrapping up un…

054fc2e

…it tests

github-actions bot added the docs and tests Improvements or additions to documentation label Jul 4, 2023

jsboige added 2 commits July 4, 2023 15:05

Merge pull request #13 from microsoft/main

5cc574e

merge upstream

Removing dependency to dataflow

09baa2c

jsboige requested a review from SergeyMenshykh July 4, 2023 13:15

jsboige added 3 commits July 6, 2023 10:22

Merge pull request #14 from microsoft/main

bd793b1

update upstream

adressing oobabooga CI warnings

b36d400

Merge pull request #15 from microsoft/main

7ac17e8

merge upstream

shawncal changed the base branch from main to feature/oobabooga July 8, 2023 03:54

shawncal merged commit 91292ab into microsoft:feature/oobabooga Jul 8, 2023
8 of 10 checks passed

jsboige mentioned this pull request Jul 8, 2023

feature/oobaboga: Fixing merge from main #1911

Merged

jsboige mentioned this pull request Jul 12, 2023

feature/oobaboga: Fixing merge from main (#1911) jsboige/semantic-kernel#5

Merged

5 tasks

dmytrostruk mentioned this pull request Jul 17, 2023

.Net: Merge feature/oobabooga branch to main #2016

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added a text completion connector to oobabooga text-generation-webui project #1357

Added a text completion connector to oobabooga text-generation-webui project #1357

jsboige commented Jun 7, 2023 •

edited

Loading

jsboige commented Jun 7, 2023 •

edited

Loading

dmytrostruk left a comment

dmytrostruk left a comment

dmytrostruk commented Jun 9, 2023 •

edited

Loading

jsboige commented Jul 3, 2023 •

edited

Loading

dmytrostruk commented Jul 4, 2023

jsboige commented Jul 7, 2023

shawncal commented Jul 8, 2023

jsboige commented Jul 8, 2023

iamianM commented Aug 1, 2023

Added a text completion connector to oobabooga text-generation-webui project #1357

Added a text completion connector to oobabooga text-generation-webui project #1357

Conversation

jsboige commented Jun 7, 2023 • edited Loading

Motivation and Context

Description

Contribution Checklist

jsboige commented Jun 7, 2023 • edited Loading

dmytrostruk left a comment

Choose a reason for hiding this comment

dmytrostruk left a comment

Choose a reason for hiding this comment

dmytrostruk commented Jun 9, 2023 • edited Loading

jsboige commented Jul 3, 2023 • edited Loading

dmytrostruk commented Jul 4, 2023

jsboige commented Jul 7, 2023

shawncal commented Jul 8, 2023

jsboige commented Jul 8, 2023

iamianM commented Aug 1, 2023

jsboige commented Jun 7, 2023 •

edited

Loading

jsboige commented Jun 7, 2023 •

edited

Loading

dmytrostruk commented Jun 9, 2023 •

edited

Loading

jsboige commented Jul 3, 2023 •

edited

Loading