Support Anthropic claude-3-haiku model with text input

[giordafrancis]

@philipws I've been unable to define a model definition structure on DocTran-<branch>-app-basereadablereadableModelmodelTable-xxxxxxxx-xxxxxxxxxxxxx for model anthropic.claude-3-haiku-20240307-v1:0. Below is my attempt to create a structure. The Text will not generate in the app.

Any ideas about what is incorrect with the structure?
{ "id": { "S": "Text-01" }, "default": { "BOOL": true }, "name": { "S": "Text Simplifier" }, "text": { "M": { "modelId": { "S": "anthropic.claude-3-haiku-20240307-v1:0" }, "parameters": { "M": { "max_tokens": { "N": "300" }, "anthropic_version": { "S": "bedrock-2023-05-31" }, "stop_sequences": { "L": [ { "S": "Human:" } ] }, "temperature": { "N": "1" }, "top_k": { "N": "250" }, "top_p": { "N": "0.999" } } }, "prePrompt": { "S": "Simplify the following text so it can be easily understood by those with a low reading age, use short sentences, explain any abbreviations or words that can have two meanings and separate the sentences in to new lines." } } }, "type": { "S": "text" } }

[philipws]

Hey @giordafrancis

The changes from Claude v2 to Claude v3. The payload sent to Claude v3 differs from v2. Therefore the payload preparation done by this app needs to be updated/expanded to accommodate v3. I will edit the issue title to be a feature request for the model.

This is also faced in other projects

• Amazon.BedrockRuntime.Model.ValidationException: "claude-3-sonnet-20240229" is not supported on this API aws/aws-sdk-net#3217
• Bedrock integration with Claude3 fails with ValidationException "claude-3-sonnet-20240229" is not supported on this API. Please use the Messages API instead. langchain-ai/langchain#18513
• BedrockChat is not using Messages API for Anthropic v3 models langchain-ai/langchain#18514

In the mean time you can use the v2 models, with example here.

From my initial look at this it appears to be because Claude v3 support different types of input in addition to text, whereas v2 was just text. So the payload needs to specify that it is indeed text.

[philipws]

Initial support has been created upstream, but not in a tagged release yet, as it'll need some testing done.

• https://github.com/aws-samples/document-translation/blob/main/infrastructure/lib/features/readable/vendor/text.anthropic.claude-3-*-*-v1.ts

A dependency (CDK) also has an open issue that affects the custom resources for this project. This means we can't deploy the example models as references. It'd be nice to have this resolved for the next release.

• (custom-resources): Package does not exist aws/aws-cdk#30067

[philipws]

@giordafrancis

Readable: Add support for Titan text and Titan image models
Readable: Add support for Anthropic 3 Sonnet & Haiku models

https://github.com/aws-samples/document-translation/releases/tag/v2.3.0

Please test and review.

[giordafrancis]

Hi @philipws thank you for this and apologies for the delay in testing. I've now upgraded to v2.3.5 and pushed to codecommit. I've added the below item to DocTran-main-app-basereadablereadableModelmodelTablexxxx. I've tried to Generate some text in Simple Readable but no luck. Any idea if I'm missing something? Thank you in advance

{ "id": { "S": "Example-Text-02" }, "default": { "BOOL": true }, "name": { "S": "Text Simplifier new prompt Claude 3 haiku" }, "text": { "M": { "modelId": { "S": "anthropic.claude-3-haiku-20240307-v1:0" }, "parameters": { "M": { "anthropic_version": { "S": "bedrock-2023-05-31" }, "max_tokens_to_sample": { "N": "1000" }, "stop_sequences": { "L": [ { "S": "Human:" } ] }, "temperature": { "N": "1" }, "top_k": { "N": "250" }, "top_p": { "N": "0.999" } } }, "prePrompt": { "S": "Your task is to take the text provided and rewrite it in a way that is easy for people with learning difficulties. Assume this individuals are adults with a reading age equivalent to young learners in grades 3-5 to read and understand. Simplify advanced vocabulary, break down long sentences, explain difficult concepts in plain language, and present the information in a clear, engaging way.The short rewritten text should convey the core ideas of the original text in an reading age appropriate manner. " } } }, "type": { "S": "text" } }

[philipws]

Hey @giordafrancis

tl:dr; Try changing max_tokens_to_sample to max_tokens.

Anthropic changed the payload structure from the v2 to v3 models. It looks like you've edited the provided example which is made for v2. It appears that max_tokens_to_sample does not exist for the messages API https://docs.anthropic.com/en/api/messages

The v3 structure looks more like the below. Simply Readable takes care of building the text part, so we can ignore that, and focus on the other parameters. One that is different is max_tokens_to_sample to max_tokens.

{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "hello world"
        }
      ]
    }
  ],
  "anthropic_version": "bedrock-2023-05-31",
  "max_tokens": 2000,
  "temperature": 1,
  "top_k": 250,
  "top_p": 0.999,
  "stop_sequences": ["\\n\\nHuman:"]
}

For reference, within the AWS Bedrock console text playground you can select "View API Request". The payload is an escaped string value in the body field. I find this very helpful in taking what I've tested in Bedrock into the app.

[giordafrancis]

@philipws Thank you, that did it and worked like a charm. Feeling slightly embarrassed for not spotting that given that I've used v3 API in the past. Also thank you for the "View API request", a super helpful pointer.