Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bedrock: L1 constructs error #29966

Closed
Samir-AWS opened this issue Apr 26, 2024 · 9 comments
Closed

bedrock: L1 constructs error #29966

Samir-AWS opened this issue Apr 26, 2024 · 9 comments
Assignees
Labels
bug This issue is a bug. p1 toolkit/migrate Related to cdk migrate

Comments

@Samir-AWS
Copy link

Describe the bug

Bedrock L1 constructs have been added via #29911 in version 2.139.0

Issue1
Unable to use cdk migrate to generate it from a CloudFormation stack

$ cdk migrate --stack-name Test2 --from-path ./test2.yml --language csharp

This command is an experimental feature.
 ❌  Migrate failed for `Test2`: Test2Stack could not be generated because Name is not a valid property for resource BedrockKnowledgeBase of type AWS::Bedrock::KnowledgeBase

Test2Stack could not be generated because Name is not a valid property for resource BedrockKnowledgeBase of type AWS::Bedrock::KnowledgeBase

Issue2
CfnKnowledgeBase class is missing a required property StorageConfiguration in CfnKnowledgeBaseProps

Issue3
I tried to deploy without this property anyway and ran into another issue-

$ cdk synth

<successfully synthesises - output omitted>

$ cdk deploy

Test: deploying... [1/1]
Test: creating CloudFormation changeset...

 ❌  Test failed: Error [ValidationError]: Template format error: Unrecognized resource types: [AWS::Bedrock::DataSource, AWS::Bedrock::KnowledgeBase]
    at Request.extractError (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:46692)
    at Request.callListeners (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:91559)
    at Request.emit (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:91007)
    at Request.emit (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:199533)
    at Request.transition (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:193085)
    at AcceptorStateMachine.runTo (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:157957)
    at /usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:158287
    at Request.<anonymous> (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:193377)
    at Request.<anonymous> (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:199608)
    at Request.callListeners (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:91727) {
  code: 'ValidationError',
  time: 2024-04-25T23:56:02.578Z,
  requestId: 'f5515800-5335-434b-9df6-d930bcf788c4',
  statusCode: 400,
  retryable: false,
  retryDelay: 236.99598614253415
}

 ❌ Deployment failed: Error [ValidationError]: Template format error: Unrecognized resource types: [AWS::Bedrock::DataSource, AWS::Bedrock::KnowledgeBase]
    at Request.extractError (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:46692)
    at Request.callListeners (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:91559)
    at Request.emit (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:91007)
    at Request.emit (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:199533)
    at Request.transition (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:193085)
    at AcceptorStateMachine.runTo (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:157957)
    at /usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:158287
    at Request.<anonymous> (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:193377)
    at Request.<anonymous> (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:199608)
    at Request.callListeners (/usr/local/Cellar/aws-cdk/2.139.0/libexec/lib/node_modules/aws-cdk/lib/index.js:385:91727) {
  code: 'ValidationError',
  time: 2024-04-25T23:56:02.578Z,
  requestId: 'f5515800-5335-434b-9df6-d930bcf788c4',
  statusCode: 400,
  retryable: false,
  retryDelay: 236.99598614253415
}

Template format error: Unrecognized resource types: [AWS::Bedrock::DataSource, AWS::Bedrock::KnowledgeBase]

Expected Behavior

  • Support for cdk migration
  • Addition of StorageConfiguration
  • Successful deployment without any validation error

Current Behavior

  • Migration is not possible for Bedrock resources
  • AWS::Bedrock::KnowledgeBase is invalid without StorageConfiguration property
  • Deployment error- not recognising AWS::Bedrock::DataSource, AWS::Bedrock::KnowledgeBase resources

Reproduction Steps

C#

        var bedrockKnowledgeBase = new CfnKnowledgeBase(this, "BedrockKnowledgeBase", new CfnKnowledgeBaseProps
        {
            Name = "bedrock-dotnet-demo-kb",
            KnowledgeBaseConfiguration = new KnowledgeBaseConfigurationProperty
            {
                Type = "VECTOR",
                VectorKnowledgeBaseConfiguration = new VectorKnowledgeBaseConfigurationProperty
                {
                    EmbeddingModelArn = $"arn:aws:bedrock:{Region}::foundation-model/amazon.titan-embed-text-v1"
                }
            },
            RoleArn = bedrockExecutionRoleForKnowledgeBase.AttrArn
        });

        var bedrockKnowledgeBaseDataStore = new CfnDataSource(this, "BedrockKnowledgeBaseDataStore", new CfnDataSourceProps
        {
            Name = "bedrock-dotnet-demo-kb-data-source",
            KnowledgeBaseId = bedrockKnowledgeBase.Ref,
            DataSourceConfiguration = new DataSourceConfigurationProperty
            {
                Type = "S3",
                S3Configuration = new S3DataSourceConfigurationProperty
                {
                    BucketArn = bedrockRagDataBucket.AttrArn,
                    InclusionPrefixes = new[] { "kb-" }
                }
            }
        });

Possible Solution

No response

Additional Information/Context

$ cdk version
2.139.0 (build 2699ffd)

DotNet Amazon.CDK.Lib NuGet version- 2.139.0

OS- MacOS Sonoma

CDK CLI Version

2.139.0

Framework Version

No response

Node.js Version

18.17.0

OS

MacOS Sonoma 14.4.1

Language

.NET

Language Version

8.0.201

Other information

No response

@Samir-AWS Samir-AWS added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Apr 26, 2024
@github-actions github-actions bot added the @aws-cdk/aws-cloudformation Related to AWS CloudFormation label Apr 26, 2024
@ssinghgai
Copy link

+1 - Observed the same behavior - "#: required key [StorageConfiguration] not found ""
image

@khushail khushail added investigating This issue is being investigated and/or work is in progress to resolve the issue. p2 and removed needs-triage This issue or PR still needs to be triaged. labels Apr 26, 2024
@khushail
Copy link
Contributor

khushail commented Apr 26, 2024

+1 - Observed the same behavior - "#: required key [StorageConfiguration] not found "" image

@Samir-AWS , Thanks for reporting this issue. I am also facing the same deployment issue as shared by @ssinghgai.

I see there is some mismatch in the Cfn Service spec mentioning the StorageConfiguration -
Screenshot 2024-04-26 at 12 01 24 PM

and the CDK Documentation for CfnKnowledgeBaseProps.

CDK imports servicespec using spec2cdk and based on the package.json, current latest CDK release is using 0.0.64 . I also see some discrepancies in documentation as well - Here is AWS Documentation and this is CDK documentation. Marking the issue as appropriate.

@khushail khushail added effort/medium Medium work item – several days of effort and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Apr 26, 2024
@pahud
Copy link
Contributor

pahud commented Apr 26, 2024

I can confirmed StorageConfiguration is available in CFN spec 171.0.0 at us-east-1.

The aws-bedrock-knowledgebase.json of awscdk-service-spec@v0.0.65 is having StorageConfiguration now. This means StorageConfiguration is available in awscdk-service-spec@v0.0.65

Looks like the most recent aws-cdk-lib@2.139.0 is still using awscdk-service-spec@0.0.64

I believe the next aws-cdk release should include awscdk-service-spec@0.0.65 that has the StorageConfiguration support.

@TheRealAmazonKendra
Copy link
Contributor

The addition of resources in cdk migrate (in cdk-from-cfn) is independent of their addition into aws-cdk-lib. We had been blocked from releasing in that repo due to an unrelated issue so no updates made it into the last cdk release. We can now release again so these updates should make it into the next release.

@TheRealAmazonKendra TheRealAmazonKendra added toolkit/migrate Related to cdk migrate and removed @aws-cdk/aws-cloudformation Related to AWS CloudFormation effort/medium Medium work item – several days of effort labels Apr 29, 2024
@TheRealAmazonKendra TheRealAmazonKendra self-assigned this Apr 29, 2024
@piradeepk piradeepk added p1 and removed p2 labels May 1, 2024
@colifran
Copy link
Contributor

colifran commented May 2, 2024

Hi @Samir-AWS. This PR has added StorageConfiguration into our generated bedrock L1s.

@colifran colifran closed this as completed May 2, 2024
Copy link

github-actions bot commented May 2, 2024

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@Samir-AWS
Copy link
Author

Thank you team for fixing this issue. StorageConfiguration is now available and I could deploy successfully in us-east-1 and us-west-2

However, the deployment issue still exists in ap-southeast-2 region. I haven't tried any other regions, but assuming those might have this issue as well.

$ cdk deploy

❌  bedrock-dotnet-demo failed: Error [ValidationError]: Template format error: Unrecognized resource types: [AWS::Bedrock::DataSource, AWS::Bedrock::KnowledgeBase]
    at Request.extractError (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:46692)
    at Request.callListeners (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:91600)
    at Request.emit (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:91048)
    at Request.emit (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:199651)
    at Request.transition (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:193203)
    at AcceptorStateMachine.runTo (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:158075)
    at /usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:158405
    at Request.<anonymous> (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:193495)
    at Request.<anonymous> (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:199726)
    at Request.callListeners (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:91768) {
  code: 'ValidationError',
  time: 2024-05-05T23:30:51.682Z,
  requestId: '3f9ec639-1b0f-4eee-8258-1a60cfc7396c',
  statusCode: 400,
  retryable: false,
  retryDelay: 8.307862464785432
}

 ❌ Deployment failed: Error [ValidationError]: Template format error: Unrecognized resource types: [AWS::Bedrock::DataSource, AWS::Bedrock::KnowledgeBase]
    at Request.extractError (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:46692)
    at Request.callListeners (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:91600)
    at Request.emit (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:91048)
    at Request.emit (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:199651)
    at Request.transition (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:193203)
    at AcceptorStateMachine.runTo (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:158075)
    at /usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:158405
    at Request.<anonymous> (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:193495)
    at Request.<anonymous> (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:199726)
    at Request.callListeners (/usr/local/Cellar/aws-cdk/2.140.0/libexec/lib/node_modules/aws-cdk/lib/index.js:387:91768) {
  code: 'ValidationError',
  time: 2024-05-05T23:30:51.682Z,
  requestId: '3f9ec639-1b0f-4eee-8258-1a60cfc7396c',
  statusCode: 400,
  retryable: false,
  retryDelay: 8.307862464785432
}

Template format error: Unrecognized resource types: [AWS::Bedrock::DataSource, AWS::Bedrock::KnowledgeBase]

cc: @colifran @TheRealAmazonKendra @pahud

@colifran
Copy link
Contributor

colifran commented May 6, 2024

@Samir-AWS It doesn't look like AWS::Bedrock::Datasource or AWS::Bedrock::KnowledgeBase are available in ap-southeast-2 - https://d2stg8d246z9di.cloudfront.net/latest/gzip/CloudFormationResourceSpecification.json. I don't actually see any bedrock resources available in ap-southeast-2. Here is the AWS CloudFormation resource specification page - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html.

@aws-cdk-automation
Copy link
Collaborator

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

@aws aws locked as resolved and limited conversation to collaborators Jul 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug This issue is a bug. p1 toolkit/migrate Related to cdk migrate
Projects
None yet
Development

No branches or pull requests

8 participants