Glue::CfnTable: Not able to create a table by providing schema_name and registry_name

[mehdimld]

Describe the bug

I've encountered an issue while trying to create a Glue table using the aws_glue.CfnTable construct in the AWS CDK Python library. According to the documentation, I should be able to specify the schema_reference using registry_name and schema_name. However, when I attempt to deploy this configuration, the deployment fails with an error indicating that these pieces of information are not being provided. Below is a screenshot of the code snippet I use and the error I get at deployment.

Expected Behavior

The table should be created with the schema referenced, as I provided both registry and schema name.

Current Behavior

Table creation fails

Reproduction Steps

Try to create a glue table using aws_glue.CfnTable, reference a schema and a registry.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.132.1

Framework Version

No response

Node.js Version

20.10.0

OS

Windows

Language

Python

Language Version

3.11

Other information

No response

[ashishdhingra]

@mehdimld Good morning. Thanks for opening the issue. The above error is returned by CloudFormation when the synthesized CF template is deployed. Could you please share the following:

• Minimal reproduction code for your CDK stack to reproduce the issue.
• Have you tried latest version of aws-cdk-lib?
• Share the output CloudFormation template of cdk synth command.

I tried reproducing the issue using CDK stack written both in TypeScript and Python. The properties RegistryName and SchemaName both appear to be set in output CFN template when executing cdk synth.

TypeScript

CDK stack

import * as cdk from 'aws-cdk-lib';
import * as glue from 'aws-cdk-lib/aws-glue';
import { Construct } from 'constructs';

export class TypescriptStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const cfnTable = new glue.CfnTable(this, 'MyCfnTable', {
      catalogId: 'testcatalogid',
      databaseName: 'testdatabasename',
      tableInput: {
        description: 'description',
        tableType: 'EXTERNAL_TABLE',
        name: 'testtablename',
        partitionKeys: [{
          name: 'name',

          // the properties below are optional
          comment: 'comment',
          type: 'type',
        }],
        storageDescriptor: {
          location: 's3://testbucket/',
          schemaReference: {
            schemaId: {
              registryName: 'testregistryName',
              schemaName: 'testschemaName',
            }
          }
        },
      }
    });
  }
}

cdk synth output

Resources:
  MyCfnTable:
    Type: AWS::Glue::Table
    Properties:
      CatalogId: testcatalogid
      DatabaseName: testdatabasename
      TableInput:
        Description: description
        Name: testtablename
        PartitionKeys:
          - Comment: comment
            Name: name
            Type: type
        StorageDescriptor:
          Location: s3://testbucket/
          SchemaReference:
            SchemaId:
              RegistryName: testregistryName
              SchemaName: testschemaName
        TableType: EXTERNAL_TABLE
    Metadata:
      aws:cdk:path: TypescriptStack/MyCfnTable
  CDKMetadata:
    Type: AWS::CDK::Metadata
...
...

Python

CDK stack

from aws_cdk import (
    # Duration,
    Stack,
    aws_glue as glue,
)
from constructs import Construct

class PythonStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # The code that defines your stack goes here

        # example resource
        cfnTable = glue.CfnTable(
            self, "MyCfnTable",
            catalog_id="testcatalogid",
            database_name="testdatabasename",
            table_input=glue.CfnTable.TableInputProperty(
                description="description",
                table_type="EXTERNAL_TABLE",
                name="testtablename",
                partition_keys=[glue.CfnTable.ColumnProperty(name="name", comment="comment", type="type")],
                storage_descriptor=glue.CfnTable.StorageDescriptorProperty(
                    location="s3://testbucket/",
                    schema_reference=glue.CfnTable.SchemaReferenceProperty(
                        schema_id=glue.CfnTable.SchemaIdProperty(
                            registry_name="testregistryName",
                            schema_name="testschemaName"
                        )
                    )
                )
            )
        )

cdk synth output

Resources:
  MyCfnTable:
    Type: AWS::Glue::Table
    Properties:
      CatalogId: testcatalogid
      DatabaseName: testdatabasename
      TableInput:
        Description: description
        Name: testtablename
        PartitionKeys:
          - Comment: comment
            Name: name
            Type: type
        StorageDescriptor:
          Location: s3://testbucket/
          SchemaReference:
            SchemaId:
              RegistryName: testregistryName
              SchemaName: testschemaName
        TableType: EXTERNAL_TABLE
    Metadata:
      aws:cdk:path: PythonStack/MyCfnTable
  CDKMetadata:
    Type: AWS::CDK::Metadata
...
...

Thanks,
Ashish

[mehdimld]

Hey @ashishdhingra, I have tested it locally since then and have not been able to reproduce the problem. It might be related to the cdk/node version of our Jenkins instance (the one running the cdk deploy in the above screenshot). Although the code snippet you provided works correctly during the synth, it fails at deployment time, asking for the schema_version_number (or schema_version_id, but providing this prevents you from providing the registry_name and schema_name). Thanks a lot for your help; we can close the issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[aws-cdk-automation]

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.