Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot create Aurora Serverless cluster using RDS Construct #929

Closed
patoncrispy opened this issue Oct 14, 2018 · 30 comments · Fixed by #10516
Closed

Cannot create Aurora Serverless cluster using RDS Construct #929

patoncrispy opened this issue Oct 14, 2018 · 30 comments · Fixed by #10516
Assignees
Labels
@aws-cdk/aws-rds Related to Amazon Relational Database effort/large Large work item – several weeks of effort feature-request A feature should be added or improved. in-progress This issue is being actively worked on. p1
Milestone

Comments

@patoncrispy
Copy link

patoncrispy commented Oct 14, 2018

There doesn't seem to be a way to create an Aurora Serverless database cluster using the RDS Construct. EngineMode is available in the underlying cfn but not in the Construct library.

Target Framework: netcoreapp2.1
Amazon.CDK: 0.12.0
Amazon.CDK.AWS.RDS: 0.12.0

@patoncrispy patoncrispy changed the title Cannot create Aurora Serverless cluster using RDS Construct [DotNetCore] - Cannot create Aurora Serverless cluster using RDS Construct Oct 14, 2018
@eladb
Copy link
Contributor

eladb commented Oct 14, 2018

Thanks for reporting. As a workaround, you should be able to use property overrides

@eladb eladb added the feature label Oct 14, 2018
@patoncrispy
Copy link
Author

Ah sweet, thanks!

@rix0rrr rix0rrr changed the title [DotNetCore] - Cannot create Aurora Serverless cluster using RDS Construct Cannot create Aurora Serverless cluster using RDS Construct Oct 16, 2018
@rix0rrr rix0rrr added feature-request A feature should be added or improved. and removed feature-request A feature should be added or improved. feature labels Nov 6, 2018
@debora-ito debora-ito added the @aws-cdk/aws-rds Related to Amazon Relational Database label Nov 8, 2018
@rix0rrr rix0rrr added the gap label Jan 4, 2019
@ApocDev
Copy link

ApocDev commented Jul 30, 2019

I set this up for our usage, it's ugly, but it may be of use to someone who needs a "quick and dirty" version of Serverless Aurora:

Please ignore the lack of comments, and possibly broken code, I removed some proprietary information. 🤕

import {Connections, ISecurityGroup, IVpc, Port, SecurityGroup, SubnetSelection} from "@aws-cdk/aws-ec2";
import {
    CfnDBCluster,
    CfnDBSubnetGroup,
    DatabaseSecret,
    Endpoint,
    SecretRotation,
    SecretRotationApplication,
    SecretRotationOptions
} from "@aws-cdk/aws-rds";
import {AttachmentTargetType, ISecretAttachmentTarget, SecretAttachmentTargetProps, SecretTargetAttachment} from "@aws-cdk/aws-secretsmanager";
import {Construct, RemovalPolicy, Token} from "@aws-cdk/core";

export interface ServerlessAuroraProps {
    readonly vpc: IVpc;
    readonly subnets: SubnetSelection;

    readonly clusterName: string;

    readonly masterUsername?: string;
    readonly securityGroup?: ISecurityGroup;
    readonly secretRotationApplication?: SecretRotationApplication;

    readonly maxCapacity: number;
}

export class ServerlessAurora extends Construct implements ISecretAttachmentTarget {
    public securityGroupId: string;
    public clusterIdentifier: string;
    public clusterEndpoint: Endpoint;
    public secret: SecretTargetAttachment;
    public connections: Connections;
    public vpc: IVpc;
    public vpcSubnets: SubnetSelection;
    public secretRotationApplication: SecretRotationApplication;
    public securityGroup: ISecurityGroup;

    constructor(scope: Construct, id: string, private props: ServerlessAuroraProps) {
        super(scope, id);

        this.vpc = props.vpc;
        this.vpcSubnets = props.subnets;
        this.secretRotationApplication = props.secretRotationApplication || SecretRotationApplication.MYSQL_ROTATION_SINGLE_USER;

        const secret = new DatabaseSecret(this, "MasterUserSecret", {
            username: props.masterUsername || "root",
        });

        const securityGroup = props.securityGroup || new SecurityGroup(this, "DatabaseSecurityGroup", {
            allowAllOutbound: true,
            description: `DB Cluster (${props.clusterName}) security group`,
            vpc: props.vpc
        });
        this.securityGroup = securityGroup;
        this.securityGroupId = securityGroup.securityGroupId;

        const cluster = new CfnDBCluster(this, "DatabaseCluster", {
            engine: "aurora",
            engineMode: "serverless",
            engineVersion: "5.6",

            dbClusterIdentifier: props.clusterName,

            masterUsername: secret.secretValueFromJson("username").toString(),
            masterUserPassword: secret.secretValueFromJson("password").toString(),

            dbSubnetGroupName: new CfnDBSubnetGroup(this, "db-subnet-group", {
                dbSubnetGroupDescription: `${props.clusterName} database cluster subnet group`,
                subnetIds: props.vpc.selectSubnets(props.subnets).subnetIds
            }).ref,

            vpcSecurityGroupIds: [securityGroup.securityGroupId],

            storageEncrypted: true,

            // Maximum here is 35 days
            backupRetentionPeriod: 35,

            scalingConfiguration: {
                autoPause: true,
                secondsUntilAutoPause: 300,
                minCapacity: 1,
                maxCapacity: props.maxCapacity
            }
        });
        cluster.applyRemovalPolicy(RemovalPolicy.DESTROY, {applyToUpdateReplacePolicy: true});

        this.clusterIdentifier = cluster.ref;
        // create a number token that represents the port of the cluster
        const portAttribute = Token.asNumber(cluster.attrEndpointPort);
        this.clusterEndpoint = new Endpoint(cluster.attrEndpointAddress, portAttribute);

        if (secret) {
            this.secret = secret.addTargetAttachment('AttachedSecret', {target: this});
        }
        const defaultPort = Port.tcp(this.clusterEndpoint.port);
        this.connections = new Connections({securityGroups: [securityGroup], defaultPort});

        // This is currently causing errors when deploying, since it uses a SAM template under the hood
        // Error: "Received malformed response from transform AWS::Serverless-2016-10-31"
        // It also adds in a warning from the CDK:
        // "This stack is using the deprecated `templateOptions.transform` property. Consider switching to `templateOptions.transforms`."
        // Which has notified *all* of our DevOps when we upgraded.
        // this.addRotationSingleUser("Rotation");
    }

    /**
     * Adds the single user rotation of the master password to this cluster.
     */
    public addRotationSingleUser(id: string, options?: SecretRotationOptions): SecretRotation {
        if (!this.secret) {
            throw new Error('Cannot add single user rotation for a cluster without secret.');
        }
        return new SecretRotation(this, id, {
            secret: this.secret,
            application: this.secretRotationApplication,
            vpc: this.vpc,
            vpcSubnets: this.vpcSubnets,
            target: this,
            automaticallyAfter: options ? options.automaticallyAfter : undefined,
        });
    }

    public asSecretAttachmentTarget(): SecretAttachmentTargetProps {
        return {
            targetId: this.clusterIdentifier,
            targetType: AttachmentTargetType.CLUSTER
        };
    }
}

@ryanvade
Copy link

Does #688 not implement this (it adds Aurora RDS)?
(Side note, @eladb your link is no longer valid, have a new one?)

@ApocDev
Copy link

ApocDev commented Aug 3, 2019

Does #688 not implement this (it adds Aurora RDS)?
(Side note, @eladb your link is no longer valid, have a new one?)

No. The DatabaseCluster doesn't expose the engineMode and scalingConfiguration properties. It also has other property requirements that are only valid for non-serverless instances. (Instance type, size, etc) Those options are not available in the serverless mode.

@ryanvade
Copy link

ryanvade commented Aug 4, 2019

I see, would it be preferable to add a "Serverless" Database Prop or instead change DatabaseClusterProps to support "Serverless" configurations?

@ApocDev
Copy link

ApocDev commented Aug 4, 2019

I think creating a separate resource is probably the best option considering most of the options on the current cluster type don't quite apply. E.g only MySQL and Postgres are available, and you can't call lambdas from stored procedures or queries, etc.

Serverless also has the "Web API" feature that may be useful to turn on from some stacks. (Though, I'm not sure that it's currently supported in Cfn)

@seawatts
Copy link

Thanks for reporting. As a workaround, you should be able to use property overrides

This link doesn't seem to work. Can you give an example of how to do this?

@shivaramani
Copy link

Thanks for reporting. As a workaround, you should be able to use property overrides

This link doesn't seem to work. Can you give an example of how to do this?

I also dont see that link working. Can you please an example of property overrides?

@yargevad
Copy link

yargevad commented Nov 8, 2019

This doc shows how to do property overrides.

@apurvis
Copy link

apurvis commented Nov 12, 2019

any hopes for a release that allows this?

@skinny85
Copy link
Contributor

@jogold is this a matter of adding an EngineMode enum with the values listed in https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html#cfn-rds-dbcluster-enginemode and an engineMode property to DatabaseClusterProps, or is this more complicated?

@jogold
Copy link
Contributor

jogold commented Nov 27, 2019

I think that it's more complicated than that and that it requires a separate class.

@aripalo
Copy link

aripalo commented Nov 27, 2019

@skinny85: It should also support EnableHttpEndpoint boolean property to allow enabling Data API.

Although this is not yet implemented in CfnDBCluster yet. See #5216 for more.

@jogold
Copy link
Contributor

jogold commented Nov 27, 2019

@skinny85 the DatabaseCluster was originally authored by @rix0rrr and @eladb I think. I only did the DatabaseInstance and added secret rotation to both. Maybe also check with them.

@bionicles
Copy link

I would use this

@asterikx
Copy link
Contributor

asterikx commented Mar 17, 2020

Is there any progress on this?

It would be really helpful to have an L2 construct for Aurora Serverless DB cluster including Data API, secret rotation, and methods to easily grant read/write access Lambdas or AppSync.

@StevenAskwith
Copy link

You can use Escape Hatches to modify the L2 Aurora Server construct into a Serverless version:

from aws_cdk import (
    aws_ec2 as ec2,
    aws_rds as rds,
    core
)

class CdkRdsPipelineStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc(
            self, "MyVpc",
            max_azs=2
        )

        aurora_sg = ec2.SecurityGroup(
            self, 'AU-SG',
            vpc=vpc,
            description="Allows MySQL connections",
        )

        # Aurora MySQL Cluster with Secrets Manager managed admin user
        aurora_cluster = rds.DatabaseCluster(
            self, "AU-CLUSTER-1",
            engine=rds.DatabaseClusterEngine.AURORA,
            engine_version="5.6.10a", # aws rds describe-db-engine-versions --engine aurora --query "DBEngineVersions[].EngineVersion" --region eu-west-1
            master_user=rds.Login(
                username='admin'
            ),
            default_database_name='MyDatabase',
            instance_props=rds.InstanceProps(
                instance_type=ec2.InstanceType('t3.small'),
                vpc=vpc,
                security_group=aurora_sg,
            ), 
            instances=1 # delete this 'Server' in an escape hatch below
        )

        # escape hatch https://docs.aws.amazon.com/cdk/latest/guide/cfn_layer.html#cfn_layer_raw
        cfn_aurora_cluster = aurora_cluster.node.default_child
        cfn_aurora_cluster.add_override("Properties.EngineMode", "serverless")
        cfn_aurora_cluster.add_override("Properties.EnableHttpEndpoint",True) # Enable Data API
        cfn_aurora_cluster.add_override("Properties.ScalingConfiguration", { 
            'AutoPause': True, 
            'MaxCapacity': 4, 
            'MinCapacity': 1, 
            'SecondsUntilAutoPause': 600
        }) 
        aurora_cluster.node.try_remove_child('Instance1') # Remove 'Server' instance that isn't required for serverless Aurora

        # Add Secrets Manager Password rotation
        aurora_cluster.add_rotation_single_user()

@ericzbeard
Copy link
Contributor

See #8686. That should be merged before making changes on this issue.

@skinny85 skinny85 added this to the RDS to 'stable' milestone Jul 8, 2020
@mikestopcontinues
Copy link

I built on some of the work by @ApocDev and @StevenAskwith, in case it helps anyone else here:

  const vpc = new Vpc(this, 'Vpc', {maxAzs: 2});

  const securityGroup = new SecurityGroup(this, 'AuroraSecurityGroup', {
    vpc,
    allowAllOutbound: true,
  });

  const secret = new DatabaseSecret(this, 'AuroraSecret', {
    username: 'root',
  });

  const cluster = new CfnDBCluster(this, 'AuroraCluster', {
    engine: 'aurora',
    engineVersion: '5.6.10a',
    engineMode: 'serverless',
    masterUsername: secret.secretValueFromJson('username').toString(),
    masterUserPassword: secret.secretValueFromJson('password').toString(),
    vpcSecurityGroupIds: [securityGroup.securityGroupId],
    deletionProtection: false,
    enableHttpEndpoint: true,
    storageEncrypted: true,
    backupRetentionPeriod: 7,

    dbSubnetGroupName: new CfnDBSubnetGroup(this, 'AuroraSubnetGroup', {
      dbSubnetGroupDescription: 'AuroraSubnetGroup',
      subnetIds: vpc.selectSubnets({subnetType: SubnetType.PRIVATE}).subnetIds,
    }).ref,

    scalingConfiguration: {
      autoPause: true,
      minCapacity: 1,
      maxCapacity: 16,
      secondsUntilAutoPause: 300,
    },
  });

@nija-at nija-at assigned skinny85 and unassigned nija-at Jul 14, 2020
@skinny85 skinny85 added p1 effort/large Large work item – several weeks of effort and removed effort/medium Medium work item – several days of effort labels Jul 21, 2020
@asterikx
Copy link
Contributor

Working construct with CDK 1.55.0

import * as cdk from '@aws-cdk/core';
import * as rds from '@aws-cdk/aws-rds';
import * as secretsmanager from '@aws-cdk/aws-secretsmanager';
import * as ec2 from '@aws-cdk/aws-ec2';
import * as kms from '@aws-cdk/aws-kms';

export interface AuroraServerlessProps {
  readonly engine: rds.IClusterEngine;
  readonly clusterIdentifier?: string;
  readonly parameterGroup?: rds.ParameterGroup;
  readonly vpc: ec2.IVpc;
  readonly vpcSubnets: ec2.SubnetSelection;
  readonly securityGroups?: ec2.ISecurityGroup[];
  readonly masterUsername?: string;
  readonly defaultDatabaseName?: string;
  readonly backup?: rds.BackupProps;
  readonly preferredMaintenanceWindow?: string;
  readonly storageEncryptionKey?: kms.IKey;
  readonly deletionProtection?: boolean;
  readonly removalPolicy?: cdk.RemovalPolicy;
  readonly scalingConfig?: rds.CfnDBCluster.ScalingConfigurationProperty;
  readonly enableDataApi?: boolean;
}

// https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html
// https://madabout.cloud/2019/09/01/aws-data-api-for-amazon-aurora-serverless/
// https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-rds/lib/cluster.ts
// https://github.com/ysku/aurora-serverless-example/blob/master/lib/database.ts
export class AuroraServerless extends cdk.Resource implements ec2.IConnectable, secretsmanager.ISecretAttachmentTarget {
  /**
   * Identifier of the cluster
   */
  public readonly clusterIdentifier: string;

  /**
   * ARN of the cluster
   */
  public readonly clusterArn: string;

  /**
   * The endpoint to use for read/write operations
   */
  public readonly clusterEndpoint: rds.Endpoint;

  /**
   * Access to the network connections
   */
  public readonly connections: ec2.Connections;

  /**
   * Security group identifier of this database
   */
  public readonly securityGroups: ec2.ISecurityGroup[];

  /**
   * The secret attached to this cluster
   */
  public readonly secret?: secretsmanager.ISecret;

  private readonly secretRotationApplication: secretsmanager.SecretRotationApplication;

  /**
   * The VPC where the DB subnet group is created.
   */
  private readonly vpc: ec2.IVpc;

  /**
   * The subnets used by the DB subnet group.
   */
  private readonly vpcSubnets: ec2.SubnetSelection;

  constructor(scope: cdk.Construct, id: string, props: AuroraServerlessProps) {
    super(scope, id);

    const {
      clusterIdentifier,
      engine,
      parameterGroup,
      storageEncryptionKey,
      backup,
      preferredMaintenanceWindow,
      deletionProtection,
      removalPolicy,
      defaultDatabaseName,
      vpc,
      vpcSubnets,
      scalingConfig,
      enableDataApi,
      masterUsername,
    } = props;

    this.vpc = vpc;
    this.vpcSubnets = vpcSubnets;

    // DB subnet group
    const { subnetIds } = vpc.selectSubnets(vpcSubnets);

    const dbSubnetGroup = new rds.CfnDBSubnetGroup(this, 'DbSubnetGroup', {
      dbSubnetGroupDescription: `CloudFormation managed DB subnet group.`,
      subnetIds,
    });

    // DB security group
    const securityGroups = props.securityGroups ?? [
      new ec2.SecurityGroup(this, 'DbSecurityGroup', {
        vpc: vpc,
        allowAllOutbound: false,
      }),
    ];
    this.securityGroups = securityGroups;

    // DB secret
    const secret = new rds.DatabaseSecret(this, 'DbSecret', {
      username: masterUsername || 'root',
    });
    this.secretRotationApplication = engine.singleUserRotationApplication;

    // bind the engine to the Cluster
    const clusterEngineBindConfig = engine.bindToCluster(this, {
      parameterGroup: parameterGroup,
    });
    const clusterParameterGroup = parameterGroup ?? clusterEngineBindConfig.parameterGroup;
    const clusterParameterGroupConfig = clusterParameterGroup?.bindToCluster({});

    // DB cluster
    const cluster = new rds.CfnDBCluster(this, 'DbCluster', {
      // basic
      engine: engine.engineType,
      engineVersion: engine.engineVersion?.fullVersion,
      engineMode: 'serverless',
      dbClusterIdentifier: clusterIdentifier,
      dbSubnetGroupName: dbSubnetGroup.ref,
      vpcSecurityGroupIds: securityGroups.map((sg) => sg.securityGroupId),
      // port: clusterEngineBindConfig.port, // Aurora Serverless always runs on default port
      dbClusterParameterGroupName: clusterParameterGroupConfig?.parameterGroupName,
      // associatedRoles: undefined,
      // admin
      masterUsername: secret.secretValueFromJson('username').toString(),
      masterUserPassword: secret.secretValueFromJson('password').toString(),
      backupRetentionPeriod: backup?.retention.toDays(),
      preferredBackupWindow: backup?.preferredWindow,
      preferredMaintenanceWindow: preferredMaintenanceWindow,
      databaseName: defaultDatabaseName,
      deletionProtection: deletionProtection,
      // encryption
      kmsKeyId: storageEncryptionKey?.keyArn,
      // storageEncrypted: storageEncryptionKey ? true : encrypted, // Aurora Serverless is always encrypted
      // serverless config
      enableHttpEndpoint: enableDataApi,
      scalingConfiguration: scalingConfig,
    });

    // Default deletion policy for AWS:RDS:DBCluster is SNAPSHOT
    // https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html
    if (removalPolicy) {
      cluster.applyRemovalPolicy(removalPolicy, {
        applyToUpdateReplacePolicy: true,
      });
    } else {
      // The CFN default doesn't cover UpdateReplacePolicy. Fix that here.
      cluster.cfnOptions.updateReplacePolicy = cdk.CfnDeletionPolicy.SNAPSHOT;
    }

    this.clusterIdentifier = cluster.ref;

    this.secret = secret.attach(this);

    this.clusterEndpoint = new rds.Endpoint(cluster.attrEndpointAddress, cdk.Token.asNumber(cluster.attrEndpointPort));

    const defaultPort = ec2.Port.tcp(this.clusterEndpoint.port);
    this.connections = new ec2.Connections({ securityGroups, defaultPort });

    const { region, account } = this.stack;
    this.clusterArn = `arn:aws:rds:${region}:${account}:cluster:${this.clusterIdentifier}`;
  }

  /**
   * Adds the single user rotation of the master password to this cluster.
   */
  // https://github.com/aws/aws-cdk/blob/26a69b1b090b49505f69ef2879b68d2382ea27ec/packages/%40aws-cdk/aws-rds/lib/cluster.ts#L542
  public addRotationSingleUser(automaticallyAfter?: cdk.Duration): secretsmanager.SecretRotation {
    if (!this.secret) {
      throw new Error('Cannot add single user rotation for a cluster without secret.');
    }

    const id = 'RotationSingleUser';
    const existing = this.node.tryFindChild(id);
    if (existing) {
      throw new Error('A single user rotation was already added to this cluster.');
    }

    return new secretsmanager.SecretRotation(this, id, {
      secret: this.secret,
      automaticallyAfter,
      application: this.secretRotationApplication,
      vpc: this.vpc,
      vpcSubnets: this.vpcSubnets,
      target: this,
    });
  }

  public asSecretAttachmentTarget(): secretsmanager.SecretAttachmentTargetProps {
    return {
      targetType: secretsmanager.AttachmentTargetType.RDS_DB_CLUSTER,
      targetId: this.clusterIdentifier,
    };
  }

  /*
  // https://github.com/aws/aws-cdk/blob/26a69b1b090b49505f69ef2879b68d2382ea27ec/packages/%40aws-cdk/aws-rds/lib/cluster.ts#L566
  public addRotationMultiUser(id: string, options: rds.RotationMultiUserOptions): secretsmanager.SecretRotation {
    if (!this.secret) {
      throw new Error('Cannot add multi user rotation for a cluster without secret.');
    }
    return new secretsmanager.SecretRotation(this, id, {
      secret: options.secret,
      masterSecret: this.secret,
      automaticallyAfter: options.automaticallyAfter,
      application: this.multiUserRotationApplication,
      vpc: this.vpc,
      vpcSubnets: this.vpcSubnets,
      target: this,
    });
  }
   */
}

@manishkpr
Copy link

this article saved my day 🙇 🙇 🙇

@Wingjam
Copy link

Wingjam commented Aug 21, 2020

Thanks @asterikx for sharing your construct, saved a lot of time!

@skinny85 skinny85 assigned shivlaks and unassigned skinny85 Sep 4, 2020
@SomayaB SomayaB added the in-progress This issue is being actively worked on. label Sep 24, 2020
@mergify mergify bot closed this as completed in #10516 Sep 30, 2020
mergify bot pushed a commit that referenced this issue Sep 30, 2020
Adds a new construct to specify Aurora Serverless clusters

This is largely a stripped down version of `Cluster` as there are many properties
within clusters that do not apply to Aurora Serverless. Some of the notable
exclusions are:
* backup windows
* maintenance windows
* associated roles (as S3 import/exports are not supported)
* iam database authentication
* exporting cloudwatch log exports
* storage encryption is always true in Aurora serverless

Added:
* Scaling options which only apply to Aurora serverless and cannot be used for
   provisioned clusters
* Enum `AuroraCapacityUnit` to specify the provisioned capacity
* Enable http endpoint for a serverless cluster

Closes #929

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@vahdet
Copy link

vahdet commented Jan 1, 2021

For those finding this issue closed after a time and looking for what to do now: An L2 construct ServerlessCluster for serverless Aurora appears in docs now: https://docs.aws.amazon.com/cdk/api/latest/docs/aws-rds-readme.html#serverless

Have fun.

@ffxsam
Copy link

ffxsam commented Aug 7, 2021

@skinny85

This still seems problematic (or at least not clearly documented).

For instance, engine.bindToCluster is required - but what is it? I don't see it documented anywhere. The docs say it's a method, but when I'm defining my cluster using rds.ServerlessCluster, it treats it as a required property.

Also, why are singleUserRotationApplication and multiUserRotationApplication needed? I've created a serverless cluster in CloudFormation YAML before and not needed these.

Lastly, it seems impossible to create a serverless cluster without NAT gateways, which are not required. If I use isolated subnets for the rds.SubnetGroup, it complains:

Error: There are no 'Private' subnet groups in this VPC. Available types: Isolated,Public

But if I create private subnets in my VPC definition, NAT gateways are automatically created. Setting natGateways: 0 complains that I should use isolated subnets—so you can see my catch-22 here.

Expand to see definition
  const secret = new rds.DatabaseSecret(stack, 'RDSSecret', {
    secretName: 'dev/rds',
    username: process.env.APP_NAME,
  });
  const subnetGroup = new rds.SubnetGroup(stack, 'RDSSubnetGroup', {
    subnetGroupName: 'rds-subnet-group',
    vpc: props.vpc,
    description: 'Private subnet group for DBs',
  });
  const db = new rds.ServerlessCluster(stack, 'Database', {
    backupRetention: Duration.days(stack.stage === 'prod' ? 30 : 7),
    clusterIdentifier: `${process.env.APP_NAME}-${stack.stage}`,
    credentials: {
      username: process.env.APP_NAME,
      secret,
    },
    subnetGroup,
    defaultDatabaseName: process.env.APP_NAME,
    deletionProtection: stack.stage === 'prod',
    enableDataApi: true,
    engine: {
      engineType: 'aurora-postgresql',
      engineVersion: {
        majorVersion: '10',
        fullVersion: '10.7',
      },
      // Why are these rotation values needed?
      multiUserRotationApplication:
        secretsmanager.SecretRotationApplication.POSTGRES_ROTATION_MULTI_USER,
      singleUserRotationApplication:
        secretsmanager.SecretRotationApplication.POSTGRES_ROTATION_SINGLE_USER,
      supportedLogTypes: ['what is this? not documented'],
      // What is this?
      bindToCluster({}, {}) {},
    },
    scaling: {
      autoPause: Duration.hours(stack.stage === 'prod' ? 0 : 1),
      minCapacity: rds.AuroraCapacityUnit.ACU_1,
      maxCapacity:
        stack.stage === 'prod'
          ? rds.AuroraCapacityUnit.ACU_8
          : rds.AuroraCapacityUnit.ACU_1,
    },
  });

@ffxsam
Copy link

ffxsam commented Aug 7, 2021

Related: #15929

@ffxsam
Copy link

ffxsam commented Aug 10, 2021

I made some progress on this, but there are still some questionable things going on.

Here's my definition of my serverless cluster using CfnDbCluster:

  const db = new rds.CfnDBCluster(stack, 'RDSDatabase', {
    backupRetentionPeriod: stack.stage === 'prod' ? 30 : 1,
    databaseName: process.env.APP_NAME,
    dbClusterIdentifier: `${process.env.APP_NAME}-${stack.stage}`,
    enableHttpEndpoint: true,
    engine: 'aurora-postgresql',
    engineMode: 'serverless',
    engineVersion: '10.14',
    masterUsername: secret.secretValueFromJson('username').toString(),
    masterUserPassword: secret.secretValueFromJson('password').toString(),
    vpcSecurityGroupIds: [dbSecGroup.securityGroupId],
    scalingConfiguration: {
      autoPause: stack.stage !== 'prod',
      maxCapacity: stack.stage === 'prod' ? 8 : 2,
      minCapacity: 2,
      secondsUntilAutoPause: 3600,
    },
  });

Here's the ServerlessCluster version of that:

  const db = new rds.ServerlessCluster(stack, 'RDSDatabase', {
    backupRetention: Duration.days(stack.stage === 'prod' ? 30 : 1),
    clusterIdentifier: `${process.env.APP_NAME}-${stack.stage}`,
    credentials: rds.Credentials.fromSecret(secret),
    defaultDatabaseName: process.env.APP_NAME,
    enableDataApi: true,
    engine: rds.DatabaseClusterEngine.AURORA_POSTGRESQL,
    scaling: {
      autoPause: Duration.hours(stack.stage === 'prod' ? 0 : 1),
      maxCapacity:
        stack.stage === 'prod'
          ? rds.AuroraCapacityUnit.ACU_8
          : rds.AuroraCapacityUnit.ACU_2,
      minCapacity: rds.AuroraCapacityUnit.ACU_2,
    },
    vpc: ec2.Vpc.fromLookup(stack, 'MainVpc', { isDefault: true }),
    securityGroups: [dbSecGroup],
  });

I get this error when trying to deploy:
Error: There are no 'Private' subnet groups in this VPC. Available types: Public

Yet using CfnDbCluster, it works fine. Why are private subnets required when they're really not? The security group is only so a dev can manually connect to the DB via SSH within SSM.. otherwise it's not needed since I'm using the Data API.

@nikita-sheremet-clearscale

@ffxsam

vpcSubnets selector must be provided to fix the error There are no 'Private' subnet groups in this VPC. Available types: Public like that (java):

import software.amazon.awscdk.core.RemovalPolicy;
import software.amazon.awscdk.core.SecretValue;
import software.amazon.awscdk.services.ec2.InstanceClass;
import software.amazon.awscdk.services.ec2.InstanceSize;
import software.amazon.awscdk.services.ec2.InstanceType;
import software.amazon.awscdk.services.ec2.SubnetSelection;
import software.amazon.awscdk.services.ec2.SubnetType;
import software.amazon.awscdk.services.ec2.Vpc;
import software.amazon.awscdk.services.ec2.VpcLookupOptions;
import software.amazon.awscdk.services.rds.AuroraPostgresClusterEngineProps;
import software.amazon.awscdk.services.rds.AuroraPostgresEngineVersion;
import software.amazon.awscdk.services.rds.Credentials;
import software.amazon.awscdk.services.rds.DatabaseCluster;
import software.amazon.awscdk.services.rds.DatabaseClusterEngine;
import software.amazon.awscdk.services.rds.DatabaseClusterProps;
import software.amazon.awscdk.services.rds.InstanceProps;

class Scratch {
  public static void main(String[] args) {
    var cluster = new DatabaseCluster(this, "PostgresDb",
        DatabaseClusterProps.builder()
            .clusterIdentifier("aurora-postgresql-cluster")
            .instanceIdentifierBase("aurora-postgresql-cluster")
            .engine(DatabaseClusterEngine.auroraPostgres(
                AuroraPostgresClusterEngineProps.builder()
                    .version(
                        AuroraPostgresEngineVersion.VER_12_4
                    )
                    .build()))
            .removalPolicy(RemovalPolicy.DESTROY)
            .deletionProtection(false)
//            .credentials(Credentials.fromSecret(databaseSecret))
            .credentials(Credentials.fromPassword("somename", SecretValue.plainText("password")))
            .instanceIdentifierBase("aurora-postgresql-instance")

            .instanceProps(
                InstanceProps.builder()
                    .instanceType(
                        InstanceType.of(InstanceClass.MEMORY4, InstanceSize.LARGE)
                    )
                    .vpc(Vpc.fromLookup(this, "DefaultVpc", VpcLookupOptions.builder().isDefault(true).build()))
                    .vpcSubnets(SubnetSelection.builder().subnetType(SubnetType.PUBLIC).build()) // <- here fixed the error
                    .publiclyAccessible(true)
                    .build())
            .build()
    );
  }
}

@skinny85
Copy link
Contributor

@ffxsam the reply by @nikita-sheremet-clearscale is spot on.

Do you have any other questions/problems with getting this working?

@ffxsam
Copy link

ffxsam commented Aug 25, 2021

@skinny85 No sir! This wound up working fine for me:

(expand for code)
  const secret = new rds.DatabaseSecret(stack, 'RDSSecret', {
    secretName: 'dev/rds',
    username: process.env.APP_NAME,
  });

  const mainVpc = ec2.Vpc.fromLookup(stack, 'MainVpc', { isDefault: true });

  const dbSecGroup = new ec2.SecurityGroup(stack, 'RDSSecurityGroup', {
    description: 'Access to DBs in VPC',
    securityGroupName: 'database-access',
    vpc: mainVpc,
    allowAllOutbound: true,
  });

  dbSecGroup.addIngressRule(dbSecGroup, ec2.Port.tcp(5432));

  const db = new rds.ServerlessCluster(stack, 'RDSDatabase', {
    backupRetention: Duration.days(stack.stage === 'prod' ? 30 : 1),
    clusterIdentifier: `${process.env.APP_NAME}-${stack.stage}`,
    credentials: rds.Credentials.fromSecret(secret),
    defaultDatabaseName: process.env.APP_NAME,
    enableDataApi: true,
    engine: rds.DatabaseClusterEngine.auroraPostgres({
      version: rds.AuroraPostgresEngineVersion.VER_10_14,
    }),
    scaling: {
      autoPause: Duration.hours(stack.stage === 'prod' ? 0 : 1),
      maxCapacity:
        stack.stage === 'prod'
          ? rds.AuroraCapacityUnit.ACU_8
          : rds.AuroraCapacityUnit.ACU_2,
      minCapacity: rds.AuroraCapacityUnit.ACU_2,
    },
    vpc: mainVpc,
    vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
    securityGroups: [dbSecGroup],
  });

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-rds Related to Amazon Relational Database effort/large Large work item – several weeks of effort feature-request A feature should be added or improved. in-progress This issue is being actively worked on. p1
Projects
None yet
Development

Successfully merging a pull request may close this issue.