pipelines: Cannot perform lookup in cross-account nested stack

[growak]

Describe the bug

When a resource is created in a NestedStack in a cross-account pipeline, the Build/Synth phase failed with he following error message: Need to perform AWS calls for account 222222222222, but the current credentials are for 111111111111.

111111111111 is the pipeline account
222222222222 is the target account where the resource should be deployed.

If the same resource is moved to the parent stack everything works.

Expected Behavior

Resource creation should work in NestedStack in cross-account pipelines.

Current Behavior

Resource creation failed in NestedStack in cross-account pipelines.

Reproduction Steps

Create a pipeline.
Create a stage in a different account.
Create a NestedStack inside de stage stack.
Create a resource in the Nested Stack.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.74.0

Framework Version

No response

Node.js Version

v16.20.0

OS

Amazon-Linux 2023

Language

Typescript

Language Version

No response

Other information

No response

[pahud]

I believe when you cdk bootstrap the account 222222222222, you need to pass --trust and --trust-for-lookup for 111111111111 as described in the doc. Does this work with you?

[growak]

Everything works fine when the resource is in a normal Stack and fails when it is in a Nested Stack. All accounts have been bootstrapped correctly.

[pahud]

@growak Thank you for your immediate response. Are you able to provide a small working sample that I can reproduce in my account? This will help us address this issue much easier.

[growak]

Do you want a code repository or can I copy paste the code in comments?

[pahud]

@growak Please feel free to just copy paste the minimal required code in the comments or in the issue description with code blocks, I will copy/paste into my IDE and try reproduce this error.

[growak]

flow.ts:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as iam from 'aws-cdk-lib/aws-iam';
import { Repository, IRepository } from 'aws-cdk-lib/aws-codecommit';
import { ComputeType } from 'aws-cdk-lib/aws-codebuild';
import { CodePipeline, CodePipelineSource, ShellStep, ManualApprovalStep, IFileSetProducer } from 'aws-cdk-lib/pipelines';

export interface FlowStackProps<S extends cdk.Stage, SP extends cdk.StageProps> extends cdk.StackProps {
  rolePolicy: iam.PolicyStatement[]
  selfMutation: boolean
  repositoryName: string
  repositoryPath?: string
  subRepositoryNames?: string[]
  repositoryBranch: string
  repositoryClone: boolean
  installCommands?: string[]
  commands?: string[]
  stagesProps: SP[],
  stageFactory: (s: Construct, n: string, p: SP) => S;
}

export class FlowStack<S extends cdk.Stage, SP extends cdk.StageProps> extends cdk.Stack {
  constructor(scope: Construct, id: string, props: FlowStackProps<S, SP>) {
    super(scope, id, props)

    // create repositories

    const repository = Repository.fromRepositoryName(this, 'Repository', props.repositoryName)
    const repositoryPath = props.repositoryPath ?  props.repositoryPath : '.'
    const temporaryPath = '../tmp'

    // create sub repositories

    const subRepositories: IRepository[] = []
    const subInputSources: Record<string, IFileSetProducer> = {}

    if (props.subRepositoryNames) {
      for(let subRespositoryName of props.subRepositoryNames) {
        const subRepository = Repository.fromRepositoryName(this, `${this.capitalize(subRespositoryName)}SubRepository` , subRespositoryName)
        subRepositories.push(subRepository)
        subInputSources[`${temporaryPath}/${subRespositoryName}`] = CodePipelineSource.codeCommit(subRepository, props.repositoryBranch)
      }
    }

    // create role policy

    const rolePolicy = props.rolePolicy.concat(props.stagesProps.map(props => new iam.PolicyStatement({
        actions: ['sts:AssumeRole'],
        // change that to a more precise role generated by CDK
        resources: [`arn:aws:iam::${props.env ? props.env.account: '*'}:role/*`]
      })
    ))

    // create the pipeline

    const installCommands = props.installCommands? props.installCommands : []
    const commands = props.commands? props.commands : []
    const primaryOutputDirectory = `${repositoryPath}/cdk.out`

    const pipeline = new CodePipeline(this, 'Pipeline', {
      selfMutation: props.selfMutation,
      dockerEnabledForSelfMutation: true,
      dockerEnabledForSynth: true,
      publishAssetsInParallel: true,
      crossAccountKeys: true,
      codeBuildDefaults: {
        buildEnvironment: {
          privileged: true,
          computeType: ComputeType.LARGE,
        },
        rolePolicy: subRepositories.map(
          r => new iam.PolicyStatement({
            actions: [ "codecommit:GitPull" ],
            resources: [ r.repositoryArn ]
          })
        ).concat(rolePolicy)
      },
      selfMutationCodeBuildDefaults: {
        buildEnvironment: {
          computeType: ComputeType.LARGE,
        }
      },
      assetPublishingCodeBuildDefaults: {
        buildEnvironment: {
          computeType: ComputeType.LARGE,
        }
      },
      synth: new ShellStep('Synth', {
        input: CodePipelineSource.codeCommit(repository, props.repositoryBranch, {
          codeBuildCloneOutput: props.repositoryClone,
        }),
        additionalInputs: subInputSources,
        installCommands: installCommands.concat([
          `rm -rf ${temporaryPath}`
        ]),
        commands: commands.concat([
          `cd ${repositoryPath}`,
          'npm ci',
          'npm run build',
          'npx cdk synth'
        ]),
        primaryOutputDirectory
      })
    });

    // add stages

    props.stagesProps.forEach((stageProps, index) => {
      const stageName = stageProps.stageName ? stageProps.stageName : `Stage${index}`
      const stage = pipeline.addStage(props.stageFactory(this, stageName, stageProps));
      if (index < props.stagesProps.length - 1) {
        stage.addPost(new ManualApprovalStep(`Manual Approval`))
      }
    })

  }

  capitalize(str: string) {
    return str.charAt(0).toUpperCase() + str.slice(1);
  }

}

main.ts:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { RootCoreStack, GlobalCoreStack, RegionalCoreStack } from '../core'

import * as r53 from 'aws-cdk-lib/aws-route53';

export interface MainStageProps extends cdk.StageProps {
  rootAccountId: string
  domainName: string
  otherDomainNames?: [string]
  stageName: string
  vpcCidr: string
}

export class MainStage extends cdk.Stage {
  constructor(scope: Construct, id: string, props: MainStageProps) {
    super(scope, id, props);

    const rootCore = new RootCoreStack(this, 'RootCore', {
      domainName: props.domainName,
      stageName: props.stageName,
      vpcCidr: props.vpcCidr,
      env: {
        account: props.rootAccountId,
        region: 'us-east-1'
      },
    })

    const globalCore = new GlobalCoreStack(this, 'GlobalCore', {
      domainName: props.domainName,
      stageName: props.stageName,
      vpcCidr: props.vpcCidr,
      env: {
        region: 'us-east-1'
      },
      crossRegionReferences: true,
    })

    const regionalCore = new RegionalCoreStack(this, 'RegionalCore', {
      domainName: props.domainName,
      stageName: props.stageName,
      vpcCidr: props.vpcCidr,
      crossRegionReferences: true,
    })

  }
}

core.ts:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as r53 from 'aws-cdk-lib/aws-route53';
import { GlobalNetworkStack, RegionalNetworkStack } from './network'

export interface CoreStackProps extends cdk.StackProps {
  domainName: string
  stageName: string
  vpcCidr: string
}

export class RootCoreStack extends cdk.Stack {

  public rootHostedZone: r53.IHostedZone

  constructor(scope: Construct, id: string, props: CoreStackProps) {
    super(scope, id, props)

    const rootDomainName = props.domainName
    this.rootHostedZone = r53.HostedZone.fromLookup(this, 'RootHostedZone', {
        domainName:rootDomainName
    })

  }

}

export class GlobalCoreStack extends cdk.Stack {

  public network: GlobalNetworkStack

  constructor(scope: Construct, id: string, props: CoreStackProps) {
    super(scope, id, props)

    this.logging = new LoggingStack(this, 'Logging', {})
    this.network = new GlobalNetworkStack(this, 'Network', {
      domainName: props.domainName,
      stageName: props.stageName,
    })

  }

}

export class RegionalCoreStack extends cdk.Stack {

  public network: RegionalNetworkStack

  constructor(scope: Construct, id: string, props: CoreStackProps) {
    super(scope, id, props)

    this.network = new RegionalNetworkStack(this, 'Network', {
      domainName: props.domainName,
      stageName: props.stageName,
      vpcCidr: props.vpcCidr
    })

  }

}

network.ts:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as r53 from 'aws-cdk-lib/aws-route53';
import * as cm from 'aws-cdk-lib/aws-certificatemanager';
import * as ec2 from 'aws-cdk-lib/aws-ec2';

export interface GlobalNetworkStackProps extends cdk.NestedStackProps {
    domainName: string
    stageName: string
}

export class GlobalNetworkStack extends cdk.NestedStack {

  public stageHostedZone: r53.IHostedZone
  public certificate: cm.Certificate

  constructor(scope: Construct, id: string, props: GlobalNetworkStackProps) {
    super(scope, id, props);

    // create certificate

    /*this.certificate = new cm.Certificate(this, 'Certificate', {
        domainName: rootDomainName,
        subjectAlternativeNames: [
            `*.${rootDomainName}`
        ],
        validation: cm.CertificateValidation.fromDns(this.rootHostedZone),
    })*/

    // create stage hosted zone

    const stageDomainName = `${props.stageName.substring(0, 3)}.${props.domainName}`
    this.stageHostedZone = new r53.HostedZone(this, 'StageHostedZone', {
        zoneName: stageDomainName
    })

    /*new r53.NsRecord(this, `StageNSRecord`, {
        recordName: stageDomainName,
        zone: this.rootHostedZone,
        values: this.stageHostedZone.hostedZoneNameServers || [],
        ttl: cdk.Duration.minutes(5)
    })*/

  }
}

export interface RegionalNetworkStackProps extends cdk.NestedStackProps {
    domainName: string
    stageName: string
    vpcCidr: string
}

export class RegionalNetworkStack extends cdk.NestedStack {

  public certificate: cm.Certificate

  constructor(scope: Construct, id: string, props: RegionalNetworkStackProps) {
    super(scope, id, props);

    // retrieve root hosted zone

    // const domainName = props.domainName
    // const rootHostedZone = r53.HostedZone.fromLookup(this, 'RootHostedZone', {domainName})

    // create certificate

    /*this.certificate = new cm.Certificate(this, 'Certificate', {
        domainName,
        subjectAlternativeNames: [
            `*.${domainName}`
        ],
        validation: cm.CertificateValidation.fromDns(rootHostedZone)
    })*/

    // vpc

    const vpc = new ec2.Vpc(this, 'Vpc', {
        ipAddresses: ec2.IpAddresses.cidr(props.vpcCidr),
        maxAzs: 2,
        subnetConfiguration: [{
            name: 'Public',
            subnetType: ec2.SubnetType.PUBLIC,
        }],
    })

  }

}

[peterwoodworth]

I was able to reproduce this.

Need to perform AWS calls for account xxx but the current credentials are for yyy

This occurs when you need to perform a lookup within a cross-account nested stack. I'm not aware of a way you can directly fix this within your app, but I think a viable workaround could be refactoring your code such that lookups are performed only in top-level stacks, with the values passed to the nested stacks once the lookup has been properly performed

I'm also not super sure what is causing this behavior with nested stacks to begin with, I'm not terribly familiar with them

[growak]

Hey Peter, thank you for your answer. I had the same issue creating VPC in a NestedStack. Were you able to reproduce that bug too? I didn't test other constructs.

[peterwoodworth]

Yes, creating a VPC can cause a lookup to occur (lookup stack AZs) depending on how you have it configured. This is likely what's occurring for you in this case

[growak]

Thank you for your answer. Isn't that a bug? Any idea how to fix it? I may try to do a PR!

[peterwoodworth]

Sorry I didn't reply to this @growak, yes, this is a bug! And, I'm not really sure at all why this is occurring. If you have any ideas, a PR would be very welcome! Otherwise, we'll try to investigate and fix this when we can

[corymhall]

I think the issue here is that the NestedStackSynthesizer does not pass the lookupRoleArn to the synthesizeTemplate method.

The solution is probably to somehow inherit the settings from the parent stack synthesizer.

[IgnacioAcunaF]

Hi, encountered exactly the same problem on a Python and a Typescript CDK pipeline.
As @corymhall mentioned, is because the NestedStackSynthesizer doesn't pass the lookupRoleArn to the synthesizeTemplate method so the inherited Stack's methods of the NestedStackSynthesizer use the local credentials instead of assuming the cdk's cross account role.

When passing the role to the synthesizeTemplate method, the NestedStack indeed assume the cross-account role and is able to perform the lookup on the target account.

Attached a PR that solves the issue.

[peterwoodworth]

@IgnacioAcunaF were you still interested in continuing the PR?

[IgnacioAcunaF]

Hi @peterwoodworth yes. I haven't been able to upload some changes the last weeks, but I think I'll be able to do it the next one.

Thanks for the reminder

[SuzakuTheKnight]

+1 to the affected users count. Watching this thread.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.