Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Lambda): Lambda environment variables that reference updated "parameters" are not updated when redeploying the stack #30101

Closed
AllanOricil opened this issue May 8, 2024 · 18 comments
Assignees
Labels
@aws-cdk/aws-lambda Related to AWS Lambda bug This issue is a bug. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.

Comments

@AllanOricil
Copy link

AllanOricil commented May 8, 2024

Describe the bug

I have a lambda full of env variables referencing a "parameter", like the one shown below

![img]

INFO: pay attention that I'm loading the right .env file as shown in the above image. I always set NODE_ENV before I ran any cdk command. And I print the .env path to make sure the right one was selected when building things locally.

![img]

The above screenshots show a template that was synthesized by CDK after I changed the .env file that provide values to my parameters. However, even after deploying this new stack, CDK did not change my env variable values in my lambda functions. In the image below you can see that "ALLOWED_ORIGINS" was not updated.

![img]

Another weird thing is the CDK doesnt even realize that this change will happen. Below you can see the output of cdk diff for the same stack.

Expected Behavior

CDK must update Lambda Env Variables values that reference parameters whenever they are changed.

Current Behavior

CDK does not update Lambda Env Variables values when they reference parameters that have changed

Reproduction Steps

  1. create a construct with one lambda

  2. create a cfn parameter that is loaded with values from a .env file. You can use dotenv. like I did.
    ![img]
    ![img]

  3. add an env variable to your lambda, and fill its value with the value from the parameter.

![img]

  1. synth and deploy it. Open the synthesized template and verify the parameter value is the same one from your .env.
  2. change the .env value
  3. synth. Again verify the synthesized template and verify that it has changed, and it again matches the one from your .env.
  4. deploy it and verify that the value was not changed in AWS
  5. conclude it is a bug

Possible Solution

Couldn't find any woraround

Additional Information/Context

No response

CDK CLI Version

2.138.0

Framework Version

No response

Node.js Version

node 18

OS

macos latest

Language

TypeScript

Language Version

typescript@v5.2.2

Other information

No response

@AllanOricil AllanOricil added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels May 8, 2024
@github-actions github-actions bot added the @aws-cdk/core Related to core CDK functionality label May 8, 2024
@AllanOricil AllanOricil changed the title (core): Lambda environment variables that reference updated "parameters" arent not updated when redeploying the stack (Lambda): Lambda environment variables that reference updated "parameters" arent not updated when redeploying the stack May 8, 2024
@github-actions github-actions bot added the @aws-cdk/aws-lambda Related to AWS Lambda label May 8, 2024
@AllanOricil
Copy link
Author

AllanOricil commented May 8, 2024

I also deleted ALL env variables in one of my functions, using the console. Then I redeployed the stack, and verified my env variables were not added back. There were a bunch of other env variables there that are just gone.

![img]

@AllanOricil
Copy link
Author

AllanOricil commented May 8, 2024

I had to invest around 1 hour to destroy and recreate my stack in order to have env variables updated

![img]

@AllanOricil
Copy link
Author

AllanOricil commented May 8, 2024

Please, create a dedicated flag in cdk that "updates env variables only".

--update-lambda-env-variables-only

Look at this! 28min to deploy + the time to destroy the stack + the time I now have to invest to update the API url in some other stacks

![img]

@piradeepk piradeepk removed the @aws-cdk/core Related to core CDK functionality label May 8, 2024
@ashishdhingra ashishdhingra self-assigned this May 8, 2024
@ashishdhingra ashishdhingra added needs-reproduction This issue needs reproduction. and removed needs-triage This issue or PR still needs to be triaged. labels May 8, 2024
@ashishdhingra
Copy link
Contributor

@AllanOricil Somehow I'm unable to reproduce the issue with CDK version 2.140.0. Used the code below:
TypescriptStack.ts

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as dotenv from 'dotenv';

export class TypescriptStack extends cdk.Stack {
  private readonly allowedOrigins: cdk.CfnParameter;

  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    dotenv.config();
    console.log("ALLOWED_ORIGINS: " + this.getEnvVar("ALLOWED_ORIGINS"));

    this.allowedOrigins = new cdk.CfnParameter(this, "allowedOrigins", {
      type: "String",
      noEcho: true,
      default: this.getEnvVar("ALLOWED_ORIGINS")
    });

    const lambdaFunction = new lambda.Function(this, 'testLambda', {
      runtime: lambda.Runtime.NODEJS_20_X,
      code: lambda.Code.fromAsset('lambda'),
      handler: 'hello.handler',
      environment: {
         ALLOWED_ORIGINS: this.getEnvVar("ALLOWED_ORIGINS")
      }
    });
  }

  getEnvVar(envVarName: string): string {
    return process.env.ALLOWED_ORIGINS ?? "";
  }
}

lambda\hello.js

exports.handler = async function(event) {
    console.log("request:", JSON.stringify(event, undefined, 2));
    return {
      statusCode: 200,
      headers: { "Content-Type": "text/plain" },
      body: `Hello, CDK! You've hit ${event.path}\n`
    };
  };

.env (on 1st deploy)

ALLOWED_ORIGINS=http://localhost:8080,http://testdomain.com

.env (on 2nd deploy)

ALLOWED_ORIGINS=http://localhost:8080,http://testdomain.com,http://makemydomain.com

package.json

{
  "name": "typescript",
  "version": "0.1.0",
  "bin": {
    "typescript": "bin/typescript.js"
  },
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "test": "jest",
    "cdk": "cdk"
  },
  "devDependencies": {
    "@types/jest": "^29.5.12",
    "@types/node": "20.12.7",
    "aws-cdk": "2.140.0",
    "jest": "^29.7.0",
    "ts-jest": "^29.1.2",
    "ts-node": "^10.9.2",
    "typescript": "~5.4.5"
  },
  "dependencies": {
    "aws-cdk-lib": "2.140.0",
    "constructs": "^10.0.0",
    "dotenv": "^16.4.5",
    "source-map-support": "^0.5.21"
  }
}

Executing cdk diff before 2nd deployment gave the below output:

Parameters
[~] Parameter allowedOrigins allowedOrigins: {"Type":"String","Default":"http://localhost:8080,http://testdomain.com","NoEcho":true} to {"Type":"String","Default":"http://localhost:8080,http://testdomain.com,http://makemydomain.com","NoEcho":true}

Resources
[~] AWS::Lambda::Function testLambda testLambdaC13632F6 
 └─ [~] Environment
     └─ [~] .Variables:
         └─ [~] .ALLOWED_ORIGINS:
             ├─ [-] http://localhost:8080,http://testdomain.com
             └─ [+] http://localhost:8080,http://testdomain.com,http://makemydomain.com

Executing cdk deploy updated the environment variable in Lambda function.

Thanks,
Ashish

@ashishdhingra ashishdhingra added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed needs-reproduction This issue needs reproduction. labels May 8, 2024
@AllanOricil
Copy link
Author

AllanOricil commented May 8, 2024

@ashishdhingra your lambda is not using the value from the parameter. It is using the env variable. Your rendered template must have a Ref pointing to the parameter.

@ashishdhingra
Copy link
Contributor

@ashishdhingra your lambda is not using the value from the parameter. It is using the env variable. Your rendered template must have a Ref pointing to the parameter.

@AllanOricil Thanks for pointing it out. I modified the code to below and the issue is now reproducible:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as dotenv from 'dotenv';

export class TypescriptStack extends cdk.Stack {
  private readonly allowedOrigins: cdk.CfnParameter;

  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    dotenv.config();
    console.log("ALLOWED_ORIGINS: " + this.getEnvVar("ALLOWED_ORIGINS"));

    this.allowedOrigins = new cdk.CfnParameter(this, "allowedOrigins", {
      type: "String",
      noEcho: true,
      default: this.getEnvVar("ALLOWED_ORIGINS")
    });

    const lambdaFunction = new lambda.Function(this, 'testLambda', {
      runtime: lambda.Runtime.NODEJS_20_X,
      code: lambda.Code.fromAsset('lambda'),
      handler: 'hello.handler',
      environment: {
         ALLOWED_ORIGINS: this.allowedOrigins.value.toString()
      }
    });
  }

  getEnvVar(envVarName: string): string {
    return process.env.ALLOWED_ORIGINS ?? "";
  }
}

It produces below CloudFormation template (as an example), with environment variable as Ref to allowedOrigins parameter:

Parameters:
  allowedOrigins:
    Type: String
    Default: http://localhost:8080,http://testdomain.com,http://makemydomain.com
    NoEcho: true
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]
Resources:
  testLambdaServiceRole955E2289:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
    Metadata:
      aws:cdk:path: TypescriptStack/testLambda/ServiceRole/Resource
  testLambdaC13632F6:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket:
          Fn::Sub: cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}
        S3Key: b12b4d45960ea908338c4544bcac8039b13736e6d6af9e8a85ef57f538e29706.zip
      Environment:
        Variables:
          ALLOWED_ORIGINS:
            Ref: allowedOrigins
      Handler: hello.handler
      Role:
        Fn::GetAtt:
          - testLambdaServiceRole955E2289
          - Arn
      Runtime: nodejs20.x
    DependsOn:
      - testLambdaServiceRole955E2289
    Metadata:
      aws:cdk:path: TypescriptStack/testLambda/Resource
      aws:asset:path: asset.b12b4d45960ea908338c4544bcac8039b13736e6d6af9e8a85ef57f538e29706
      aws:asset:is-bundled: false
      aws:asset:property: Code
...
...

Running cdk diff (before 2nd deployment), produces the following output:

ALLOWED_ORIGINS: http://localhost:8080,http://testdomain.com,http://makemydomain.com
Stack TypescriptStack
Hold on while we create a read-only change set to get a diff with accurate replacement information (use --no-change-set to use a less accurate but faster template-only diff)
Parameters
[~] Parameter allowedOrigins allowedOrigins: {"Type":"String","Default":"http://localhost:8080,http://testdomain.com","NoEcho":true} to {"Type":"String","Default":"http://localhost:8080,http://testdomain.com,http://makemydomain.com","NoEcho":true}


✨  Number of stacks with differences: 1

I'm unsure though if this is a CDK issue or CloudFormation issue since the synthesized template is handled by CloudFormation, and CloudFormation allows declaring parameters.

Need to discuss this with team.

Thanks,
Ashish

@AllanOricil
Copy link
Author

@ashishdhingra keep us updated
Thank you!

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label May 9, 2024
@AllanOricil AllanOricil changed the title (Lambda): Lambda environment variables that reference updated "parameters" arent not updated when redeploying the stack (Lambda): Lambda environment variables that reference updated "parameters" are not updated when redeploying the stack May 9, 2024
@AllanOricil
Copy link
Author

Sorry, I thought I had found another related issue. But I confused an execution record with a config record.

@skinny85
Copy link
Contributor

skinny85 commented May 9, 2024

@AllanOricil why do you need to use a Parameter at all? In general, CloudFormation Parameters are discouraged when using CDK.

@AllanOricil
Copy link
Author

AllanOricil commented May 9, 2024

@skinny85

For my use case, I can definitely remove it. Thanks for pointing that out 🤙

However, I did it like that because I wanted to verify if it was possible to override an env variable if necessary using cli parameters. This is useful in complex scenarios (I did not find one yet) which the value of the env variable in the lambda has to be evaluated at deploy time, with information outside of the cdk project.

With a parameter like that, I can deploy my stack as so:

DYNAMIC_VARIABLE=$(...)
cdk synth stack-name -e --parameters myDynamicVariable $DYNAMIC_VARIABLE

If AWS does not recommend it. What about removing this feature? For example, if one tries to do what I did, throw an exception and stop the synthesis. However I do think it could be useful. I just don't know a real use case to show as an example.

@skinny85
Copy link
Contributor

skinny85 commented May 9, 2024

@AllanOricil No worries! A few thoughts:

  1. There's no reason to use CloudFormation Parameters if all you need is to read an environment variable. You can always just read an environment variable directly in your CDK code:

    const dynamicVariable = process.env.DYNAMIC_VARIABLE ?? 'DEFAULT_VALUE';
    // use dynamicVariable in your code...
  2. The CDK Context is a good mechanism if you need to reference something outside your code.

In general, the results of synthesizing your CDK app should be deterministic, and avoid things like external environment variables as much as possible.

Do you need to use environment variables perhaps because you are using the same CDK Stack instance for multiple deployed Stacks in CloudFormation? That's not the best practice when using CDK.

Let me know if this makes sense!

@ashishdhingra
Copy link
Contributor

For testing purposes, without using CDK:

  • Deployed the below CloudFormation stack from AWS console (manually uploaded the lambda code ZIP file to S3 bucket):
Parameters:
  allowedOrigins:
    Type: String
    Default: http://localhost:8080,http://testdomain.com
    NoEcho: true
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]
Resources:
  testLambdaServiceRole955E2289:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
  testLambdaC13632F6:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: testbucket-issue1880
        S3Key: b12b4d45960ea908338c4544bcac8039b13736e6d6af9e8a85ef57f538e29706.zip
      Environment:
        Variables:
          ALLOWED_ORIGINS:
            Ref: allowedOrigins
      Handler: hello.handler
      Role:
        Fn::GetAtt:
          - testLambdaServiceRole955E2289
          - Arn
      Runtime: nodejs20.x
    DependsOn:
      - testLambdaServiceRole955E2289

This created Lambda along with environment variable ALLOWED_ORIGINS having value http://localhost:8080,http://testdomain.com.

  • Modified the value of CloudFormation parameter allowedOrigins to http://localhost:8080,http://testdomain.com,http://makemydomain.com. Deployed the below template (updated CloudFormation stack to use a new template):
Parameters:
  allowedOrigins:
    Type: String
    Default: http://localhost:8080,http://testdomain.com,http://makemydomain.com
    NoEcho: true
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]
Resources:
  testLambdaServiceRole955E2289:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
  testLambdaC13632F6:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: testbucket-issue1880
        S3Key: b12b4d45960ea908338c4544bcac8039b13736e6d6af9e8a85ef57f538e29706.zip
      Environment:
        Variables:
          ALLOWED_ORIGINS:
            Ref: allowedOrigins
      Handler: hello.handler
      Role:
        Fn::GetAtt:
          - testLambdaServiceRole955E2289
          - Arn
      Runtime: nodejs20.x
    DependsOn:
      - testLambdaServiceRole955E2289
  • During AWS console workflow, in Specify stack details step showing there was a checkbox Use previous value in Parameters section. By default, this checkbox was checked. In the last step Review testlambdastack, it displayed message No updates are to be performed..

  • Went back to Specify stack details step and unchecked the checkbox Use previous value. Unchecking the checkbox blanked out the value (could be a UI issue since it should have read updated value from uploaded YAML file). Hence, manually specified value http://localhost:8080,http://testdomain.com,http://makemydomain.com for allowedOrigins parameter.

  • Going to last Review testlambdastack now displayed Lambda function to be modified in Change set preview section.

  • Submitting the changes updated the Lambda function with environment variable ALLOWED_ORIGINS changed to http://localhost:8080,http://testdomain.com,http://makemydomain.com .

So, by default, any changes to CloudFormation parameter are not used (based on observation in AWS console UI).

If we look at the CDK guide Deploying with parameters, is explicitly specifies that By default, the AWS CDK retains values of parameters from previous deployments and uses them in subsequent deployments if they are not specified explicitly. Use the --no-previous-parameters flag to require all parameters to be specified. (this may be due to limitation of CloudFormation).

Going with the above understanding, using cdk deploy --no-previous-parameters on 2nd deployment updated the value of environment variable in Lambda configuration.

@AllanOricil Please use cdk deploy --no-previous-parameters to use the updated value of CloudFormation parameter.

Thanks,
Ashish

@ashishdhingra ashishdhingra added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label May 9, 2024
@AllanOricil
Copy link
Author

@ashishdhingra Good job!

I didnt know about this flag.

Can't it become true by default? Like, it seems a little bit weird for that it isn't true by default in cdk. The only way to know about it is to be aware how this feature works in Cloudformation.

It seems counter intuitive based on how many people from CDK team itself, and a former one, couldn't spot the issue right away.

WARNING: Please, understand I'm not being trying to be mean here. It is really just an observation.

Can you think on a reason for no doing that?

1 similar comment
@AllanOricil
Copy link
Author

@ashishdhingra Good job!

I didnt know about this flag.

Can't it become true by default? Like, it seems a little bit weird for that it isn't true by default in cdk. The only way to know about it is to be aware how this feature works in Cloudformation.

It seems counter intuitive based on how many people from CDK team itself, and a former one, couldn't spot the issue right away.

WARNING: Please, understand I'm not being trying to be mean here. It is really just an observation.

Can you think on a reason for no doing that?

@ashishdhingra
Copy link
Contributor

Can't it become true by default? Like, it seems a little bit weird for that it isn't true by default in cdk. The only way to know about it is to be aware how this feature works in Cloudformation.

It seems counter intuitive based on how many people from CDK team itself, and a former one, couldn't spot the issue right away.

WARNING: Please, understand I'm not being trying to be mean here. It is really just an observation.

Can you think on a reason for no doing that?

@AllanOricil Based on my observation in last #30101 (comment), one possible reasoning is one-to-one parity with CloudFormation UI in AWS console. By default, in AWS console, checkbox Use previous value is checked. This is just an assumption. Also, if we change the default behavior, it might break some small set of customers where by default it is currently assumed that new value for CfnParameter would not be used by default.

@ashishdhingra ashishdhingra added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels May 9, 2024
@AllanOricil
Copy link
Author

I will close this. Thank you all for the help.

Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@aws-cdk-automation
Copy link
Collaborator

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

@aws aws locked as resolved and limited conversation to collaborators Jul 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
@aws-cdk/aws-lambda Related to AWS Lambda bug This issue is a bug. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.
Projects
None yet
Development

No branches or pull requests

5 participants