sam local invoke - Timed out while attempting to establish a connection to the container

[jdesrochers]

Description:

The following command started to error Timed out while attempting to establish a connection to the container
sam local invoke MyTestLambda --event my-test-event.json --debug --docker-network host
The project that is experiencing this issue had not been touched since version 1.6.2 of the aws-sam-cli. When upgrading to the lastest version 1.48.0 we started to see this issue. After trying a number of versions we found this issue started after version 1.12.0.

Steps to reproduce:

1. Install aws-sam-cli version 1.48.0 or any version after 1.12.0
2. Run sam local invoke MyTestLambda --event my-test-event.json --debug --docker-network host using a template.yaml similar to the following. NOTE: the template below has a bunch of stuff renamed from the original and values obfuscated to hide some things. This is just to give an idea of what was being used.

AWSTemplateFormatVersion: 2010-09-09
Description: my-test
Transform:
- AWS::Serverless-2016-10-31
Resources:
  MyTestDLQueue:
    Type: AWS::SQS::Queue
    Properties:
      FifoQueue: true
      QueueName: MyTestDLQueue.fifo
  MyTestQueue:
    Type: AWS::SQS::Queue
    DependsOn: MyTestDLQueue
    Properties:
      FifoQueue: true
      QueueName: MyTestQueue.fifo
      RedrivePolicy:
        deadLetterTargetArn:
          Fn::GetAtt:
          - MyTestDLQueue
          - Arn
        maxReceiveCount: 5
  MyTestQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    DependsOn:
    - MyTestQueue
    - MyTestDLQueue
    Properties:
      Queues:
      - Ref: MyTestQueue
      - Ref: MyTestDLQueue
      PolicyDocument:
        Statement:
        - Effect: Allow
          Action:
          - sqs:SendMessage
          - sqs:ReceiveMessage
          Resource:
          - Fn::GetAtt:
            - MyTestQueue
            - Arn
          - Fn::GetAtt:
            - MyTestDLQueue
            - Arn
          Principal:
            AWS:
            - Ref: AWS::AccountId
  MyTestLambdaRole:
    Type: AWS::IAM::Role
    DependsOn: MyTestQueue
    Properties:
      RoleName: MyTestLambdaRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: /
      Policies:
      - PolicyName: logs
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action:
            - logs:*
            Resource: arn:aws:logs:*:*:*
      - PolicyName: sqs
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action:
            - sqs:*
            Resource:
            - Fn::GetAtt:
              - MyTestQueue
              - Arn
            - Fn::GetAtt:
              - MyTestDLQueue
              - Arn
      - PolicyName: s3
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action:
            - s3:CopyObject
            - s3:GetObject
            - s3:GetObjectTagging
            - s3:GetObjectVersion
            - s3:ListBucket
            - s3:PutObject
            - s3:PutObjectTagging
            - s3:PutObjectAcl
            Resource: '*'
      - PolicyName: ec2
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action:
            - ec2:DescribeNetworkInterfaces
            - ec2:CreateNetworkInterface
            - ec2:DeleteNetworkInterface
            - ec2:DescribeInstances
            - ec2:AttachNetworkInterface
            Resource: '*'
  MyTestLambda:
    Type: AWS::Serverless::Function
    DependsOn: MyTestLambdaRole
    Properties:
      Description: A Lambda function that logs the payload of messages sent to an
        associated SQS queue.
      FunctionName: MyTestLambda
      Role:
        Fn::GetAtt:
        - MyTestLambdaRole
        - Arn
      Runtime: nodejs12.x
      Handler: cjs/index.handler
      Events:
        SQSQueueEvent:
          Type: SQS
          Properties:
            Queue:
              Fn::GetAtt:
              - MyTestQueue
              - Arn
      Environment:
        Variables:
          S3_BUCKET: my.test.s3.bucket
      MemorySize: 128
      Timeout: 25
      VpcConfig:
        SecurityGroupIds:
        - sg-**********
        SubnetIds:
        - subnet-**********
      CodeUri: MyTestLambda
  MyTestLambdaLogGroup:
    Type: AWS::Logs::LogGroup
    DependsOn: MyTestLambda
    Properties:
      RetentionInDays: 14
      LogGroupName: /aws/lambda/MyTestLambda

Observed result:

2022-05-05 16:01:27,737 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2022-05-05 16:01:27,737 | Using config file: samconfig.toml, config environment: default
2022-05-05 16:01:27,737 | Expand command line arguments to:
2022-05-05 16:01:27,737 | --template_file=~/my-test/.aws-sam/build/template.yaml --event=events/company-verification-event-sqs-fifo.json --docker_network=host --function_logical_id=MyTestLambda --no_event --layer_cache_basedir=~/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1 
2022-05-05 16:01:27,737 | local invoke command is called
2022-05-05 16:01:27,762 | No Parameters detected in the template
2022-05-05 16:01:27,784 | There is no customer defined id or cdk path defined for resource MyTestDLQueue, so we will use the resource logical id as the resource id
2022-05-05 16:01:27,784 | There is no customer defined id or cdk path defined for resource MyTestQueue, so we will use the resource logical id as the resource id
2022-05-05 16:01:27,784 | There is no customer defined id or cdk path defined for resource MyTestQueuePolicy, so we will use the resource logical id as the resource id
2022-05-05 16:01:27,784 | There is no customer defined id or cdk path defined for resource MyTestLambdaRole, so we will use the resource logical id as the resource id
2022-05-05 16:01:27,784 | Sam customer defined id is more priority than other IDs. Customer defined id for resource MyTestLambda is MyTestLambda
2022-05-05 16:01:27,784 | There is no customer defined id or cdk path defined for resource MyTestLambdaLogGroup, so we will use the resource logical id as the resource id
2022-05-05 16:01:27,785 | 0 stacks found in the template
2022-05-05 16:01:27,785 | No Parameters detected in the template
2022-05-05 16:01:27,800 | There is no customer defined id or cdk path defined for resource MyTestDLQueue, so we will use the resource logical id as the resource id
2022-05-05 16:01:27,800 | There is no customer defined id or cdk path defined for resource MyTestQueue, so we will use the resource logical id as the resource id
2022-05-05 16:01:27,800 | There is no customer defined id or cdk path defined for resource MyTestQueuePolicy, so we will use the resource logical id as the resource id
2022-05-05 16:01:27,800 | There is no customer defined id or cdk path defined for resource MyTestLambdaRole, so we will use the resource logical id as the resource id
2022-05-05 16:01:27,800 | Sam customer defined id is more priority than other IDs. Customer defined id for resource MyTestLambda is MyTestLambda
2022-05-05 16:01:27,800 | There is no customer defined id or cdk path defined for resource MyTestLambdaLogGroup, so we will use the resource logical id as the resource id
2022-05-05 16:01:27,801 | 6 resources found in the stack 
2022-05-05 16:01:27,801 | Found Serverless function with name='MyTestLambda' and CodeUri='MyTestLambda'
2022-05-05 16:01:27,801 | --base-dir is not presented, adjusting uri MyTestLambda relative to ~/my-test/.aws-sam/build/template.yaml
2022-05-05 16:01:27,823 | Found one Lambda function with name 'MyTestLambda'
2022-05-05 16:01:27,823 | Invoking cjs/index.handler (nodejs12.x)
2022-05-05 16:01:27,823 | Environment variables overrides data is standard format
2022-05-05 16:01:27,823 | Loading AWS credentials from session with profile 'None'
2022-05-05 16:01:27,837 | Resolving code path. Cwd=~/my-test/.aws-sam/build, CodeUri=~/my-test/.aws-sam/build/MyTestLambda
2022-05-05 16:01:27,837 | Resolved absolute path to code is ~/my-test/.aws-sam/build/MyTestLambda
2022-05-05 16:01:27,837 | Code ~/my-test/.aws-sam/build/MyTestLambda is not a zip/jar file
2022-05-05 16:01:27,867 | Skip pulling image and use local one: public.ecr.aws/sam/emulation-nodejs12.x:rapid-1.48.0-x86_64.
2022-05-05 16:01:27,867 | Mounting ~/my-test/.aws-sam/build/MyTestLambda as /var/task:ro,delegated inside runtime container
2022-05-05 16:02:28,254 | Cleaning all decompressed code dirs
2022-05-05 16:02:28,254 | Timed out while attempting to establish a connection to the container. You can increase this timeout by setting the SAM_CLI_CONTAINER_CONNECTION_TIMEOUT environment variable. The current timeout is 300.0 (seconds).

As you can see in the logs we did try increasing the SAM_CLI_CONTAINER_CONNECTION_TIMEOUT without luck

Expected result:

Lambda should have been invoked and it was not due to container connection timeout

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

• OS: MacOS Monterey Version 12.2.1 (21D62)
• sam --version: 1.48.0
  NOTE: This started happening with version 1.13.1
• AWS region: N/A this is a local issue

[jfuss]

@jdesrochers --docker-network host attaches the container to the Docker VM (on Mac and Windows). So when you provide that to the CLI, the CLI cannot connect to the container through http. This was changed in 1.13.

Paths forward:

• Set a customer network that allows access to the VM host
• Use --container-host (see Testing Api Gateway endpoint locally returning No response from invoke container and Invalid lambda response received in Dockerised sam app #2492 for more details).
• Drop --docker-network host (you shouldn't need this option unless you are doing "fancy" docker things).

[jdesrochers]

Thanks @jfuss I was able to replace --docker-network host with --container-host localhost. Now we are able to run locally with the latest version of the aws-sam-cli. Thanks for the help. Closing this ticket.

[5t33]

FYI - this doesn't work for me. I'm able to connect to localhost s3 at http://localhost:4566 from my browser. However, I'm not able to connect to it using boto3. I run the command like so:

sam local invoke ... --container-host localhost