Testing Api Gateway endpoint locally returning No response from invoke container and Invalid lambda response received in Dockerised sam app

[alliesground]

Description:

I'm dockerising sam app. When accessing api endpoint for HelloWorld locally, it responds with No response from invoke container for HelloWorldFunction and Invalid lambda response received: Lambda response must be valid json.

It works fine whithout dockerisation.

Steps to reproduce:

Dockerfile

FROM ruby:2.7.0-alpine

RUN apk add --update --no-cache \
    build-base \
    postgresql-dev \
    postgresql-client \
    python3 \
    py3-pip \
    util-linux \
    python3-dev

RUN pip3 install aws-sam-cli

ARG USER
ARG HOME
ARG UID

RUN apk add --update \
    sudo

RUN echo "Welcome home: $USER => $UID"

RUN adduser -S -D -G users -u $UID $USER
RUN addgroup -S sudo
RUN echo "%sudo ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/sudo 
RUN adduser $USER sudo

RUN echo "Welcome home: $USER"

WORKDIR ${HOME}

EXPOSE 3001

ENTRYPOINT ["sh", "./entrypoint.sh"]

docker-compose.yml

version: '3.8'
services:
  sam_app:
    build:
      context: ./sam-app
      args:
        - HOME
        - USER
        - UID
    user: "${UID}:100"
    command: ["$PWD"]
    ports:
      - "3001:3001"
    volumes:
      - ./sam-app:$HOME
      - /var/run/docker.sock:/var/run/docker.sock

Entrypoint.sh

BASEDIR="$1"

echo "Basedir => ${BASEDIR}"

sudo sam local start-api \
  --template ./template.yaml \
  --host 0.0.0.0 \
  --port 3001 \
  --docker-volume-basedir "${BASEDIR}/sam-app/" \
  --docker-network drink_default \
  --debug

hello_world/app.rb

def lambda_handler(event:, context:)
  {
    statusCode: 200,
    body: {
      message: "Hello World!",
      # location: response.body
    }.to_json
  }
end

Observed result:

am_app_1  | 2020-12-21 01:31:22,493 | Constructed String representation of Event to invoke Lambda. Event: {"body": null, "headers": {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9", "Accept-Encoding": "gzip, deflate, br", "Accept-Language": "en-GB,en;q=0.9", "Connection": "keep-alive", "Host": "localhost:3001", "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36", "X-Forwarded-Port": "3001", "X-Forwarded-Proto": "http"}, "httpMethod": "GET", "isBase64Encoded": false, "multiValueHeaders": {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Accept-Encoding": ["gzip, deflate, br"], "Accept-Language": ["en-GB,en;q=0.9"], "Connection": ["keep-alive"], "Host": ["localhost:3001"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"], "X-Forwarded-Port": ["3001"], "X-Forwarded-Proto": ["http"]}, "multiValueQueryStringParameters": null, "path": "/hello", "pathParameters": null, "queryStringParameters": null, "requestContext": {"accountId": "123456789012", "apiId": "1234567890", "domainName": "localhost:3001", "extendedRequestId": null, "httpMethod": "GET", "identity": {"accountId": null, "apiKey": null, "caller": null, "cognitoAuthenticationProvider": null, "cognitoAuthenticationType": null, "cognitoIdentityPoolId": null, "sourceIp": "172.22.0.1", "user": null, "userAgent": "Custom User Agent String", "userArn": null}, "path": "/hello", "protocol": "HTTP/1.1", "requestId": "dd99d1bb-cc40-42f8-a2f2-eeefcc22d111", "requestTime": "21/Dec/2020:01:30:38 +0000", "requestTimeEpoch": 1608514238, "resourceId": "123456", "resourcePath": "/hello", "stage": "Prod"}, "resource": "/hello", "stageVariables": null, "version": "1.0"}
sam_app_1  | 2020-12-21 01:31:22,493 | Found one Lambda function with name 'HelloWorldFunction'
sam_app_1  | 2020-12-21 01:31:22,494 | Invoking app.lambda_handler (ruby2.7)
sam_app_1  | 2020-12-21 01:31:22,494 | No environment variables found for function 'HelloWorldFunction'
sam_app_1  | 2020-12-21 01:31:22,494 | Environment variables overrides data is standard format
sam_app_1  | 2020-12-21 01:31:22,494 | Loading AWS credentials from session with profile 'None'
sam_app_1  | 2020-12-21 01:31:24,549 | Resolving code path. Cwd=/home/sameer/projects/drink/sam-app/, CodeUri=hello_world/
sam_app_1  | 2020-12-21 01:31:24,549 | Resolved absolute path to code is /home/sameer/projects/drink/sam-app/hello_world
sam_app_1  | 2020-12-21 01:31:24,550 | Code /home/sameer/projects/drink/sam-app/hello_world is not a zip/jar file
sam_app_1  | 2020-12-21 01:31:24,574 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-ruby2.7:rapid-1.15.0.
sam_app_1  | 
sam_app_1  | 2020-12-21 01:31:24,574 | Mounting /home/sameer/projects/drink/sam-app/hello_world as /var/task:ro,delegated inside runtime container
sam_app_1  | 2020-12-21 01:31:25,386 | Starting a timer for 3 seconds for function 'HelloWorldFunction'
sam_app_1  | 2020-12-21 01:31:26,707 | Cleaning all decompressed code dirs
sam_app_1  | 2020-12-21 01:31:26,707 | No response from invoke container for HelloWorldFunction
sam_app_1  | 2020-12-21 01:31:26,707 | Invalid lambda response received: Lambda response must be valid json
sam_app_1  | 2020-12-21 01:31:26 172.22.0.1 - - [21/Dec/2020 01:31:26] "GET /hello HTTP/1.1" 502 -
sam_app_1  | 2020-12-21 01:31:26 172.22.0.1 - - [21/Dec/2020 01:31:26] "GET /favicon.ico HTTP/1.1" 403 -

Expected result:

Json output on browser {message: "hello world"}

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: ubuntu VERSION="20.04.1 LTS (Focal Fossa)"
2. sam --version: 1.15.0
3. Homebrew: VERSION="2.6.2"
4. docker: VERSION="19.03.8"

Add --debug flag to command you are running

[jfuss]

@alliesground SAM CLI is meant to be run from your machine and interact with docker, not run SAM CLI within Docker. My suggestion here is to do that instead of Dockerize.

What's the use case on wanting to do this? Why do you need to be running docker within docker?

[rbliss]

I also get this exact error. I will chime in on my use case which may or may not be similar:

In my case, I'm using VSCode's Dev Containers, to create an isolated dev environment SAM is running in. My setup is nearly identical, using a volume to expose the host's docker socket, and SAM's --docker-volume-basedir flag to direct SAM to the appropriate location. Hitting the endpoint gives me the same No response from invoke container for HelloWorldLambda

Unfortunately, SAM's --debug flag doesn't go into much detail about what went on with the lambda container it spins up to serve the request. I started tracing through the CLI source to see what happened, but haven't had time to finish my search.

[alliesground]

@jfuss my use case is similar to that of @rbliss i.e to create an isolated dev environment, which includes react-app and db, all composed with docker-compose.

[alayor]

I'm also having the same problem. My use case is exactly the same as @alliesground.

I followed the instructions on this post. It makes me think this used to work before.
https://medium.com/monsoon-engineering/running-aws-sam-in-a-docker-container-2491596672c2

[gpomykala]

I receive same error when trying to hit a lambda spun with 'sam local start-lambda --docker-network host'. It used to work with older version of SAM CLI. I am not doing anything extraordinary, just a basic integration test of a lambda client hitting lambda hosted in SAM CLI

[jfuss]

@gpomykala Your issue is logged here: #2436 (comment) and workarounds here: #2436 (comment)

@rbliss @alayor @alliesground
We have seen this happen when SAM CLI cannot communicate to the container. Instead of exec-ing within the container, we updated to move to publishing a port and then calling into the container of http. This could be the cause here as well. SAM CLI will communicate to the container over localhost only, so there is probably some docker network trickery going on. I highly suggest to not use SAM CLI with the container like this. It's not really a use case we directly support or recommend.

[alayor]

@jfuss Thank you for your response and links.
I wonder if there could be a way to change the "communicate to the container over localhost" setting.
I just wanted to test if changing "localhost" to "docker.for.mac.localhost" would make it work.
I know this is not recommended, but I figured I could ask anyway.

[jfuss]

@alayor I am open to ways to make this work if we can find a solution that works across OS. I am not sure if moving to the docker.for.mac.localhost (or host.docker.internal which I think might work across OS).

Just mapping this out:

• SAM runs in Container A on a bridge docker network (by default).
• Executing a sam local command will start up a container and publish ports to localhost (not sure if this is localhost of the machine or on the bridge docker network).
• SAM attempts to call localhost: to communicate to the container and fails to get a response.

If in step two, this does publish a port on the machines localhost, then I would assume what you are suggesting could work. Super hacky and seems much easier to just run the CLI on your machine and not have to care about docker at all. Could be a personal preference though.

[alliesground]

@jfuss thanks for your response. It was just an itch of mine to try to dockerize sam-cli especially after reading some articles on the possibilities. Anyways I will move forward with your advice and run CLI on the host machine and maybe try to connect it with other tech stacks running in the container.

[ilyasotkov]

I was able to get SAM local invoke to work inside Dev Container by replacing hardcoded localhost here

https://github.com/aws/aws-sam-cli/blob/develop/samcli/local/docker/container.py#L41

with host.docker.internal. I used this guide for attaching the VS Code debugger: https://github.com/aws/aws-toolkit-vscode/blob/a380685696e39247d6ec0f8f1e9928ba2562f550/docs/debugging-python-lambda-functions.md, also replacing localhost with host.docker.internal in launch.json.

@jfuss I'm running this on macOS, AFAIK this won't work on linux out of the box due to docker/for-linux#264, but a SAM CLI option to specify host would be nice.

[jasonterando]

If in step two, this does publish a port on the machines localhost, then I would assume what you are suggesting could work. Super hacky and seems much easier to just run the CLI on your machine and not have to care about docker at all. Could be a personal preference though.

Given that SAM CLI is launching a container to run SAM template, would it make sense to consider documenting how to launch the image via Docker Compose (or similar tool)?

Probably ought to explain why some of us want to run SAM under Docker (regardless if it's a good idea, possible, etc.) One of the nice things about developing with Docker is that it helps to avoid workstation bloat and conflicts by, for example, having to install yet another package manager (i.e. Homebrew); which, in turn, steps on our environment's default Python version. This is not to pick on Homebrew, it's probably great; although like many tools of its ilk it is much easier to install than to remove when you no longer need it (like having to manually edit ~/bashrc). At the very least, let Python developers know in the docs there's the option of running pip3 install aws-sam-cli.

On a side note, I tried ilyasotkov's workaround/hack in Linux and it didn't work for me. For anybody interested, here's the Dockerfile I set up:

FROM python:alpine
RUN apk update && \
    apk upgrade && \
    apk add bash && \
    apk add --no-cache --virtual build-deps build-base gcc && \
    pip install aws-sam-cli && \
    apk del build-deps

# Replace localhost with host.docker.internal in AWS SAM
RUN sed -i 's/localhost/host.docker.internal/g' /usr/local/lib/python3.9/site-packages/samcli/local/docker/container.py

RUN mkdir /app
WORKDIR /app
EXPOSE 3000
COPY ./.docker/entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

And the entrypoint.sh...

#!/bin/bash

# Trick to get host.docker.internal working on Linux Docker
# From https://dev.to/bufferings/access-host-from-a-docker-container-4099
HOST_DOMAIN="host.docker.internal"
ping -q -c1 $HOST_DOMAIN > /dev/null 2>&1
if [ $? -ne 0 ]; then
    HOST_IP=$(route | awk 'FNR==3 {print $2}')
    echo -e "$HOST_IP\t$HOST_DOMAIN" >> /etc/hosts
fi

set -o errexit
cd /app
/usr/local/bin/sam local start-api \
  --template ./template.yaml \
  --host 0.0.0.0 \
  --docker-volume-basedir /app \
  --docker-network my-demo-app_local  \
  --skip-pull-image

[xiaoxinghu]

Another common use case for dockerizing sam: CI pipelines. Like @jasonterando said, it's a pain to maintain packages/tools installed on build agents. Different teams need their tools in particular ways. It's more like dependencies for your projects. The tools break over time, so you need to keep your setup predictable. That's what docker provides. If it's "not a good idea" to use sam within docker, then I run out of ideas to make my CI pipeline stable over long period of time. I can't trust my scripts/code to still work 1 year from now, after countless patching/upgrading of the build agents. Not to mention that normally it's not easy to tweak infrastructure setups within big corporates, it's a long and risky process. Sometimes impossible (e.g. they refuse to install arbitrary software on agents, because they gave you docker already).

[ilyasotkov]

I highly suggest to not use SAM CLI with the container like this. It's not really a use case we directly support or recommend.

@jfuss Doesn't the presence of -v, --docker-volume-basedir TEXT option for sam local invoke contradict with that statement? The docs for it say

If Docker is running on a remote machine, you must mount the path where the AWS SAM file exists on the Docker machine, and modify this value to match the remote machine.

So remote Docker hosts are already partially supported.

[sapessi]

Adding the pm-review label.

If I read the comments correctly, it sounds like the primary use-case here is to use the SAM CLI inside a Dockerized development environment, a-la VSCode remote. Is that correct? Could you tell us a little bit more about the rest of the environment setup (OS, IDE, etc) so that we can make sure we plan accordingly and have the right conversation on our side.

Let us take this away and think about it for a bit.

[jasonterando]

Adding the pm-review label.

If I read the comments correctly, it sounds like the primary use-case here is to use the SAM CLI inside a Dockerized development environment, a-la VSCode remote. Is that correct? Could you tell us a little bit more about the rest of the environment setup (OS, IDE, etc) so that we can make sure we plan accordingly and have the right conversation on our side.

Let us take this away and think about it for a bit.
Hi, thanks for the dialogue.

For DevOps, we like to launch containers that execute build and then unit and integration testing. It keeps us from having to clutter up our CI/CD server with external dependencies for which we have to keep track of for updates, etc. For example, I can deploy a PHP application which runs composer install and PHPUnit without having to have PHP, composer, etc. installed on the CI/CD server, and have to worry about keeping them up-to-date. In the case of SAM, it'd be nice to start off with a base image of AWS SAM CLI, add in TypeScript, do my testing and then build by deployment all from a container, without having to deal with installing dependencies on my CI/CD server.

For development purposes, it would be slick to be able to set up a Docker Compose configuration that launches the SAM-configured service with accompanying dependencies (like localstack for simulating communication with Amazon resources like S3 and Kinesis), mocked external services, etc.. Using Docker Compose for this is largely OS-agnostic, without having to install everything on the developer's machine. Alternatively I can stand up Bash, PowerShell, etc. scripts to launch and integrate things, but it's more clunky, and in the case of installing the SAM CLI, may involve installing things like Homebrew which make changes to the developer's Python environment.

[ilyasotkov]

I used the docker-in-docker setup to be able to attach VS Code's debugger to a SAM CLI-managed container when using VS Code dev containers. This repo demonstrates this (with the above localhost-replacing hack already applied):

https://github.com/ilyasotkov/vscode-devcontainer-aws-sam-local

I'm running this on a macOS machine but it could equally be Ubuntu or Windows as long as it has Docker Desktop installed.

[chaoyangnz]

downgrade to 1.12.0. works

[BenVosper]

Thanks for looking into this @sapessi. Have you got any updates you might be able to share on progress?

I'd like to echo the points @jasonterando made on the use-cases for this functionality. Essentially, we'd like to be able to run the CLI via docker to allow:

• Reproducible build for our continuous integration / deployment workflows
• Reliable cross-platform development enviromnents, to make our apps built with the CLI to be as easy as possible to develop

[dog5tar]

I got the same error as the description of the issue.

I have aws-sam-cli installed via brew.
I have MySQL running from the docker container.

sam local start-api --docker-network host connects to MySQL no problem but does not invoke lambdas.

[jamesorlakin]

Other than downgrading to 1.12.0, is there a workaround for newer versions when not doing anything network related? I'm keen to try to use warm containers in the newer versions to profile performance without the Java classloader penalties, but I'm only doing sam local start-api and getting the invoke errors as with others which makes local development virtually impossible.

I would assume this would be a more widespread issue if it didn't work for anyone, so would there be something about my environment that would cause this?

I'm using Docker Toolbox (unfortunately I can't use Desktop due to Hyper-V issues) which means I'm stuck with Docker 19.03. As I'm on Windows, this is all running in a VirtualBox VM and I'm invoking SAM on the Windows side. I've not done anything to the network configuration with regards to Docker, but creating a network with docker network create test and then doing --docker-network test does not help in case it's defaulting to host networking of some sort. This is a fairly standard Maven app; there's no Dockerfile image building if that matters.

[ppejovic]

I have a working compose setup where in the ENTRYPOINT I'm hacking the localhost IP in /etc/hosts of the container running sam local start-lambda:

function hackLocalhostIp {
    local localhost_ip="$(getent hosts host.docker.internal | awk '{ print $1 }')"
    # Docker mounts /etc/hosts so we can't update the file in place with sed
    cp /etc/hosts tmpfile
    sed "/^127.0.0.1/ s/.*/$localhost_ip\tlocalhost/g" < tmpfile > /etc/hosts 
    rm tmpfile
}

[neugeeug]

I have similar problem calling lambda locally:

sam build && sam local invoke --debug --docker-network host -e events/event.json

And can't see output at all:
2021-02-24 15:35:28,692 | No response from invoke container for MyTestLambda

Lambda has been written in nodejs14.x.
Environment:

$ sam --version    
SAM CLI, version 1.18.2

I tried to downgrade to version 1.12.0 but got a message that node14.x is not supported.
Could you please give an estimate when are you going to provide a fix.

[sapessi]

Thank you all for the feedback. I have added this to our backlog and we'll post updates here as we prioritize it.

[alexdrans]

I fixed this like this - Essentially, make sure your SAM local and database are running on the same Docker network that it not host.

> sam --version
SAM CLI, version 1.20.0

version: '3.5'
services:
  postgres:
    image: postgres:10.12
    ...
    networks:
      - api

networks:
  api:

Use docker-compose up -d, then docker network ls to get the network name or look at the logs for the network created

DATABASE_URL=postgres://user:password@postgres:5432/databaseName - Notice the host changes from localhost to postgres

sam local start-api --docker-network=app-name_api

[kstro21]

I'm trying the [Unstable] Nightly Release v1.21.1.dev202103302031 - 2021-03-30 which adds the --container-host option include when #2700 was merged, but it still does not work. In my case, the containers are running in a remote host but the ports are being bound to 127.0.0.1 making them not accessible from my computer thus resulting in No response from invoking container for MyLambdaFunction.

Is there an option so the ports are not bound to localhost inside the container?

This is the command I'm using

sam local start-lambda \
-t template.yaml \
--profile sandbox \
--region us-east-1 \
--env-vars .envs.json \
--host 0.0.0.0 \
--parameter-overrides Environment=sandbox \
--container-host 192.168.99.102 \
--warm-containers LAZY \
--skip-pull-image

This started failing after upgrading to 1.8 and newer versions.

[kstro21]

Ok, I have new information on the issue.

aws-sam-cli/samcli/local/docker/container.py

Lines 153 to 158 in 86f88cb

	kwargs["ports"] = {self.RAPID_PORT_CONTAINER: ("127.0.0.1", self.rapid_port_host)}
	if self._exposed_ports:
	kwargs["ports"].update(
	{container_port: ("127.0.0.1", host_port) for container_port, host_port in self._exposed_ports.items()}
	)

Can you see how '127.0.0.1' is hardcoded on the code? Why? If I remove the 127.0.0.1 from those 2 lines, everything work as expected.

[xazhao]

@kstro21 Hi we hardcoded 127.0.0.1 for security purpose because we don't want it bind ports to all interfaces by default.

I'm trying to reproduce your issue. I have a few questions. You have SAM CLI and your project on your local host while Docker engine is running on a remote host right? Does Docker have access to your local SAM files and how is your Docker file sharing setup?

[kstro21]

Hey, @xazhao, yes, I have SAM CLI and my project on my local computer, and the Docker engine is running on a remote host, Docker has access to my SAM files through a mounted remoted directory. But if you are in Linux, you can easily reproduce it using what @jamesorlakin is using, Docker Toolbox which will run the Docker containers inside a VirtualBox Virtual Machine.

Let me know if that helps.

[rv2673]

@xazhao
I do understand that the 127.0.0.1 is sensible for security, though what about having a flag to be able to change to which ip/host it binds for those who need it with a 'remote' docker host?

In our setup we have the following:

• A docker build container where sam cli runs running on a machine.
• A priviliged docker (rootless)dind container running on the same machine.
• Both share a volume where the code repository is on
• The containers are connected to a docker network so that the docker machine is available with hostname 'docker'

The docker host is remote from the perspective of the build container.

I slightly modified sam cli in our build container image to get it to work currently:

• Use the version that contains the following PR feat: add option --container-host to commands local start-api, local start-lambda and local invoke #2700 (which sadly was reverted)
• Made it the container host flag also work for warm containers (it missed propagation to container instance in the run method of runtime. line 115. Which caused the host used for calling to None). (feature was reverted before i could submit a PR)
• I removed the hardcoded 127.0.0.1 so that it binds to all interfaces, which is safe in this case since it is contained in a virtual docker network(and only one interface). More ideally this would be done trough a flag. Or use the container-host flag name/ip.

[xazhao]

@rv2673 Using --container-host to replace 127.0.0.1 is a better idea in my opinion since it won't bind to all interfaces. It would support your use case and use case of @kstro21 .
Thanks for looking into the warm container issue. We realized that issue before last release so we decided to revert the change. We would like to release a completed function as possible.
Currently I'm preparing a PR to include following changes:

1. Reverted PR [feat: add option --container-host to commands local start-api, local start-lambda and local invoke #2700]
2. Fix warm container issue
3. Replace 127.0.0.1 with the new option --container-host so it can support remote Docker host.

[timoaudi]

For those who are wondering how to downgrade to 1.12.0 – this worked for me:

$ brew uninstall aws-sam-cli
$ pip3 install --user 'aws-sam-cli==1.12.0'

[xazhao]

@kstro21 @rv2673 In our latest nightly release, a new option --container-host-interface is added so you can specify what interface you want to bind ports to. Feel free to try it with remote Docker host.

[kstro21]

@xazhao, thanks, it works.

What's the difference between --container-host and --container-host-interface? For me are the same. This is how I have to use them

sam local start-api \
-t template.yaml \
--region us-east-1 \
--env-vars .envs.json \
--parameter-overrides Environment=sandbox \
--container-host 192.168.99.102 \
--container-host-interface 192.168.99.102 \
--warm-containers LAZY \
--skip-pull-image

As you can see, both options have the same value.

[xazhao]

@kstro21 In your case, yes these two options have the same value.
In some cases, for example, we want to use it in a docker container, --container-host will be host.docker.internal and --container-host-interface will be 127.0.0.1 since we want it bind to loopback interface because of security reason.
--container-host is the host of locally emulated Lambda container while --container-host-interface is used to specify interface it binds ports to. They two options should work independently.
Hopefully this makes sense to you.

[rv2673]

@xazhao Sorry was off for a few days. But just tested it, and it works great.

[psdkumar]

I fixed this like this - Essentially, make sure your SAM local and database are running on the same Docker network that it not host.

> sam --version
SAM CLI, version 1.20.0

version: '3.5'
services:
  postgres:
    image: postgres:10.12
    ...
    networks:
      - api

networks:
  api:

Use docker-compose up -d, then docker network ls to get the network name or look at the logs for the network created

DATABASE_URL=postgres://user:password@postgres:5432/databaseName - Notice the host changes from localhost to postgres

sam local start-api --docker-network=app-name_api

@alexdrans This worked for me. Thanks!

---

For anyone who finds this:

I replaced app-name_api with whatever is visible after running docker network ls. For example, in my case, it was local-setup_postgres

[kingram6865]

Is there a resolution to this issue?
Or a breadcrumb I can grab to decipher this problem?

Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instan
tly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2021-08-06 21:14:29  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
Invoking app.lambdaHandler (nodejs12.x)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-nodejs12.x:rapid-1.24.1.

Mounting C:\\tmp\samtest\hello-world as /var/task:ro,delegated inside runtime container
Function 'HelloWorldFunction' timed out after 3 seconds
No response from invoke container for HelloWorldFunction
Invalid lambda response received: Lambda response must be valid json
2021-08-06 21:15:02 127.0.0.1 - - [06/Aug/2021 21:15:02] "←[35m←[1mGET /hello HTTP/1.1←[0m" 502 -
2021-08-06 21:15:02 127.0.0.1 - - [06/Aug/2021 21:15:02] "←[31m←[1mGET /favicon.ico HTTP/1.1←[0m" 403 -

[jomach]

I had the same issue.
running on a mac with visual studio code and dev containers.
Solved by just adding: --container-host host.docker.internal

[kingram6865]

I had the same issue.
running on a mac with visual studio code and dev containers.
Solved by just adding: --container-host host.docker.internal

That was a pretty good tip but the issue is still unresolved:

$ sam local start-api --host 192.168.1.83 --container-host host.docker.internal
Mounting HelloWorldFunction at http://192.168.1.83:3000/hello [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2021-08-17 17:09:42  * Running on http://192.168.1.83:3000/ (Press CTRL+C to quit)
Invoking app.lambdaHandler (nodejs12.x)
Image was not found.
Building image..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Failed to build Docker Image
NoneType: None
Exception on /hello [GET]
Traceback (most recent call last):
  File "/PUBLIC/DOWNLOADS/.venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/PUBLIC/DOWNLOADS/.venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/PUBLIC/DOWNLOADS/.venv/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/PUBLIC/DOWNLOADS/.venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/PUBLIC/DOWNLOADS/.venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/PUBLIC/DOWNLOADS/.venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/PUBLIC/DOWNLOADS/aws-sam-cli/samcli/local/apigw/local_apigw_service.py", line 317, in _request_handler
    self.lambda_runner.invoke(route.function_name, event, stdout=stdout_stream_writer, stderr=self.stderr)
  File "/PUBLIC/DOWNLOADS/aws-sam-cli/samcli/commands/local/lib/local_lambda.py", line 137, in invoke
    container_host_interface=self.container_host_interface,
  File "/PUBLIC/DOWNLOADS/aws-sam-cli/samcli/lib/telemetry/metric.py", line 217, in wrapped_func
    return_value = func(*args, **kwargs)
  File "/PUBLIC/DOWNLOADS/aws-sam-cli/samcli/local/lambdafn/runtime.py", line 176, in invoke
    container = self.create(function_config, debug_context, container_host, container_host_interface)
  File "/PUBLIC/DOWNLOADS/aws-sam-cli/samcli/local/lambdafn/runtime.py", line 86, in create
    container_host_interface=container_host_interface,
  File "/PUBLIC/DOWNLOADS/aws-sam-cli/samcli/local/docker/lambda_container.py", line 87, in __init__
    image = LambdaContainer._get_image(lambda_image, runtime, packagetype, imageuri, layers)
  File "/PUBLIC/DOWNLOADS/aws-sam-cli/samcli/local/docker/lambda_container.py", line 213, in _get_image
    return lambda_image.build(runtime, packagetype, image, layers)
  File "/PUBLIC/DOWNLOADS/aws-sam-cli/samcli/local/docker/lambda_image.py", line 144, in build
    self._build_image(image if image else image_name, image_tag, downloaded_layers, stream=stream_writer)
  File "/PUBLIC/DOWNLOADS/aws-sam-cli/samcli/local/docker/lambda_image.py", line 245, in _build_image
    raise ImageBuildException("Error building docker image: {}".format(log["error"]))
samcli.commands.local.cli_common.user_exceptions.ImageBuildException: Error building docker image: The command '/bin/sh -c chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1
2021-08-17 17:12:01 192.168.1.161 - - [17/Aug/2021 17:12:01] "GET /hello HTTP/1.1" 502 -
2021-08-17 17:12:02 192.168.1.161 - - [17/Aug/2021 17:12:02] "GET /favicon.ico HTTP/1.1" 403 -
2021-08-17 17:18:29 192.168.1.161 - - [17/Aug/2021 17:18:29] "GET / HTTP/1.1" 403 -
2021-08-17 17:18:29 192.168.1.161 - - [17/Aug/2021 17:18:29] "GET /favicon.ico HTTP/1.1" 403 -

Navigating to the url:

Navigating to the url with the proper endpoint:

This is a mess. I just want to have an environment to test my lambdas before deploying them.

[matthewcn56]

I am having the same issue as @kingram6865

[eichelkrauta]

hey friends above that were having the "Internal Server error" response, I think I had the same thing come up, maybe this would help?

I started off having the "empty response" issue no matter what I tried. After a lot of trial and error, eventually I landed on the below combination of things:

This here is the command I am using:
sam local start-api --warm-containers EAGER --host 0.0.0.0 --container-host host.docker.internal

The combination of the --host and --container-host is what made it work for me. I noticed that when I did a docker ps it was publishing 0.0.0.0:3000->3000, which tipped me off to add the --host 0.0.0.0. So that got me to the point where I was seeing the internal server error. After adding the magical --container-host host.docker.internal as well, that's when it all started to work for me.