Skip to content
This repository has been archived by the owner on Jan 23, 2024. It is now read-only.

ERROR: Encountered errors while bringing up the project. #1212

Closed
walterra opened this issue Jul 19, 2021 · 8 comments
Closed

ERROR: Encountered errors while bringing up the project. #1212

walterra opened this issue Jul 19, 2021 · 8 comments

Comments

@walterra
Copy link

I'm trying to run apm integration testing on a local machine with macOS 11.4.

I'm seeing the following error when trying to run scripts/compose.py:

 $ ./scripts/compose.py start 7.14.0 --bc
found latest build candidate for 7.14 - https://staging.elastic.co/7.14.0-682a8012/summary-7.14.0.html at https://staging.elastic.co/latest/7.14.json
Skipping download of elasticsearch-7.14.0-docker-image.tar.gz, local file is current
Skipping download of apm-server-7.14.0-docker-image.tar.gz, local file is current
Skipping download of kibana-7.14.0-docker-image.tar.gz, local file is current
Starting stack services..

Pulling wait-service ... done
Starting localtesting_7.14.0_elasticsearch ... done
Starting localtesting_7.14.0_kibana        ... done

ERROR: for wait-service  Container "1a436786542a" is unhealthy.
ERROR: for apm-server  Container "31ff405ee763" is unhealthy.
ERROR: Encountered errors while bringing up the project.
Traceback (most recent call last):
  File "./scripts/compose.py", line 31, in <module>
    main()
  File "./scripts/compose.py", line 17, in main
    setup()
  File "/Users/walterra/dev/apm-integration-testing/scripts/modules/cli.py", line 193, in __call__
    self.args.func()
  File "/Users/walterra/dev/apm-integration-testing/scripts/modules/cli.py", line 680, in start_handler
    self.run_docker_compose_process(docker_compose_cmd + up_params)
  File "/Users/walterra/dev/apm-integration-testing/scripts/modules/cli.py", line 442, in run_docker_compose_process
    subprocess.check_call(docker_compose_cmd)
  File "/usr/local/anaconda3/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker-compose', '-f', '/Users/walterra/dev/apm-integration-testing/docker-compose.yml', 'up', '-d']' returned non-zero exit status 1.
@kuisathaverat
Copy link
Contributor

you have to check the logs of the container 31ff405ee763 

docker logs -f 31ff405ee763

@walterra
Copy link
Author

For some reason I see the ES container shutting down, but there aren't any obvious errors in its log, not sure why it's stopping:

docker ps --all

CONTAINER ID   IMAGE                                                  COMMAND                  CREATED             STATUS                         PORTS                      NAMES
43aacf6eca2a   docker.elastic.co/beats/elastic-agent:7.14.0           "/usr/bin/tini -- /u…"   About an hour ago   Exited (1) About an hour ago                              localtesting_7.14.0_elastic-agent
4fc930014da2   apm-integration-testing_apm-server                     "/apmpkg"                About an hour ago   Exited (1) 3 minutes ago                                  localtesting_7.14.0_apm-server-managed
ceae74f6234f   postgres:10                                            "docker-entrypoint.s…"   About an hour ago   Exited (0) 39 minutes ago                                 localtesting_7.14.0_postgres
ace604b064ab   redis:4                                                "docker-entrypoint.s…"   About an hour ago   Exited (0) 39 minutes ago                                 localtesting_7.14.0_redis
31ff405ee763   docker.elastic.co/kibana/kibana:7.14.0                 "/bin/tini -- /usr/l…"   About an hour ago   Up 38 minutes (unhealthy)      127.0.0.1:5601->5601/tcp   localtesting_7.14.0_kibana
1a436786542a   docker.elastic.co/elasticsearch/elasticsearch:7.14.0   "/bin/tini -- /usr/l…"   About an hour ago   Exited (137) 4 minutes ago                                localtesting_7.14.0_elasticsearch

docker logs -f 1a436786542a
...
{"type": "server", "timestamp": "2021-07-19T13:53:58,571Z", "level": "INFO", "component": "o.e.i.g.DatabaseRegistry", "cluster.name": "docker-cluster", "node.name": "1a436786542a", "message": "downloading geoip database [GeoLite2-Country.mmdb] to [/tmp/elasticsearch-3210457199444842828/geoip-databases/XIGAHHCYSLa5cEoQ2uSZFQ/GeoLite2-Country.mmdb.tmp.gz]", "cluster.uuid": "utXHWfENRbaz01uA9KtJnA", "node.id": "XIGAHHCYSLa5cEoQ2uSZFQ"  }
{"type": "server", "timestamp": "2021-07-19T13:53:58,572Z", "level": "INFO", "component": "o.e.i.g.DatabaseRegistry", "cluster.name": "docker-cluster", "node.name": "1a436786542a", "message": "downloading geoip database [GeoLite2-ASN.mmdb] to [/tmp/elasticsearch-3210457199444842828/geoip-databases/XIGAHHCYSLa5cEoQ2uSZFQ/GeoLite2-ASN.mmdb.tmp.gz]", "cluster.uuid": "utXHWfENRbaz01uA9KtJnA", "node.id": "XIGAHHCYSLa5cEoQ2uSZFQ"  }
{"type": "server", "timestamp": "2021-07-19T13:53:58,573Z", "level": "INFO", "component": "o.e.i.g.DatabaseRegistry", "cluster.name": "docker-cluster", "node.name": "1a436786542a", "message": "downloading geoip database [GeoLite2-City.mmdb] to [/tmp/elasticsearch-3210457199444842828/geoip-databases/XIGAHHCYSLa5cEoQ2uSZFQ/GeoLite2-City.mmdb.tmp.gz]", "cluster.uuid": "utXHWfENRbaz01uA9KtJnA", "node.id": "XIGAHHCYSLa5cEoQ2uSZFQ"  }
{"type": "server", "timestamp": "2021-07-19T13:53:58,853Z", "level": "INFO", "component": "o.e.i.g.DatabaseRegistry", "cluster.name": "docker-cluster", "node.name": "1a436786542a", "message": "database file changed [/tmp/elasticsearch-3210457199444842828/geoip-databases/XIGAHHCYSLa5cEoQ2uSZFQ/GeoLite2-Country.mmdb], reload database...", "cluster.uuid": "utXHWfENRbaz01uA9KtJnA", "node.id": "XIGAHHCYSLa5cEoQ2uSZFQ"  }
{"type": "server", "timestamp": "2021-07-19T13:53:58,950Z", "level": "INFO", "component": "o.e.i.g.DatabaseRegistry", "cluster.name": "docker-cluster", "node.name": "1a436786542a", "message": "database file changed [/tmp/elasticsearch-3210457199444842828/geoip-databases/XIGAHHCYSLa5cEoQ2uSZFQ/GeoLite2-ASN.mmdb], reload database...", "cluster.uuid": "utXHWfENRbaz01uA9KtJnA", "node.id": "XIGAHHCYSLa5cEoQ2uSZFQ"  }
{"type": "server", "timestamp": "2021-07-19T13:53:59,300Z", "level": "INFO", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "1a436786542a", "message": "geoip database [GeoLite2-ASN.mmdb] is up to date, updated timestamp", "cluster.uuid": "utXHWfENRbaz01uA9KtJnA", "node.id": "XIGAHHCYSLa5cEoQ2uSZFQ"  }
{"type": "server", "timestamp": "2021-07-19T13:53:59,614Z", "level": "INFO", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "1a436786542a", "message": "geoip database [GeoLite2-City.mmdb] is up to date, updated timestamp", "cluster.uuid": "utXHWfENRbaz01uA9KtJnA", "node.id": "XIGAHHCYSLa5cEoQ2uSZFQ"  }
{"type": "server", "timestamp": "2021-07-19T13:53:59,616Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "docker-cluster", "node.name": "1a436786542a", "message": "Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.monitoring-es-7-2021.07.19][0]]]).", "cluster.uuid": "utXHWfENRbaz01uA9KtJnA", "node.id": "XIGAHHCYSLa5cEoQ2uSZFQ"  }
{"type": "server", "timestamp": "2021-07-19T13:53:59,820Z", "level": "INFO", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "1a436786542a", "message": "geoip database [GeoLite2-Country.mmdb] is up to date, updated timestamp", "cluster.uuid": "utXHWfENRbaz01uA9KtJnA", "node.id": "XIGAHHCYSLa5cEoQ2uSZFQ"  }
{"type": "server", "timestamp": "2021-07-19T13:53:59,905Z", "level": "INFO", "component": "o.e.i.g.DatabaseRegistry", "cluster.name": "docker-cluster", "node.name": "1a436786542a", "message": "database file changed [/tmp/elasticsearch-3210457199444842828/geoip-databases/XIGAHHCYSLa5cEoQ2uSZFQ/GeoLite2-City.mmdb], reload database...", "cluster.uuid": "utXHWfENRbaz01uA9KtJnA", "node.id": "XIGAHHCYSLa5cEoQ2uSZFQ"  }

docker logs -f 31ff405ee763
...
{"type":"response","@timestamp":"2021-07-19T13:57:49+00:00","tags":["api"],"pid":1207,"method":"get","statusCode":503,"req":{"url":"/api/status","method":"get","headers":{"host":"kibana:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"172.18.0.3","userAgent":"curl/7.61.1"},"res":{"statusCode":503,"responseTime":38,"contentLength":32094},"message":"GET /api/status 503 38ms - 31.3KB"}
{"type":"log","@timestamp":"2021-07-19T13:57:59+00:00","tags":["warning","plugins","licensing"],"pid":1207,"message":"License information could not be obtained from Elasticsearch due to ConnectionError: getaddrinfo ENOTFOUND elasticsearch error"}
{"type":"response","@timestamp":"2021-07-19T13:57:59+00:00","tags":["api"],"pid":1207,"method":"get","statusCode":503,"req":{"url":"/api/status","method":"get","headers":{"host":"kibana:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"172.18.0.3","userAgent":"curl/7.61.1"},"res":{"statusCode":503,"responseTime":35,"contentLength":32089},"message":"GET /api/status 503 35ms - 31.3KB"}
{"type":"log","@timestamp":"2021-07-19T13:58:09+00:00","tags":["warning","plugins","licensing"],"pid":1207,"message":"License information could not be obtained from Elasticsearch due to ConnectionError: getaddrinfo ENOTFOUND elasticsearch error"}
{"type":"response","@timestamp":"2021-07-19T13:58:09+00:00","tags":["api"],"pid":1207,"method":"get","statusCode":503,"req":{"url":"/api/status","method":"get","headers":{"host":"kibana:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"172.18.0.3","userAgent":"curl/7.61.1"},"res":{"statusCode":503,"responseTime":36,"contentLength":32088},"message":"GET /api/status 503 36ms - 31.3KB"}
{"type":"log","@timestamp":"2021-07-19T13:58:18+00:00","tags":["warning","plugins","licensing"],"pid":1207,"message":"License information could not be obtained from Elasticsearch due to ConnectionError: getaddrinfo ENOTFOUND elasticsearch error"}
{"type":"log","@timestamp":"2021-07-19T13:58:18+00:00","tags":["warning","plugins","licensing"],"pid":1207,"message":"License information could not be obtained from Elasticsearch due to ConnectionError: getaddrinfo ENOTFOUND elasticsearch error"}
{"type":"log","@timestamp":"2021-07-19T13:58:19+00:00","tags":["warning","plugins","licensing"],"pid":1207,"message":"License information could not be obtained from Elasticsearch due to ConnectionError: getaddrinfo ENOTFOUND elasticsearch error"}
{"type":"response","@timestamp":"2021-07-19T13:58:19+00:00","tags":["api"],"pid":1207,"method":"get","statusCode":503,"req":{"url":"/api/status","method":"get","headers":{"host":"kibana:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"172.18.0.3","userAgent":"curl/7.61.1"},"res":{"statusCode":503,"responseTime":35,"contentLength":32087},"message":"GET /api/status 503 35ms - 31.3KB"}
{"type":"log","@timestamp":"2021-07-19T13:58:29+00:00","tags":["warning","plugins","licensing"],"pid":1207,"message":"License information could not be obtained from Elasticsearch due to ConnectionError: getaddrinfo ENOTFOUND elasticsearch error"}
{"type":"response","@timestamp":"2021-07-19T13:58:29+00:00","tags":["api"],"pid":1207,"method":"get","statusCode":503,"req":{"url":"/api/status","method":"get","headers":{"host":"kibana:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"172.18.0.3","userAgent":"curl/7.61.1"},"res":{"statusCode":503,"responseTime":35,"contentLength":32085},"message":"GET /api/status 503 35ms - 31.3KB"}
{"type":"log","@timestamp":"2021-07-19T13:58:39+00:00","tags":["warning","plugins","licensing"],"pid":1207,"message":"License information could not be obtained from Elasticsearch due to ConnectionError: getaddrinfo ENOTFOUND elasticsearch error"}
{"type":"response","@timestamp":"2021-07-19T13:58:39+00:00","tags":["api"],"pid":1207,"method":"get","statusCode":503,"req":{"url":"/api/status","method":"get","headers":{"host":"kibana:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"172.18.0.3","userAgent":"curl/7.61.1"},"res":{"statusCode":503,"responseTime":37,"contentLength":32089},"message":"GET /api/status 503 37ms - 31.3KB"}

@kuisathaverat
Copy link
Contributor

Is the container killed?

docker inspect 31ff405ee763

@kuisathaverat
Copy link
Contributor

I just ran the same command locally and it works, so I will say that you did not have enough memory to run all containers, Could you check that you allow Docker to have 10GB of RAM at least?

 ./scripts/compose.py start 7.14.0 --bc
found latest build candidate for 7.14 - https://staging.elastic.co/7.14.0-682a8012/summary-7.14.0.html at https://staging.elastic.co/latest/7.14.json
('downloading', u'https://staging.elastic.co/7.14.0-682a8012/downloads/kibana/kibana-7.14.0-docker-image.tar.gz')
('downloading', u'https://staging.elastic.co/7.14.0-682a8012/downloads/apm-server/apm-server-7.14.0-docker-image.tar.gz')
('downloading', u'https://staging.elastic.co/7.14.0-682a8012/downloads/elasticsearch/elasticsearch-7.14.0-docker-image.tar.gz')
853eca8ac50a: Loading layer [==================================================>]  137.8MB/137.8MB
1e4f6dd575a5: Loading layer [==================================================>]  52.74kB/52.74kB
8e88d6c274e4: Loading layer [==================================================>]  4.096kB/4.096kB
69e88ea82730: Loading layer [==================================================>]  4.096kB/4.096kB
730a7ab8694e: Loading layer [==================================================>]  138.8MB/138.8MB
a2c7a150b5cc: Loading layer [==================================================>]  2.048kB/2.048kB
7e68ff4d82ca: Loading layer [==================================================>]  15.87kB/15.87kB
e78c629d7125: Loading layer [==================================================>]  1.125MB/1.125MB
28e5e29aaf4f: Loading layer [==================================================>]  6.144kB/6.144kB
92373e1d1710: Loading layer [==================================================>]  374.8kB/374.8kB
Loaded image: docker.elastic.co/apm/apm-server:7.14.0
e4202438862c: Loading layer [==================================================>]    289MB/289MB
7073d05af65b: Loading layer [==================================================>]  26.62kB/26.62kB
857c814d2e55: Loading layer [==================================================>]  3.584kB/3.584kB
38928b76e6fd: Loading layer [==================================================>]  20.34MB/20.34MB
740defa3d0ba: Loading layer [==================================================>]  56.83kB/56.83kB
686a4a76ad59: Loading layer [=================>                                 ]  298.6MB/865.9MB
686a4a76ad59: Loading layer [========================>                          ]  427.3MB/865.9MB
686a4a76ad59: Loading layer [==========================>                        ]  300.3MB/564.1MB
686a4a76ad59: Loading layer [============================================>      ]    771MB/865.9MB
686a4a76ad59: Loading layer [=================================================> ]  856.8MB/865.9MB
686a4a76ad59: Loading layer [=================================================> ]  860.1MB/865.9MB
686a4a76ad59: Loading layer [=================================================> ]  863.4MB/865.9MB
686a4a76ad59: Loading layer [==================================================>]  865.9MB/865.9MB
fd33bbe2bea3: Loading layer [==================================================>]  2.048kB/2.048kB
9bae31291da7: Loading layer [==================================================>]  4.096kB/4.096kB
c637c333eba8: Loading layer [==================================================>]  19.46kB/19.46kB
253d8d0aba19: Loading layer [==================================================>]  4.096kB/4.096kB
e35ca91339fc: Loading layer [==================================================>]  479.2kB/479.2kB
2cb7f14ce5a8: Loading layer [==================================================>]  309.8kB/309.8kB
Loaded image: docker.elastic.co/kibana/kibana:7.14.0
Starting/Building stack services..

Pulling wait-service ... done
Docker Compose is now in the Docker CLI, try `docker compose up`

Creating network "apm-integration-testing" with the default driver
Creating volume "apm-integration-testing_esdata" with local driver
Creating volume "apm-integration-testing_pgdata" with local driver
Creating localtesting_7.14.0_elasticsearch ... done
Creating localtesting_7.14.0_kibana        ... done
Creating localtesting_7.14.0_apm-server    ... done
Creating wait                              ... done

@walterra
Copy link
Author

Oh great, docker just had the default 2GB, I bumped it up and now it worked too! Thank you so much, it's been almost 5 years since I touched docker 😅 . I'll now continue with trying to bring up all services and see if that works.

@walterra
Copy link
Author

I was now able to bring up all services including the load generators, thanks again!

@kuisathaverat
Copy link
Contributor

I happened me too several times

@kuisathaverat kuisathaverat reopened this Jul 19, 2021
@cachedout
Copy link
Contributor

We are currently tracking what we now believe to be the root cause of this issue in the Kibana repo. I'm going to close this so that we can follow it there:

elastic/kibana#110583

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cachedout @walterra @kuisathaverat