good start

[stephenc01]

I was unable to use this as is. There are a lot of hard coded urls in the building of the container and seems for everything to run on one host

Maybe if I get time I can look into fixing the containers. but eod the container names should be vars and not hard coded.

[nook24]

Hi @stephenc01,

thanks for your feedback.

There are a lot of hard coded urls in the building of the containe

Could you please add a list of URLs that are hardcoded? Our idea was, to make all hostnames adjustable through the environment file. For example:

OITC_NAEMON_HOSTNAME=naemon
SE_GRAPHITE_ADDRESS=carbon-c-relay
CCR_CARBON_CACHE_1=carbon-cache1:2003

Could be possible that we have missed one or two.

I have build a distributed setup using Docker Swarm, you can find my notes about this here: https://gist.github.com/nook24/75b8a07d19989de6fcc122c78044ce82

[RedBeardDotDev]

@nook24 - I've experienced the same issue.

I've changed all the container/service names to make them easier to find on my small server where I run a couple of dozen other containers.

The services in my compose.yml are:

~/docker/openitcockpit$ docker compose config --services
oitc-carbon-c-relay
oitc-carbon-cache1
oitc-carbon-cache2
oitc-checkmk
oitc-gearmand
oitc-grafana
oitc-graphite-web
oitc-mod_gearman_worker
oitc-mysql
oitc-naemon
oitc-postfix
oitc-puppeteer
oitc-redis
oitc-statusengine-worker
oitc-victoria-metrics
openitcockpit

My openitcockpit.env file contains lines like:

MYSQL_HOST=oitc-mysql
OITC_REDIS_HOST=oitc-redis
OITC_NAEMON_HOSTNAME=oitc-naemon
SE_MYSQL_HOST=oitc-mysql

and

MYSQL_ROOT_PASSWORD=<a-super-secret-password>
MYSQL_DATABASE=openitcockpit
MYSQL_USER=openitcockpit
MYSQL_PASSWORD=<another-super-secret-password>
MYSQL_PORT=3306
MYSQL_HOST=oitc-mysql

I run linuxserver.io's swag, which effectively puts it behind an nginx reverse proxy.

When I run docker compose up -d oitc-mysql it starts the oitc-mysql service and initialises mysql. I can connect to it using the credentials above from another host on the same network. So it seems the oitc-mysql service is respecting the openitcockpit.env values.

When I try running docker compose up -d for the openitcockpit and oitc-statusengine-worker services it looks like they don't use the values from the openitcockpit.env file.

When I run docker compose logs openitcockpit I get a series of:

openitcockpit  | MySQL not available. Retrying...
openitcockpit  | MySQL not available. Retrying...
openitcockpit  | MySQL not available. Retrying...
openitcockpit  | MySQL not available. Retrying...
openitcockpit  | MySQL not available. Retrying...
openitcockpit  | MySQL not available. Retrying...
openitcockpit  | MySQL not available. Retrying...
openitcockpit  | MySQL not available. Retrying...

Until it gives up, crashes then restarts.

I noticed that my drive was filling up and it looks like this file is the culprit:

~/docker/openitcockpit$ sudo ls -lSh /var/lib/docker/overlay2/2ef8a1ac68be88a65503eb3a489b4df3c4a566def8f76de9940fb5bc329c5827/diff
total 67G
-rw-r--r-- 1 root root  67G Dec  6 21:32 stdout
...
...

The contents of which is:

PHP Warning:  GearmanWorker::work(): GEARMAN_UNKNOWN_STATE in /opt/openitc/statusengine3/worker/src/GearmanWorker.php on line 93
PHP Warning:  GearmanWorker::work(): GEARMAN_UNKNOWN_STATE in /opt/openitc/statusengine3/worker/src/GearmanWorker.php on line 93
PHP Warning:  GearmanWorker::work(): GEARMAN_UNKNOWN_STATE in /opt/openitc/statusengine3/worker/src/GearmanWorker.php on line 93
PHP Warning:  GearmanWorker::work(): GEARMAN_UNKNOWN_STATE in /opt/openitc/statusengine3/worker/src/GearmanWorker.php on line 93
PHP Warning:  GearmanWorker::work(): GEARMAN_UNKNOWN_STATE in /opt/openitc/statusengine3/worker/src/GearmanWorker.php on line 93

This fills up at a GB every few seconds until the drive filles up.

Checking the contents of /opt/openitc/etc/mysql/mysql.cnf, it looks like it's not reading the openitcockpit.env file:

root@openitcockpit:/opt/openitc/etc/mysql# cat mysql.cnf
; Automatically generated for openITCOCKPIT scripts. DO NOT TOUCH!
[client]
database = openitcockpit
host = mysql
user = openitcockpit
password = openitcockpit_password
port = 3306

Shouldn't those details be the ones from openitcockpit.env? And presumably the stdout for statusengine-worker shouldn't fill up that fast?

[nook24]

Happy new Year @RedBeardDotDev - thanks for reporting.

I found a few places where hardcoded hostnames where used instead of dynamic values from the openitcockpit.env file.

I nuked my docker test environment and pushed a new nightly version. I also prefixed all my containers with oitc- and I'm using this openitcockpit.env file

MYSQL_ROOT_PASSWORD=root_password
MYSQL_DATABASE=openitcockpit
MYSQL_USER=openitcockpit
MYSQL_PASSWORD=openitcockpit_password
MYSQL_HOST=oitc-mysql
OITC_REDIS_HOST=oitc-redis
OITC_GEARMAN_ADDRESS=oitc-gearmand
OITC_GRAPHITE_WEB_ADDRESS=oitc-graphite-web
OITC_GRAFANA_ADMIN_PASSWORD=bGsPMxURCjg4esgJ
OITC_GRAFANA_URL=http://oitc-grafana:3033
OITC_PUPPETEER_ADDRESS=http://oitc-puppeteer:8084/
OITC_NAEMON_HOSTNAME=oitc-naemon
OITC_STATUSENGINE_WORKER_HOSTNAME=oitc-statusengine-worker
OITC_MAIL_RELAYHOST=oitc-postfix
MOD_GEARMAN_GEARMAN_SERVER=oitc-gearmand:4730
MOD_GEARMAN_WORKER_GEARMAN_SERVER=oitc-gearmand:4730
SE_GEARMAN_ADDRESS=oitc-gearmand
SE_REDIS_ADDRESS=oitc-redis
SE_MYSQL_HOST=oitc-mysql
SE_GRAPHITE_ADDRESS=oitc-carbon-c-relay
SE_BROKER_GEARMAN_SERVER=oitc-gearmand:4730
VICTORIA_METRICS_HOST=oitc-victoriametrics
GF_SECURITY_ADMIN_PASSWORD=bGsPMxURCjg4esgJ
CCR_CARBON_CACHE_1=oitc-carbon-cache1:2003
CCR_CARBON_CACHE_2=oitc-carbon-cache2:2003

[Ubuntu][15:10]root@dziegler-docker~/openITCOCKPIT-ee-docker# docker compose config --services
oitc-carbon-c-relay
oitc-carbon-cache1
oitc-carbon-cache2
oitc-checkmk
oitc-gearmand
oitc-grafana
oitc-graphite-web
oitc-mod_gearman_worker
oitc-mysql
oitc-naemon
oitc-redis
oitc-openitcockpit
oitc-postfix
oitc-puppeteer
oitc-statusengine-worker
oitc-victoria-metrics

Would be good if you could also give it a shot

[RedBeardDotDev]

Thanks @nook24 .. I don't have a ton of time (busy moving house) but I've spun up a quick stack with the :latest images. It's looking pretty good so far.

The only message I'm getting is from oitc-naemon:

oitc-naemon               | 2024-01-02 23:23:59,663 INFO exited: naemon (exit status 1; not expected)
oitc-naemon               | 2024-01-02 23:24:00,665 INFO spawned: 'naemon' with pid 11860
oitc-naemon               | 2024-01-02 23:24:00,724 INFO exited: naemon (exit status 1; not expected)
oitc-naemon               | 2024-01-02 23:24:02,726 INFO spawned: 'naemon' with pid 11879
oitc-naemon               | 2024-01-02 23:24:03,786 INFO success: naemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

That might be something else, so I'm not too worried about it.

[nook24]

You need to use the :nightly tag for (at least) this 3 images:

openitcockpit/naemon:nightly
openitcockpit/victoria-metrics:latest
openitcockpit/openitcockpit-ce:nightly

Also please make sure you have set all environment variables from this comment #1 (comment) in your openitcockpit.env - as some of them are completely new.

oitc-naemon               | 2024-01-02 23:23:59,663 INFO exited: naemon (exit status 1; not expected)
oitc-naemon               | 2024-01-02 23:24:00,665 INFO spawned: 'naemon' with pid 11860
oitc-naemon               | 2024-01-02 23:24:00,724 INFO exited: naemon (exit status 1; not expected)
oitc-naemon               | 2024-01-02 23:24:02,726 INFO spawned: 'naemon' with pid 11879
oitc-naemon               | 2024-01-02 23:24:03,786 INFO success: naemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

This is the Naemon process which is crashing right away - not so good. That's probably because you are using the current stable version (latest tag) where the environment variable was missing.

As soon as you switch to nightly, this should be fixed.