Excessive error log warnings with healthcheck --connect #430
Comments
grooverdan
added a commit
to grooverdan/mariadb-docker
that referenced
this issue
May 9, 2022
The --connect test does a tcp connection. There is no mysql@::1 or mysql@127.0.0.1 user created so the access is denied, but the healtcheck is correct. The entries in the error log because of the default log_warnings=2. To avoid hitting max_connect_errors (default 100), a non-connection way is to look at the proc status for listening sockets. While normally not a direct correlation to a connection, mariadb listens and is followed by accepting connections immedately without anything that can fail inbetween. We move the previous connect to mariadb_connect for compatibility if users want to continue with that mechanism. We pre-emtively create tcp listen as a mechanism and redirect connect to that. Closes MariaDB#430
|
Started concept on 003ee86 Its not good enough though as a health-cmd run as root cannot process the visiblity of the pid 1 file descriptors Alternate And the bad side is it won't part by process id. Also assuming port 3306. Parsing out Still looking for alternatives. |
grooverdan
added a commit
to grooverdan/mariadb-docker
that referenced
this issue
May 10, 2022
The --connect test does a tcp connection. There is no
mysql@::1 or mysql@127.0.0.1 user created so the access
is denied, but the healtcheck is correct.
The entries in the error log because of the default
log_warnings=2.
To avoid hitting max_connect_errors (default 100), a non-connection
way is to look at the proc status for listening sockets.
While normally not a direct correlation to a connection, mariadb
listens and is followed by accepting connections immedately without
anything that can fail inbetween.
We move the previous connect to mariadb_connect for compatibility
if users want to continue with that mechanism. We pre-emtively create
tcp listen as a mechanism and redirect connect to that.
Also tested:
lsof -t -p 1 -a -iTCP -sTCP:LISTEN cannot resolve symlink /proc/1/fd/{num}
when run as root.
ss -nlHt state listening "( sport = :3306 )" could work, but no way
to tie to pid 1, and the 3306 port isn't fixed.
Closes MariaDB#430
|
Attempted new technique of looking at Seems to be working: procfs(5) for details on |
|
@grooverdan Wow, that was fast! My comment was just posted a few hours ago. Thank you and thanks to all of the contributors for being so much involved for the open source community! |
|
You're welcome. A new release is coming out soon so I'm looking to have this ready for it. I need to think a bit more about cases where it won't work. And about the stability of a 0A state as the Linux kernel exposes. Pros:
Cons:
@jerdoe your bug report comment was a contribution. Welcome to the community. |
|
Unfortunately I didn't submit my change to get the warnings reduced on the server in time for the coming release (MariaDB/server#2213). I hope to get them in the release afterwards and the container changes to accommodate them. |
|
Can anyone explain to regular humans how to solve the flooding problem? I'm struggling with the same problem now, using the healthcheck script! Thanks! |
grooverdan
added a commit
to grooverdan/mariadb-docker
that referenced
this issue
May 16, 2023
grooverdan
added a commit
to grooverdan/mariadb-docker
that referenced
this issue
May 17, 2023
grooverdan
added a commit
to grooverdan/mariadb-docker
that referenced
this issue
May 17, 2023
mysql@{127.0.0.1,::1} users aren't granted any privileges and have
accounts locked and passwords expired.
Closes MariaDB#430
grooverdan
added a commit
to grooverdan/mariadb-docker
that referenced
this issue
Jun 15, 2023
mysql@{127.0.0.1,::1} users aren't granted any privileges and have
accounts locked and passwords expired.
Even expired passwords need to comply with --plugin-load-add=simple_password_check
so a generated password is created and saved in
$DATADIR/.my-healthcheck.cnf.
The healthcheck --connect will increment the server status variable Aborted_connects
for each check, however now connection_error* counts are changed.
This also prevents any invalid password errors showing up in the
container log.
Closes MariaDB#430
grooverdan
added a commit
to grooverdan/mariadb-docker
that referenced
this issue
Jun 15, 2023
…tcheck --connect
healtcheck_connect@{127.0.0.1,::1} users aren't granted any privileges and have
accounts locked and passwords expired.
Even expired passwords need to comply with --plugin-load-add=simple_password_check
so a generated password is created and saved in
$DATADIR/.my-healthcheck.cnf.
The healthcheck --connect will increment the server status variable Aborted_connects
for each check, however now connection_error* counts are changed.
This also prevents any invalid password errors showing up in the
container log.
Closes MariaDB#430
grooverdan
added a commit
to grooverdan/mariadb-docker
that referenced
this issue
Jun 15, 2023
…tcheck --connect
healtcheck_connect@{127.0.0.1,::1} users aren't granted any privileges and have
accounts locked and passwords expired.
Even expired passwords need to comply with --plugin-load-add=simple_password_check
so a generated password is created and saved in
$DATADIR/.my-healthcheck.cnf.
The healthcheck --connect will increment the server status variable Aborted_connects
for each check, however now connection_error* counts are changed.
This also prevents any invalid password errors showing up in the
container log.
Closes MariaDB#430
grooverdan
added a commit
to grooverdan/mariadb-docker
that referenced
this issue
Jun 20, 2023
…althcheck --connect
healthcheck@{127.0.0.1,::1,localhost} users are granted USAGE by default, which
is enough for the non-replication healthchecks in healtcheck.sh.
The env variable MARIADB_HEALTHCHECK_GRANTS can replace USAGE with any
comma separated set of grants.
On initialization a generated password is created and saved in
$DATADIR/.my-healthcheck.cnf along with the server port and socket. If the
command args or default configuration file changes this may become out
of date. Because the password is generated in configuration file the
'#', comment, and '=' characters cannot be part of this password.
The healthcheck.cnf configuration file also sets protocol=tcp to
semi enforce --connect being a standard part of the test. This is
required as starts of the service under --skip-networking should
never be considered healthy.
The healthcheck script also has the --default-extra-file set to this
.my-healthcheck.cnf file if it exists so that all healthcheck scripts
use the authentication here by default.
The compatibility with old instances, without the .my-healtcheck.cnf is
preserved.
The healthcheck --connect will increment the server status variable Aborted_connects
for each check, however now connection_error* counts are changed.
This also prevents any invalid password errors showing up in the
container log.
Closes MariaDB#430
grooverdan
added a commit
to grooverdan/mariadb-docker
that referenced
this issue
Jun 20, 2023
…althcheck --connect
healthcheck@{127.0.0.1,::1,localhost} users are granted USAGE by default, which
is enough for the non-replication healthchecks in healtcheck.sh.
The env variable MARIADB_HEALTHCHECK_GRANTS can replace USAGE with any
comma separated set of grants.
On initialization a generated password is created and saved in
$DATADIR/.my-healthcheck.cnf along with the server port and socket. If the
command args or default configuration file changes this may become out
of date. Because the password is generated in configuration file the
'#', comment, and '=' characters cannot be part of this password.
The healthcheck.cnf configuration file also sets protocol=tcp to
semi enforce --connect being a standard part of the test. This is
required as starts of the service under --skip-networking should
never be considered healthy.
The healthcheck script also has the --default-extra-file set to this
.my-healthcheck.cnf file if it exists so that all healthcheck scripts
use the authentication here by default.
The compatibility with old instances, without the .my-healtcheck.cnf is
preserved.
The healthcheck --connect will increment the server status variable Aborted_connects
for each check, however now connection_error* counts are changed.
This also prevents any invalid password errors showing up in the
container log.
Closes MariaDB#430
grooverdan
added a commit
to grooverdan/mariadb-docker
that referenced
this issue
Jun 20, 2023
…althcheck --connect
healthcheck@{127.0.0.1,::1,localhost} users are granted USAGE by default, which
is enough for the non-replication healthchecks in healtcheck.sh.
The env variable MARIADB_HEALTHCHECK_GRANTS can replace USAGE with any
comma separated set of grants.
On initialization a generated password is created and saved in
$DATADIR/.my-healthcheck.cnf along with the server port and socket. If the
command args or default configuration file changes this may become out
of date. Because the password is generated in configuration file the
'#', comment, and '=' characters cannot be part of this password.
The healthcheck.cnf configuration file also sets protocol=tcp to
enforce indirectly that --connect being a standard part of the test. This is
required as starts of the service under --skip-networking should
never be considered healthy.
The healthcheck script also has the --defaults-extra-file set to this
.my-healthcheck.cnf file, if it exists (backwards compatible on
previously created datadirs), so that all new healthcheck invokations
use the authentication here by default.
The compatibility with old instances, without the .my-healthcheck.cnf is
preserved by non setting --defaults-extra-file.
The healthcheck --connect will increment the server status variable Aborted_connects
for each check, however now connection_error* counts are changed.
This also prevents any invalid password errors showing up in the
container log.
Closes MariaDB#430
|
so released now - see #512 (comment) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
From @jerdoe in #94 (comment)
--connectis using TCP somariadb@localhostisn't sufficient.log_warnings=1is one work around.Options:
log_warnings=1by default, don't want to play around too much with server defaults in containers.mysql@127.0.0.1/mysql@::1users. need to give empty password otherwise same error. Which conflicts with password validation plugins.The text was updated successfully, but these errors were encountered: