Failed to get cluster primary addresses - Classic protocol error: MySQL Error 2013 (HY000): Lost connection to MySQL server at \'waiting for initial communication packet\', system error: 110

[nobuto-m]

Steps to reproduce

1. juju deploy mysql-k8s --channel 8.0/edge

Expected behavior

All MySQL units get green.

Actual behavior

MySQL units can be stuck with the following status in the initial deployment.

$ juju status -m openstack cinder-mysql
Model      Controller            Cloud/Region             Version  SLA          Timestamp
openstack  mysunbeam-controller  mysunbeam-k8s/localhost  3.5.3    unsupported  02:05:50Z

SAAS       Status  Store  URL
microceph  active  local  admin/openstack-machines.microceph

App           Version                  Status       Scale  Charm      Channel   Rev  Address        Exposed  Message
cinder-mysql  8.0.37-0ubuntu0.22.04.3  maintenance      3  mysql-k8s  8.0/edge  174  10.152.183.91  no       joining the cluster

Unit             Workload     Agent  Address     Ports  Message
cinder-mysql/0*  active       idle   10.1.58.84         Primary
cinder-mysql/1   maintenance  idle   10.1.48.85         joining the cluster
cinder-mysql/2   waiting      idle   10.1.51.16         waiting to get cluster primary from peers

Versions

Operating system: jammy

Juju CLI: 3.5.3-genericlinux-amd64

Juju agent: 3.5.3

Charm revision: 8.0/edge 174

microk8s: MicroK8s v1.28.12 revision 7038

Log output

Juju debug log:
mysql_charm.log

unit-cinder-mysql-2: 02:05:57 WARNING unit.cinder-mysql/2.juju-log Failed to check if cluster metadata exists from_instance='cinder-mysql-2.cinder-mysql-endpoints.openstack.svc.cluster.local'
unit-cinder-mysql-2: 02:06:38 WARNING unit.cinder-mysql/2.juju-log Failed to check if cluster metadata exists from_instance='cinder-mysql-0.cinder-mysql-endpoints.openstack.svc.cluster.local'
unit-cinder-mysql-2: 02:07:19 WARNING unit.cinder-mysql/2.juju-log Failed to get cluster primary addresses
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cinder-mysql-2/charm/src/mysql_k8s_helpers.py", line 616, in _run_mysqlsh_script
    stdout, _ = process.wait_output()
  File "/var/lib/juju/agents/unit-cinder-mysql-2/charm/venv/ops/pebble.py", line 1771, in wait_output
    raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['/usr/bin/mysqlsh', '--no-wizard', '--python', '--verbose=1', '-f', '/tmp/script.py', ';', 'rm', '/tmp/script.py'], stdout='', stderr='Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nverbose: 2024-08-15T02:06:39Z: Loading startup files...\nverbose: 2024-08-15T02:06:39Z: Loading plugins...\nverbose: 2024-08-15T02:06:39Z: Connecting to MySQL at: clusteradmin@cinder-mysql-0.cinder-mysql-endpoints.openstack.svc.cluster.local\nTraceback (most recent call last):\n  File "<string>", line 1, in <module>\nValueError: Shell.connect_to_primary: X protocol error: MySQL server has gone away\nClassic protocol error: MySQL Error 2013 (HY000): Lost connection to MySQL server at \'waiting for initial communication packet\', system error: 110\n\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cinder-mysql-2/charm/lib/charms/mysql/v0/mysql.py", line 2117, in get_cluster_primary_address
    output = self._run_mysqlsh_script("\n".join(get_cluster_primary_commands))
  File "/var/lib/juju/agents/unit-cinder-mysql-2/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 724, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-cinder-mysql-2/charm/src/mysql_k8s_helpers.py", line 619, in _run_mysqlsh_script
    raise MySQLClientError
charms.mysql.v0.mysql.MySQLClientError

Additional context

[github-actions]

https://warthogs.atlassian.net/browse/DPE-5164