Unable to update to 1.23 from 1.22

[AlexGustafsson]

I'm unable to update from 1.22 to 1.23.

First, I drained the node.

kubectl drain delta --ignore-daemonsets

Then, I updated packages on the node.

sudo apt-get update && sudo apt-get upgrade

Then, I tried to refresh the snap.

sudo snap refresh microk8s --channel=1.23/stable

I then received the following output. I've tried to run it again, but it's consistent. All the referenced IPs are reachable.

error: cannot perform the following tasks:
- Run configure hook of "microk8s" snap if present (run hook "configure": 
-----
++ ips+='\nIP.4 = 192.168.100.103'
++ sep='\n'
++ /snap/microk8s/2765/bin/sed -i 's/#MOREIPS/IP.3 = 10.0.1.22\nIP.4 = 192.168.100.103/g' /var/snap/microk8s/2765/certs/csr.conf.rendered
++ '[' -f /var/snap/microk8s/2765/certs/csr.conf ']'
++ local force
++ /snap/microk8s/2765/usr/bin/cmp -s /var/snap/microk8s/2765/certs/csr.conf.rendered /var/snap/microk8s/2765/certs/csr.conf
++ force=false
++ false
++ '[' '!' -f /var/snap/microk8s/2765/certs/front-proxy-client.crt ']'
+++ /snap/microk8s/2765/usr/bin/openssl x509 -noout -issuer
++ '[' 'issuer=CN = front-proxy-ca' == 'issuer=CN = 127.0.0.1' ']'
++ echo 0
+ '[' 0 == 1 ']'
+ '[' -e /var/snap/microk8s/2765/args/containerd-template.toml ']'
+ grep -e 'stream_server_address = ""' /var/snap/microk8s/2765/args/containerd-template.toml
+ grep -e '\-\-allow-privileged' /var/snap/microk8s/2765/args/kubelet
+ '[' -f /root/snap/microk8s/common/istio-auth.lock ']'
+ '[' -f /root/snap/microk8s/common/istio-auth.lock ']'
+ '[' -f /var/snap/microk8s/2765/credentials/kubelet.config ']'
+ '[' -f /var/snap/microk8s/2765/credentials/proxy.config ']'
+ '[' -f /var/snap/microk8s/2765/credentials/scheduler.config ']'
+ '[' -f /var/snap/microk8s/2765/credentials/controller.config ']'
+ '[' -e /var/snap/microk8s/2765/args/containerd-env ']'
+ grep -e KATA_PATH /var/snap/microk8s/2765/args/containerd-env
# KATA_PATH=
PATH=$PATH:$KATA_PATH
+ '[' -e /var/snap/microk8s/2765/args/containerd-template.toml ']'
+ grep -e io.containerd.kata.v2 /var/snap/microk8s/2765/args/containerd-template.toml
      runtime_type = "io.containerd.kata.v2"
+ for dir in ${SNAP_DATA}/credentials/ ${SNAP_DATA}/certs/ ${SNAP_DATA}/args/ ${SNAP_DATA}/var/lock
+ chmod -R ug+rwX /var/snap/microk8s/2765/credentials/
+ chmod -R o-rwX /var/snap/microk8s/2765/credentials/
+ for dir in ${SNAP_DATA}/credentials/ ${SNAP_DATA}/certs/ ${SNAP_DATA}/args/ ${SNAP_DATA}/var/lock
+ chmod -R ug+rwX /var/snap/microk8s/2765/certs/
+ chmod -R o-rwX /var/snap/microk8s/2765/certs/
+ for dir in ${SNAP_DATA}/credentials/ ${SNAP_DATA}/certs/ ${SNAP_DATA}/args/ ${SNAP_DATA}/var/lock
+ chmod -R ug+rwX /var/snap/microk8s/2765/args/
+ chmod -R o-rwX /var/snap/microk8s/2765/args/
+ for dir in ${SNAP_DATA}/credentials/ ${SNAP_DATA}/certs/ ${SNAP_DATA}/args/ ${SNAP_DATA}/var/lock
+ chmod -R ug+rwX /var/snap/microk8s/2765/var/lock
+ chmod -R o-rwX /var/snap/microk8s/2765/var/lock
+ getent group microk8s
+ getent group microk8s
+ chgrp microk8s -R /var/snap/microk8s/2765/credentials/ /var/snap/microk8s/2765/certs/ /var/snap/microk8s/2765/args/ /var/snap/microk8s/2765/var/lock/ /var/snap/microk8s/2765/var/kubernetes/backend/
+ try_copy_users_to_snap_microk8s
+ getent group microk8s
+ getent group snap_microk8s
+ echo 'One of the microk8s or snap_microk8s groups is missing'
One of the microk8s or snap_microk8s groups is missing
+ '[' -e /var/snap/microk8s/2765/opt/cni/bin/flanneld ']'
+ '[' -f /var/snap/microk8s/2765/args/flanneld ']'
+ grep -e etcd.socket:2379 /var/snap/microk8s/2765/args/etcd
+ grep -e basic-auth-file /var/snap/microk8s/2765/args/kube-apiserver
+ grep -e service-account-issuer /var/snap/microk8s/2765/args/kube-apiserver
--service-account-issuer='https://kubernetes.default.svc'
+ grep -e service-account-signing-key-file /var/snap/microk8s/2765/args/kube-apiserver
--service-account-signing-key-file=${SNAP_DATA}/certs/serviceaccount.key
+ grep -e RemoveSelfLink /var/snap/microk8s/2765/args/kube-apiserver
--feature-gates=RemoveSelfLink=false
+ grep '\-\-enable\-v2' /var/snap/microk8s/2765/args/etcd
--enable-v2=true
+ '[' -L /var/snap/microk8s/2765/bin/cilium ']'
+ '[' -e /var/snap/microk8s/2765/var/lock/clustered.lock ']'
+ '[' -e /var/snap/microk8s/2765/args/flannel-template.conflist ']'
+ grep -e cniVersion /var/snap/microk8s/2765/args/flannel-template.conflist
    "cniVersion": "0.3.1",
+ '[' '!' -f /var/snap/microk8s/2765/args/cluster-agent ']'
+ grep -e '\-\-timeout' /var/snap/microk8s/2765/args/cluster-agent
--timeout 240
+ mkdir -p /var/snap/microk8s/2765/juju/share/juju /var/snap/microk8s/2765/juju-home
+ chmod -R ug+rwX /var/snap/microk8s/2765/juju /var/snap/microk8s/2765/juju-home
+ chmod -R o-rwX /var/snap/microk8s/2765/juju /var/snap/microk8s/2765/juju-home
+ getent group microk8s
+ chgrp microk8s -R /var/snap/microk8s/2765/juju /var/snap/microk8s/2765/juju-home
+ grep -e '\-\-ip-masq' /var/snap/microk8s/2765/args/flanneld
--ip-masq=true
+ grep -e '\-\-cluster-cidr=10.152.183.0/24' /var/snap/microk8s/2765/args/kube-proxy
+ '[' -e /var/snap/microk8s/2765/var/lock/stopped.lock ']'
+ '[' -e /var/snap/microk8s/2765/var/lock/lite.lock ']'
+ '[' -e /var/snap/microk8s/2765/args/kubelite ']'
+ grep -e '\-\-insecure\-port' /var/snap/microk8s/2765/args/kube-apiserver
--insecure-port=0
+ skip_opt_in_config insecure-port kube-apiserver
+ local opt=--insecure-port
+ local config_file=/var/snap/microk8s/2765/args/kube-apiserver
+ run_with_sudo /snap/microk8s/2765/bin/sed -i /--insecure-port/d /var/snap/microk8s/2765/args/kube-apiserver
+ '[' -n '' ']'
+ '[' /snap/microk8s/2765/bin/sed == preserve_env ']'
+ sudo /snap/microk8s/2765/bin/sed -i /--insecure-port/d /var/snap/microk8s/2765/args/kube-apiserver
+ '[' -e /var/snap/microk8s/2765/var/lock/ha-cluster ']'
+ run_with_sudo preserve_env /snap/microk8s/2765/usr/bin/python3 /snap/microk8s/2765/scripts/cluster/distributed_op.py remove_argument kube-apiserver --insecure-port
+ '[' -n '' ']'
+ '[' preserve_env == preserve_env ']'
+ shift
+ sudo -E /snap/microk8s/2765/usr/bin/python3 /snap/microk8s/2765/scripts/cluster/distributed_op.py remove_argument kube-apiserver --insecure-port
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?
Removing argument --insecure-port from nodes.
Could not query for nodes
Command '['/snap/microk8s/2765/microk8s-kubectl.wrapper', 'get', 'no', '-o', 'json']' returned non-zero exit status 1.
-----)

Kubelite is indeed listening on port 16443:

sudo lsof -i -P -n | grep 16443

kubelite  1043821            root   19u  IPv6 80579851      0t0  TCP *:16443 (LISTEN)

The node is still available to the cluster.

kubectl get nodes

delta     Ready,SchedulingDisabled   <none>   156d   v1.22.4-3+adc4115d990346

inspection-report-20211208_170331.tar.gz

[ktsakalozos]

Hi @AlexGustafsson, thank you for reporting this. We will have a fix for this soon. A workaround is to delte the --insecure-port argument from the /var/snap/microk8s/current/args/kube-apiserver file before doing the refresh command.

[ktsakalozos]

The fix will be released in a few hours.

[AlexGustafsson]

It works now, thank you for the quick fix.