[Release-1.28] - Custom agent/etc/containerd/certs.d/* files are deleted by k3s on startup

[brandond]

Backport fix for Custom agent/etc/containerd/certs.d/* files are deleted by k3s on startup

• Custom agent/etc/containerd/certs.d/* files are deleted by k3s on startup #10787

[fmoral2]

Validated on Version:

-$ k3s version v1.28.13+k3s-d5b7bac7 (d5b7bac7)

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
ubuntu
AMD

Cluster Configuration:
-1 node server

Steps to validate the fix

1. Install k3s
2. Create your own host dir at /var/lib/rancher/k3s/agent/etc/containerd/certs.d/MYREGISTRY and add custom hosts.toml and/or certs.
3. Add an entry for MYREGISTRY in registries.yaml and restart k3s, note that the custom files are cleaned up and replaced with the K3s-managed hosts.toml
4. Edit the file at /var/lib/rancher/k3s/agent/etc/containerd/certs.d/MYREGISTRY/hosts.toml to remove the DO NOT EDIT header and restart k3s; note that it is reverted to the managed content.
5. Remove MYREGISTRY from registries.yaml and restart k3s. Note that /var/lib/rancher/k3s/agent/etc/containerd/certs.d/MYREGISTRY is removed.

Reproduction Issue:

k3s version v1.30.4+k3s1 (98262b5d)



:~$  sudo mkdir -p /var/lib/rancher/k3s/agent/etc/containerd/certs.d/fran.test

:~$  sudo tee /var/lib/rancher/k3s/agent/etc/containerd/certs.d/fran.test/hosts.toml <<EOF
server = "https://fran.test:443"
[host."https://fran.test:443"]
  capabilities = ["pull", "resolve"]
EOF

:~$   ls /var/lib/rancher/k3s/agent/etc/containerd/certs.d/fran.test
ls: cannot access '/var/lib/rancher/k3s/agent/etc/containerd/certs.d/fran.test': No such file or directory

 


:~$  sudo systemctl restart k3s


:~$ sudo tee /etc/rancher/k3s/registries.yaml <<EOF
mirrors:
  "fran.test:443":
    endpoint:
      - "https://fran.test:443"
EOF


:~$  sudo systemctl restart k3s

 



~$  sudo sed -i '/DO NOT EDIT/d' /var/lib/rancher/k3s/agent/etc/containerd/certs.d/fran.test/hosts.toml


~$ cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/fran.test/hosts.toml
cat: /var/lib/rancher/k3s/agent/etc/containerd/certs.d/fran.test/hosts.toml: No such file or directory



sudo sed -i '/fran.test.test/d' /etc/rancher/k3s/registries.yaml

sudo systemctl restart k3s

 $ ls /var/lib/rancher/k3s/agent/etc/containerd/certs.d/
ls: cannot access '/var/lib/rancher/k3s/agent/etc/containerd/certs.d/': No such file or directory

Validation Results:

  

Step 2: 
 
 sudo mkdir -p /var/lib/rancher/k3s/agent/etc/containerd/certs.d/localhost:5000

$  sudo tee /var/lib/rancher/k3s/agent/etc/containerd/certs.d/localhost:5000/hosts.toml <<EOF
server = "https://localhost:5000"
[host."https://localhost:5000"]
  capabilities = ["pull", "resolve"]
EOF

~$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/localhost:5000/hosts.toml
server = "https://localhost:5000"
[host."https://localhost:5000"]
  capabilities = ["pull", "resolve"]


$  sudo systemctl restart k3s

~$ sudo ls /var/lib/rancher/k3s/agent/etc/containerd/certs.d/localhost:5000
hosts.toml

 



Step 3:

~$  sudo tee /etc/rancher/k3s/registries.yaml <<EOF
mirrors:
  "localhost:5000":
    endpoint:
      - "https://localhost:5000"
EOF


$ sudo cat /etc/rancher/k3s/registries.yaml 
mirrors:
  "localhost:5000":
    endpoint:
      - "https://localhost:5000"


 

$  sudo systemctl restart k3s

 
 

~$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/localhost:5000/hosts.toml
# File generated by k3s. DO NOT EDIT.

server = "http://localhost:5000/v2"
capabilities = ["pull", "resolve", "push"]


[host]
[host."https://localhost:5000/v2"]
  capabilities = ["pull", "resolve"]





 Step 4: 
sudo sed -i '/DO NOT EDIT/d' /var/lib/rancher/k3s/agent/etc/containerd/certs.d/localhost:5000/hosts.toml

$  sudo systemctl restart k3s


$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/localhost:5000/hosts.toml
# File generated by k3s. DO NOT EDIT.

server = "http://localhost:5000/v2"
capabilities = ["pull", "resolve", "push"]


[host]
[host."https://localhost:5000/v2"]
  capabilities = ["pull", "resolve"]







~$  sudo truncate -s 0 /etc/rancher/k3s/registries.yaml 

$ cat /etc/rancher/k3s/registries.yaml
 <EMPTY>


~$ sudo systemctl restart k3s




$ sudo ls /var/lib/rancher/k3s/agent/etc/containerd/certs.d/  
<EMPTY>