Perform O/S patching where scheduled patching has not worked.

[steelhead31]

Investigate the machines highlighted by Nagios, that have O/S patches outstanding. These patches should be applied automatically via Cron, but on a number of machines, this has not happened.

[steelhead31]

test-skytap-ubuntu2004-ppc64le-1

*Nothing Running In Jenkins
*Docker Installed - No Docker Containers Running
*77 Updates Required ( 45 Critical )
*Docker Requires Upgrade As Part Of (apt upgrade)
*Docker Version :  20.10.21
*Crontab For Apt Upgrades Entry In Place ( 1am Daily )

[steelhead31]

test-scaleway-ubuntu1604-x64-1

*Nothing Running In Jenkins
*Docker Installed - No Docker Containers Running
*2 Updates Required ( 0 Critical )
*Docker Version 20.10.7
*Crontab For Apt Upgrades Entry In Place ( 1am Daily )

[steelhead31]

test-osuosl-ubuntu2004-ppc64le-1

*Nothing Running In Jenkins
*Docker Installed - No Docker Containers Running
*77 Updates Required ( 45 Critical )
*Docker Requires Upgrade As Part Of (apt upgrade)
*Docker Version :  20.10.21
*Crontab For Apt Upgrades Entry In Place ( 1am Daily )

[sxa]

@steelhead31 For the comments such as these:

*77 Updates Required ( 45 Critical )
*Crontab For Apt Upgrades Entry In Place ( 1am Daily )

Does that mean that you have manually put the crontab in place today, or that it was already in place? It's not quite clear from the wording. We should take a look at why that wasn't set up via the playbooks if you had to put it in place yourself. (Related: This is why #695 is increasingly important)

[steelhead31]

@steelhead31 For the comments such as these:

*77 Updates Required ( 45 Critical ) *Crontab For Apt Upgrades Entry In Place ( 1am Daily )

Does that mean that you have manually put the crontab in place today, or that it was already in place? It's not quite clear from the wording. We should take a look at why that wasn't set up via the playbooks if you had to put it in place yourself. (Related: This is why #695 is increasingly important)

The entry is definitely in place, I'll note the ones that don't have it if I find any... the 4 so far, have all had issues with a hung apt-upgrade process, ( which I've needed to kill, and then do a dpkg configure -a ) to correct.. the subsequent apt-get upgrades have then run ok.. so I suspect an issue occurred at some point during a previous scheduled run, and then Nagios has highlighted the issue occurring.

[steelhead31]

test-osuosl-ubuntu1804-ppc64le-2

*Nothing Running In Jenkins
*Docker Installed - No Docker Containers Running
*Docker version 18.06.3-ce, build d7080c1 ( Requires upgrade as part of apt-get )
*54 Updates Required ( 43 Critical ) 
*Post Upgrade Docker version v23.0.3, build 3e7cbfd
*Crontab For Apt Upgrades Entry In Place ( 1am Daily )

[steelhead31]

test-osuosl-ubuntu1804-ppc64le-1

*Nothing Running In Jenkins
*Docker Installed - No Docker Containers Running
*Docker version 18.06.3-ce, build d7080c1 ( Requires upgrade as part of apt-get )
*54 Updates Required ( 43 Critical ) 
*Post Upgrade Docker version v23.0.3, build 3e7cbfd
*Crontab For Apt Upgrades Entry In Place ( 1am Daily )

[steelhead31]

test-osuosl-ubuntu1604-ppc64le-2

*Nothing Running In Jenkins
*Docker Installed - No Docker Containers Running
*Docker version 18.06.3-ce, build d7080c1 ( Requires upgrade as part of apt-get )
*2 Updates Required ( 0 Critical ) 
*Post Upgrade Docker version v20.10.7, build f0df350
*Crontab For Apt Upgrades Entry In Place ( 1am Daily )

[steelhead31]

test-osuosl-ubuntu1604-ppc64le-1

*Nothing Running In Jenkins
*Docker Installed - No Docker Containers Running
*Docker version 18.06.3-ce, build d7080c1 ( Requires upgrade as part of apt-get )
*2 Updates Required ( 0 Critical ) 
*Post Upgrade Docker version v20.10.7, build f0df350
*Crontab For Apt Upgrades Entry In Place ( 1am Daily )

[steelhead31]

test-ibmcloud-ubuntu1604-x64-1

*Nothing Running In Jenkins
*Docker Installed - No Docker Containers Running
*Docker version 18.09.7, build 2d0083d ( does not require upgrade )
*2 Updates Required ( 0 Critical ) 
*Crontab For Apt Upgrades Entry In Place ( 1am Daily )

[steelhead31]

docker-skytap-ubuntu2004-ppc64le-1 / dockerhost-skytap-ubuntu2204-ppc64le-1 / (20.61.136.212)

*Nothing Running In Jenkins
*Docker Installed - Docker version 20.10.21, build 20.10.21-0ubuntu1~20.04.1
*Docker Containers Running 
d098d6ec1203   aqa_deb11   "/usr/sbin/sshd -D"   6 weeks ago    Up 6 weeks   21/tcp, 0.0.0.0:2216->22/tcp, :::2216->22/tcp   deb11.2216
086cb9fa222e   aqa_deb11   "/usr/sbin/sshd -D"   3 months ago   Up 7 weeks   21/tcp, 0.0.0.0:2215->22/tcp, :::2215->22/tcp   deb11.2215
4cd9f59fb76d   aqa_deb11   "/usr/sbin/sshd -D"   3 months ago   Up 7 weeks   21/tcp, 0.0.0.0:2211->22/tcp, :::2211->22/tcp   deb11_2211
61ce9f5ad9e8   aqa_deb11   "/usr/sbin/sshd -D"   3 months ago   Up 7 weeks   21/tcp, 0.0.0.0:2212->22/tcp, :::2212->22/tcp   deb11_2212
fd3b078ef89b   aqa_u2204   "/usr/sbin/sshd -D"   3 months ago   Up 7 weeks   0.0.0.0:2213->22/tcp, :::2213->22/tcp           u2204.2213
2bbf9f1edad1   aqa_u2204   "/usr/sbin/sshd -D"   3 months ago   Up 7 weeks   0.0.0.0:2214->22/tcp, :::2214->22/tcp           u2204.2214
*62 Updates Required ( 33 Critical )

Stop ( docker ps --format '{{.ID}}' )
Start
 docker start d098d6ec1203
 docker start 086cb9fa222e
 docker start 4cd9f59fb76d
 docker start 61ce9f5ad9e8
 docker start fd3b078ef89b
 docker start 2bbf9f1edad1

*Crontab For Apt Upgrades Entry In Place ( 1am Daily )

[steelhead31]

dockerhost-osuosl-ubuntu2004-ppc64le-1

*Nothing Running In Jenkins
*Docker Installed: Docker version 20.10.21, build 20.10.21-0ubuntu120.04.1
*Docker Running:
root@test-osuosl-ubuntu2004-ppc64le-1:# docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS          PORTS                                   NAMES
1fec8d7cc871   b287507f1c49   "/bin/sh -c 'dnf ins…"   21 minutes ago   Up 21 minutes                                           elegant_moser
31408d1a6757   aqa_f33        "/usr/sbin/sshd -D"      2 months ago     Up 4 weeks      0.0.0.0:2227->22/tcp, :::2227->22/tcp    F33_2227
246f6654e0d3   aqa_u2010      "/usr/sbin/sshd -D"      2 months ago     Up 4 weeks      0.0.0.0:2225->22/tcp, :::2225->22/tcp   U2010_2225
21e24ce1a89b   aqa_u1804      "/usr/sbin/sshd -D"      2 months ago     Up 4 weeks      0.0.0.0:2223->22/tcp, :::2223->22/tcp   U1804_2223

*62 Updates Required ( 36 Critical ) 
*Crontab For Apt Upgrades Entry In Place ( 1am Daily )
*Docker Installed - Docker version 20.10.21, build 20.10.21-0ubuntu1~20.04.1

[steelhead31]

dockerhost-marist-ubuntu2204-s390x-1

 *Nothing Running In Jenkins
 *Docker Installed: Docker version 20.10.21, build 20.10.21-0ubuntu122.04.2
 *Docker Running:
oot@dockerhost1:# docker ps
CONTAINER ID   IMAGE         COMMAND               CREATED        STATUS        PORTS                                   NAMES
cd3deaedf8f1   aqa_sles12    "/usr/sbin/sshd -D"   7 months ago   Up 6 months   0.0.0.0:2225->22/tcp, :::2225->22/tcp   SLES12_2225
4bf9b46e25bc   sles15_test   "/usr/sbin/sshd -D"   8 months ago   Up 6 months   0.0.0.0:2224->22/tcp, :::2224->22/tcp   sles15_container
a2ca5cf6e8b0   aqa_sles15    "/bin/bash"           8 months ago   Up 6 months   0.0.0.0:2211->22/tcp, :::2211->22/tcp   sles15.2211

*52 Updates Required ( 27 Critical ) 
*Crontab For Apt Upgrades Entry In Place ( 1am Daily )
*Docker Installed - Docker version 20.10.21, build 20.10.21-0ubuntu1~22.04.2

[steelhead31]

dockerhost-equinix-ubuntu2004-armv8-1

*Nothing Running In Jenkins
*Docker Installed: Docker version 20.10.22, build 3a2c30b
*Docker Running:
ae83604739c6   aqa_f35        "/usr/sbin/sshd -D"   8 weeks ago     Up 3 weeks   0.0.0.0:2234->22/tcp, :::2234->22/tcp   F35_2234
463b24ca83fb   aqa_alp313     "/usr/sbin/sshd -D"   2 months ago    Up 3 weeks   0.0.0.0:2223->22/tcp, :::2223->22/tcp   alp313.2223
c26ce4e29415   aqa_u2004      "/usr/sbin/sshd -D"   2 months ago    Up 3 weeks   0.0.0.0:2232->22/tcp, :::2232->22/tcp   U2004_2232
93d2b4e13a22   aqa_u2010      "/usr/sbin/sshd -D"   2 months ago    Up 3 weeks   0.0.0.0:2231->22/tcp, :::2231->22/tcp   U2010_2231
d27b3b8525aa   aqa_alp314     "/usr/sbin/sshd -D"   2 months ago    Up 3 weeks   0.0.0.0:2224->22/tcp, :::2224->22/tcp   alp314.2224
07149945267e   aqa_u2004      "/usr/sbin/sshd -D"   2 months ago    Up 3 weeks   0.0.0.0:2226->22/tcp, :::2226->22/tcp   U2004_2226
6e5b2bb3eab7   aqa_u2204      "/usr/sbin/sshd -D"   9 months ago    Up 3 weeks   0.0.0.0:2222->22/tcp, :::2222->22/tcp   U2204_2222
e7660b5abce5   u2004arm32     "/usr/sbin/sshd -D"   17 months ago   Up 3 weeks   0.0.0.0:2233->22/tcp, :::2233->22/tcp   U2004arm32.2233
1bfd2a038c8d   aqa_u2004.32   "/usr/sbin/sshd -D"   23 months ago   Up 3 weeks   0.0.0.0:2324->22/tcp, :::2324->22/tcp   U2004_2324.32
db94894974b0   aqa_u2004.32   "/usr/sbin/sshd -D"   23 months ago   Up 3 weeks   0.0.0.0:2323->22/tcp, :::2323->22/tcp   U2004_2323.32
412c7c344fe5   aqa_u2004.32   "/usr/sbin/sshd -D"   23 months ago   Up 3 weeks   0.0.0.0:2322->22/tcp, :::2322->22/tcp   U2004_2322.32
fd3994e02f15   aqa_arm32      "/usr/sbin/sshd -D"   24 months ago   Up 3 weeks   0.0.0.0:3224->22/tcp, :::3224->22/tcp   u2004.arm32.3224
a96fd12a60b1   aqa_u1804      "/usr/sbin/sshd -D"   2 years ago     Up 3 weeks   0.0.0.0:2230->22/tcp, :::2230->22/tcp   U1804_2230

*55 Updates Required ( 0 Critical ) 
*Crontab For Apt Upgrades Entry In Place ( 1am Daily )

[steelhead31]

All O/S upgrades completed, notes above show the state and works completed.