Curl v8 breaks wget on MacOS

[Haroon-Khel]

Currently in our playbooks we install the latest Curl onto a machine if it is detected that its current version is less than 7.58.0.

test-macstadium-macos1014-x64-1 and -3 are two such machines, they now have the latest curl

test-macstadium-macos1014-x64-1:~ administrator$ curl --version
curl 8.0.1 (x86_64-apple-darwin18.7.0) libcurl/8.0.1 (SecureTransport) OpenSSL/1.1.1t zlib/1.2.11 brotli/1.0.9 zstd/1.5.5 libidn2/2.3.4 libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3
Release-Date: 2023-03-20
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Initially this broke wget

test-macstadium-macos1014-x64-1:~ administrator$ wget 'https://api.adoptium.net/v3/binary/latest/8/ga/mac/x64/jdk/hotspot/normal/eclipse?project=jdk'
dyld: Library not loaded: /usr/local/opt/libunistring/lib/libunistring.2.dylib
  Referenced from: /usr/local/bin/wget
  Reason: image not found
Abort trap: 6

I reinstalled wget via brew, which installed it with Openssl3

...
perl ./Configure --prefix=/usr/local/Cellar/openssl@3/3.1.0 --openssldir=/usr/local/etc/openssl@3 --libdir=/usr/local/Cellar/openssl@3/3.1.0/lib no-ssl3 no-ssl3-method no-zlib darwin64-x86_64-cc enable-ec_ni
==> make

==> make install MANDIR=/usr/local/Cellar/openssl@3/3.1.0/share/man MANSUFFIX=ssl
==> make test

🍺  /usr/local/Cellar/openssl@3/3.1.0: 6,487 files, 29.9MB, built in 11 minutes 55 seconds
==> Installing wget
==> ./configure --prefix=/usr/local/Cellar/wget/1.21.3_1 --sysconfdir=/usr/local/etc --with-ssl=openssl --with-libssl-prefix=/usr/local/opt/openssl@3 --disable-pcre --disable-pcre2 --without-libpsl --without-inc
==> make install
...

and now wget works fine. However, the get_url ansible module now fails to download jdks

TASK [adoptopenjdk_install : Download latest JDK 18 release (macOS)] ***********
FAILED - RETRYING: Download latest JDK 18 release (macOS) (3 retries left).
FAILED - RETRYING: Download latest JDK 18 release (macOS) (2 retries left).
FAILED - RETRYING: Download latest JDK 18 release (macOS) (1 retries left).
fatal: [test-macstadium-macos1014-x64-1]: FAILED! => {"attempts": 3, "changed": false, "dest": "/tmp/18-installer.pkg", "elapsed": 0, "msg": "Request failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>", "url": "https://api.adoptium.net/v3/installer/latest/18/ga/mac/x64/jdk/hotspot/normal/eclipse?project=jdk"}

(failed this way before and after reinstalling wget, but not before curl was updated to 8)

I think it has to do with the version of openssl ansible is using for its get_url module

(default openssl before openssl3 through brew was installed and I symlinked it to /usr/local/bin/openssl)
test-macstadium-macos1014-x64-1:~ administrator$ /usr/bin/openssl version
LibreSSL 2.6.5

test-macstadium-macos1014-x64-1:~ administrator$ /usr/local/bin/openssl version
OpenSSL 3.1.0 14 Mar 2023 (Library: OpenSSL 3.1.0 14 Mar 2023)

test-macstadium-macos1014-x64-1:~ administrator$ /usr/bin/openssl s_client -connect api.adoptium.net:443
...

    Start Time: 1683306301
    Timeout   : 7200 (sec)
    Verify return code: 10 (certificate has expired)

est-macstadium-macos1014-x64-1:~ administrator$ /usr/local/bin/openssl s_client -connect api.adoptium.net:443
...
SSL handshake has read 4704 bytes and written 444 bytes
Verification: OK