Old CA certificates in AmazonLinux 2023

[TheRealBro]

During minor upgrade from 2.9 to 2.10 the container base OS was silently switched from AmazonLinux 2 to AmazonLinux 2023.

Not that silent though, as CA root cert bundle in AL2023 is utterly broken.

See amazonlinux/amazon-linux-2023#471

While I don't know why that switch of base OS happened in a minor change, it definitely might break things in deployments.

[marcin-bojar]

Yep, 2.10.0 has broken our CI/CD pipeline. Got ConnectionError: certificate has expired

Reverting to 2.9.0 fixes the issue

[bbarani]

@TheRealBro We are looking in to this issue and will track the upstream issue as well.

The decision to move to AL2023 was finalized for 2.10.0 release to support Node18 version based on the proposal here.

CC: @peterzhuamazon @prudhvigodithi

[TheRealBro]

Understanding the need for Node 18, those big changes should end up in a major release - not in a minor version upgrade. I guess that is what SemVer defines as well. justmy2cents

[bbarani]

@TheRealBro We had to migrate to Node18 since the End-of-Life Date of Node16 version was suddenly changed to September 11th, 2023. Having said that, OpenSearch dashboards still should support older version of node ( i.e. node 14 and node 16 by switching the node version locally) but we ship only the actively supported version (node 18) as part of the 2.10.0 distribution.

[prudhvigodithi]

Hey @TheRealBro @marcin-bojar, I'm curious can you please elaborate or share the error how the application is being impacted? I assume you are using docker and the container exits ?

[TheRealBro]

@xoxys do you still have the logs?

[TheRealBro]

New finding ... your documentation on DockerHub became outdated as well with that change.
https://hub.docker.com/r/opensearchproject/opensearch

Note: OpenSearch images use [Amazon Linux 2](https://aws.amazon.com/amazon-linux-2/) as the base image. If you are using [Docker Desktop](https://www.docker.com/products/docker-desktop/), then we recommend configuring Docker to use a minimum of 4 GB of your system memory.

[TheRealBro]

Hey @TheRealBro @marcin-bojar, I'm curious can you please elaborate or share the error how the application is being impacted? I assume you are using docker and the container exits ?

Sure. If you run OpenSearch with Certificates from an "official CA" and not self-signed PKI, than somebody might come to the idea of copying or mounting the system-provided CA file

/etc/pki/tls/certs/ca-bundle.crt

into the config directory and change the mandatory value for

plugins.security.ssl.transport.pemtrustedcas_filepath

to that file.

If you do so with AmazonLinux 2 based containers, OpenSearch starts.

If you do the same with AmazonLinux 2023 based containers, OpenSearch fails to start, as the certificate file seems to exceed some header sizes and it tries to load the "bloated" CA file with old certs from AL2023.

We saw this message in our logging after we switched to the new version of your container. Replacing the CA file with a correct one (not containing a lot of old certs) fixed the issue. But still, this is wrong having such old certs in a distro / container.

["javax.net.ssl.SSLProtocolException: The size of the handshake message (40539) exceeds the maximum allowed size (32768)"

[peterzhuamazon]

New finding ... your documentation on DockerHub became outdated as well with that change. https://hub.docker.com/r/opensearchproject/opensearch

Note: OpenSearch images use [Amazon Linux 2](https://aws.amazon.com/amazon-linux-2/) as the base image. If you are using [Docker Desktop](https://www.docker.com/products/docker-desktop/), then we recommend configuring Docker to use a minimum of 4 GB of your system memory.

Thanks @TheRealBro , we just updated the descriptions now.

[Divyaasm]

Hi @bbarani @peterzhuamazon , is there any action item from our end related to the issue.

[peterzhuamazon]

Pending: amazonlinux/amazon-linux-2023#471

[jordarlu]

I am closing this issue as no longer expired CA certs in AL2023, AL2 bundle cert.
Please refer the validation on amazonlinux/amazon-linux-2023#471 (comment) for AL2023, and amazonlinux/amazon-linux-2023#471 (comment) for AL2.
please feel free to reopen this issue if needed.
thanks!