statefulset: Use stateful sets for posgres deployments (PROJQUAY-6672) #927
Conversation
jonathankingfc
commented
May 30, 2024
jonathankingfc
commented
- Swap Postgres and Clair Postgres Deployments to StatefulSets
jonathankingfc
force-pushed
the
PROJQUAY-6672
branch
10 times, most recently
from
7e3cab8
to
fb9e6ba
Compare
jonathankingfc
force-pushed
the
PROJQUAY-6672
branch
from
fb9e6ba
to
84e5ac2
Compare
- Swap Postgres and Clair Postgres Deployments to StatefulSets
| @@ -1,5 +1,5 @@ | |||
| apiVersion: apps/v1 | |||
| kind: Deployment | |||
| kind: StatefulSet | |||
There was a problem hiding this comment.
how will this work with existing deployments? On upgrade will it replace the deployment with a statefulset?
There was a problem hiding this comment.
Yes, it will replace the deployment with a stateful set. This is so that we do not have issues with overlapping deployments that would cause issues on the upgrade job.
| replicas: 1 | ||
| strategy: | ||
| type: Recreate |
There was a problem hiding this comment.
This will switch the deployment to a rolling upgrade, which could cause the old and new pod to run at the same time. Will that cause issues?
There was a problem hiding this comment.
Since it is a stateful set, it will only run one pod at a time
There was a problem hiding this comment.
In theory, issues can occur where Quay is operating on a Clair that has a split-brain during a deployment (i.e. the security worker requests the indexing state from a new pod, sees it has changed then makes an indexing request but it goes to an old pod). In practice, this is how it works in production and we've never seen problems so I think it's probably fine but something to document.
| @@ -1,5 +1,5 @@ | |||
| apiVersion: apps/v1 | |||
| kind: Deployment | |||
| kind: StatefulSet | |||
There was a problem hiding this comment.
The original issue seems to be caused by the old and the new db pods being matched by the same service, making the requests against the db go to either or randomly. Does switching to stateful sets fix this issue?
There was a problem hiding this comment.
Yes, since their will be only one underlying pod attached to the service
|
/retest |
1 similar comment
|
/retest |
|
@jonathankingfc: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
