Container status not updated after termination cause by pod deletion

[howardjohn]

What happened?

Reproduction steps below, but at a high level:

1. I have a pod with 2 containers. I kill the pod. terminationGracePeriodSeconds is 30s
2. Container 1 terminates immediately. Container 2 terminates after 30s.
3. Immediately after the removal, pod is updated with deletionTimestamp
4. Container 1 terminates, but pod status does NOT change at all.
5. 30s later, container 2 terminates and pod dies.

What did you expect to happen?

After container 1 terminates, the containerStatus is updated to indicate the terminated container is no longer running.

How can we reproduce it (as minimally and precisely as possible)?

Apply this pod:

apiVersion: v1
kind: Pod
metadata:
  name: foo
spec:
  containers:
  - name: echo
    image: gcr.io/istio-testing/app:latest
  - name: sleep
    image: howardjohn/shell
    args:
    - /bin/sleep
    - infinity

echo container exits immediately, sleep does not until SIGKILL.

Next, run kubectl delete foo.

Observe that after a few seconds (up to 30s), the container status is not updated:

  containerStatuses:
  - containerID: containerd://399d45c45585232ef561c61267e40793146a6dd1bda93131abc311611dc4bae7
    image: gcr.io/istio-testing/app:latest
    imageID: gcr.io/istio-testing/app@sha256:1c0f400e31d492c2af4a2370943f3697a9a65823a179971ec720e510fba7bf78
    lastState: {}
    name: echo
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-12-09T00:44:44Z"
  - containerID: containerd://55b438c97c5aad6c10256ff90a42e2edd50295697dccef9d3bcae0a5dde96bb8
    image: docker.io/howardjohn/shell:latest
    imageID: docker.io/howardjohn/shell@sha256:9139585d3569e38dffbf035716b7ce1b1c6578e6276b10b82b09e1187f7917ff
    lastState: {}
    name: sleep
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-12-09T00:44:45Z"

Kubelet logs, with -v6:
kubelet.txt

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T21:30:26Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}

Also tested on GKE 1.21

Cloud provider

Kind/GKE

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[howardjohn]

/sig node

[matthyx]

@howardjohn so you are initiating a race condition between terminationGracePeriodSeconds and kubectl delete pod?

[howardjohn]

I am not quite sure what you mean by the race? I simply run kubectl delete pod and observe that container status does not match reality - the terminated container A is reflected in the status until container B terminates (which happens to be triggered by grace period exhausting; I imagine it would be the same if it was just delayed X seconds form initial SIGTERM).

[matthyx]

Ah, I see, sorry I thought you were killing the first container manually and then initiating a kubectl delete pod... I need coffee!

[jonyhy96]

/assign
Thanks! Will try to reproduce and fix it

[ehashman]

/triage accepted

[jonyhy96]

After dig into source code, find the reason why pod status is not update is:

1. kubelet killpod will wait until all containers stop
2. kubelet doesn't update pod status after killpod

need some advice before fix it

[jonyhy96]

One approach is makekillpod returns a channel, for each success killContainer we update the pod status.

[howardjohn]

One question I have is is this a bug or a performance optimization? I do have a (potential) use case for reading this status but not sure how much overhead there is for adding more writes on each pod kill

[jonyhy96]

One question I have is is this a bug or a performance optimization? I do have a (potential) use case for reading this status but not sure how much overhead there is for adding more writes on each pod kill

IMO, if exit time interval between containers are long then it is a bug. But maybe In most cases this may cause some overhead.

[jonyhy96]

Since we have another user's use case about using the status during containers shutdown #107183
May need maintainer of sig-node to decide wether we should update the pod status during containers shutdown

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[vaibhav2107]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[vaibhav2107]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[songjiaxun]

/reopen

[k8s-ci-robot]

@songjiaxun: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[songjiaxun]

/remove-lifecycle rotten

[SergeyKanzhelev]

/reopen

[k8s-ci-robot]

@SergeyKanzhelev: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.