Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rule for docker image pull error #117

Merged
merged 1 commit into from
Jun 21, 2017

Conversation

ajitak
Copy link
Contributor

@ajitak ajitak commented Jun 14, 2017

Add a new rule which catches docker image corruption. It will address kubernetes/kubernetes#47219


This change is Reviewable

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 14, 2017
@ajitak
Copy link
Contributor Author

ajitak commented Jun 15, 2017

/assign @Random-Liu
/assign @dchen1107

"rules": []
"rules": [
{
"type": "permanent",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this problem is permanent, you may want to add a condition for it.

Copy link
Contributor Author

@ajitak ajitak Jun 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed it to temporary.

{
"type": "permanent",
"reason": "CorruptDockerImage",
"pattern": "container start failed: ErrImagePull: (.+): directory not empty"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a kubelet log. Are your should this is docker log?

Copy link
Contributor Author

@ajitak ajitak Jun 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like I had published outdated changes. Fixed it.

@dchen1107
Copy link
Member

ping ...

Thanks!

@ajitak
Copy link
Contributor Author

ajitak commented Jun 17, 2017

@dchen1107: waiting for @Random-Liu to review.

@Random-Liu
Copy link
Member

Random-Liu commented Jun 20, 2017

@ajitak Have you tried the rule for journald? Does it work?\

I guess it will not.

@ajitak
Copy link
Contributor Author

ajitak commented Jun 20, 2017

I have not been able inject messages in journald. I tested regex with docker log string to test the pattern.

@marun
Copy link

marun commented Jun 21, 2017

@Random-Liu @ajitak: What's required to move this forward, or can it be moved out of 1.7?

@ajitak
Copy link
Contributor Author

ajitak commented Jun 21, 2017

@Random-Liu Updated the regex and tested it.

When I run kubectl describe node node-name command, I see following under events:
3m 4s 2 {docker-monitor gke-demo-cluster-default-pool-fdda9125-7gr3} Warning CorruptDockerImage time="2017-06-21T20:25:42.481887487Z" level=error msg="Error trying v2 registry: failed to register layer: rename /var/lib/docker/image/aufs/layerdb/tmp/layer-632022140 /var/lib/docker/image/aufs/layerdb/sha256/011b303988d241a4ae28a6b82b0d8262751ef02910f0ae2265cb637504b72e36: directory not empty"

@Random-Liu Random-Liu added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 21, 2017
@Random-Liu Random-Liu merged commit b6205ea into kubernetes:master Jun 21, 2017
Random-Liu added a commit that referenced this pull request Jun 21, 2017
Add a new rule which catches docker image corruption (cherrypick #117 to v0.4)
@ajitak ajitak deleted the npd-config branch June 21, 2017 23:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants