devmapper driver flake

[bparees]

failed to register layer: devicemapper: Error running deviceResume dm_task_run failed
F0624 14:23:42.011832       1 helpers.go:107] error: build error: failed to register layer: devicemapper: Error running deviceResume dm_task_run failed
error: The build extended-test-build-no-outputname-moynj-m22ea/test-docker-1 status is "Failed"

as seen in:
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/2563/consoleFull

Not sure what component label to use for this, @danmcp.

[jwhonce]

@rhvgoyal Do you have any insight here?

[rhvgoyal]

We have a known race in suspend/resume path and this can trigger when deferred device removal is enabled.

Shishir has queued up a patch for that upstream. This error message has similar signature and I suspect that patch might fix it.

moby/moby#23497

[rhvgoyal]

cc @shishir-a412ed

[shishir-a412ed]

@jwhonce I agree this looks like the race condition issue.
moby/moby#23418

A fix is already proposed upstream and internally (projectatomic/docker)

Upstream: moby/moby#23497
Projectatomic/docker: projectatomic/docker#171

[bparees]

recurred here:
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/2798/consoleFull

[jwhonce]

The current plan is to get a build by @runcom for testing projectatomic/docker#171 (comment)

[runcom]

@jwhonce is that for RHEL? if it is @lsm5 might give you that because I don't have any RHEL build machine (I never had, fedora only)

[bparees]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/3199/consoleFull

[smarterclayton]

Do we have a plan to get a fix tested?

[jwhonce]

@bparees Which OS and Docker version are the jenkins tests using?

[bparees]

@jwhonce

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)

$ uname -a
Linux ip-172-18-10-184.ec2.internal 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.3.x86_64
 Go version:      go1.6.2
 Git commit:      969e2ce-unsupported
 Built:           Tue Jul  5 17:10:57 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.3.x86_64
 Go version:      go1.6.2
 Git commit:      969e2ce-unsupported
 Built:           Tue Jul  5 17:10:57 2016
 OS/Arch:         linux/amd64

[jwhonce]

@shishir-a412ed @rhvgoyal Looking at the git logs @shishir-a412ed 's patch should be included in the version of docker being run here. Any other ideas?

[bparees]

@jwhonce @shishir-a412ed @rhvgoyal is this another form of the same bug?

F0730 11:25:29.985953       1 helpers.go:108] error: build error: open /dev/mapper/docker-202:2-50512749-36414ad9688ecd4e76f41c686c014523ab0f06bdc075ed2e3381c5b58ed858ce: no such file or directory

as seen here:
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/4391/consoleFull