Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extended.[k8s.io] Service endpoints latency should not be very high [Conformance] #15024

Closed
stevekuznetsov opened this issue Jul 3, 2017 · 8 comments
Closed

Extended.[k8s.io] Service endpoints latency should not be very high [Conformance] #15024

stevekuznetsov opened this issue Jul 3, 2017 · 8 comments
Assignees
@derekwaynecarr @ingvagabund
Labels
component/kubernetes kind/test-flake Categorizes issue or PR as related to test flakes. priority/P2

Comments

@stevekuznetsov
Copy link
Contributor

Seeing failures like this one:

Stacktrace

/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/service_latency.go:117
Not all RC/pod/service trials succeeded: got 1 errors
50, 90, 99 percentiles: 169.718538ms 262.214396ms 460.529611ms
/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/service_latency.go:115
Standard Output 
[BeforeEach] [Top Level]
  /go/src/github.com/openshift/origin/test/extended/util/test.go:52
[BeforeEach] [k8s.io] Service endpoints latency
  /go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:120
STEP: Creating a kubernetes client
Jul  3 12:19:08.253: INFO: >>> kubeConfig: /etc/origin/master/admin.kubeconfig

STEP: Building a namespace api object
Jul  3 12:19:08.311: INFO: About to run a Kube e2e test, ensuring namespace is privileged
STEP: Waiting for a default service account to be provisioned in namespace
[It] should not be very high [Conformance]
  /go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/service_latency.go:117
STEP: creating replication controller svc-latency-rc in namespace e2e-tests-svc-latency-x0dp3
Jul  3 12:19:11.475: INFO: Created: latency-svc-jfb18
Jul  3 12:19:11.486: INFO: Got endpoints: latency-svc-jfb18 [17.089967ms]
Jul  3 12:19:11.500: INFO: Created: latency-svc-8kkls
Jul  3 12:19:11.513: INFO: Created: latency-svc-7jqhh
Jul  3 12:19:11.524: INFO: Got endpoints: latency-svc-8kkls [37.588434ms]
Jul  3 12:19:11.533: INFO: Created: latency-svc-pdlgf
Jul  3 12:19:11.533: INFO: Created: latency-svc-jkk2j
Jul  3 12:19:11.535: INFO: Got endpoints: latency-svc-7jqhh [48.09913ms]
Jul  3 12:19:11.546: INFO: Created: latency-svc-r3pvd
Jul  3 12:19:11.554: INFO: Created: latency-svc-bgzw4
Jul  3 12:19:11.554: INFO: Got endpoints: latency-svc-jkk2j [67.021442ms]
Jul  3 12:19:11.559: INFO: Got endpoints: latency-svc-pdlgf [72.279025ms]
Jul  3 12:19:11.573: INFO: Created: latency-svc-39c9n
Jul  3 12:19:11.581: INFO: Got endpoints: latency-svc-r3pvd [93.682904ms]
Jul  3 12:19:11.582: INFO: Created: latency-svc-ph8v4
Jul  3 12:19:11.586: INFO: Got endpoints: latency-svc-bgzw4 [98.31208ms]
Jul  3 12:19:11.590: INFO: Created: latency-svc-gfvls
Jul  3 12:19:11.600: INFO: Created: latency-svc-w24t6
Jul  3 12:19:11.606: INFO: Got endpoints: latency-svc-39c9n [117.812789ms]
Jul  3 12:19:11.609: INFO: Created: latency-svc-wv6vq
Jul  3 12:19:11.615: INFO: Got endpoints: latency-svc-ph8v4 [126.940569ms]
Jul  3 12:19:11.621: INFO: Created: latency-svc-2v629
Jul  3 12:19:11.625: INFO: Got endpoints: latency-svc-gfvls [137.478434ms]
Jul  3 12:19:11.628: INFO: Created: latency-svc-rgh0t
Jul  3 12:19:11.635: INFO: Got endpoints: latency-svc-wv6vq [148.980817ms]
Jul  3 12:19:11.636: INFO: Created: latency-svc-ljjkz
Jul  3 12:19:11.637: INFO: Got endpoints: latency-svc-w24t6 [148.799974ms]
Jul  3 12:19:11.638: INFO: Got endpoints: latency-svc-2v629 [152.000902ms]
Jul  3 12:19:11.647: INFO: Created: latency-svc-5cbcw
Jul  3 12:19:11.655: INFO: Got endpoints: latency-svc-rgh0t [168.797504ms]
Jul  3 12:19:11.656: INFO: Created: latency-svc-jlfsr
Jul  3 12:19:11.662: INFO: Got endpoints: latency-svc-ljjkz [176.439487ms]
Jul  3 12:19:11.668: INFO: Created: latency-svc-nvtwm
Jul  3 12:19:11.672: INFO: Got endpoints: latency-svc-5cbcw [183.349741ms]
Jul  3 12:19:11.680: INFO: Created: latency-svc-j060k
Jul  3 12:19:11.686: INFO: Created: latency-svc-q6h4v
Jul  3 12:19:11.688: INFO: Got endpoints: latency-svc-nvtwm [152.985997ms]
Jul  3 12:19:11.689: INFO: Got endpoints: latency-svc-jlfsr [164.219881ms]
Jul  3 12:19:11.694: INFO: Created: latency-svc-mrcg2
Jul  3 12:19:11.700: INFO: Got endpoints: latency-svc-j060k [145.93327ms]
Jul  3 12:19:11.714: INFO: Got endpoints: latency-svc-q6h4v [154.609489ms]
Jul  3 12:19:11.714: INFO: Created: latency-svc-32s04
Jul  3 12:19:11.714: INFO: Created: latency-svc-zx67n
Jul  3 12:19:11.720: INFO: Created: latency-svc-d2g6w
Jul  3 12:19:11.723: INFO: Got endpoints: latency-svc-mrcg2 [141.831582ms]
Jul  3 12:19:11.730: INFO: Created: latency-svc-nzqvt
Jul  3 12:19:11.735: INFO: Got endpoints: latency-svc-zx67n [149.092286ms]
Jul  3 12:19:11.737: INFO: Created: latency-svc-zlx1h
Jul  3 12:19:11.749: INFO: Created: latency-svc-9xlp5
Jul  3 12:19:11.755: INFO: Got endpoints: latency-svc-d2g6w [148.88548ms]
Jul  3 12:19:11.760: INFO: Got endpoints: latency-svc-32s04 [144.736992ms]
Jul  3 12:19:11.763: INFO: Created: latency-svc-pj378
Jul  3 12:19:11.767: INFO: Created: latency-svc-js7lx
Jul  3 12:19:11.769: INFO: Got endpoints: latency-svc-nzqvt [144.280438ms]
Jul  3 12:19:11.782: INFO: Got endpoints: latency-svc-9xlp5 [145.205607ms]
Jul  3 12:19:11.787: INFO: Got endpoints: latency-svc-zlx1h [151.40401ms]
Jul  3 12:19:11.787: INFO: Created: latency-svc-gq5rk
Jul  3 12:19:11.789: INFO: Got endpoints: latency-svc-pj378 [151.118524ms]
Jul  3 12:19:11.793: INFO: Got endpoints: latency-svc-js7lx [137.975238ms]
Jul  3 12:19:11.794: INFO: Created: latency-svc-k4t6r
Jul  3 12:19:11.800: INFO: Created: latency-svc-5n7k4
Jul  3 12:19:11.805: INFO: Created: latency-svc-k6v7h
Jul  3 12:19:11.813: INFO: Got endpoints: latency-svc-gq5rk [58.086951ms]
Jul  3 12:19:11.815: INFO: Created: latency-svc-909fb
Jul  3 12:19:11.820: INFO: Got endpoints: latency-svc-k4t6r [148.21801ms]
Jul  3 12:19:11.825: INFO: Created: latency-svc-h2lt4
Jul  3 12:19:11.830: INFO: Got endpoints: latency-svc-k6v7h [141.925298ms]
Jul  3 12:19:11.835: INFO: Got endpoints: latency-svc-5n7k4 [146.921956ms]
Jul  3 12:19:11.836: INFO: Created: latency-svc-w6891
Jul  3 12:19:11.839: INFO: Created: latency-svc-1ks7c
Jul  3 12:19:11.846: INFO: Created: latency-svc-x9x4f
Jul  3 12:19:11.854: INFO: Created: latency-svc-8x1vk
Jul  3 12:19:11.859: INFO: Got endpoints: latency-svc-909fb [158.879282ms]
Jul  3 12:19:11.861: INFO: Got endpoints: latency-svc-h2lt4 [146.979968ms]
Jul  3 12:19:11.868: INFO: Got endpoints: latency-svc-w6891 [144.712963ms]
Jul  3 12:19:11.871: INFO: Created: latency-svc-hcftg
Jul  3 12:19:11.873: INFO: Got endpoints: latency-svc-1ks7c [137.430421ms]
Jul  3 12:19:11.877: INFO: Created: latency-svc-7l4pt
Jul  3 12:19:11.884: INFO: Got endpoints: latency-svc-x9x4f [124.145635ms]
Jul  3 12:19:11.888: INFO: Got endpoints: latency-svc-8x1vk [225.326492ms]
Jul  3 12:19:11.890: INFO: Created: latency-svc-z0vrj
Jul  3 12:19:11.901: INFO: Created: latency-svc-nk61v
Jul  3 12:19:11.901: INFO: Created: latency-svc-9jv5h
Jul  3 12:19:11.909: INFO: Got endpoints: latency-svc-hcftg [139.413865ms]
Jul  3 12:19:11.913: INFO: Created: latency-svc-hgzz3
Jul  3 12:19:11.922: INFO: Created: latency-svc-mk3zr
Jul  3 12:19:11.930: INFO: Got endpoints: latency-svc-nk61v [141.246635ms]
Jul  3 12:19:11.947: INFO: Created: latency-svc-brbkf
Jul  3 12:19:11.950: INFO: Got endpoints: latency-svc-z0vrj [163.552494ms]
Jul  3 12:19:11.951: INFO: Got endpoints: latency-svc-7l4pt [168.426779ms]
Jul  3 12:19:11.951: INFO: Got endpoints: latency-svc-9jv5h [157.561767ms]
Jul  3 12:19:11.960: INFO: Created: latency-svc-h2f7b
Jul  3 12:19:11.965: INFO: Got endpoints: latency-svc-hgzz3 [151.501611ms]
Jul  3 12:19:11.968: INFO: Created: latency-svc-9z06l
Jul  3 12:19:11.971: INFO: Got endpoints: latency-svc-mk3zr [150.961324ms]
Jul  3 12:19:11.977: INFO: Created: latency-svc-r93vc
Jul  3 12:19:11.979: INFO: Got endpoints: latency-svc-brbkf [144.269317ms]
Jul  3 12:19:11.989: INFO: Created: latency-svc-j7s84
Jul  3 12:19:11.990: INFO: Got endpoints: latency-svc-h2f7b [159.186515ms]
Jul  3 12:19:11.995: INFO: Got endpoints: latency-svc-9z06l [135.536068ms]
Jul  3 12:19:12.001: INFO: Created: latency-svc-tqswn
Jul  3 12:19:12.007: INFO: Got endpoints: latency-svc-r93vc [146.323354ms]
Jul  3 12:19:12.011: INFO: Created: latency-svc-5vzr0
Jul  3 12:19:12.018: INFO: Created: latency-svc-cslr7
Jul  3 12:19:12.021: INFO: Got endpoints: latency-svc-j7s84 [153.352451ms]
Jul  3 12:19:12.038: INFO: Created: latency-svc-m5r8x
Jul  3 12:19:12.057: INFO: Created: latency-svc-b11bc
Jul  3 12:19:12.061: INFO: Got endpoints: latency-svc-5vzr0 [177.01163ms]
Jul  3 12:19:12.063: INFO: Created: latency-svc-fv965
Jul  3 12:19:12.066: INFO: Got endpoints: latency-svc-cslr7 [178.559871ms]
Jul  3 12:19:12.069: INFO: Got endpoints: latency-svc-tqswn [196.378242ms]
Jul  3 12:19:12.072: INFO: Created: latency-svc-nwmjs
Jul  3 12:19:12.079: INFO: Got endpoints: latency-svc-m5r8x [170.053042ms]
Jul  3 12:19:12.080: INFO: Created: latency-svc-n980b
Jul  3 12:19:12.092: INFO: Created: latency-svc-k85kz
Jul  3 12:19:12.095: INFO: Got endpoints: latency-svc-b11bc [164.165091ms]
Jul  3 12:19:12.103: INFO: Created: latency-svc-9zxs5
Jul  3 12:19:12.104: INFO: Got endpoints: latency-svc-fv965 [153.501487ms]
Jul  3 12:19:12.108: INFO: Got endpoints: latency-svc-nwmjs [157.142096ms]
Jul  3 12:19:12.111: INFO: Got endpoints: latency-svc-n980b [160.573662ms]
Jul  3 12:19:12.118: INFO: Created: latency-svc-hf013
Jul  3 12:19:12.126: INFO: Created: latency-svc-3g40m
Jul  3 12:19:12.128: INFO: Got endpoints: latency-svc-k85kz [163.171142ms]
Jul  3 12:19:12.137: INFO: Got endpoints: latency-svc-9zxs5 [165.686374ms]
Jul  3 12:19:12.139: INFO: Created: latency-svc-8jhwq
Jul  3 12:19:12.150: INFO: Created: latency-svc-19wc0
Jul  3 12:19:12.160: INFO: Created: latency-svc-bq2dh
Jul  3 12:19:12.165: INFO: Created: latency-svc-khk9n
Jul  3 12:19:12.169: INFO: Got endpoints: latency-svc-3g40m [179.087581ms]
Jul  3 12:19:12.170: INFO: Got endpoints: latency-svc-hf013 [190.697044ms]
Jul  3 12:19:12.179: INFO: Created: latency-svc-06g8q
Jul  3 12:19:12.189: INFO: Created: latency-svc-ffbtf
Jul  3 12:19:12.192: INFO: Got endpoints: latency-svc-19wc0 [184.522794ms]
Jul  3 12:19:12.202: INFO: Created: latency-svc-7s5rl
Jul  3 12:19:12.206: INFO: Got endpoints: latency-svc-bq2dh [184.997512ms]
Jul  3 12:19:12.208: INFO: Got endpoints: latency-svc-8jhwq [213.158303ms]
Jul  3 12:19:12.219: INFO: Created: latency-svc-n94x6
Jul  3 12:19:12.222: INFO: Got endpoints: latency-svc-06g8q [156.052333ms]
Jul  3 12:19:12.223: INFO: Got endpoints: latency-svc-khk9n [161.303794ms]
Jul  3 12:19:12.226: INFO: Created: latency-svc-9gdkx
Jul  3 12:19:12.227: INFO: Got endpoints: latency-svc-ffbtf [157.886881ms]
Jul  3 12:19:12.238: INFO: Created: latency-svc-6xmn9
Jul  3 12:19:12.241: INFO: Got endpoints: latency-svc-n94x6 [146.526243ms]
Jul  3 12:19:12.244: INFO: Got endpoints: latency-svc-7s5rl [165.110648ms]
Jul  3 12:19:12.253: INFO: Created: latency-svc-rjf1m
Jul  3 12:19:12.263: INFO: Created: latency-svc-z2337
Jul  3 12:19:12.263: INFO: Got endpoints: latency-svc-9gdkx [159.542396ms]
Jul  3 12:19:12.272: INFO: Got endpoints: latency-svc-6xmn9 [163.836222ms]
Jul  3 12:19:12.274: INFO: Created: latency-svc-kzm18
Jul  3 12:19:12.279: INFO: Got endpoints: latency-svc-rjf1m [168.069035ms]
Jul  3 12:19:12.283: INFO: Created: latency-svc-8b4s2
Jul  3 12:19:12.291: INFO: Created: latency-svc-13ffw
Jul  3 12:19:12.291: INFO: Got endpoints: latency-svc-z2337 [163.27626ms]
Jul  3 12:19:12.304: INFO: Created: latency-svc-cbl8n
Jul  3 12:19:12.310: INFO: Got endpoints: latency-svc-kzm18 [173.037082ms]
Jul  3 12:19:12.315: INFO: Created: latency-svc-4fkl3
Jul  3 12:19:12.317: INFO: Got endpoints: latency-svc-8b4s2 [148.185689ms]
Jul  3 12:19:12.319: INFO: Got endpoints: latency-svc-13ffw [148.918087ms]
Jul  3 12:19:12.326: INFO: Created: latency-svc-p7673
Jul  3 12:19:12.332: INFO: Got endpoints: latency-svc-cbl8n [139.972523ms]
Jul  3 12:19:12.340: INFO: Created: latency-svc-rklhr
Jul  3 12:19:12.346: INFO: Got endpoints: latency-svc-4fkl3 [139.384218ms]
Jul  3 12:19:12.350: INFO: Created: latency-svc-tkbms
Jul  3 12:19:12.354: INFO: Got endpoints: latency-svc-p7673 [145.613502ms]
Jul  3 12:19:12.359: INFO: Created: latency-svc-xc6l6
Jul  3 12:19:12.372: INFO: Created: latency-svc-sdkxk
Jul  3 12:19:12.375: INFO: Got endpoints: latency-svc-tkbms [43.700432ms]
Jul  3 12:19:12.383: INFO: Created: latency-svc-4c66c
Jul  3 12:19:12.383: INFO: Got endpoints: latency-svc-rklhr [161.094196ms]
Jul  3 12:19:12.401: INFO: Got endpoints: latency-svc-xc6l6 [174.149782ms]
Jul  3 12:19:12.403: INFO: Got endpoints: latency-svc-sdkxk [162.177999ms]
Jul  3 12:19:12.413: INFO: Created: latency-svc-pjzkm
Jul  3 12:19:12.414: INFO: Got endpoints: latency-svc-4c66c [169.718538ms]
Jul  3 12:19:12.441: INFO: Created: latency-svc-rmt5m
Jul  3 12:19:12.468: INFO: Created: latency-svc-h1cf4
Jul  3 12:19:12.472: INFO: Got endpoints: latency-svc-pjzkm [208.922566ms]
Jul  3 12:19:12.507: INFO: Created: latency-svc-lpc5q
Jul  3 12:19:12.530: INFO: Got endpoints: latency-svc-rmt5m [258.404277ms]
Jul  3 12:19:12.556: INFO: Created: latency-svc-0nv06
Jul  3 12:19:12.564: INFO: Got endpoints: latency-svc-h1cf4 [284.918504ms]
Jul  3 12:19:12.577: INFO: Created: latency-svc-27wz3
Jul  3 12:19:12.602: INFO: Created: latency-svc-0wmk0
Jul  3 12:19:12.615: INFO: Got endpoints: latency-svc-lpc5q [323.902167ms]
Jul  3 12:19:12.648: INFO: Got endpoints: latency-svc-0nv06 [337.717968ms]
Jul  3 12:19:12.669: INFO: Created: latency-svc-h2820
Jul  3 12:19:12.683: INFO: Created: latency-svc-qhbzw
Jul  3 12:19:12.699: INFO: Created: latency-svc-8bhjr
Jul  3 12:19:12.741: INFO: Got endpoints: latency-svc-0wmk0 [422.051726ms]
Jul  3 12:19:12.759: INFO: Created: latency-svc-x12w8
Jul  3 12:19:12.777: INFO: Got endpoints: latency-svc-qhbzw [554.73846ms]
Jul  3 12:19:12.778: INFO: Got endpoints: latency-svc-27wz3 [460.529611ms]
Jul  3 12:19:12.780: INFO: Got endpoints: latency-svc-h2820 [434.667314ms]
Jul  3 12:19:12.781: INFO: Got endpoints: latency-svc-8bhjr [426.962279ms]
Jul  3 12:19:12.790: INFO: Created: latency-svc-b98fn
Jul  3 12:19:12.794: INFO: Created: latency-svc-glg52
Jul  3 12:19:12.797: INFO: Created: latency-svc-wktzp
Jul  3 12:19:12.802: INFO: Got endpoints: latency-svc-x12w8 [426.210595ms]
Jul  3 12:19:12.822: INFO: Created: latency-svc-zw505
Jul  3 12:19:12.835: INFO: Got endpoints: latency-svc-b98fn [433.946653ms]
Jul  3 12:19:12.838: INFO: Got endpoints: latency-svc-wktzp [434.558083ms]
Jul  3 12:19:12.840: INFO: Got endpoints: latency-svc-glg52 [456.613925ms]
Jul  3 12:19:12.851: INFO: Created: latency-svc-666lv
Jul  3 12:19:12.858: INFO: Got endpoints: latency-svc-zw505 [443.652016ms]
Jul  3 12:19:12.865: INFO: Created: latency-svc-t5g97
Jul  3 12:19:12.874: INFO: Created: latency-svc-xbc1x
Jul  3 12:19:12.881: INFO: Created: latency-svc-cftl6
Jul  3 12:19:12.889: INFO: Got endpoints: latency-svc-666lv [416.902454ms]
Jul  3 12:19:12.893: INFO: Created: latency-svc-4pxmf
Jul  3 12:19:12.901: INFO: Got endpoints: latency-svc-t5g97 [370.913886ms]
Jul  3 12:19:12.909: INFO: Got endpoints: latency-svc-xbc1x [345.058195ms]
Jul  3 12:19:12.910: INFO: Created: latency-svc-tmw5v
Jul  3 12:19:12.910: INFO: Created: latency-svc-gkn4c
Jul  3 12:19:12.917: INFO: Got endpoints: latency-svc-cftl6 [301.90533ms]
Jul  3 12:19:12.922: INFO: Got endpoints: latency-svc-4pxmf [274.475664ms]
Jul  3 12:19:12.930: INFO: Created: latency-svc-8cms4
Jul  3 12:19:12.937: INFO: Created: latency-svc-0gxmx
Jul  3 12:19:12.946: INFO: Got endpoints: latency-svc-tmw5v [204.968543ms]
Jul  3 12:19:12.947: INFO: Got endpoints: latency-svc-gkn4c [169.257322ms]
Jul  3 12:19:12.957: INFO: Created: latency-svc-wjzvp
Jul  3 12:19:12.961: INFO: Got endpoints: latency-svc-8cms4 [183.472352ms]
Jul  3 12:19:12.973: INFO: Created: latency-svc-d0p1q
Jul  3 12:19:12.980: INFO: Created: latency-svc-nvd4h
Jul  3 12:19:12.986: INFO: Created: latency-svc-kbtrl
Jul  3 12:19:12.986: INFO: Got endpoints: latency-svc-0gxmx [205.93804ms]
Jul  3 12:19:12.986: INFO: Got endpoints: latency-svc-wjzvp [205.71604ms]
Jul  3 12:19:13.004: INFO: Created: latency-svc-cvwgf
Jul  3 12:19:13.011: INFO: Created: latency-svc-rhl57
Jul  3 12:19:13.011: INFO: Got endpoints: latency-svc-nvd4h [175.908082ms]
Jul  3 12:19:13.012: INFO: Got endpoints: latency-svc-kbtrl [173.506098ms]
Jul  3 12:19:13.014: INFO: Got endpoints: latency-svc-d0p1q [212.400679ms]
Jul  3 12:19:13.021: INFO: Created: latency-svc-1p12w
Jul  3 12:19:13.030: INFO: Created: latency-svc-wrn4n
Jul  3 12:19:13.039: INFO: Created: latency-svc-4b628
Jul  3 12:19:13.043: INFO: Got endpoints: latency-svc-rhl57 [185.617708ms]
Jul  3 12:19:13.044: INFO: Got endpoints: latency-svc-cvwgf [203.57756ms]
Jul  3 12:19:13.044: INFO: Got endpoints: latency-svc-1p12w [142.63839ms]
Jul  3 12:19:13.054: INFO: Created: latency-svc-jq0vq
Jul  3 12:19:13.071: INFO: Created: latency-svc-0cxpm
Jul  3 12:19:13.081: INFO: Got endpoints: latency-svc-4b628 [171.753392ms]
Jul  3 12:19:13.084: INFO: Created: latency-svc-r2ltz
Jul  3 12:19:13.085: INFO: Got endpoints: latency-svc-wrn4n [195.926034ms]
Jul  3 12:19:13.091: INFO: Created: latency-svc-wzppn
Jul  3 12:19:13.097: INFO: Got endpoints: latency-svc-0cxpm [174.944002ms]
Jul  3 12:19:13.103: INFO: Got endpoints: latency-svc-jq0vq [186.0635ms]
Jul  3 12:19:13.104: INFO: Created: latency-svc-c23f6
Jul  3 12:19:13.110: INFO: Got endpoints: latency-svc-r2ltz [164.077973ms]
Jul  3 12:19:13.115: INFO: Created: latency-svc-3z0rk
Jul  3 12:19:13.121: INFO: Got endpoints: latency-svc-wzppn [174.62925ms]
Jul  3 12:19:13.133: INFO: Created: latency-svc-0r5kq
Jul  3 12:19:13.134: INFO: Got endpoints: latency-svc-c23f6 [173.177171ms]
Jul  3 12:19:13.137: INFO: Created: latency-svc-vjcq1
Jul  3 12:19:13.140: INFO: Got endpoints: latency-svc-3z0rk [153.983482ms]
Jul  3 12:19:13.153: INFO: Created: latency-svc-6h1nw
Jul  3 12:19:13.163: INFO: Created: latency-svc-0k4pt
Jul  3 12:19:13.170: INFO: Created: latency-svc-sfbhn
Jul  3 12:19:13.172: INFO: Got endpoints: latency-svc-vjcq1 [160.920347ms]
Jul  3 12:19:13.179: INFO: Got endpoints: latency-svc-0r5kq [192.945217ms]
Jul  3 12:19:13.184: INFO: Created: latency-svc-5tfcl
Jul  3 12:19:13.186: INFO: Got endpoints: latency-svc-6h1nw [174.568382ms]
Jul  3 12:19:13.195: INFO: Created: latency-svc-48mrm
Jul  3 12:19:13.202: INFO: Got endpoints: latency-svc-0k4pt [187.707178ms]
Jul  3 12:19:13.209: INFO: Created: latency-svc-x1hn6
Jul  3 12:19:13.213: INFO: Got endpoints: latency-svc-sfbhn [169.906085ms]
Jul  3 12:19:13.216: INFO: Created: latency-svc-wpdp9
Jul  3 12:19:13.222: INFO: Got endpoints: latency-svc-5tfcl [178.593082ms]
Jul  3 12:19:13.230: INFO: Created: latency-svc-9k9vd
Jul  3 12:19:13.236: INFO: Got endpoints: latency-svc-48mrm [192.249131ms]
Jul  3 12:19:13.251: INFO: Created: latency-svc-kwbr7
Jul  3 12:19:13.251: INFO: Created: latency-svc-mq4qh
Jul  3 12:19:13.257: INFO: Got endpoints: latency-svc-9k9vd [159.926706ms]
Jul  3 12:19:13.264: INFO: Created: latency-svc-39t7h
Jul  3 12:19:13.264: INFO: Got endpoints: latency-svc-wpdp9 [179.09122ms]
Jul  3 12:19:13.270: INFO: Created: latency-svc-prk3m
Jul  3 12:19:13.270: INFO: Got endpoints: latency-svc-x1hn6 [189.133469ms]
Jul  3 12:19:13.277: INFO: Got endpoints: latency-svc-mq4qh [173.24646ms]
Jul  3 12:19:13.283: INFO: Created: latency-svc-57rt0
Jul  3 12:19:13.286: INFO: Got endpoints: latency-svc-kwbr7 [175.74976ms]
Jul  3 12:19:13.294: INFO: Got endpoints: latency-svc-39t7h [172.916712ms]
Jul  3 12:19:13.297: INFO: Created: latency-svc-mgjlf
Jul  3 12:19:13.300: INFO: Got endpoints: latency-svc-prk3m [165.603325ms]
Jul  3 12:19:13.306: INFO: Got endpoints: latency-svc-57rt0 [165.79596ms]
Jul  3 12:19:13.310: INFO: Created: latency-svc-r4p6n
Jul  3 12:19:13.319: INFO: Got endpoints: latency-svc-mgjlf [147.020188ms]
Jul  3 12:19:13.323: INFO: Created: latency-svc-zm03n
Jul  3 12:19:13.332: INFO: Created: latency-svc-8r72v
Jul  3 12:19:13.337: INFO: Got endpoints: latency-svc-r4p6n [157.81303ms]
Jul  3 12:19:13.343: INFO: Created: latency-svc-0hjqw
Jul  3 12:19:13.351: INFO: Created: latency-svc-9k8pm
Jul  3 12:19:13.360: INFO: Got endpoints: latency-svc-8r72v [158.334222ms]
Jul  3 12:19:13.361: INFO: Got endpoints: latency-svc-zm03n [174.463723ms]
Jul  3 12:19:13.368: INFO: Created: latency-svc-fhq88
Jul  3 12:19:13.376: INFO: Got endpoints: latency-svc-0hjqw [162.512277ms]
Jul  3 12:19:13.378: INFO: Created: latency-svc-3rp5v
Jul  3 12:19:13.384: INFO: Got endpoints: latency-svc-9k8pm [161.782222ms]
Jul  3 12:19:13.387: INFO: Created: latency-svc-p6xfv
Jul  3 12:19:13.393: INFO: Created: latency-svc-x17r8
Jul  3 12:19:13.400: INFO: Got endpoints: latency-svc-fhq88 [163.943628ms]
Jul  3 12:19:13.404: INFO: Got endpoints: latency-svc-3rp5v [147.228535ms]
Jul  3 12:19:13.407: INFO: Created: latency-svc-hvpjq
Jul  3 12:19:13.417: INFO: Got endpoints: latency-svc-p6xfv [152.883822ms]
Jul  3 12:19:13.420: INFO: Created: latency-svc-cv704
Jul  3 12:19:13.428: INFO: Got endpoints: latency-svc-x17r8 [158.103505ms]
Jul  3 12:19:13.430: INFO: Created: latency-svc-blxdz
Jul  3 12:19:13.438: INFO: Created: latency-svc-7zwhv
Jul  3 12:19:13.441: INFO: Got endpoints: latency-svc-hvpjq [164.748465ms]
Jul  3 12:19:13.443: INFO: Created: latency-svc-gwpcv
Jul  3 12:19:13.456: INFO: Got endpoints: latency-svc-cv704 [169.833565ms]
Jul  3 12:19:13.465: INFO: Created: latency-svc-11cz6
Jul  3 12:19:13.481: INFO: Created: latency-svc-f46qv
Jul  3 12:19:13.483: INFO: Got endpoints: latency-svc-blxdz [188.74402ms]
Jul  3 12:19:13.492: INFO: Got endpoints: latency-svc-7zwhv [192.265329ms]
Jul  3 12:19:13.495: INFO: Created: latency-svc-50xxj
Jul  3 12:19:13.497: INFO: Got endpoints: latency-svc-gwpcv [191.186867ms]
Jul  3 12:19:13.500: INFO: Created: latency-svc-13th0
Jul  3 12:19:13.502: INFO: Got endpoints: latency-svc-11cz6 [182.919377ms]
Jul  3 12:19:13.507: INFO: Created: latency-svc-9tjt9
Jul  3 12:19:13.513: INFO: Got endpoints: latency-svc-f46qv [175.615772ms]
Jul  3 12:19:13.517: INFO: Created: latency-svc-hrmvq
Jul  3 12:19:13.517: INFO: Got endpoints: latency-svc-13th0 [156.54849ms]
Jul  3 12:19:13.520: INFO: Got endpoints: latency-svc-50xxj [159.645ms]
Jul  3 12:19:13.524: INFO: Created: latency-svc-d2rrc
Jul  3 12:19:13.538: INFO: Created: latency-svc-rmqzx
Jul  3 12:19:13.543: INFO: Got endpoints: latency-svc-hrmvq [159.123449ms]
Jul  3 12:19:13.548: INFO: Created: latency-svc-21673
Jul  3 12:19:13.548: INFO: Got endpoints: latency-svc-9tjt9 [172.531433ms]
Jul  3 12:19:13.556: INFO: Created: latency-svc-n6m3s
Jul  3 12:19:13.563: INFO: Got endpoints: latency-svc-d2rrc [162.969251ms]
Jul  3 12:19:13.575: INFO: Created: latency-svc-s1ljc
Jul  3 12:19:13.580: INFO: Got endpoints: latency-svc-rmqzx [175.746839ms]
Jul  3 12:19:13.588: INFO: Got endpoints: latency-svc-21673 [170.993332ms]
Jul  3 12:19:13.589: INFO: Created: latency-svc-3k2xc
Jul  3 12:19:13.595: INFO: Got endpoints: latency-svc-n6m3s [166.953157ms]
Jul  3 12:19:13.602: INFO: Got endpoints: latency-svc-s1ljc [160.778044ms]
Jul  3 12:19:13.608: INFO: Created: latency-svc-1l713
Jul  3 12:19:13.608: INFO: Created: latency-svc-tvh4q
Jul  3 12:19:13.619: INFO: Created: latency-svc-thf55
Jul  3 12:19:13.623: INFO: Created: latency-svc-hfp17
Jul  3 12:19:13.631: INFO: Created: latency-svc-k3ts8
Jul  3 12:19:13.638: INFO: Created: latency-svc-cz6js
Jul  3 12:19:13.646: INFO: Created: latency-svc-s4v2c
Jul  3 12:19:13.653: INFO: Got endpoints: latency-svc-3k2xc [197.145289ms]
Jul  3 12:19:13.657: INFO: Created: latency-svc-6jjgm
Jul  3 12:19:13.675: INFO: Created: latency-svc-cc1k3
Jul  3 12:19:13.677: INFO: Got endpoints: latency-svc-hfp17 [175.002584ms]
Jul  3 12:19:13.678: INFO: Got endpoints: latency-svc-1l713 [195.004011ms]
Jul  3 12:19:13.681: INFO: Got endpoints: latency-svc-thf55 [183.271641ms]
Jul  3 12:19:13.691: INFO: Got endpoints: latency-svc-tvh4q [198.933544ms]
Jul  3 12:19:13.713: INFO: Got endpoints: latency-svc-k3ts8 [200.214015ms]
Jul  3 12:19:13.717: INFO: Created: latency-svc-k6zmp
Jul  3 12:19:13.731: INFO: Created: latency-svc-pqjnr
Jul  3 12:19:13.736: INFO: Created: latency-svc-099n9
Jul  3 12:19:13.750: INFO: Created: latency-svc-6zxzp
Jul  3 12:19:13.750: INFO: Got endpoints: latency-svc-6jjgm [207.197205ms]
Jul  3 12:19:13.751: INFO: Got endpoints: latency-svc-s4v2c [230.699473ms]
Jul  3 12:19:13.751: INFO: Got endpoints: latency-svc-cc1k3 [97.650634ms]
Jul  3 12:19:13.760: INFO: Got endpoints: latency-svc-k6zmp [196.809856ms]
Jul  3 12:19:13.760: INFO: Got endpoints: latency-svc-cz6js [243.017757ms]
Jul  3 12:19:13.771: INFO: Created: latency-svc-v9ptc
Jul  3 12:19:13.774: INFO: Created: latency-svc-dmzmh
Jul  3 12:19:13.784: INFO: Got endpoints: latency-svc-099n9 [195.969442ms]
Jul  3 12:19:13.785: INFO: Created: latency-svc-hh3v8
Jul  3 12:19:13.789: INFO: Got endpoints: latency-svc-pqjnr [208.667617ms]
Jul  3 12:19:13.794: INFO: Got endpoints: latency-svc-6zxzp [198.960882ms]
Jul  3 12:19:13.804: INFO: Created: latency-svc-khmp0
Jul  3 12:19:13.816: INFO: Created: latency-svc-1jl4r
Jul  3 12:19:13.816: INFO: Got endpoints: latency-svc-dmzmh [268.056919ms]
Jul  3 12:19:13.818: INFO: Got endpoints: latency-svc-v9ptc [215.965034ms]
Jul  3 12:19:13.827: INFO: Created: latency-svc-1r2tk
Jul  3 12:19:13.833: INFO: Got endpoints: latency-svc-khmp0 [155.983328ms]
Jul  3 12:19:13.836: INFO: Got endpoints: latency-svc-hh3v8 [157.509684ms]
Jul  3 12:19:13.852: INFO: Got endpoints: latency-svc-1jl4r [171.543028ms]
Jul  3 12:19:13.858: INFO: Created: latency-svc-4sp6g
Jul  3 12:19:13.859: INFO: Created: latency-svc-fsl1b
Jul  3 12:19:13.865: INFO: Got endpoints: latency-svc-1r2tk [173.481446ms]
Jul  3 12:19:13.870: INFO: Created: latency-svc-008qg
Jul  3 12:19:13.872: INFO: Created: latency-svc-clfrg
Jul  3 12:19:13.883: INFO: Created: latency-svc-srb3t
Jul  3 12:19:13.900: INFO: Got endpoints: latency-svc-4sp6g [148.741692ms]
Jul  3 12:19:13.922: INFO: Created: latency-svc-gvgtl
Jul  3 12:19:13.928: INFO: Got endpoints: latency-svc-fsl1b [94.928144ms]
Jul  3 12:19:13.933: INFO: Got endpoints: latency-svc-008qg [182.15375ms]
Jul  3 12:19:13.936: INFO: Created: latency-svc-jfjh8
Jul  3 12:19:13.941: INFO: Got endpoints: latency-svc-clfrg [181.023201ms]
Jul  3 12:19:13.948: INFO: Created: latency-svc-vj6kk
Jul  3 12:19:13.953: INFO: Got endpoints: latency-svc-srb3t [192.87702ms]
Jul  3 12:19:13.957: INFO: Created: latency-svc-8m3hm
Jul  3 12:19:13.962: INFO: Got endpoints: latency-svc-gvgtl [177.92015ms]
Jul  3 12:19:13.978: INFO: Created: latency-svc-hvmcx
Jul  3 12:19:13.978: INFO: Got endpoints: latency-svc-jfjh8 [188.918851ms]
Jul  3 12:19:13.980: INFO: Got endpoints: latency-svc-vj6kk [185.303758ms]
Jul  3 12:19:13.985: INFO: Created: latency-svc-kk52x
Jul  3 12:19:13.987: INFO: Got endpoints: latency-svc-8m3hm [170.213234ms]
Jul  3 12:19:14.000: INFO: Created: latency-svc-ps9rl
Jul  3 12:19:14.003: INFO: Got error: The POST operation against Service could not be completed at this time, please try again.
Jul  3 12:19:14.010: INFO: Got endpoints: latency-svc-hvmcx [192.02128ms]
Jul  3 12:19:14.010: INFO: Created: latency-svc-cbzbv
Jul  3 12:19:14.013: INFO: Got endpoints: latency-svc-kk52x [262.214396ms]
Jul  3 12:19:14.019: INFO: Created: latency-svc-6269g
Jul  3 12:19:14.024: INFO: Got endpoints: latency-svc-ps9rl [188.169421ms]
Jul  3 12:19:14.035: INFO: Got endpoints: latency-svc-cbzbv [182.561447ms]
Jul  3 12:19:14.040: INFO: Got endpoints: latency-svc-6269g [175.151965ms]
Jul  3 12:19:14.040: INFO: Latencies: [37.588434ms 43.700432ms 48.09913ms 58.086951ms 67.021442ms 72.279025ms 93.682904ms 94.928144ms 97.650634ms 98.31208ms 117.812789ms 124.145635ms 126.940569ms 135.536068ms 137.430421ms 137.478434ms 137.975238ms 139.384218ms 139.413865ms 139.972523ms 141.246635ms 141.831582ms 141.925298ms 142.63839ms 144.269317ms 144.280438ms 144.712963ms 144.736992ms 145.205607ms 145.613502ms 145.93327ms 146.323354ms 146.526243ms 146.921956ms 146.979968ms 147.020188ms 147.228535ms 148.185689ms 148.21801ms 148.741692ms 148.799974ms 148.88548ms 148.918087ms 148.980817ms 149.092286ms 150.961324ms 151.118524ms 151.40401ms 151.501611ms 152.000902ms 152.883822ms 152.985997ms 153.352451ms 153.501487ms 153.983482ms 154.609489ms 155.983328ms 156.052333ms 156.54849ms 157.142096ms 157.509684ms 157.561767ms 157.81303ms 157.886881ms 158.103505ms 158.334222ms 158.879282ms 159.123449ms 159.186515ms 159.542396ms 159.645ms 159.926706ms 160.573662ms 160.778044ms 160.920347ms 161.094196ms 161.303794ms 161.782222ms 162.177999ms 162.512277ms 162.969251ms 163.171142ms 163.27626ms 163.552494ms 163.836222ms 163.943628ms 164.077973ms 164.165091ms 164.219881ms 164.748465ms 165.110648ms 165.603325ms 165.686374ms 165.79596ms 166.953157ms 168.069035ms 168.426779ms 168.797504ms 169.257322ms 169.718538ms 169.833565ms 169.906085ms 170.053042ms 170.213234ms 170.993332ms 171.543028ms 171.753392ms 172.531433ms 172.916712ms 173.037082ms 173.177171ms 173.24646ms 173.481446ms 173.506098ms 174.149782ms 174.463723ms 174.568382ms 174.62925ms 174.944002ms 175.002584ms 175.151965ms 175.615772ms 175.746839ms 175.74976ms 175.908082ms 176.439487ms 177.01163ms 177.92015ms 178.559871ms 178.593082ms 179.087581ms 179.09122ms 181.023201ms 182.15375ms 182.561447ms 182.919377ms 183.271641ms 183.349741ms 183.472352ms 184.522794ms 184.997512ms 185.303758ms 185.617708ms 186.0635ms 187.707178ms 188.169421ms 188.74402ms 188.918851ms 189.133469ms 190.697044ms 191.186867ms 192.02128ms 192.249131ms 192.265329ms 192.87702ms 192.945217ms 195.004011ms 195.926034ms 195.969442ms 196.378242ms 196.809856ms 197.145289ms 198.933544ms 198.960882ms 200.214015ms 203.57756ms 204.968543ms 205.71604ms 205.93804ms 207.197205ms 208.667617ms 208.922566ms 212.400679ms 213.158303ms 215.965034ms 225.326492ms 230.699473ms 243.017757ms 258.404277ms 262.214396ms 268.056919ms 274.475664ms 284.918504ms 301.90533ms 323.902167ms 337.717968ms 345.058195ms 370.913886ms 416.902454ms 422.051726ms 426.210595ms 426.962279ms 433.946653ms 434.558083ms 434.667314ms 443.652016ms 456.613925ms 460.529611ms 554.73846ms]
Jul  3 12:19:14.040: INFO: 50 %ile: 169.718538ms
Jul  3 12:19:14.040: INFO: 90 %ile: 262.214396ms
Jul  3 12:19:14.040: INFO: 99 %ile: 460.529611ms
Jul  3 12:19:14.040: INFO: Total sample count: 199
[AfterEach] [k8s.io] Service endpoints latency
  /go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:121
STEP: Collecting events from namespace "e2e-tests-svc-latency-x0dp3".
STEP: Found 5 events.
Jul  3 12:19:14.048: INFO: At 2017-07-03 12:19:08 -0400 EDT - event for svc-latency-rc: {replication-controller } SuccessfulCreate: Created pod: svc-latency-rc-1qqrb
Jul  3 12:19:14.048: INFO: At 2017-07-03 12:19:08 -0400 EDT - event for svc-latency-rc-1qqrb: {default-scheduler } Scheduled: Successfully assigned svc-latency-rc-1qqrb to ip-172-18-8-151.ec2.internal
Jul  3 12:19:14.048: INFO: At 2017-07-03 12:19:09 -0400 EDT - event for svc-latency-rc-1qqrb: {kubelet ip-172-18-8-151.ec2.internal} Pulled: Container image "gcr.io/google_containers/pause-amd64:3.0" already present on machine
Jul  3 12:19:14.048: INFO: At 2017-07-03 12:19:10 -0400 EDT - event for svc-latency-rc-1qqrb: {kubelet ip-172-18-8-151.ec2.internal} Created: Created container
Jul  3 12:19:14.048: INFO: At 2017-07-03 12:19:10 -0400 EDT - event for svc-latency-rc-1qqrb: {kubelet ip-172-18-8-151.ec2.internal} Started: Started container
Jul  3 12:19:14.057: INFO: POD                       NODE                          PHASE    GRACE  CONDITIONS
Jul  3 12:19:14.057: INFO: docker-registry-2-qlj9f   ip-172-18-8-151.ec2.internal  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-07-03 11:48:46 -0400 EDT  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-07-03 11:48:55 -0400 EDT  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-07-03 11:48:46 -0400 EDT  }]
Jul  3 12:19:14.057: INFO: registry-console-1-s93tx  ip-172-18-8-151.ec2.internal  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-07-03 11:48:35 -0400 EDT  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-07-03 11:48:45 -0400 EDT  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-07-03 11:48:35 -0400 EDT  }]
Jul  3 12:19:14.058: INFO: router-2-m9d1b            ip-172-18-8-151.ec2.internal  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-07-03 11:49:00 -0400 EDT  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-07-03 11:49:20 -0400 EDT  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-07-03 11:49:00 -0400 EDT  }]
Jul  3 12:19:14.058: INFO: svc-latency-rc-1qqrb      ip-172-18-8-151.ec2.internal  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2017-07-03 12:19:08 -0400 EDT  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2017-07-03 12:19:10 -0400 EDT  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2017-07-03 12:19:08 -0400 EDT  }]
Jul  3 12:19:14.058: INFO: 
Jul  3 12:19:14.063: INFO: 
Logging node info for node ip-172-18-8-151.ec2.internal
Jul  3 12:19:14.066: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:ip-172-18-8-151.ec2.internal,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/ip-172-18-8-151.ec2.internal,UID:ac6a6553-6005-11e7-a26d-0e291487f9de,ResourceVersion:26264,Generation:0,CreationTimestamp:2017-07-03 11:38:34 -0400 EDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,kubernetes.io/hostname: ip-172-18-8-151.ec2.internal,region: infra,zone: default,},Annotations:map[string]string{volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,},Spec:NodeSpec{PodCIDR:,ExternalID:ip-172-18-8-151.ec2.internal,ProviderID:aws:////i-06c00e874607b784d,Unschedulable:false,Taints:[],},Status:NodeStatus{Capacity:ResourceList{alpha.kubernetes.io/nvidia-gpu: {{0 0} {<nil>} 0 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},memory: {{16388919296 0} {<nil>} 16004804Ki BinarySI},pods: {{40 0} {<nil>} 40 DecimalSI},},Allocatable:ResourceList{alpha.kubernetes.io/nvidia-gpu: {{0 0} {<nil>} 0 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},memory: {{16284061696 0} {<nil>} 15902404Ki BinarySI},pods: {{40 0} {<nil>} 40 DecimalSI},},Phase:,Conditions:[{OutOfDisk False 2017-07-03 12:19:12 -0400 EDT 2017-07-03 11:38:34 -0400 EDT KubeletHasSufficientDisk kubelet has sufficient disk space available} {MemoryPressure False 2017-07-03 12:19:12 -0400 EDT 2017-07-03 11:38:34 -0400 EDT KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2017-07-03 12:19:12 -0400 EDT 2017-07-03 11:38:34 -0400 EDT KubeletHasNoDiskPressure kubelet has no disk pressure} {Ready True 2017-07-03 12:19:12 -0400 EDT 2017-07-03 11:48:28 -0400 EDT KubeletReady kubelet is posting ready status}],Addresses:[{LegacyHostIP 172.18.8.151} {InternalIP 172.18.8.151} {Hostname ip-172-18-8-151.ec2.internal}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:f9370ed252a14f73b014c1301a9b6d1b,SystemUUID:EC202F4A-C749-F005-2133-29DB80CCE7A7,BootID:323904ba-9a36-4462-8a2d-ced0eb85db60,KernelVersion:3.10.0-514.26.1.el7.x86_64,OSImage:Red Hat Enterprise Linux Server 7.3 (Maipo),ContainerRuntimeVersion:docker://1.12.6,KubeletVersion:v1.6.1+5115d708d7,KubeProxyVersion:v1.6.1+5115d708d7,OperatingSystem:linux,Architecture:amd64,},Images:[{[openshift/origin-federation:95c9d10 openshift/origin-federation:latest] 1162721964} {[openshift/origin-docker-registry:95c9d10 openshift/origin-docker-registry:latest] 1058762453} {[openshift/origin-gitserver:95c9d10 openshift/origin-gitserver:latest] 1045166197} {[openshift/openvswitch:95c9d10 openshift/openvswitch:latest] 1023440709} {[openshift/node:95c9d10 openshift/node:latest] 1021758968} {[openshift/origin-keepalived-ipfailover:95c9d10 openshift/origin-keepalived-ipfailover:latest] 998562265} {[openshift/origin-service-catalog:95c9d10 openshift/origin-service-catalog:latest] 996631457} {[openshift/origin-haproxy-router:95c9d10 openshift/origin-haproxy-router:latest] 992792252} {[openshift/origin-f5-router:95c9d10 openshift/origin-f5-router:latest] 971758015} {[openshift/origin:95c9d10 openshift/origin:latest] 971758015} {[openshift/origin-recycler:95c9d10 openshift/origin-recycler:latest] 971758015} {[openshift/origin-deployer:95c9d10 openshift/origin-deployer:latest] 971758015} {[openshift/origin-sti-builder:95c9d10 openshift/origin-sti-builder:latest] 971758015} {[openshift/origin-docker-builder:95c9d10 openshift/origin-docker-builder:latest] 971758015} {[openshift/origin-cluster-capacity:95c9d10 openshift/origin-cluster-capacity:latest] 931856892} {[docker.io/openshift/origin-release@sha256:a5a76325a03ae33db7e554690abd6e05a10621d542bf80279db473e769a95aa6 docker.io/openshift/origin-release:golang-1.7] 853115907} {[docker.io/openshift/origin-haproxy-router@sha256:c9374f410e32907be1fa1d14d77e58206ef0be949a63a635e6f3bafa77b35726 docker.io/openshift/origin-haproxy-router:v1.5.1] 738600544} {[docker.io/openshift/origin-deployer@sha256:77ac551235d8edf43ccb2fbd8fa5384ad9d8b94ba726f778fced18710c5f74f0 docker.io/openshift/origin-deployer:v1.5.1] 617474229} {[docker.io/centos/mongodb-32-centos7@sha256:aa37993c3be2d4731db79c0c7aba11db3e9352b1adc586c7e9054f57808789c0] 567697574} {[172.30.227.185:5000/extended-test-build-valuefrom-mhn58-1dtqf/test@sha256:a7305ab56ccfff1c54e251fe90287d641d2b5ee11454c2dac6fe0c1835ea6a00 172.30.227.185:5000/extended-test-build-valuefrom-mhn58-1dtqf/test:latest] 557300467} {[docker.io/centos/php-70-centos7@sha256:dfb7fea3a8aa8d0dcb6589696a6125b88050236213bab019cd0f7b86f98d33bb docker.io/centos/php-70-centos7:latest] 539284292} {[172.30.227.185:5000/extended-test-build-valuefrom-50hwk-zrq66/test@sha256:a7656964bca610e861a0446330ca760f900cdf19fc5d4a7bd59ff643d3dcc303 172.30.227.185:5000/extended-test-build-valuefrom-50hwk-zrq66/test:latest] 539277017} {[docker.io/centos/ruby-23-centos7@sha256:b013adb164ca694c7c36df8538ed1b14647cad625c935f21ea479709c2cd578e] 506478866} {[docker.io/centos/ruby-22-centos7@sha256:fb5bf6dcd15e89f5b18f2022f7c4c76d0ee442958085cbe09758250898231866 docker.io/centos/ruby-22-centos7:latest] 491818467} {[docker.io/centos/s2i-base-centos7@sha256:0ca53884fa702cef54b971a1120ca7c5e22e34396c43d7d790a5cff45241e8fa docker.io/centos/s2i-base-centos7:latest] 429926950} {[docker.io/openshift/origin-docker-registry@sha256:cfe82b08f94c015d31664573f3caa4307ffc7941c930cc7ae9419d68cec32ed5 docker.io/openshift/origin-docker-registry:v1.5.1] 428360819} {[gcr.io/google_containers/redis@sha256:f066bcf26497fbc55b9bf0769cb13a35c0afa2aa42e737cc46b7fb04b23a2f25 gcr.io/google_containers/redis:e2e] 418929769} {[docker.io/cockpit/kubernetes@sha256:44303d8588d6e3ece0d42563c7d8e64ed8423b65c947000922a8bcc9b2d696b9 docker.io/cockpit/kubernetes:latest] 397845496} {[openshift/origin-egress-http-proxy:95c9d10 openshift/origin-egress-http-proxy:latest] 396614319} {[openshift/origin-egress-router:95c9d10 openshift/origin-egress-router:latest] 365348493} {[openshift/origin-base:95c9d10 openshift/origin-base:latest] 363672960} {[docker.io/openshift/origin-base@sha256:3848ab52436662e4193f34063bbfd259c0c09cbe91562acec7dd6eb510ca2e94 docker.io/openshift/origin-base:latest] 363024868} {[openshift/origin-pod:95c9d10 openshift/origin-pod:latest] 213256219} {[docker.io/openshift/origin-pod@sha256:3e24f54cb9196fcebfb9a5ec8ad66a12db1f6057144683fc87e1a44e9d1b6aee docker.io/openshift/origin-pod:latest] 213256214} {[openshift/origin-source:95c9d10 openshift/origin-source:latest] 192548907} {[172.30.227.185:5000/extended-test-docker-build-pullsecret-6p8z3-p2btb/image1@sha256:d58314404b46991c7e03ad714f6b433e44f36fc97b2888bd4bd71b3cd0a86844 172.30.227.185:5000/extended-test-docker-build-pullsecret-6p8z3-p2btb/image1:latest] 192548537} {[docker.io/centos@sha256:aebf12af704307dfa0079b3babdca8d7e8ff6564696882bcb5d11f1d461f9ee9 docker.io/centos:7 docker.io/centos:centos7 docker.io/centos:latest] 192548537} {[gcr.io/google_containers/jessie-dnsutils@sha256:2460d596912244b5f8973573f7150e7264b570015f4becc2d0096f0bd1d17e36 gcr.io/google_containers/jessie-dnsutils:e2e] 190122856} {[gcr.io/google_containers/nginx-slim@sha256:8b4501fe0fe221df663c22e16539f399e89594552f400408303c42f3dd8d0e52 gcr.io/google_containers/nginx-slim:0.8] 110461313} {[docker.io/nginx@sha256:72c7191585e9b79cde433c89955547685db00f3a8595a750339549f6acef7702 docker.io/nginx:latest] 107448913} {[gcr.io/google_containers/nginx-slim@sha256:dd4efd4c13bec2c6f3fe855deeab9524efe434505568421d4f31820485b3a795 gcr.io/google_containers/nginx-slim:0.7] 86838142} {[gcr.io/google_containers/nettest@sha256:8af3a0e8b8ab906b0648dd575e8785e04c19113531f8ffbaab9e149aa1a60763 gcr.io/google_containers/nettest:1.7] 24051275} {[gcr.io/google_containers/hostexec@sha256:cab8d4e2526f8f767c64febe4ce9e0f0e58cd35fdff81b3aadba4dd041ba9f00 gcr.io/google_containers/hostexec:1.2] 13185747} {[gcr.io/google_containers/dnsutils@sha256:cd9182f6d74e616942db1cef6f25e1e54b49ba0330c2e19d3ec061f027666cc0 gcr.io/google_containers/dnsutils:e2e] 8893907} {[gcr.io/google_containers/netexec@sha256:56c53846f44ea214e4aa5df37c9c50331f0b09e64a32cc7cf17c7e1808d38eef gcr.io/google_containers/netexec:1.7] 8016035} {[gcr.io/google_containers/serve_hostname@sha256:a49737ee84a3b94f0b977f32e60c5daf11f0b5636f1f7503a2981524f351c57a gcr.io/google_containers/serve_hostname:v1.4] 6222101} {[docker.io/openshift/hello-openshift@sha256:f66e70806ec29dd943d1822fd659a241184028c08a184c317e69ae8bc671316e docker.io/openshift/hello-openshift:latest] 5635113} {[openshift/hello-openshift:95c9d10 openshift/hello-openshift:latest] 5635113} {[gcr.io/google_containers/porter@sha256:076acdada33f35b917c9eebe89eba95923601302beac57274985e418b70067e2 gcr.io/google_containers/porter:cd5cb5791ebaa8641955f0e8c2a9bed669b1eaab] 5010921} {[gcr.io/google_containers/fakegitserver@sha256:e974692bb4d422a4e9ea6ff9df85fa36f189010703400496fea44aac6589d0dd gcr.io/google_containers/fakegitserver:0.1] 5007469}],VolumesInUse:[],VolumesAttached:[],},}
Jul  3 12:19:14.066: INFO: 
Logging kubelet events for node ip-172-18-8-151.ec2.internal
Jul  3 12:19:14.069: INFO: 
Logging pods the kubelet thinks is on node ip-172-18-8-151.ec2.internal
Jul  3 12:19:14.076: INFO: router-2-m9d1b started at 2017-07-03 11:49:00 -0400 EDT (0+1 container statuses recorded)
Jul  3 12:19:14.076: INFO: 	Container router ready: true, restart count 0
Jul  3 12:19:14.076: INFO: registry-console-1-s93tx started at 2017-07-03 11:48:35 -0400 EDT (0+1 container statuses recorded)
Jul  3 12:19:14.076: INFO: 	Container registry-console ready: true, restart count 0
Jul  3 12:19:14.076: INFO: svc-latency-rc-1qqrb started at 2017-07-03 12:19:08 -0400 EDT (0+1 container statuses recorded)
Jul  3 12:19:14.076: INFO: 	Container svc-latency-rc ready: true, restart count 0
Jul  3 12:19:14.076: INFO: docker-registry-2-qlj9f started at 2017-07-03 11:48:46 -0400 EDT (0+1 container statuses recorded)
Jul  3 12:19:14.076: INFO: 	Container registry ready: true, restart count 0
Jul  3 12:19:14.447: INFO: 
Latency metrics for node ip-172-18-8-151.ec2.internal
Jul  3 12:19:14.447: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.99 Latency:2m2.535565s}
Jul  3 12:19:14.447: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.9 Latency:2m1.931808s}
Jul  3 12:19:14.447: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.5 Latency:2m1.023595s}
Jul  3 12:19:14.447: INFO: {Operation:create Method:pod_worker_latency_microseconds Quantile:0.9 Latency:2m0.024386s}
Jul  3 12:19:14.447: INFO: {Operation:create Method:pod_worker_latency_microseconds Quantile:0.99 Latency:2m0.024386s}
Jul  3 12:19:14.447: INFO: {Operation:update Method:pod_worker_latency_microseconds Quantile:0.9 Latency:30.602199s}
Jul  3 12:19:14.447: INFO: {Operation:update Method:pod_worker_latency_microseconds Quantile:0.99 Latency:30.602199s}
Jul  3 12:19:14.447: INFO: {Operation:update Method:pod_worker_latency_microseconds Quantile:0.5 Latency:30.326595s}
Jul  3 12:19:14.447: INFO: {Operation:stop_container Method:docker_operations_latency_microseconds Quantile:0.99 Latency:30.118457s}
Jul  3 12:19:14.447: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.99 Latency:10.977294s}
STEP: Dumping a list of prepulled images on each node
Jul  3 12:19:14.452: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "e2e-tests-svc-latency-x0dp3" for this suite.
Jul  3 12:19:44.556: INFO: namespace: e2e-tests-svc-latency-x0dp3, resource: bindings, ignored listing per whitelist
@stevekuznetsov stevekuznetsov added component/kubernetes kind/test-flake Categorizes issue or PR as related to test flakes. priority/P1 labels Jul 3, 2017
@stevekuznetsov stevekuznetsov mentioned this issue Jul 3, 2017
Merged
@sjenning
Copy link
Contributor

sjenning commented Jul 5, 2017

1 out of 200 samples is failing.

Jul  3 12:19:14.003: INFO: Got error: The POST operation against Service could not be completed at this time, please try again.

@bparees
Copy link
Contributor

bparees commented Jul 13, 2017

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_gce/4299/

@bparees bparees mentioned this issue Jul 13, 2017
Merged
@knobunc knobunc mentioned this issue Jul 19, 2017
Merged
@knobunc
Copy link
Contributor

knobunc commented Jul 19, 2017

#15229 flaked, logs: https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/3282/

@sjenning
Copy link
Contributor

sjenning commented Dec 1, 2017

@ingvagabund PTAL

@sjenning sjenning removed their assignment Dec 1, 2017
@ingvagabund
Copy link
Member

ingvagabund commented Dec 5, 2017 •
edited
Loading

AWS:

GCE:

@ingvagabund
Copy link
Member

ingvagabund commented Dec 5, 2017 •
edited
Loading

This is not an urgent issue so I will wait once the 3.8 rpms are released so I can run the test_pull_request_origin_extended_conformance_install_update job and get the exactly same environment. The GCE does not seem to produce the failure so far but it's a flake, I will check more jobs to see if it occurs.

@ingvagabund
Copy link
Member

Blocked on #17263

@smarterclayton
Copy link
Contributor

This test no longer run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@derekwaynecarr derekwaynecarr

@ingvagabund ingvagabund

Labels
component/kubernetes kind/test-flake Categorizes issue or PR as related to test flakes. priority/P2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@sjenning @smarterclayton @bparees @derekwaynecarr @stevekuznetsov @ingvagabund @knobunc