TestConcurrentBuildImageChangeTriggerControllers flake

[bparees]

--- FAIL: TestConcurrentBuildImageChangeTriggerControllers (65.27s)
	controllers.go:312: Timed out waiting for watch: initial build added
	etcd.go:99: dumping etcd to "/tmp/openshift/test-integration//etcd-dump-runtime.call32.json"

as seen in https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/11008/consoleFull#55367059856bf4006e4b05b79524e5923

appears to be caused by the imagechangecontroller never starting up because it's waiting for the shared informer cache to sync:

I0207 02:58:32.423681   21876 factory.go:331] Waiting for the bc caches to sync before starting the imagechange buildconfig controller worker
I0207 02:58:32.423687   21876 factory.go:331] Waiting for the bc caches to sync before starting the imagechange buildconfig controller worker
I0207 02:58:32.424077   21876 factory.go:331] Waiting for the bc caches to sync before starting the imagechange buildconfig controller worker

@deads2k can you offer some insight as to why the cache would sometimes fail to sync?

the logic for setting up the caches and sync check are here:
https://github.com/openshift/origin/blob/master/pkg/cmd/server/origin/run_components.go#L305-L314
https://github.com/openshift/origin/blob/master/pkg/build/controller/factory/factory.go#L302-L339

[bparees]

@smarterclayton fyi.

[soltysh]

One another here: https://ci.openshift.redhat.com/jenkins/job/fork_ami/322/consoleFull

[bparees]

yeah, looks identical... BC caches never reported they were synced. Need @liggitt or @deads2k to provide some shared informer expertise on the code i linked above.

[smarterclayton]

And @ncdc On Feb 10, 2017, at 6:43 PM, Ben Parees <notifications@github.com> wrote: yeah, looks identical... BC caches never reported they were synced. Need @liggitt <https://github.com/liggitt> or @deads2k <https://github.com/deads2k> to provide some shared informer expertise on the code i linked above. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12899 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p3WfDZNHboFTfUneiR80syO7uY4eks5rbPYJgaJpZM4L8l4e> .

[bparees]

turns out i think this is the same issue as #12017, the imagechangecontroller is seeing the event before its buildconfig cache contains the newly created buildconfig, so it doesn't take any action on the imagestream event (because it can't find a buildconfig that cares about that event).

not sure what's up w/ all the "waiting for sync" messages, but i think they are a red herring.

[0xmichalis]

Recurred in https://ci.dev.openshift.redhat.com/jenkins/job/test_pull_requests_ose/1154/consoleFull

[bparees]

@Kargakis did you confirm the ose branch your pull is against has the fix that closed this issue? It went in pretty recently.

[0xmichalis]

Yeah master hasn't been updated in a while so it's likely that the fix is missing from there.