Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CLI cluster manager #start tests are flaky #15888

Closed
kimjoar opened this issue Jan 8, 2018 · 0 comments
Closed

CLI cluster manager #start tests are flaky #15888

kimjoar opened this issue Jan 8, 2018 · 0 comments
Assignees
@spalger
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc test

Comments

@kimjoar
Copy link
Contributor

kimjoar commented Jan 8, 2018 •
edited
Loading

Both of there tests are flaky:

kibana/src/cli/cluster/__tests__/worker.js

Lines 161 to 184 in f8e0adf

it('creates a fork and waits for it to come online', async function () {
const worker = setup({ log: new Log(false, true) });
sinon.spy(worker, 'on');
await worker.start();
sinon.assert.calledOnce(cluster.fork);
sinon.assert.calledWith(worker.on, 'fork:online');
});
it('listens for cluster and process "exit" events', async function () {
const worker = setup({ log: new Log(false, true) });
sinon.spy(process, 'on');
sinon.spy(cluster, 'on');
await worker.start();
sinon.assert.calledOnce(cluster.on);
sinon.assert.calledWith(cluster.on, 'exit');
sinon.assert.calledOnce(process.on);
sinon.assert.calledWith(process.on, 'exit');
});

The first one failed 7 times the last week and I created a PR to skip it for now (#15889). The second failed 3 times in a week and I created #16045 to skip it.

Full error for first test:

  1) CLI cluster manager #start when not started creates a fork and waits for it to come online:
     expected fork to be called once but was called twice
    fork({ kbnWorkerArgv: "[]", kbnWorkerType: undefined }) => [MockClusterFork] {
  _events: { disconnect: function () {}, message: function () {}, online: function () {} },
  _eventsCount: 3,
  _maxListeners: undefined,
  domain: null,
  emit: function emit() {},
  isDead: function spy() {},
  on: function on() {},
  process: { kill: function spy() {} },
  removeListener: function removeListener() {},
  send: function stub() {}
} at /var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/src/cli/cluster/worker.js:150:25

    fork({ kbnWorkerArgv: "[]", kbnWorkerType: undefined }) => [MockClusterFork] {
  _events: { disconnect: function () {}, message: function () {}, online: function () {} },
  _eventsCount: 3,
  _maxListeners: undefined,
  domain: null,
  emit: function emit() {},
  isDead: function spy() {},
  on: function on() {},
  process: { kill: function spy() {} },
  removeListener: function removeListener() {},
  send: function stub() {}
} at /var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/src/cli/cluster/worker.js:150:25
  AssertError: expected fork to be called once but was called twice
      fork({ kbnWorkerArgv: "[]", kbnWorkerType: undefined }) => [MockClusterFork] {
    _events: { disconnect: function () {}, message: function () {}, online: function () {} },
    _eventsCount: 3,
    _maxListeners: undefined,
    domain: null,
    emit: function emit() {},
    isDead: function spy() {},
    on: function on() {},
    process: { kill: function spy() {} },
    removeListener: function removeListener() {},
    send: function stub() {}
  } at src/cli/cluster/worker.js:150:25
  
      fork({ kbnWorkerArgv: "[]", kbnWorkerType: undefined }) => [MockClusterFork] {
    _events: { disconnect: function () {}, message: function () {}, online: function () {} },
    _eventsCount: 3,
    _maxListeners: undefined,
    domain: null,
    emit: function emit() {},
    isDead: function spy() {},
    on: function on() {},
    process: { kill: function spy() {} },
    removeListener: function removeListener() {},
    send: function stub() {}
  } at src/cli/cluster/worker.js:150:25
      at Object.fail (node_modules/sinon/lib/sinon/assert.js:90:29)
      at failAssertion (node_modules/sinon/lib/sinon/assert.js:51:24)
      at Object.assert.(anonymous function) [as calledOnce] (node_modules/sinon/lib/sinon/assert.js:74:21)
      at Context.<anonymous> (src/cli/cluster/__tests__/worker.js:168:22)
      at undefined.next (native)
      at step (src/cli/cluster/__tests__/worker.js:8:1)
      at src/cli/cluster/__tests__/worker.js:8:1

Full error for second test:

1) CLI cluster manager #start when not started listens for cluster and process "exit" events:
   expected on to be called once but was called twice
  on(exit, function () {}) => [EventEmitter] {
SCHED_NONE: 1,
SCHED_RR: 2,
Worker: function Worker() {},
_events: {
  exit: [function () {}, function () {}, function () {}, function () {}, function () {}, function () {}, function () {}]
},
_eventsCount: 1,
_maxListeners: undefined,
disconnect: function () {},
domain: null,
fork: function fork() {},
isMaster: true,
isWorker: false,
on: function on() {},
schedulingPolicy: 2,
settings: {
  args: ["jenkins:unit"],
  exec: "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/src/cli",
  execArgv: [],
  silent: false
},
setupMaster: function () {},
workers: {
  6: [Worker] {
    _events: {
      disconnect: function () {},
      message: [function () {}, function () {}],
      online: function () {}
    },
    _eventsCount: 3,
    _maxListeners: undefined,
    domain: null,
    exitedAfterDisconnect: undefined,
    id: 6,
    process: [ChildProcess] {
      _channel: [Pipe] {
        buffering: false,
        onread: function () {},
        sockets: { got: {  }, send: {  } },
        writeQueueSize: 0
      },
      _closesGot: 0,
      _closesNeeded: 2,
      _disconnect: function finish() {},
      _events: {
        disconnect: function g() {},
        error: function () {},
        exit: function g() {},
        internalMessage: [function () {}, function () {}],
        message: function () {}
      },
      _eventsCount: 5,
      _handle: [Process] { onexit: function () {}, owner: [Circular], pid: 24514 },
      _handleQueue: null,
      _maxListeners: undefined,
      _pendingMessage: null,
      _send: function signature
  if () {},
      connected: true,
      disconnect: function () {},
      domain: null,
      exitCode: null,
      killed: false,
      pid: 24514,
      send: function () {},
      signalCode: null,
      spawnargs: ["/tmp/kibana/node/6.12.2/bin/node", "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/src/cli", "jenkins:unit"],
      spawnfile: "/tmp/kibana/node/6.12.2/bin/node",
      stderr: null,
      stdin: null,
      stdio: [null, null, null, null],
      stdout: null
    },
    state: "online",
    suicide: undefined
  }
}
at BinderFor.on (/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/src/utils/binder.js:15:8)

  on(exit, function () {}) => [EventEmitter] {
SCHED_NONE: 1,
SCHED_RR: 2,
Worker: function Worker() {},
_events: {
  exit: [function () {}, function () {}, function () {}, function () {}, function () {}, function () {}, function () {}]
},
_eventsCount: 1,
_maxListeners: undefined,
disconnect: function () {},
domain: null,
fork: function fork() {},
isMaster: true,
isWorker: false,
on: function on() {},
schedulingPolicy: 2,
settings: {
  args: ["jenkins:unit"],
  exec: "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/src/cli",
  execArgv: [],
  silent: false
},
setupMaster: function () {},
workers: {
  6: [Worker] {
    _events: {
      disconnect: function () {},
      message: [function () {}, function () {}],
      online: function () {}
    },
    _eventsCount: 3,
    _maxListeners: undefined,
    domain: null,
    exitedAfterDisconnect: undefined,
    id: 6,
    process: [ChildProcess] {
      _channel: [Pipe] {
        buffering: false,
        onread: function () {},
        sockets: { got: {  }, send: {  } },
        writeQueueSize: 0
      },
      _closesGot: 0,
      _closesNeeded: 2,
      _disconnect: function finish() {},
      _events: {
        disconnect: function g() {},
        error: function () {},
        exit: function g() {},
        internalMessage: [function () {}, function () {}],
        message: function () {}
      },
      _eventsCount: 5,
      _handle: [Process] { onexit: function () {}, owner: [Circular], pid: 24514 },
      _handleQueue: null,
      _maxListeners: undefined,
      _pendingMessage: null,
      _send: function signature
  if () {},
      connected: true,
      disconnect: function () {},
      domain: null,
      exitCode: null,
      killed: false,
      pid: 24514,
      send: function () {},
      signalCode: null,
      spawnargs: ["/tmp/kibana/node/6.12.2/bin/node", "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/src/cli", "jenkins:unit"],
      spawnfile: "/tmp/kibana/node/6.12.2/bin/node",
      stderr: null,
      stdin: null,
      stdio: [null, null, null, null],
      stdout: null
    },
    state: "online",
    suicide: undefined
  }
}
at BinderFor.on (/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/src/utils/binder.js:15:8)
AssertError: expected on to be called once but was called twice
    on(exit, function () {}) => [EventEmitter] {
  SCHED_NONE: 1,
  SCHED_RR: 2,
  Worker: function Worker() {},
  _events: {
    exit: [function () {}, function () {}, function () {}, function () {}, function () {}, function () {}, function () {}]
  },
  _eventsCount: 1,
  _maxListeners: undefined,
  disconnect: function () {},
  domain: null,
  fork: function fork() {},
  isMaster: true,
  isWorker: false,
  on: function on() {},
  schedulingPolicy: 2,
  settings: {
    args: ["jenkins:unit"],
    exec: "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/src/cli",
    execArgv: [],
    silent: false
  },
  setupMaster: function () {},
  workers: {
    6: [Worker] {
      _events: {
        disconnect: function () {},
        message: [function () {}, function () {}],
        online: function () {}
      },
      _eventsCount: 3,
      _maxListeners: undefined,
      domain: null,
      exitedAfterDisconnect: undefined,
      id: 6,
      process: [ChildProcess] {
        _channel: [Pipe] {
          buffering: false,
          onread: function () {},
          sockets: { got: {  }, send: {  } },
          writeQueueSize: 0
        },
        _closesGot: 0,
        _closesNeeded: 2,
        _disconnect: function finish() {},
        _events: {
          disconnect: function g() {},
          error: function () {},
          exit: function g() {},
          internalMessage: [function () {}, function () {}],
          message: function () {}
        },
        _eventsCount: 5,
        _handle: [Process] { onexit: function () {}, owner: [Circular], pid: 24514 },
        _handleQueue: null,
        _maxListeners: undefined,
        _pendingMessage: null,
        _send: function signature
    if () {},
        connected: true,
        disconnect: function () {},
        domain: null,
        exitCode: null,
        killed: false,
        pid: 24514,
        send: function () {},
        signalCode: null,
        spawnargs: ["/tmp/kibana/node/6.12.2/bin/node", "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/src/cli", "jenkins:unit"],
        spawnfile: "/tmp/kibana/node/6.12.2/bin/node",
        stderr: null,
        stdin: null,
        stdio: [null, null, null, null],
        stdout: null
      },
      state: "online",
      suicide: undefined
    }
  }
} at BinderFor.on (src/utils/binder.js:15:8)

    on(exit, function () {}) => [EventEmitter] {
  SCHED_NONE: 1,
  SCHED_RR: 2,
  Worker: function Worker() {},
  _events: {
    exit: [function () {}, function () {}, function () {}, function () {}, function () {}, function () {}, function () {}]
  },
  _eventsCount: 1,
  _maxListeners: undefined,
  disconnect: function () {},
  domain: null,
  fork: function fork() {},
  isMaster: true,
  isWorker: false,
  on: function on() {},
  schedulingPolicy: 2,
  settings: {
    args: ["jenkins:unit"],
    exec: "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/src/cli",
    execArgv: [],
    silent: false
  },
  setupMaster: function () {},
  workers: {
    6: [Worker] {
      _events: {
        disconnect: function () {},
        message: [function () {}, function () {}],
        online: function () {}
      },
      _eventsCount: 3,
      _maxListeners: undefined,
      domain: null,
      exitedAfterDisconnect: undefined,
      id: 6,
      process: [ChildProcess] {
        _channel: [Pipe] {
          buffering: false,
          onread: function () {},
          sockets: { got: {  }, send: {  } },
          writeQueueSize: 0
        },
        _closesGot: 0,
        _closesNeeded: 2,
        _disconnect: function finish() {},
        _events: {
          disconnect: function g() {},
          error: function () {},
          exit: function g() {},
          internalMessage: [function () {}, function () {}],
          message: function () {}
        },
        _eventsCount: 5,
        _handle: [Process] { onexit: function () {}, owner: [Circular], pid: 24514 },
        _handleQueue: null,
        _maxListeners: undefined,
        _pendingMessage: null,
        _send: function signature
    if () {},
        connected: true,
        disconnect: function () {},
        domain: null,
        exitCode: null,
        killed: false,
        pid: 24514,
        send: function () {},
        signalCode: null,
        spawnargs: ["/tmp/kibana/node/6.12.2/bin/node", "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/src/cli", "jenkins:unit"],
        spawnfile: "/tmp/kibana/node/6.12.2/bin/node",
        stderr: null,
        stdin: null,
        stdio: [null, null, null, null],
        stdout: null
      },
      state: "online",
      suicide: undefined
    }
  }
} at BinderFor.on (src/utils/binder.js:15:8)
    at Object.fail (node_modules/sinon/lib/sinon/assert.js:90:29)
    at failAssertion (node_modules/sinon/lib/sinon/assert.js:51:24)
    at Object.assert.(anonymous function) [as calledOnce] (node_modules/sinon/lib/sinon/assert.js:74:21)
    at Context.<anonymous> (src/cli/cluster/__tests__/worker.js:222:32)
    at undefined.next (native)
    at step (src/cli/cluster/__tests__/worker.js:33:191)
    at src/cli/cluster/__tests__/worker.js:33:361
@kimjoar kimjoar added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc build labels Jan 8, 2018
@kimjoar kimjoar mentioned this issue Jan 8, 2018
Merged
@spalger spalger self-assigned this Jan 11, 2018
@spalger spalger added test and removed build labels Jan 11, 2018
@kimjoar kimjoar changed the title CLI cluster manager #start test is flaky CLI cluster manager #start tests are flaky Jan 15, 2018
@spalger spalger self-assigned this Feb 14, 2018
@azasypkin azasypkin mentioned this issue Sep 6, 2018
Merged
@liza-mae liza-mae added failed-test A test failure on a tracked branch, potentially flaky-test and removed flaky-failing-test labels Nov 1, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spalger spalger

Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc test
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kimjoar @spalger @azasypkin @liza-mae