Failing test: Jest Integration Tests.src/core/server/saved_objects/migrationsv2/integration_tests - migration from 7.7.2-xpack with 100k objects copies all the document of the previous index to the new one #96895

kibanamachine · 2021-04-12T21:01:12Z

A test failed on a tracked branch

TypeError: this.licenseService.stop is not a function
    at MonitoringPlugin.stop (/dev/shm/workspace/parallel/4/kibana/x-pack/plugins/monitoring/server/plugin.ts:234:27)
    at PluginWrapper.stop (/dev/shm/workspace/parallel/4/kibana/src/core/server/plugins/plugin.ts:131:27)
    at PluginsSystem.stopPlugins (/dev/shm/workspace/parallel/4/kibana/src/core/server/plugins/plugins_system.ts:201:48)
    at PluginsService.stop (/dev/shm/workspace/parallel/4/kibana/src/core/server/plugins/plugins_service.ts:139:5)
    at Server.stop (/dev/shm/workspace/parallel/4/kibana/src/core/server/server.ts:274:5)
    at Root.shutdown (/dev/shm/workspace/parallel/4/kibana/src/core/server/root/index.ts:72:5)
    at Root.start (/dev/shm/workspace/parallel/4/kibana/src/core/server/root/index.ts:54:7)
    at async Promise.all (index 1)
    at startServers (/dev/shm/workspace/parallel/4/kibana/src/core/server/saved_objects/migrationsv2/integration_tests/migration_7.7.2_xpack_100k.test.ts:88:5)
    at Object.<anonymous> (/dev/shm/workspace/parallel/4/kibana/src/core/server/saved_objects/migrationsv2/integration_tests/migration_7.7.2_xpack_100k.test.ts:106:5)

First failure: Jenkins Build

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-04-21T21:02:11Z

Pinging @elastic/kibana-core (Team:Core)

spalger · 2021-04-21T21:05:47Z

After we cut the 7.13 branch this test started failing with:

00:27:03.482  Summary of all failing tests
00:27:03.483   FAIL  src/core/server/saved_objects/migrationsv2/integration_tests/migration_7.7.2_xpack_100k.test.ts (379.315 s)
00:27:03.483    ● migration from 7.7.2-xpack with 100k objects › copies all the document of the previous index to the new one
00:27:03.483  
00:27:03.483      Network Error
00:27:03.483  
00:27:03.483        at createError (node_modules/axios/lib/core/createError.js:16:15)
00:27:03.483        at XMLHttpRequest.handleError (node_modules/axios/lib/adapters/xhr.js:84:14)
00:27:03.483        at Request.onRequestResponse (../../../kibana/node_modules/request/request.js:1059:10)
00:27:03.483            at HTTPParser.execute (<anonymous>)

It's unclear what might be causing this, but the test is reliant on the version and was consistently failing in 7.13, so I only skipped there. CC @rudolf

Skipped

7.13: 2b57ef3

kibanamachine · 2021-04-21T22:00:01Z

New failure: Jenkins Build

pgayvallet · 2021-04-22T08:23:09Z

@spalger Just checkout'ed the 7.13 branch and ran the test 20 times locally without being able to reproduce this Network Error. All runs did pass.

Do you know if anything on CI may be causing that? Of course we don't have an usable stack trace, so that doesn't help a bit about where/how exactly the error did happen.

The only notable difference between 7.13 and master that could impact this is the version of @elastic/elasticsearch we're using: npm:@elastic/elasticsearch-canary@^8.0.0-canary.4 vs npm:@elastic/elasticsearch-canary@7.13.0-canary.1

@delvedor Is there any differences in the transport layer between these two versions? Was any library bumped between those?

afharo · 2021-04-22T08:48:43Z

Looking at the extended logs in Jenkins, it looks like a Cross-origin issue? 🤔

22:22:46    console.error
22:22:46      Error: Cross origin http://localhost forbidden
22:22:46          at dispatchError (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/xhr-utils.js:62:19)
22:22:46          at Object.validCORSHeaders (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/xhr-utils.js:74:5)
22:22:46          at receiveResponse (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js:796:19)
22:22:46          at Request.<anonymous> (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js:657:36)
22:22:46          at Request.emit (events.js:315:20)
22:22:46          at Request.onRequestResponse (/dev/shm/workspace/kibana/node_modules/request/request.js:1059:10)
22:22:46          at ClientRequest.emit (events.js:315:20)
22:22:46          at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:641:27)
22:22:46          at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
22:22:46          at HTTPParser.execute (<anonymous>)
22:22:46          at TLSSocket.socketOnData (_http_client.js:509:22)
22:22:46          at TLSSocket.emit (events.js:315:20)
22:22:46          at addChunk (internal/streams/readable.js:309:12)
22:22:46          at readableAddChunk (internal/streams/readable.js:284:9)
22:22:46          at TLSSocket.Readable.push (internal/streams/readable.js:223:10)
22:22:46          at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23) undefined
22:22:46  
22:22:46        at VirtualConsole.<anonymous> (../../../kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/virtual-console.js:29:45)
22:22:46        at Request.onRequestResponse (../../../kibana/node_modules/request/request.js:1059:10)
22:22:46            at HTTPParser.execute (<anonymous>)
22:22:46  
22:22:46    console.error
22:22:46      Error: Cross origin http://localhost forbidden
22:22:46          at dispatchError (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/xhr-utils.js:62:19)
22:22:46          at Object.validCORSHeaders (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/xhr-utils.js:74:5)
22:22:46          at receiveResponse (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js:796:19)
22:22:46          at Request.<anonymous> (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js:657:36)
22:22:46          at Request.emit (events.js:315:20)
22:22:46          at Request.onRequestResponse (/dev/shm/workspace/kibana/node_modules/request/request.js:1059:10)
22:22:46          at ClientRequest.emit (events.js:315:20)
22:22:46          at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:641:27)
22:22:46          at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
22:22:46          at HTTPParser.execute (<anonymous>)
22:22:46          at TLSSocket.socketOnData (_http_client.js:509:22)
22:22:46          at TLSSocket.emit (events.js:315:20)
22:22:46          at addChunk (internal/streams/readable.js:309:12)
22:22:46          at readableAddChunk (internal/streams/readable.js:284:9)
22:22:46          at TLSSocket.Readable.push (internal/streams/readable.js:223:10)
22:22:46          at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23) undefined
22:22:46  
22:22:46        at VirtualConsole.<anonymous> (../../../kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/virtual-console.js:29:45)
22:22:46        at Request.onRequestResponse (../../../kibana/node_modules/request/request.js:1059:10)
22:22:46            at HTTPParser.execute (<anonymous>)
22:22:46  
22:22:46    console.error
22:22:46      Error: Cross origin http://localhost forbidden
22:22:46          at dispatchError (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/xhr-utils.js:62:19)
22:22:46          at Object.validCORSHeaders (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/xhr-utils.js:74:5)
22:22:46          at receiveResponse (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js:796:19)
22:22:46          at Request.<anonymous> (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js:657:36)
22:22:46          at Request.emit (events.js:315:20)
22:22:46          at Request.onRequestResponse (/dev/shm/workspace/kibana/node_modules/request/request.js:1059:10)
22:22:46          at ClientRequest.emit (events.js:315:20)
22:22:46          at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:641:27)
22:22:46          at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
22:22:46          at HTTPParser.execute (<anonymous>)
22:22:46          at TLSSocket.socketOnData (_http_client.js:509:22)
22:22:46          at TLSSocket.emit (events.js:315:20)
22:22:46          at addChunk (internal/streams/readable.js:309:12)
22:22:46          at readableAddChunk (internal/streams/readable.js:284:9)
22:22:46          at TLSSocket.Readable.push (internal/streams/readable.js:223:10)
22:22:46          at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23) undefined
22:22:46  
22:22:46        at VirtualConsole.<anonymous> (../../../kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/virtual-console.js:29:45)
22:22:46        at Request.onRequestResponse (../../../kibana/node_modules/request/request.js:1059:10)
22:22:46            at HTTPParser.execute (<anonymous>)
22:22:46  
22:22:46    console.error
22:22:46      Error: Cross origin http://localhost forbidden
22:22:46          at dispatchError (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/xhr-utils.js:62:19)
22:22:46          at Object.validCORSHeaders (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/xhr-utils.js:74:5)
22:22:46          at receiveResponse (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js:796:19)
22:22:46          at Request.<anonymous> (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js:657:36)
22:22:46          at Request.emit (events.js:315:20)
22:22:46          at Request.onRequestResponse (/dev/shm/workspace/kibana/node_modules/request/request.js:1059:10)
22:22:46          at ClientRequest.emit (events.js:315:20)
22:22:46          at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:641:27)
22:22:46          at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
22:22:46          at HTTPParser.execute (<anonymous>)
22:22:46          at TLSSocket.socketOnData (_http_client.js:509:22)
22:22:46          at TLSSocket.emit (events.js:315:20)
22:22:46          at addChunk (internal/streams/readable.js:309:12)
22:22:46          at readableAddChunk (internal/streams/readable.js:284:9)
22:22:46          at TLSSocket.Readable.push (internal/streams/readable.js:223:10)
22:22:46          at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23) undefined
22:22:46  
22:22:46        at VirtualConsole.<anonymous> (../../../kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/virtual-console.js:29:45)
22:22:46        at Request.onRequestResponse (../../../kibana/node_modules/request/request.js:1059:10)
22:22:46            at HTTPParser.execute (<anonymous>)
22:22:46  
22:22:46           │ info [o.e.c.m.MetadataIndexTemplateService] [a68d2d2a522a] adding template [.kibana-event-log-7.13.0-template] for index patterns [.kibana-event-log-7.13.0-*]
22:22:46           │ info [o.e.c.m.MetadataIndexTemplateService] [a68d2d2a522a] adding template [.kibana_security_session_index_template_1] for index patterns [.kibana_security_session_1]
22:22:46           │ info [o.e.c.m.MetadataCreateIndexService] [a68d2d2a522a] [.kibana-event-log-7.13.0-000001] creating index, cause [api], templates [.kibana-event-log-7.13.0-template], shards [1]/[1]
22:22:46           │ info [o.e.c.r.a.AllocationService] [a68d2d2a522a] updating number_of_replicas to [0] for indices [.kibana-event-log-7.13.0-000001]
22:22:46           │ info [o.e.x.i.IndexLifecycleTransition] [a68d2d2a522a] moving index [.kibana-event-log-7.13.0-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [kibana-event-log-policy]
22:22:46           │ info [o.e.c.r.a.AllocationService] [a68d2d2a522a] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana-event-log-7.13.0-000001][0]]]).
22:22:46           │ info [o.e.c.m.MetadataCreateIndexService] [a68d2d2a522a] [.kibana_security_session_1] creating index, cause [api], templates [.kibana_security_session_index_template_1], shards [1]/[0]
22:22:46           │ info [o.e.x.i.IndexLifecycleTransition] [a68d2d2a522a] moving index [.kibana-event-log-7.13.0-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [kibana-event-log-policy]
22:22:46           │ info [o.e.c.r.a.AllocationService] [a68d2d2a522a] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_security_session_1][0]]]).
22:22:46           │ info [o.e.x.i.IndexLifecycleTransition] [a68d2d2a522a] moving index [.kibana-event-log-7.13.0-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [kibana-event-log-policy]
22:22:49           │ info [o.e.x.m.p.NativeController] [a68d2d2a522a] Native controller process has stopped - no new native processes can be started
22:22:49           │ info [o.e.n.Node] [a68d2d2a522a] stopping ...
22:22:49           │ info [o.e.x.w.WatcherService] [a68d2d2a522a] stopping watch service, reason [shutdown initiated]
22:22:49           │ info [o.e.x.w.WatcherLifeCycleService] [a68d2d2a522a] watcher has stopped and shutdown
22:22:49           │ info [o.e.n.Node] [a68d2d2a522a] stopped
22:22:49           │ info [o.e.n.Node] [a68d2d2a522a] closing ...
22:22:53           │ info [o.e.c.InternalClusterInfoService] [a68d2d2a522a] failed to retrieve stats for node [bO9XOU1xTvGpOOCwlMG0bQ]: [a68d2d2a522a][127.0.0.1:6143][cluster:monitor/nodes/stats[n]]
22:22:53           │ info [o.e.c.InternalClusterInfoService] [a68d2d2a522a] failed to retrieve shard stats from node [bO9XOU1xTvGpOOCwlMG0bQ]: [a68d2d2a522a][127.0.0.1:6143][indices:monitor/stats[n]]
22:23:00           │ info [o.e.n.Node] [a68d2d2a522a] closed
22:23:00           │ info [es] stopped
22:23:00           │ info [es] cleanup complete
22:23:10   FAIL  src/core/server/saved_objects/migrationsv2/integration_tests/migration_7.7.2_xpack_100k.test.ts (379.315 s)
22:23:10    migration from 7.7.2-xpack with 100k objects
22:23:10      ✕ copies all the document of the previous index to the new one (1149 ms)
22:23:10  
22:23:10    ● migration from 7.7.2-xpack with 100k objects › copies all the document of the previous index to the new one
22:23:10  
22:23:10      Network Error
22:23:10  
22:23:10        at createError (node_modules/axios/lib/core/createError.js:16:15)
22:23:10        at XMLHttpRequest.handleError (node_modules/axios/lib/adapters/xhr.js:84:14)
22:23:10        at Request.onRequestResponse (../../../kibana/node_modules/request/request.js:1059:10)
22:23:10            at HTTPParser.execute (<anonymous>)

delvedor · 2021-04-22T14:19:01Z

@delvedor Is there any differences in the transport layer between these two versions? Was any library bumped between those?

@pgayvallet no differences.

spalger · 2021-04-22T19:06:42Z

@pgayvallet want to try that in a PR that unskips this test? If it's not reproducible in a PR then I'm good moving forward with un-skipping. Otherwise we'll have something we can add additional debug logging to to figure out what's going on.

mshustov · 2021-04-23T08:21:25Z

I'm wondering why jest-environment-jsdom affects the server-side test env. Is it applicable for all the tests by default?

spalger · 2021-04-26T19:50:02Z

I'm wondering why jest-environment-jsdom affects the server-side test env. Is it applicable for all the tests by default?

Unfortunately Jest has no concept of what a server-side test is, we could use different config files for server/browser tests but for now all tests extend the base config which sets the Jest env to jest-environment-jsdom, which provides a global XHR implementation that libraries like request automatically use.

kibanamachine · 2021-05-01T01:52:40Z

New failure: Jenkins Build

kibanamachine · 2021-05-18T20:45:40Z

New failure: Jenkins Build

tylersmalley · 2021-05-18T21:51:29Z

Also skipping in master as there are quite a few failures here.

master: a105c7a

mistic · 2021-07-30T13:35:31Z

This test started to failed again and looks flaky. I'm skipping it for now.

It failed for example at #15905

Summary of all failing tests
12:54:13   FAIL  src/core/server/saved_objects/migrationsv2/integration_tests/migration_7.7.2_xpack_100k.test.ts (271.198 s)
12:54:13  
12:54:13  
12:54:13    ● Test suite failed to run
12:54:13  
12:54:13      Failed to download https://storage.googleapis.com/headless_shell/chromium-d163fd7-darwin_x64.zip
12:54:13  
12:54:13        53 |               }
12:54:13        54 |             } catch (err) {
12:54:13      > 55 |               const message = new Error(`Failed to download ${url}`);
12:54:13           |                               ^
12:54:13        56 |               logger.error(err);
12:54:13        57 |               throw message;
12:54:13        58 |             }
12:54:13  
12:54:13        at x-pack/plugins/reporting/server/browsers/download/ensure_downloaded.ts:55:31
12:54:13            at runMicrotasks (<anonymous>)
12:54:13            at async Promise.all (index 0)
12:54:13        at x-pack/plugins/reporting/server/browsers/download/ensure_downloaded.ts:37:7
12:54:13            at async Promise.all (index 0)
12:54:13        at ensureDownloaded (x-pack/plugins/reporting/server/browsers/download/ensure_downloaded.ts:32:3)
12:54:13        at ensureBrowserDownloaded (x-pack/plugins/reporting/server/browsers/download/ensure_downloaded.ts:21:3)
12:54:13        at backgroundInstall (x-pack/plugins/reporting/server/browsers/install.ts:42:7)
12:54:13  
12:54:13

master/8.0: f21c7e1
7.x/7.15: e23a033

kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Apr 12, 2021

botelastic bot added the needs-team Issues missing a team label label Apr 12, 2021

spalger added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Apr 21, 2021

botelastic bot removed the needs-team Issues missing a team label label Apr 21, 2021

spalger added a commit that referenced this issue Apr 21, 2021

skip suite failing in 7.13 (#96895)

2b57ef3

spalger added blocker skipped-test v7.13.0 labels Apr 21, 2021

tylersmalley pushed a commit that referenced this issue May 18, 2021

skip suite failing in 7.13 (#96895)

a105c7a

mshustov added the triage_needed label May 19, 2021

yctercero pushed a commit to yctercero/kibana that referenced this issue May 25, 2021

skip suite failing in 7.13 (elastic#96895)

32093be

joshdover assigned Bamieh Jun 10, 2021

Bamieh mentioned this issue Jun 28, 2021

[Migrations V2] Unskip migration_7.7.2_xpack_100k #103435

Merged

Bamieh closed this as completed in #103435 Jun 29, 2021

mistic added a commit that referenced this issue Jul 30, 2021

skip flaky suite (#96895)

f21c7e1

mistic added a commit that referenced this issue Jul 30, 2021

skip flaky suite (#96895)

e23a033

mistic added v8.0.0 and removed v7.13.0 labels Jul 30, 2021

mistic reopened this Jul 30, 2021

mshustov mentioned this issue Aug 2, 2021

Disable reporting for all "oss: false" integration tests #107360

Merged

mshustov closed this as completed in #107360 Aug 2, 2021

streamich pushed a commit to vadimkibana/kibana that referenced this issue Aug 8, 2021

skip flaky suite (elastic#96895)

5e281d4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failing test: Jest Integration Tests.src/core/server/saved_objects/migrationsv2/integration_tests - migration from 7.7.2-xpack with 100k objects copies all the document of the previous index to the new one #96895

Failing test: Jest Integration Tests.src/core/server/saved_objects/migrationsv2/integration_tests - migration from 7.7.2-xpack with 100k objects copies all the document of the previous index to the new one #96895

kibanamachine commented Apr 12, 2021 •

edited

Loading

elasticmachine commented Apr 21, 2021

spalger commented Apr 21, 2021

kibanamachine commented Apr 21, 2021

pgayvallet commented Apr 22, 2021 •

edited

Loading

afharo commented Apr 22, 2021

delvedor commented Apr 22, 2021

spalger commented Apr 22, 2021

mshustov commented Apr 23, 2021

spalger commented Apr 26, 2021

kibanamachine commented May 1, 2021

kibanamachine commented May 18, 2021

tylersmalley commented May 18, 2021 •

edited

Loading

mistic commented Jul 30, 2021

Failing test: Jest Integration Tests.src/core/server/saved_objects/migrationsv2/integration_tests - migration from 7.7.2-xpack with 100k objects copies all the document of the previous index to the new one #96895

Failing test: Jest Integration Tests.src/core/server/saved_objects/migrationsv2/integration_tests - migration from 7.7.2-xpack with 100k objects copies all the document of the previous index to the new one #96895

Comments

kibanamachine commented Apr 12, 2021 • edited Loading

elasticmachine commented Apr 21, 2021

spalger commented Apr 21, 2021

kibanamachine commented Apr 21, 2021

pgayvallet commented Apr 22, 2021 • edited Loading

afharo commented Apr 22, 2021

delvedor commented Apr 22, 2021

spalger commented Apr 22, 2021

mshustov commented Apr 23, 2021

spalger commented Apr 26, 2021

kibanamachine commented May 1, 2021

kibanamachine commented May 18, 2021

tylersmalley commented May 18, 2021 • edited Loading

mistic commented Jul 30, 2021

kibanamachine commented Apr 12, 2021 •

edited

Loading

pgayvallet commented Apr 22, 2021 •

edited

Loading

tylersmalley commented May 18, 2021 •

edited

Loading