Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Saved object migrations #20243

Merged
merged 123 commits into from
Sep 17, 2018
Merged
Show file tree
Hide file tree
Changes from 107 commits
Commits
Show all changes
123 commits
Select commit Hold shift + click to select a range
4411f7a
Move index management concerns into the saved object client
chrisdavies Jun 8, 2018
69a2632
Change ordering of saved object mixin so it can access elasticsearch
chrisdavies Jun 8, 2018
dc28022
Merge branch 'master' of github.com:elastic/kibana into migrations/sa…
chrisdavies Jun 11, 2018
d219188
Remove patchKibanaIndex from health_check test
chrisdavies Jun 11, 2018
ff5cb68
Move index init logic into saved object client factory
chrisdavies Jun 11, 2018
2d232c4
Fix initialze_saved_object_indices TypeScript error
chrisdavies Jun 11, 2018
85593a6
Simplify patch index logic a bit
chrisdavies Jun 11, 2018
0e49e4e
Remove types/jest from package.json in attempt to get build passing
chrisdavies Jun 12, 2018
494db0d
Merge branch 'master' of github.com:elastic/kibana into migrations/sa…
chrisdavies Jun 12, 2018
db8e0a9
Fix loosen strictness of migration mapping validations
chrisdavies Jun 12, 2018
ceb1529
Merge branch 'master' of github.com:elastic/kibana into migrations/sa…
chrisdavies Jun 12, 2018
a098084
Fix saved objects mixin ordering
chrisdavies Jun 13, 2018
027cc10
Making onBeforeWrite accessible to custom factories
kobelb Jun 14, 2018
29ecffc
Merge master
chrisdavies Jun 15, 2018
10ad3ba
Merge pull request #1 from kobelb/migrations/saved_objects
chrisdavies Jun 15, 2018
8d75d0a
Merge brandon's pr
chrisdavies Jun 15, 2018
723f2a3
Organizational tweak, add tests, add doc comments, tweak readme
chrisdavies Jun 15, 2018
a726fe0
Fix saved objects mixin's active mappings call
chrisdavies Jun 15, 2018
7f566f5
Merge master
chrisdavies Jun 25, 2018
b4bb4bc
Add core document transform logic
chrisdavies Jun 26, 2018
72a9a0a
Remove index patching from the saved object pre-write
chrisdavies Jun 26, 2018
87e4b75
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Jul 3, 2018
89f15c1
Add readers for retrieving docs from an index or array
chrisdavies Jul 3, 2018
7b4aaa4
Make more consistent calls to build_active_mappings
chrisdavies Jul 3, 2018
6c4b19e
Fix index export
chrisdavies Jul 3, 2018
43d865c
Add migration cluster coordination logic
chrisdavies Jul 3, 2018
171e635
Add migration bulk document writer
chrisdavies Jul 3, 2018
d87cc0c
Add support to saved / raw docs for supplementary properties
chrisdavies Jul 3, 2018
f616ba5
Logic to migrate saved object documents to the latest version
chrisdavies Jul 5, 2018
f32d20c
Remove knowledge of plugins from core migrations
chrisdavies Jul 5, 2018
c70838a
Fix saved objects mixin
chrisdavies Jul 5, 2018
bb7b7d0
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Jul 5, 2018
68db719
Initial index migration logic
chrisdavies Jul 12, 2018
7b8e399
Merge master
chrisdavies Jul 12, 2018
54c98e2
Fix problem with "this" being undefined
chrisdavies Jul 12, 2018
9d6f72d
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Jul 17, 2018
b3f65dc
Run migrations on import / export
chrisdavies Jul 17, 2018
a72d2eb
Add validation to the saved object client (server side)
chrisdavies Jul 18, 2018
4d2c573
Merge document validator code into migrations
chrisdavies Jul 18, 2018
c6cb8a3
Add doc validation to Kibana index migrations
chrisdavies Jul 18, 2018
bdedfb0
Remove the migration lock index logic
chrisdavies Jul 18, 2018
6cd918e
Handle null uiExports in Kibana migrator
chrisdavies Jul 18, 2018
b708c3a
Improve migration logging
chrisdavies Jul 18, 2018
c3fd212
Reorg of migration type definitions, removal of unused code
chrisdavies Jul 18, 2018
d52a8ee
A bit of refactoring and commenting
chrisdavies Jul 20, 2018
2583486
Update migrations readme
chrisdavies Jul 20, 2018
e9e9353
Fix a bug w/ index migrator
chrisdavies Jul 20, 2018
7162c74
Fix shakespeare functional test
chrisdavies Jul 20, 2018
b9838e8
Update es_archiver to migrate archives when they are loaded
chrisdavies Jul 20, 2018
7456364
Merge master
chrisdavies Jul 20, 2018
dba1662
Fix index migrator tests
chrisdavies Jul 20, 2018
3b4347b
Add basic integration tests for migration core logic
chrisdavies Jul 20, 2018
03581c5
Remove ref to docValidator in saved objects mixin.
chrisdavies Jul 23, 2018
43cecc5
Skip migrations if elasticsearch plugin is disabled.
chrisdavies Jul 23, 2018
60b6e58
Fix es_archiver stubs and alias tests
chrisdavies Jul 23, 2018
7a40d86
Move migration tests into saved_object folder.
chrisdavies Jul 23, 2018
f5f0809
Fix es_archiver alias stub
chrisdavies Jul 23, 2018
c6ad734
Update saved object repo tests to check for migrations
chrisdavies Jul 23, 2018
d02c67b
Fix flyout and dashboard import tests
chrisdavies Jul 24, 2018
ce171e2
Update server reload logging test to handle
chrisdavies Jul 24, 2018
586870a
Make saved objects bulk_get test aware of migrations
chrisdavies Jul 24, 2018
8471d03
Modify migrations to not set migrationVersion on docs
chrisdavies Jul 24, 2018
ff1494a
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Jul 24, 2018
aeaee8b
Merge master
chrisdavies Jul 24, 2018
89e2634
Merge master
chrisdavies Jul 24, 2018
531ceb9
Fix RBAC tests broken by migrations
chrisdavies Jul 25, 2018
5a9b936
Change migrations to pass invalid / non saved object
chrisdavies Jul 25, 2018
1e43afa
Remove obvious comments from elastic_index
chrisdavies Jul 25, 2018
644c221
Tweak the migrations readme.
chrisdavies Jul 25, 2018
b8841f5
Expand the document ownership example in the migration readme
chrisdavies Jul 25, 2018
38b0eb9
Tewak migration readme's code section to have less specificity
chrisdavies Jul 25, 2018
57c192a
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Aug 1, 2018
c38d764
Tweak migration coordinator log message
chrisdavies Aug 1, 2018
3f521a9
Add better error messages for invalid migration definitions
chrisdavies Aug 1, 2018
e03365e
Fix bug in the index up to date check for migrations. If there
chrisdavies Aug 1, 2018
b905634
Tidy up migration log messages
chrisdavies Aug 6, 2018
095d47b
Add debug logs of what saved objects are being migrated
chrisdavies Aug 6, 2018
d7a9b84
Refactor migrations per feedback, tweak logs, a bit of clean up,
chrisdavies Aug 7, 2018
71ad10f
Add logic to determine if mapping changes require an index migration
chrisdavies Aug 7, 2018
4a8f955
Fix migration integration tests, tweak readme
chrisdavies Aug 7, 2018
10b0222
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Aug 7, 2018
2011f1c
Require migrations if the index is not aliased
chrisdavies Aug 7, 2018
e34e4f8
Tweak migration result to have a distinct result for patch operations
chrisdavies Aug 7, 2018
753e820
Merge master
chrisdavies Aug 22, 2018
de53060
Fix ui_settings tests to handle kibana index being aliased.
chrisdavies Aug 22, 2018
4671f11
Remove kbn_server code that was duplicated while merging master
chrisdavies Aug 22, 2018
d7cc69f
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Aug 23, 2018
05887e0
Consolodate es + kibana launching logic that is the source
chrisdavies Aug 23, 2018
e03a863
Remove misleading comment
chrisdavies Aug 23, 2018
130457e
Move input validation to the top of startTestServers
chrisdavies Aug 23, 2018
3b18879
Merge branch 'flakytests/start_es_and_kibana' into migrations/core
chrisdavies Aug 23, 2018
b1ba1b3
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Aug 23, 2018
78e7227
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Aug 24, 2018
d9fc199
Merge master
chrisdavies Aug 29, 2018
4cacb87
Address initial feedback.
chrisdavies Aug 29, 2018
c6896d2
Merge master
chrisdavies Sep 4, 2018
475defa
Tweak comments, test the edge case where a saved object property has …
chrisdavies Sep 4, 2018
9b45b74
Remove the `.only` from index_migrator.test.ts
chrisdavies Sep 4, 2018
26e04e6
Code refinements per code review feedback, renaming various
chrisdavies Sep 5, 2018
51b3e5b
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Sep 5, 2018
5c2a5fd
Fix document migrator TypeScript error resultant from changing
chrisdavies Sep 5, 2018
1cd609f
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Sep 5, 2018
3ed23ba
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Sep 6, 2018
d4471ba
Finish unit tests for elastic_index
chrisdavies Sep 6, 2018
9f37115
Merge master
chrisdavies Sep 6, 2018
30541c6
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Sep 7, 2018
2b2dd34
Get rid of obsolete server start code.
chrisdavies Sep 7, 2018
f5f8451
Add a comment to saved object repository
chrisdavies Sep 7, 2018
dfbd4d3
Make a few adjustmens per Tyler's feedback
chrisdavies Sep 10, 2018
685fe2b
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Sep 10, 2018
a47096e
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Sep 11, 2018
c0b1970
Move saved object conversion logic to its own file, and
chrisdavies Sep 11, 2018
317242d
Centralize saved object logic to convert to / from raw document
chrisdavies Sep 11, 2018
329bc41
Fix broken saved object tests
chrisdavies Sep 11, 2018
fd099c9
Fix broken rbac tests
chrisdavies Sep 11, 2018
a1c9807
Log the failed document and transform if the transform fails
chrisdavies Sep 12, 2018
a1b5442
Move saved object to / from raw logic into its own saved object folder
chrisdavies Sep 12, 2018
dbcf3d7
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Sep 12, 2018
0dbf496
Merge master
chrisdavies Sep 13, 2018
5ea544e
Fix bug with importing future docs reporting a poor error.
chrisdavies Sep 14, 2018
efa0065
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Sep 14, 2018
119ebc4
Tweak the error message when a future doc is imported
chrisdavies Sep 17, 2018
0408cbf
Merge branch 'master' of github.com:elastic/kibana into migrations/core
chrisdavies Sep 17, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -235,9 +235,11 @@
"@types/prop-types": "^15.5.3",
"@types/react": "^16.3.14",
"@types/react-dom": "^16.0.5",
"@types/semver": "^5.5.0",
"@types/sinon": "^5.0.1",
"@types/react-redux": "^6.0.6",
"@types/redux": "^3.6.31",
"@types/redux-actions": "^2.2.1",
"@types/sinon": "^5.0.0",
"@types/strip-ansi": "^3.0.0",
"@types/supertest": "^2.0.5",
"@types/type-detect": "^4.0.1",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,16 @@ Object {
],
"type": "log",
},
Object {
"@timestamp": "## @timestamp ##",
"message": "The elasticsearch plugin is disabled. Skipping migrations.",
"pid": "## PID ##",
"tags": Array [
"warning",
"migration",
],
"type": "log",
},
Object {
"@timestamp": "## @timestamp ##",
"message": "Server running at http://localhost:8274",
Expand Down
2 changes: 0 additions & 2 deletions src/core_plugins/elasticsearch/lib/__tests__/health_check.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ const NoConnections = require('elasticsearch').errors.NoConnections;
import mappings from './fixtures/mappings';
import healthCheck from '../health_check';
import kibanaVersion from '../kibana_version';
import * as patchKibanaIndexNS from '../patch_kibana_index';

const esPort = 9220;
const esUrl = `http://elastic:changement@localhost:9220`;
Expand All @@ -49,7 +48,6 @@ describe('plugins/elasticsearch', () => {

// Stub the Kibana version instead of drawing from package.json.
sandbox.stub(kibanaVersion, 'get').returns(COMPATIBLE_VERSION_NUMBER);
sandbox.stub(patchKibanaIndexNS, 'patchKibanaIndex');

// setup the plugin stub
plugin = {
Expand Down
218 changes: 0 additions & 218 deletions src/core_plugins/elasticsearch/lib/__tests__/patch_kibana_index.js

This file was deleted.

9 changes: 1 addition & 8 deletions src/core_plugins/elasticsearch/lib/health_check.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ import Promise from 'bluebird';
import elasticsearch from 'elasticsearch';
import kibanaVersion from './kibana_version';
import { ensureEsVersion } from './ensure_es_version';
import { patchKibanaIndex } from './patch_kibana_index';

const NoConnections = elasticsearch.errors.NoConnections;

Expand Down Expand Up @@ -63,13 +62,7 @@ export default function (plugin, server) {
function check() {
const healthCheck =
waitForPong(callAdminAsKibanaUser, config.get('elasticsearch.url'))
.then(waitForEsVersion)
.then(() => patchKibanaIndex({
callCluster: callAdminAsKibanaUser,
log: (...args) => server.log(...args),
indexName: config.get('kibana.index'),
kibanaIndexMappingsDsl: server.getKibanaIndexMappingsDsl()
}));
.then(waitForEsVersion);

return healthCheck
.then(setGreenStatus)
Expand Down
Loading