Saved object migrations

[chrisdavies]

This PR introduces Kibana index migrations, per issue 15100.

Implementation details can be found in the readme.

Briefly, migrations are the mechanism for upgrading Kibana saved object documents and mappings from one version of Kibana to another.

Currently, the bulk of the migration logic is in place, and is expected to be relatively stable (thus ready for an initial review). But there are still tests to be written, particularly integration tests.

Testing

Some scenarios to consider:

• Run Kibana with no migrations defined?
• Define some migrations and boot Kibana without any .kibana index?
• Define some migrations and boot Kibana on an existing index?
• Add subsequent migrations, do they all apply?
• Write a migration that throws an exception. Does the system do what you'd expect?
• Run multiple Kibana instances that attempt to run migrations.
• Disable a plugin that has migrations defined
  • Can you import documents associated with that plugin?
  • Can you re-enable the plugin?
  • Do things work as expected?
• Can you write a migration that changes a document's type from, say type: 'foo' to type: 'bar'?

To test this, the best thing to do is to write your own test migrations, probably adding them to an existing plugin.

I've done this in the following way in src/core_plugins/kibana:

Modify mappings.json to have a new, test document type:

"foo": {
  "properties": {
    "meaning": { "type": "integer" }
  }
},

Start Kibana in REPL mode and create some docs:

yarn run --repl

var callCluster = kbnServer.server.plugins.elasticsearch.getCluster('admin').callWithInternalUser;

callCluster('index', { index: '.kibana', type: 'doc', id: 'foo:a', body: { type: 'foo', foo: { meaning: 1 } } });
callCluster('index', { index: '.kibana', type: 'doc', id: 'foo:b', body: { type: 'foo', foo: { meaning: 2 } } });
callCluster('index', { index: '.kibana', type: 'doc', id: 'foo:c', body: { type: 'foo', foo: { meaning: 3 } } });

To index.js, import lodash (for convenience):

import _ from 'lodash';

Modify the uiExports object to have some migrations:

migrations: {
  foo: {
    '1.0.0': doc => _.set(doc, 'attributes.meaning', doc.attributes.meaning + 1),
    '1.0.1': doc => _.set(doc, 'attributes.meaning', doc.attributes.meaning * 2),
  },
},

Save and wait for Kibana to reload. It should run migrations, and you should be able to query and see that the docs were updated:

var callCluster = kbnServer.server.plugins.elasticsearch.getCluster('admin').callWithInternalUser;

callCluster('search', { index: '.kibana', type: 'doc', body: { query: { term: { type: 'foo' } } } });

As you tinker, you'll likely get into a state where migrations fail, in which case, there will be an invalid index lingering around that will prevent migrations from running again. So, this little CURL is your friend (replacing the index name as appropriate):

curl -X DELETE "http://localhost:9200/.kibana_5"

Or if you have Kibana running in REPL mode:

var callCluster = kbnServer.server.plugins.elasticsearch.getCluster('admin').callWithInternalUser;
callCluster('indices.delete', { index: '.kibana_5' });

Right now, Kibana won't attempt to re-run migrations until you've restarted it. It would be fairly simple to have the migration polling mechanism detect that the index was removed, and automatically attempt to re-run migrations. The question is, should this be a manual thing (e.g. restart) or should it be automatic?

[tylersmalley]

Went through the recent changes and everything there looks good. While doing some final testing I noticed that an invalid (future) version for a saved objects throws a 500. This is particularly problematic for the import as we should provide a clearer explanation for the failure.

[chrisdavies]

@tylersmalley That's a good catch. I'll take a look at boomifying it.

[chrisdavies]

@tylersmalley Should be good. I wonder if there are other errors in the stack that are worth boomifying. I also caught a path that could cause an infinite loop! Yeesh.

[tylersmalley]

@chrisdavies getting Document "b7331100-afec-11e8-a3b4-dddba010a6a3" has property "visualization" with version "7.1.0" but only "0.0.0" or lower is supported by this version of Kibana.. I assume it's because the document being updated doesn't have a migrationVersion yet. Would it be safe to default that to the current version of Kibana if it's not defined?

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[tylersmalley]

Much better: Document "b7331100-afec-11e8-a3b4-dddba010a6a3" has property "visualization" which belongs to a more recent version of Kibana (7.1.0).