[APM] Make UI indices space aware (support for spaces)

[cauemarcondes]

related to #126176

A few more changes had to be made to fully support the new structure of the saved object.

I added a new migration to the saved object to add the new layer (apmIndices), this will also apply when saved objects are imported.

migrateLegacyAPMIndicesToSpaceAware doesn't have to care about the structure of the saved object, since it will be already using the latest one, that happened during saved object migration.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sorenlouv]

lgtm but I'll defer to @jportner

[cauemarcondes]

@elasticmachine merge upstream

[jportner]

I pulled this PR down to test it --

I exported an apm-indices object from 8.1.0 and attempted to import it into Kibana using the main branch, and I got an error:

That's because Kibana didn't change the shape of the object's attributes before creating it, and the mappings are set to dynamic: strict by default.

Then I switched to this branch and attempted to import it again, and it worked 🎉
I tested importing in the Default space and a custom space, and changing the object in the custom space.
I also tested the on-start migration.

---

Since I noticed the mappings, it reminded me of this: if you don't need to search or aggregate on these fields, then you should exclude them and use dynamic: false to reduce the size of the .kibana index mapping.

From our Saved Objects docs:

Do not use field mappings like you would use data types for the columns of a SQL database. Instead, field mappings are analogous to a SQL index. Only specify field mappings for the fields you wish to search on or query. By specifying dynamic: false in any level of your mappings, Elasticsearch will accept and store any other fields even if they are not specified in your mappings.

Since Elasticsearch has a default limit of 1000 fields per index, plugins should carefully consider the fields they add to the mappings. Similarly, Saved Object types should never use dynamic: true as this can cause an arbitrary amount of fields to be added to the .kibana index.

diff --git a/x-pack/plugins/apm/server/saved_objects/apm_indices.ts b/x-pack/plugins/apm/server/saved_objects/apm_indices.ts
index 725fdfeb63a..3f02e90a95e 100644
--- a/x-pack/plugins/apm/server/saved_objects/apm_indices.ts
+++ b/x-pack/plugins/apm/server/saved_objects/apm_indices.ts
@@ -8,7 +8,6 @@
 import { SavedObjectsType } from 'src/core/server';
 import { i18n } from '@kbn/i18n';
 import { updateApmOssIndexPaths } from './migrations/update_apm_oss_index_paths';
-import { ApmIndicesConfigName } from '..';
 
 export interface APMIndices {
   apmIndices?: {
@@ -22,32 +21,14 @@ export interface APMIndices {
   isSpaceAware?: boolean;
 }
 
-const properties: {
-  apmIndices: {
-    properties: {
-      [Property in ApmIndicesConfigName]: { type: 'keyword' };
-    };
-  };
-  isSpaceAware: { type: 'boolean' };
-} = {
-  apmIndices: {
-    properties: {
-      sourcemap: { type: 'keyword' },
-      error: { type: 'keyword' },
-      onboarding: { type: 'keyword' },
-      span: { type: 'keyword' },
-      transaction: { type: 'keyword' },
-      metric: { type: 'keyword' },
-    },
-  },
-  isSpaceAware: { type: 'boolean' },
-};
-
 export const apmIndices: SavedObjectsType = {
   name: 'apm-indices',
   hidden: false,
   namespaceType: 'single',
-  mappings: { properties },
+  mappings: {
+    dynamic: false,
+    properties: {}, // several fields exist, but we don't need to search or aggregate on them, so we exclude them from the mappings
+  },
   management: {
     importableAndExportable: true,
     icon: 'apmApp',

I also noticed what I think is an AuthZ problem for minimally privileged users but I'll talk to you about that offline!

Edit: we spoke about this, I opened a separate issue for it here: #126483

[cauemarcondes]

Since I noticed the mappings, it reminded me of this: if you don't need to search or aggregate on these fields, then you should exclude them and use dynamic: false to reduce the size of the .kibana index mapping.

@jportner I just applied the changes you suggested. I decided to fix the other bug on a new PR since it is not related to this change.

[jportner]

@jportner I just applied the changes you suggested. I decided to fix the other bug on a new PR since it is not related to this change.

In that case, I think this PR is good to go!

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: d75a6cc
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
apm-indices	9	-	-9

History

• 💚 Build #26525 succeeded ba0f252
• 💚 Build #26309 succeeded 4a15d88

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream