# [Security Solution] Data Quality dashboard storage metrics

![storage_metrics_animated](https://user-images.githubusercontent.com/4459398/233871314-6894b380-63ac-4622-b64f-965752a96019.gif) _Above: The new storage metrics treemap updates as indices are checked_ ![storage_metrics](https://user-images.githubusercontent.com/4459398/233880225-8242733a-4bd6-40b3-bffa-e283ce0d77cd.png) _Above: Storage metrics in the Data Quality dashboard_ ## Summary This PR introduces [storage metrics](elastic/security-team#6047) to the _Data Quality_ dashboard - Multiple views are enhanced to display the size of indices - A new interactive treemap visualizes the relative sizes of indices - Markdown reports include the size of indices - The Data Quality dashboard `Beta` tag is removed - Inline action buttons replace the `Take action` popover - The Global stats panel remains visible when the `Select one or more ILM phases` help is displayed - Code coverage is improved throughout the dashboard ## Details ### Multiple views enhanced to display the size of indices The following views have been enhanced to display the `Size` of indices, per the screenshots below: - The pattern table's `Size` column displays the size of a single index ![04_size_column](https://user-images.githubusercontent.com/4459398/233870161-d86eadbd-9f01-4ed6-aa6f-98f6044a4f57.png) - The pattern table's `Size` tooltip ![05_size_column_tooltip](https://user-images.githubusercontent.com/4459398/233868732-08059ba9-5e4b-4f68-a152-eb4b41db6f96.png) - The pattern rollup's `Size` stat displays the total size of indices in a pattern ![06_pattern_rollups_size](https://user-images.githubusercontent.com/4459398/233868817-babc96eb-c0aa-4b7f-bb45-54e3039d06f2.png) - The pattern rollup's `Size` stat tooltip ![07_pattern_rollups_size_tooltip](https://user-images.githubusercontent.com/4459398/233868858-14a43aa2-324f-40bd-a185-1cb7ac15c81b.png) - The global stats rollup `Size` stat displays the total size of all the patterns ![08_global_stats_rollup_size](https://user-images.githubusercontent.com/4459398/233868900-e3cbc00b-3b5a-4756-8246-cb31a1b8bac8.png) - The global stats rollup `Size` stat tooltip ![09_global_stats_rollup_size_tooltip](https://user-images.githubusercontent.com/4459398/233868952-b9c27432-c8a4-4ad5-9dda-5e1aa903758c.png) ### New interactive treemap A new interactive treemap visualizes the relative sizes of indices: - The color of indices in the treemap and its legend update as the data is checked ![storage_metrics_animated](https://user-images.githubusercontent.com/4459398/233871314-6894b380-63ac-4622-b64f-965752a96019.gif) - Clicking on an index in the treemap or the legend expands (and scrolls to) the index ### Markdown reports include the `Size` of indices Markdown reports are enhanced to include the new `Size` statistic in: - Pattern markdown tables | Result | Index | Docs | Incompatible fields | ILM Phase | Size | |--------|-------|------|---------------------|-----------|------| | ❌ | auditbeat-7.14.2-2023.04.09-000001 | 48,077 (4.3%) | 12 | `hot` | 41.3MB | | ❌ | auditbeat-7.3.2-2023.04.09-000001 | 48,068 (4.3%) | 8 | `hot` | 31MB | | ❌ | auditbeat-7.11.2-2023.04.09-000001 | 48,064 (4.3%) | 12 | `hot` | 40.8MB | - Pattern rollup markdown tables | Incompatible fields | Indices checked | Indices | Size | Docs | |---------------------|-----------------|---------|------|------| | 164 | 26 | 26 | 899.3MB | 1,118,155 | - The global stats markdown table | Incompatible fields | Indices checked | Indices | Size | Docs | |---------------------|-----------------|---------|------|------| | 166 | 32 | 32 | 9.2GB | 20,779,245 | ### Data Quality dashboard `Beta` tag removed The Data Quality dashboard `Beta` tag is removed from the following views: - The `Dashboards` page **Before:** ![11_dashboards_before](https://user-images.githubusercontent.com/4459398/233869434-d4d2ed14-4e6f-4eab-bae6-a9c9b976e20f.png) **After:** ![12_dashboards_after](https://user-images.githubusercontent.com/4459398/233869088-9dc62d7d-44cb-46cb-8880-976a7b7e9c56.png) - Security Solution side navigation **Before:** ![13_side_navigation_before](https://user-images.githubusercontent.com/4459398/233869467-e7725285-1199-40e1-ac65-054bea8b02f6.png) **After:** ![14_side_navigation_after](https://user-images.githubusercontent.com/4459398/233869146-7b89cb47-3509-478e-8675-9f1653749b18.png) - The Data Quality dashboard page header **Before:** ![15_page_header_before](https://user-images.githubusercontent.com/4459398/233869404-0b04c2ec-3d2e-4ba8-9520-68013f80e43a.png) **After:** ![16_page_header_after](https://user-images.githubusercontent.com/4459398/233869219-b54ee61e-07b7-470d-a668-b4f5ed4327e6.png) ### Inline action buttons replace the `Take action` popover Inline `Add to new case` and `Copy to clipboard` action buttons replace the `Take action` popover, the previous home of these actions: **Before:** ![17_actions_before](https://user-images.githubusercontent.com/4459398/233869306-0182145f-affc-4ad1-b63f-72e43d34234c.png) **After:** ![18_actions_after](https://user-images.githubusercontent.com/4459398/233869345-754b7448-9d28-4253-9186-5b2389acf4ff.png) ### Global stats panel remains visible when the `Select one or more ILM phases` help is displayed The Global stats panel now remains visible when the `Select one or more ILM phases` help is displayed: **Before:** ![19_select_ilm_phases_before](https://user-images.githubusercontent.com/4459398/233869754-2067fa5d-7153-407b-aa45-65332b16bc7a.png) **After:** ![20_select_ilm_phases_after](https://user-images.githubusercontent.com/4459398/233869762-38d069de-3191-4e28-8692-df42ab3b21a5.png) ### Code coverage improvements Code coverage is improved throughout the dashboard, as measured by running the following command: ```sh node scripts/jest --watch x-pack/packages/kbn-ecs-data-quality-dashboard --coverage ```
elastic · Apr 25, 2023 · 6c4b5ca · 6c4b5ca
1 parent 675ed0e
commit 6c4b5ca
Show file tree

Hide file tree

Showing 128 changed files with 12,098 additions and 849 deletions.
diff --git a/...uality-dashboard/impl/data_quality/compare_fields_table/ecs_allowed_values/index.test.tsx b/...uality-dashboard/impl/data_quality/compare_fields_table/ecs_allowed_values/index.test.tsx
@@ -0,0 +1,53 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { render, screen } from '@testing-library/react';
+import React from 'react';
+
+import { mockAllowedValues } from '../../mock/allowed_values/mock_allowed_values';
+import { TestProviders } from '../../mock/test_providers/test_providers';
+import { EcsAllowedValues } from '.';
+
+describe('EcsAllowedValues', () => {
+  describe('when `allowedValues` exists', () => {
+    beforeEach(() => {
+      render(
+        <TestProviders>
+          <EcsAllowedValues allowedValues={mockAllowedValues} />
+        </TestProviders>
+      );
+    });
+
+    test('it renders the allowed values', () => {
+      expect(screen.getByTestId('ecsAllowedValues')).toHaveTextContent(
+        mockAllowedValues.map(({ name }) => `${name}`).join('')
+      );
+    });
+
+    test('it does NOT render the placeholder', () => {
+      expect(screen.queryByTestId('ecsAllowedValuesEmpty')).not.toBeInTheDocument();
+    });
+  });
+
+  describe('when `allowedValues` is undefined', () => {
+    beforeEach(() => {
+      render(
+        <TestProviders>
+          <EcsAllowedValues allowedValues={undefined} />
+        </TestProviders>
+      );
+    });
+
+    test('it does NOT render the allowed values', () => {
+      expect(screen.queryByTestId('ecsAllowedValues')).not.toBeInTheDocument();
+    });
+
+    test('it renders the placeholder', () => {
+      expect(screen.getByTestId('ecsAllowedValuesEmpty')).toBeInTheDocument();
+    });
+  });
+});
diff --git a/...get_common_table_columns/helpers.test.tsx → ...e/get_common_table_columns/index.test.tsx b/...get_common_table_columns/helpers.test.tsx → ...e/get_common_table_columns/index.test.tsx
@@ -10,57 +10,57 @@ import { omit } from 'lodash/fp';
 import React from 'react';
 
 import { SAME_FAMILY } from '../../data_quality_panel/same_family/translations';
-import { TestProviders } from '../../mock/test_providers';
 import {
   eventCategory,
   eventCategoryWithUnallowedValues,
-} from '../../mock/enriched_field_metadata';
+} from '../../mock/enriched_field_metadata/mock_enriched_field_metadata';
+import { TestProviders } from '../../mock/test_providers/test_providers';
+import {
+  DOCUMENT_VALUES_ACTUAL,
+  ECS_DESCRIPTION,
+  ECS_MAPPING_TYPE_EXPECTED,
+  ECS_VALUES_EXPECTED,
+  FIELD,
+  INDEX_MAPPING_TYPE_ACTUAL,
+} from '../translations';
 import { EnrichedFieldMetadata } from '../../types';
 import { EMPTY_PLACEHOLDER, getCommonTableColumns } from '.';
 
 describe('getCommonTableColumns', () => {
   test('it returns the expected column configuration', () => {
-    const columns = getCommonTableColumns().map((x) => omit('render', x));
-
-    expect(columns).toEqual([
-      {
-        field: 'indexFieldName',
-        name: 'Field',
-        sortable: true,
-        truncateText: false,
-        width: '20%',
-      },
+    expect(getCommonTableColumns().map((x) => omit('render', x))).toEqual([
+      { field: 'indexFieldName', name: FIELD, sortable: true, truncateText: false, width: '20%' },
       {
         field: 'type',
-        name: 'ECS mapping type (expected)',
+        name: ECS_MAPPING_TYPE_EXPECTED,
         sortable: true,
         truncateText: false,
         width: '15%',
       },
       {
         field: 'indexFieldType',
-        name: 'Index mapping type (actual)',
+        name: INDEX_MAPPING_TYPE_ACTUAL,
         sortable: true,
         truncateText: false,
         width: '15%',
       },
       {
         field: 'allowed_values',
-        name: 'ECS values (expected)',
+        name: ECS_VALUES_EXPECTED,
         sortable: false,
         truncateText: false,
         width: '15%',
       },
       {
         field: 'indexInvalidValues',
-        name: 'Document values (actual)',
+        name: DOCUMENT_VALUES_ACTUAL,
         sortable: false,
         truncateText: false,
         width: '15%',
       },
       {
         field: 'description',
-        name: 'ECS description',
+        name: ECS_DESCRIPTION,
         sortable: false,
         truncateText: false,
         width: '20%',
@@ -141,7 +141,7 @@ describe('getCommonTableColumns', () => {
         const withTypeMismatchDifferentFamily: EnrichedFieldMetadata = {
           ...eventCategory, // `event.category` is a `keyword` per the ECS spec
           indexFieldType, // this index has a mapping of `text` instead of `keyword`
-          isInSameFamily: false, // `text` and `keyword` are not in the same family
+          isInSameFamily: false, // `text` and `wildcard` are not in the same family
         };
 
         render(
@@ -159,29 +159,18 @@ describe('getCommonTableColumns', () => {
     });
 
     describe('when the index field matches the ECS type', () => {
-      const indexFieldType = 'keyword';
-
       test('it renders the expected type with success styling', () => {
         const columns = getCommonTableColumns();
         const indexFieldTypeColumnRender = columns[2].render;
 
-        const withTypeMismatchDifferentFamily: EnrichedFieldMetadata = {
-          ...eventCategory, // `event.category` is a `keyword` per the ECS spec
-          indexFieldType, // exactly matches the ECS spec
-          isInSameFamily: true, // `keyword` is a member of the `keyword` family
-        };
-
         render(
           <TestProviders>
             {indexFieldTypeColumnRender != null &&
-              indexFieldTypeColumnRender(
-                withTypeMismatchDifferentFamily.indexFieldType,
-                withTypeMismatchDifferentFamily
-              )}
+              indexFieldTypeColumnRender(eventCategory.indexFieldType, eventCategory)}
           </TestProviders>
         );
 
-        expect(screen.getByTestId('codeSuccess')).toHaveTextContent(indexFieldType);
+        expect(screen.getByTestId('codeSuccess')).toHaveTextContent(eventCategory.indexFieldType);
       });
     });
   });

diff --git a/...e_mappings_table_columns/helpers.test.tsx → ...ble_mappings_table_columns/index.test.tsx b/...e_mappings_table_columns/helpers.test.tsx → ...ble_mappings_table_columns/index.test.tsx
@@ -10,8 +10,8 @@ import { omit } from 'lodash/fp';
 import React from 'react';
 
 import { SAME_FAMILY } from '../../data_quality_panel/same_family/translations';
-import { TestProviders } from '../../mock/test_providers';
-import { eventCategory } from '../../mock/enriched_field_metadata';
+import { TestProviders } from '../../mock/test_providers/test_providers';
+import { eventCategory } from '../../mock/enriched_field_metadata/mock_enriched_field_metadata';
 import { EnrichedFieldMetadata } from '../../types';
 import { EMPTY_PLACEHOLDER, getIncompatibleMappingsTableColumns } from '.';