[Security Solution] Data Quality dashboard storage metrics #155581
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andrew-goldstein
added
release_note:enhancement
backport:skip
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
Files by Code Ownerelastic/security-threat-hunting-explore
|
andrew-goldstein
force-pushed
the
data-quality-storage-metrics
branch
from
ba5f1d2
to
64b3bb6
Compare
semd
reviewed
Apr 24, 2023
andrew-goldstein
force-pushed
the
data-quality-storage-metrics
branch
from
2413edd
to
01b06fc
Compare
| expect(getValidValues(field)).toEqual([ | ||
| 'authentication', | ||
| 'configuration', | ||
| // no entry for 'database' |
There was a problem hiding this comment.
these inline comments for what's changed are helpful 👌🏾
| updatePatternIndexNames={updatePatternIndexNames} | ||
| updatePatternRollup={updatePatternRollup} | ||
| /> | ||
| {i !== patterns.length - 1 ? ( |
There was a problem hiding this comment.
it's the same thing, but you could do patterns[i + 1] ? (<Spacer>) : null or maybe just patterns[i + 1] && <Spacer>
| value != null ? numeral(value).format(defaultBytesFormat) : EMPTY_STAT; | ||
|
|
||
| const ilmPhases = ['hot', 'warm', 'unmanaged']; | ||
| const patterns = ['.alerts-security.alerts-default', 'auditbeat-*', 'packetbeat-*']; |
There was a problem hiding this comment.
Nit and def not necessary, but with discussions around alerts index name changes, may be better to use the ${DEFAULT_ALERTS_INDEX}-default here.
There was a problem hiding this comment.
i'm reluctant to use ${DEFAULT_ALERTS_INDEX}-default in the context of these tests (and others) because artifacts of the current value, .alerts-security.alerts-default, are ubiquitous in the mock (stats and ILM) data, and in most of the tests.
In most contexts, there's nothing magic about .alerts-security.alerts-default other than it's currently a "foreign key" in the mock data. If the value behind the ${DEFAULT_ALERTS_INDEX}-default changes, tests may start failing (with false positives) unless the mock data is manually updated via search & replace.
There was a problem hiding this comment.
Yea, took me a sec to realize all of these are just in test files, so doesn't really matter what it is as long as it aligns with the mocks. Ignore me...I've been seeing the index pattern everywhere so I'm very sensitive to it 😂
| pattern: string; | ||
| flattenedBuckets: FlattenedBucket[]; | ||
| }): LegendItem[] => | ||
| sortBy( |
There was a problem hiding this comment.
You can use orderBy to sort in descending order without the additional reverse. Not sure which is more performant though
| error.toString() != null | ||
| ? error.toString() | ||
| : i18n.AN_ERROR_OCCURRED_CHECKING_INDEX(indexName), | ||
| error: error != null ? error.toString() : i18n.AN_ERROR_OCCURRED_CHECKING_INDEX(indexName), |
| return sortedPatterns.reduce<IndexToCheck[]>((acc, pattern) => { | ||
| const indexNames = patternIndexNames[pattern] ?? []; | ||
| const indexNames = patternIndexNames[pattern]; |
There was a problem hiding this comment.
you can use orderBy in this function as well
| `| ${RESULT} | ${INDEX} | ${DOCS} | ${INCOMPATIBLE_FIELDS} | ${ILM_PHASE} | | ||
| |--------|-------|------|---------------------|-----------|`; | ||
| `| ${RESULT} | ${INDEX} | ${DOCS} | ${INCOMPATIBLE_FIELDS} | ${ILM_PHASE} | ${SIZE} | | ||
| |--------|-------|------|---------------------|-----------|------|`; |
There was a problem hiding this comment.
SUPER random thought, but you can have a space-builder so it's easier to get an idea of the numerical spacing.
Something like ${addSpace(8)} and const addSpace = (spaces: number) => Array(spaces).fill('-').join('');. DEFINITELY over-engineered, but couldn't help myself
| ? allIndexes.reduce<number>((acc, incompatible) => acc + Number(incompatible), 0) | ||
| : undefined; | ||
| if (sortByDirection === 'desc') { | ||
| return sortBy(sortByColumn, summaryTableItems).reverse(); |
There was a problem hiding this comment.
I'll stop commenting on these after this.. re the orderBy vs sort().reverse(). And once again, not sure which would be more performant
| <EuiFlexGroup data-test-subj="storageTreemap" gutterSize="none"> | ||
| <ChartFlexItem grow={true} $maxChartHeight={maxChartHeight} $minChartHeight={minChartHeight}> | ||
| {flattenedBuckets.length === 0 ? ( | ||
| <NoData reason="" /> |
There was a problem hiding this comment.
Do you need to pass the reason=""? or keep it the same as line 143?
|
|
||
| const NoDataComponent: React.FC<Props> = ({ reason }) => ( | ||
| <EuiFlexGroup alignItems="center" gutterSize="none"> | ||
| <EuiFlexItem grow={true}> |
There was a problem hiding this comment.
nit: you should be able to just pass grow
| * 2.0. | ||
| */ | ||
|
|
||
| export const mockUnallowedValuesResponse = [ |
There was a problem hiding this comment.
nit: missed the 'n' in unallowed for the name of this file :) and file path :)
| @@ -31,7 +35,21 @@ export const getTotalDocsCount = ( | |||
|
|
|||
| // only return the total when all `PatternRollup`s have a `docsCount`: | |||
| return allRollupsHaveDocsCount | |||
| ? allRollups.reduce((acc, { docsCount }) => acc + (docsCount ?? 0), 0) | |||
| ? allRollups.reduce((acc, { docsCount }) => acc + Number(docsCount), 0) | |||
There was a problem hiding this comment.
Is there ever a scenario where we would want to show partial results if one of the indices didn't return a value? Not necessary now, but just a thought. We'll see how often users run into this scenario...
michaelolo24
approved these changes
Apr 24, 2023
There was a problem hiding this comment.
The test coverage here is excellent. Thank you for adding them. Desk tested locally across multiple spaces with all ilm phases selected and de-selected and it works great! The tooltips all have the text as expected and there are no signs of the beta tag as far as I can tell. Nice work! 🚀 Just added some minor nits and questions.
 _Above: The new storage metrics treemap updates as indices are checked_  _Above: Storage metrics in the Data Quality dashboard_ ## Summary This PR introduces [storage metrics](elastic/security-team#6047) to the _Data Quality_ dashboard - Multiple views are enhanced to display the size of indices - A new interactive treemap visualizes the relative sizes of indices - Markdown reports include the size of indices - The Data Quality dashboard `Beta` tag is removed - Inline action buttons replace the `Take action` popover - The Global stats panel remains visible when the `Select one or more ILM phases` help is displayed - Code coverage is improved throughout the dashboard ## Details ### Multiple views enhanced to display the size of indices The following views have been enhanced to display the `Size` of indices, per the screenshots below: - The pattern table's `Size` column displays the size of a single index  - The pattern table's `Size` tooltip  - The pattern rollup's `Size` stat displays the total size of indices in a pattern  - The pattern rollup's `Size` stat tooltip  - The global stats rollup `Size` stat displays the total size of all the patterns  - The global stats rollup `Size` stat tooltip  ### New interactive treemap A new interactive treemap visualizes the relative sizes of indices: - The color of indices in the treemap and its legend update as the data is checked  - Clicking on an index in the treemap or the legend expands (and scrolls to) the index ### Markdown reports include the `Size` of indices Markdown reports are enhanced to include the new `Size` statistic in: - Pattern markdown tables | Result | Index | Docs | Incompatible fields | ILM Phase | Size | |--------|-------|------|---------------------|-----------|------| | ❌ | auditbeat-7.14.2-2023.04.09-000001 | 48,077 (4.3%) | 12 | `hot` | 41.3MB | | ❌ | auditbeat-7.3.2-2023.04.09-000001 | 48,068 (4.3%) | 8 | `hot` | 31MB | | ❌ | auditbeat-7.11.2-2023.04.09-000001 | 48,064 (4.3%) | 12 | `hot` | 40.8MB | - Pattern rollup markdown tables | Incompatible fields | Indices checked | Indices | Size | Docs | |---------------------|-----------------|---------|------|------| | 164 | 26 | 26 | 899.3MB | 1,118,155 | - The global stats markdown table | Incompatible fields | Indices checked | Indices | Size | Docs | |---------------------|-----------------|---------|------|------| | 166 | 32 | 32 | 9.2GB | 20,779,245 | ### Data Quality dashboard `Beta` tag removed The Data Quality dashboard `Beta` tag is removed from the following views: - The `Dashboards` page **Before:**  **After:**  - Security Solution side navigation **Before:**  **After:**  - The Data Quality dashboard page header **Before:**  **After:**  ### Inline action buttons replace the `Take action` popover Inline `Add to new case` and `Copy to clipboard` action buttons replace the `Take action` popover, the previous home of these actions: **Before:**  **After:**  ### Global stats panel remains visible when the `Select one or more ILM phases` help is displayed The Global stats panel now remains visible when the `Select one or more ILM phases` help is displayed: **Before:**  **After:**  ### Code coverage improvements Code coverage is improved throughout the dashboard, as measured by running the following command: ```sh node scripts/jest --watch x-pack/packages/kbn-ecs-data-quality-dashboard --coverage ```
…on/public/overview/links.ts`
andrew-goldstein
force-pushed
the
data-quality-storage-metrics
branch
from
01b06fc
to
177eed0
Compare
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Module Count
Async chunks
Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: |
nikitaindik
pushed a commit
to nikitaindik/kibana
that referenced
this pull request
Apr 25, 2023
…55581) # [Security Solution] Data Quality dashboard storage metrics  _Above: The new storage metrics treemap updates as indices are checked_  _Above: Storage metrics in the Data Quality dashboard_ ## Summary This PR introduces [storage metrics](elastic/security-team#6047) to the _Data Quality_ dashboard - Multiple views are enhanced to display the size of indices - A new interactive treemap visualizes the relative sizes of indices - Markdown reports include the size of indices - The Data Quality dashboard `Beta` tag is removed - Inline action buttons replace the `Take action` popover - The Global stats panel remains visible when the `Select one or more ILM phases` help is displayed - Code coverage is improved throughout the dashboard ## Details ### Multiple views enhanced to display the size of indices The following views have been enhanced to display the `Size` of indices, per the screenshots below: - The pattern table's `Size` column displays the size of a single index  - The pattern table's `Size` tooltip  - The pattern rollup's `Size` stat displays the total size of indices in a pattern  - The pattern rollup's `Size` stat tooltip  - The global stats rollup `Size` stat displays the total size of all the patterns  - The global stats rollup `Size` stat tooltip  ### New interactive treemap A new interactive treemap visualizes the relative sizes of indices: - The color of indices in the treemap and its legend update as the data is checked  - Clicking on an index in the treemap or the legend expands (and scrolls to) the index ### Markdown reports include the `Size` of indices Markdown reports are enhanced to include the new `Size` statistic in: - Pattern markdown tables | Result | Index | Docs | Incompatible fields | ILM Phase | Size | |--------|-------|------|---------------------|-----------|------| | ❌ | auditbeat-7.14.2-2023.04.09-000001 | 48,077 (4.3%) | 12 | `hot` | 41.3MB | | ❌ | auditbeat-7.3.2-2023.04.09-000001 | 48,068 (4.3%) | 8 | `hot` | 31MB | | ❌ | auditbeat-7.11.2-2023.04.09-000001 | 48,064 (4.3%) | 12 | `hot` | 40.8MB | - Pattern rollup markdown tables | Incompatible fields | Indices checked | Indices | Size | Docs | |---------------------|-----------------|---------|------|------| | 164 | 26 | 26 | 899.3MB | 1,118,155 | - The global stats markdown table | Incompatible fields | Indices checked | Indices | Size | Docs | |---------------------|-----------------|---------|------|------| | 166 | 32 | 32 | 9.2GB | 20,779,245 | ### Data Quality dashboard `Beta` tag removed The Data Quality dashboard `Beta` tag is removed from the following views: - The `Dashboards` page **Before:**  **After:**  - Security Solution side navigation **Before:**  **After:**  - The Data Quality dashboard page header **Before:**  **After:**  ### Inline action buttons replace the `Take action` popover Inline `Add to new case` and `Copy to clipboard` action buttons replace the `Take action` popover, the previous home of these actions: **Before:**  **After:**  ### Global stats panel remains visible when the `Select one or more ILM phases` help is displayed The Global stats panel now remains visible when the `Select one or more ILM phases` help is displayed: **Before:**  **After:**  ### Code coverage improvements Code coverage is improved throughout the dashboard, as measured by running the following command: ```sh node scripts/jest --watch x-pack/packages/kbn-ecs-data-quality-dashboard --coverage ```
|
@andrew-goldstein could we use the following screenshot in the 8.8 release blog? |
yes @dhru42, thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Security Solution] Data Quality dashboard storage metrics
Above: The new storage metrics treemap updates as indices are checked
Above: Storage metrics in the Data Quality dashboard
Summary
This PR introduces storage metrics to the Data Quality dashboard
Multiple views are enhanced to display the size of indices
A new interactive treemap visualizes the relative sizes of indices
Markdown reports include the size of indices
The Data Quality dashboard
Betatag is removedInline action buttons replace the
Take actionpopoverThe Global stats panel remains visible when the
Select one or more ILM phaseshelp is displayedCode coverage is improved throughout the dashboard
Details
Multiple views enhanced to display the size of indices
The following views have been enhanced to display the
Sizeof indices, per the screenshots below:Sizecolumn displays the size of a single indexSizetooltipSizestat displays the total size of indices in a patternSizestat tooltipSizestat displays the total size of all the patternsSizestat tooltipNew interactive treemap
A new interactive treemap visualizes the relative sizes of indices:
Markdown reports include the
Sizeof indicesMarkdown reports are enhanced to include the new
Sizestatistic in:hothothotData Quality dashboard
Betatag removedThe Data Quality dashboard
Betatag is removed from the following views:DashboardspageBefore:
After:
Before:
After:
Before:
After:
Inline action buttons replace the
Take actionpopoverInline
Add to new caseandCopy to clipboardaction buttons replace theTake actionpopover, the previous home of these actions:Before:
After:
Global stats panel remains visible when the
Select one or more ILM phaseshelp is displayedThe Global stats panel now remains visible when the
Select one or more ILM phaseshelp is displayed:Before:
After:
Code coverage improvements
Code coverage is improved throughout the dashboard, as measured by running the following command: