Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Logs UI] [Alerting] "Group by" functionality #68250

Merged
merged 35 commits into from
Jun 30, 2020
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
a02b17d
Server side (executor) support for group by
Kerry350 Jun 2, 2020
dd92f5f
Add UI functionality
Kerry350 Jun 4, 2020
2f9f6cd
Ensure array entries
Kerry350 Jun 4, 2020
4246222
Merge remote-tracking branch 'upstream/master' into 67465-logs-alert-…
Kerry350 Jun 4, 2020
7563fac
Merge remote-tracking branch 'upstream/master' into 67465-logs-alert-…
Kerry350 Jun 5, 2020
9be25e6
Merge remote-tracking branch 'upstream/master' into 67465-logs-alert-…
Kerry350 Jun 5, 2020
b773abe
Amend current executor tests
Kerry350 Jun 5, 2020
8726524
Merge branch 'master' into 67465-logs-alert-group-by
elasticmachine Jun 8, 2020
7461571
Merge remote-tracking branch 'upstream/master' into 67465-logs-alert-…
Kerry350 Jun 10, 2020
54d0144
Merge remote-tracking branch 'upstream/master' into 67465-logs-alert-…
Kerry350 Jun 11, 2020
c9ef022
Server side amendments
Kerry350 Jun 12, 2020
27296a9
Client side changes
Kerry350 Jun 12, 2020
b29ee1a
Merge remote-tracking branch 'upstream/master' into 67465-logs-alert-…
Kerry350 Jun 12, 2020
3901a07
Ensure "more than" is handled and ensure correct component is exporte…
Kerry350 Jun 12, 2020
0ceb3fe
Remove hit total capping due to document count total dependency
Kerry350 Jun 12, 2020
f0e11f6
Spread aggs properly
Kerry350 Jun 12, 2020
7806cf3
Alter handling of composite aggs
Kerry350 Jun 12, 2020
971e4bd
Merge remote-tracking branch 'upstream/master' into 67465-logs-alert-…
Kerry350 Jun 12, 2020
c9e56a2
Add response runtime type check
Kerry350 Jun 12, 2020
16608ee
Fix type to account for undefined group_key
Kerry350 Jun 12, 2020
b4bcde8
Merge remote-tracking branch 'upstream/master' into 67465-logs-alert-…
Kerry350 Jun 15, 2020
8bc54ff
Use separate functions for grouped and ungrouped ES queries
Kerry350 Jun 15, 2020
8093dca
Cast a wider net for group results and add "must" filters to a sub ag…
Kerry350 Jun 15, 2020
0ae6fe3
Merge remote-tracking branch 'upstream/master' into 67465-logs-alert-…
Kerry350 Jun 15, 2020
c40d459
Merge remote-tracking branch 'upstream/master' into 67465-logs-alert-…
Kerry350 Jun 15, 2020
aed4063
Merge remote-tracking branch 'upstream/master' into 67465-logs-alert-…
Kerry350 Jun 16, 2020
902bd37
Amend executor tests
Kerry350 Jun 16, 2020
8cae6b6
Ensure inner filtering scopes back to unpadded range
Kerry350 Jun 16, 2020
b67ad73
Merge remote-tracking branch 'upstream/master' into 67465-logs-alert-…
Kerry350 Jun 29, 2020
b50de34
Update x-pack/plugins/infra/server/lib/alerting/log_threshold/log_thr…
Kerry350 Jun 29, 2020
34bbece
Update x-pack/plugins/infra/server/lib/alerting/log_threshold/log_thr…
Kerry350 Jun 29, 2020
827071d
Review changes
Kerry350 Jun 29, 2020
1b4b99f
Merge branch '67465-logs-alert-group-by' of github.com:Kerry350/kiban…
Kerry350 Jun 29, 2020
a6dfeb6
Tweak filtering of fields (remove searchable requirement)
Kerry350 Jun 29, 2020
17b49f8
Update x-pack/plugins/infra/public/components/alerting/logs/expressio…
Kerry350 Jun 29, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ export const Editor: React.FC<Props> = (props) => {
const supportedFields = useMemo(() => {
if (sourceStatus?.logIndexFields) {
return sourceStatus.logIndexFields.filter((field) => {
return (field.type === 'string' || field.type === 'number') && field.searchable;
return field.type === 'string' || field.type === 'number';
Kerry350 marked this conversation as resolved.
Show resolved Hide resolved
});
} else {
return [];
Expand All @@ -143,7 +143,7 @@ export const Editor: React.FC<Props> = (props) => {
const groupByFields = useMemo(() => {
if (sourceStatus?.logIndexFields) {
return sourceStatus.logIndexFields.filter((field) => {
return field.type === 'string' && field.searchable;
return field.type === 'string' && field.aggregatable;
});
} else {
return [];
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -153,8 +153,9 @@ const buildFiltersFromCriteria = (params: LogDocumentCountAlertParams, timestamp
const { timeSize, timeUnit, criteria } = params;
const interval = `${timeSize}${timeUnit}`;
const intervalAsSeconds = getIntervalInSeconds(interval);
const intervalAsMs = intervalAsSeconds * 1000;
const to = Date.now();
const from = to - intervalAsSeconds * 1000;
const from = to - intervalAsMs;

const positiveComparators = getPositiveComparators();
const negativeComparators = getNegativeComparators();
Expand All @@ -179,7 +180,21 @@ const buildFiltersFromCriteria = (params: LogDocumentCountAlertParams, timestamp
},
};

return { rangeFilter, mustFilters, mustNotFilters };
// For group by scenarios we'll pad the time range by 1 x the interval size on the left (lte) and right (gte), this is so
// a wider net is cast to "capture" the groups. This is to account for scenarios where we want ascertain if
// there were "no documents" (less than 1 for example). In these cases we may be missing documents to build the groups
// and match / not match the criteria.
const groupedRangeFilter = {
range: {
[timestampField]: {
gte: from - intervalAsMs,
lte: to + intervalAsMs,
format: 'epoch_millis',
},
},
};

return { rangeFilter, groupedRangeFilter, mustFilters, mustNotFilters };
};

const getGroupedESQuery = (
Expand All @@ -195,28 +210,11 @@ const getGroupedESQuery = (

const timestampField = sourceConfiguration.fields.timestamp;

const { rangeFilter, mustFilters, mustNotFilters } = buildFiltersFromCriteria(
const { rangeFilter, groupedRangeFilter, mustFilters, mustNotFilters } = buildFiltersFromCriteria(
params,
timestampField
);

// For group by scenarios we'll pad the interval by 20% on the left (lte) and right (gte), this is so
// a wider net is cast to "capture" the groups. This is to account for scenarios where we want ascertain if
// there were "no documents" (less than 1 for example). In these cases we may be missing documents to build the groups
// and match (or not match) the criteria.
const interval = rangeFilter.range[timestampField].lte - rangeFilter.range[timestampField].gte;
const twentyPercentOfInterval = (20 / 100) * interval;

const paddedRangeFilter = {
range: {
[timestampField]: {
gte: rangeFilter.range[timestampField].gte - twentyPercentOfInterval,
lte: rangeFilter.range[timestampField].lte + twentyPercentOfInterval,
format: 'epoch_millis',
},
},
};

const aggregations = {
groups: {
composite: {
Expand All @@ -232,7 +230,7 @@ const getGroupedESQuery = (
filter: {
bool: {
// Scope the inner filtering back to the unpadded range
must: [rangeFilter, ...mustFilters],
filter: [rangeFilter, ...mustFilters],
},
},
},
Expand All @@ -243,7 +241,7 @@ const getGroupedESQuery = (
const body = {
query: {
bool: {
filter: [paddedRangeFilter],
filter: [groupedRangeFilter],
...(mustNotFilters.length > 0 && { must_not: mustNotFilters }),
},
},
Expand Down Expand Up @@ -320,7 +318,6 @@ const buildCriterionQuery = (criterion: Criterion): Filter | undefined => {
},
},
};
break;
case 'match': {
return {
match: {
Expand Down Expand Up @@ -400,12 +397,12 @@ const getUngroupedResults = async (query: object, callCluster: AlertServices['ca
return decodeOrThrow(UngroupedSearchQueryResponseRT)(await callCluster('search', query));
};

const getGroupedResults = async (query: any, callCluster: AlertServices['callCluster']) => {
const getGroupedResults = async (query: object, callCluster: AlertServices['callCluster']) => {
let compositeGroupBuckets: GroupedSearchQueryResponse['aggregations']['groups']['buckets'] = [];
let lastAfterKey: GroupedSearchQueryResponse['aggregations']['groups']['after_key'] | undefined;

while (true) {
const queryWithAfterKey = { ...query };
const queryWithAfterKey: any = { ...query };
queryWithAfterKey.body.aggregations.groups.composite.after = lastAfterKey;
const groupResponse: GroupedSearchQueryResponse = decodeOrThrow(GroupedSearchQueryResponseRT)(
await callCluster('search', queryWithAfterKey)
Expand Down