Skip to content

Commit

Permalink
Reduces field capabilities event loop block times by scaling linearly…
Browse files Browse the repository at this point in the history
… using hashes (#75718)

## Summary

On the `security_solution` project from different customers we have been getting reports of scaling issues and excessive NodeJS event blocking times. After in-depth tracing through some of the index and field capabilities calls we identified two of the "hot paths" running through `field_capabilities` to where it is using double looped arrays rather than hashes. By switching these two hot spots out for hashes we are now able to reduce the event loop block times by an order of magnitude. 

Before this PR you can see event loop block times as high as:

```ts
field_cap: 575.131ms
```

And after this PR you will see event loop block times drop by an order of magnitude to:

```ts
field_cap: 31.783ms
```

when you're calling into indexes as large as `filebeat-*`. This number can be higher if you're concatenating several large indexes together trying to get capabilities from each one all at once. We already only call `getFieldCapabilities` with one index at a time to spread out event block times.

The fix is to use a hash within two key areas within these two files:

```ts
src/plugins/data/server/index_patterns/fetcher/lib/field_capabilities/field_capabilities.ts
src/plugins/data/server/index_patterns/fetcher/lib/field_capabilities/field_caps_response.ts
```

This effect happens during the query of `SourceQuery`/`IndexFields` within `security_solution` but you should be able to trigger it with any application code who calls into those code paths with large index sizes such as `filebeat-*` anywhere in Kibana.

An explanation of how to see the block times for before and after
---

Add, `console.time('field_cap');` and `console.timeEnd('field_cap');` to where the synchronize code is for testing the optimizations of before and after.

For example around lines 45 with the original code:
```ts
  const esFieldCaps: FieldCapsResponse = await callFieldCapsApi(callCluster, indices);
  console.time('field_cap'); // <--- start timer
  const fieldsFromFieldCapsByName = keyBy(readFieldCapsResponse(esFieldCaps), 'name');

  const allFieldsUnsorted = Object.keys(fieldsFromFieldCapsByName)
    .filter((name) => !name.startsWith('_'))
    .concat(metaFields)
    .reduce(concatIfUniq, [] as string[])
    .map<FieldDescriptor>((name) =>
      defaults({}, fieldsFromFieldCapsByName[name], {
        name,
        type: 'string',
        searchable: false,
        aggregatable: false,
        readFromDocValues: false,
      })
    )
    .map(mergeOverrides);

  const sorted = sortBy(allFieldsUnsorted, 'name');
  console.timeEnd('field_cap'); // <--- outputs the end timer
  return sorted;

```

And around lines 45 with this pull request:

```ts
  const esFieldCaps: FieldCapsResponse = await callFieldCapsApi(callCluster, indices);
  console.time('field_cap'); // <--- start timer
  const fieldsFromFieldCapsByName = keyBy(readFieldCapsResponse(esFieldCaps), 'name');

  const allFieldsUnsorted = Object.keys(fieldsFromFieldCapsByName)
    .filter((name) => !name.startsWith('_'))
    .concat(metaFields)
    .reduce<{ names: string[]; hash: Record<string, string> }>(
      (agg, value) => {
        // This is intentionally using a "hash" and a "push" to be highly optimized with very large indexes
        if (agg.hash[value] != null) {
          return agg;
        } else {
          agg.hash[value] = value;
          agg.names.push(value);
          return agg;
        }
      },
      { names: [], hash: {} }
    )
    .names.map<FieldDescriptor>((name) =>
      defaults({}, fieldsFromFieldCapsByName[name], {
        name,
        type: 'string',
        searchable: false,
        aggregatable: false,
        readFromDocValues: false,
      })
    )
    .map(mergeOverrides);

  const sorted = sortBy(allFieldsUnsorted, 'name');
  console.timeEnd('field_cap'); // <--- outputs the end timer
  return sorted;

```

And then reload the security solutions application web page or generically anything that is going to call filebeat-* index or another large index or you could concatenate several indexes together as well to test out the performance. For security solutions we can just visit any page such as this one below which has a filebeat-* index:

```
http://localhost:5601/app/security/timelines/
```

Be sure to load it _twice_ for testing as NodeJS will sometimes report better numbers the second time as it does optimizations after the first time it encounters some code paths.

You should begin to see numbers similar to this in the before:

```ts
field_cap: 575.131ms
```

This indicates that it is blocking the event loop for around half a second before this fix. If an application adds additional indexes on-top of `filebeat`, or if it tries to execute other code after this (which we do in security solutions) then the block times will climb even higher.


However, after this fix, the m^n are changed to use hashing so this only climb by some constant * n where n is your fields and for filebeat-* it will should very low around:

```ts
field_cap: 31.783ms
```

### Checklist

Unit tests already present, so this shouldn't break anything 🤞 .
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
  • Loading branch information
FrankHassanabad authored Aug 24, 2020
1 parent c587058 commit 9dfcde2
Show file tree
Hide file tree
Showing 2 changed files with 39 additions and 15 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,6 @@ import { FieldCapsResponse, readFieldCapsResponse } from './field_caps_response'
import { mergeOverrides } from './overrides';
import { FieldDescriptor } from '../../index_patterns_fetcher';

export function concatIfUniq<T>(arr: T[], value: T) {
return arr.includes(value) ? arr : arr.concat(value);
}

/**
* Get the field capabilities for field in `indices`, excluding
* all internal/underscore-prefixed fields that are not in `metaFields`
Expand All @@ -49,8 +45,20 @@ export async function getFieldCapabilities(
const allFieldsUnsorted = Object.keys(fieldsFromFieldCapsByName)
.filter((name) => !name.startsWith('_'))
.concat(metaFields)
.reduce(concatIfUniq, [] as string[])
.map<FieldDescriptor>((name) =>
.reduce<{ names: string[]; hash: Record<string, string> }>(
(agg, value) => {
// This is intentionally using a "hash" and a "push" to be highly optimized with very large indexes
if (agg.hash[value] != null) {
return agg;
} else {
agg.hash[value] = value;
agg.names.push(value);
return agg;
}
},
{ names: [], hash: {} }
)
.names.map<FieldDescriptor>((name) =>
defaults({}, fieldsFromFieldCapsByName[name], {
name,
type: 'string',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,12 @@ export interface FieldCapsResponse {
*/
export function readFieldCapsResponse(fieldCapsResponse: FieldCapsResponse): FieldDescriptor[] {
const capsByNameThenType = fieldCapsResponse.fields;
const kibanaFormattedCaps: FieldDescriptor[] = Object.keys(capsByNameThenType).map(
(fieldName) => {

const kibanaFormattedCaps = Object.keys(capsByNameThenType).reduce<{
array: FieldDescriptor[];
hash: Record<string, FieldDescriptor>;
}>(
(agg, fieldName) => {
const capsByType = capsByNameThenType[fieldName];
const types = Object.keys(capsByType);

Expand All @@ -119,7 +123,7 @@ export function readFieldCapsResponse(fieldCapsResponse: FieldCapsResponse): Fie
// ignore the conflict and carry on (my wayward son)
const uniqueKibanaTypes = uniq(types.map(castEsToKbnFieldTypeName));
if (uniqueKibanaTypes.length > 1) {
return {
const field = {
name: fieldName,
type: 'conflict',
esTypes: types,
Expand All @@ -134,22 +138,34 @@ export function readFieldCapsResponse(fieldCapsResponse: FieldCapsResponse): Fie
{}
),
};
// This is intentionally using a "hash" and a "push" to be highly optimized with very large indexes
agg.array.push(field);
agg.hash[fieldName] = field;
return agg;
}

const esType = types[0];
return {
const field = {
name: fieldName,
type: castEsToKbnFieldTypeName(esType),
esTypes: types,
searchable: isSearchable,
aggregatable: isAggregatable,
readFromDocValues: shouldReadFieldFromDocValues(isAggregatable, esType),
};
// This is intentionally using a "hash" and a "push" to be highly optimized with very large indexes
agg.array.push(field);
agg.hash[fieldName] = field;
return agg;
},
{
array: [],
hash: {},
}
);

// Get all types of sub fields. These could be multi fields or children of nested/object types
const subFields = kibanaFormattedCaps.filter((field) => {
const subFields = kibanaFormattedCaps.array.filter((field) => {
return field.name.includes('.');
});

Expand All @@ -161,9 +177,9 @@ export function readFieldCapsResponse(fieldCapsResponse: FieldCapsResponse): Fie
.map((_, index, parentFieldNameParts) => {
return parentFieldNameParts.slice(0, index + 1).join('.');
});
const parentFieldCaps = parentFieldNames.map((parentFieldName) => {
return kibanaFormattedCaps.find((caps) => caps.name === parentFieldName);
});
const parentFieldCaps = parentFieldNames.map(
(parentFieldName) => kibanaFormattedCaps.hash[parentFieldName]
);
const parentFieldCapsAscending = parentFieldCaps.reverse();

if (parentFieldCaps && parentFieldCaps.length > 0) {
Expand All @@ -188,7 +204,7 @@ export function readFieldCapsResponse(fieldCapsResponse: FieldCapsResponse): Fie
}
});

return kibanaFormattedCaps.filter((field) => {
return kibanaFormattedCaps.array.filter((field) => {
return !['object', 'nested'].includes(field.type);
});
}

0 comments on commit 9dfcde2

Please sign in to comment.