[Security Solution] Detection Rule Management is very slow

[MikePaquette]

Kibana version: 7.13.0 BC1

Elasticsearch version: 7.13.0 BC1

Server OS version: Elastic Cloud ESS

Browser version: Chrome Version 89.0.4389.128 (Official Build) (x86_64)

Browser OS version: macOS Mojave 10.14.6

Original install method (e.g. download page, yum, from source, etc.): Elastic Cloud Security Deployment

Describe the bug: Exported rules file is incomplete - contains missing rules.

Steps to reproduce:

1. Launch Kibana
2. Go to Security-Detections->Manage Detection Rules
3. Set page size to 600
4. look at rules, select rules, try various rule operations

Expected behavior:
Started working with rules soon (minutes) after first loading the prebuilt rules.

• Rule Management is slow, causing frustration for detection engineer (me).
• Changing page size in rules to 600 took about 12 seconds - I expected 1-2 seconds
• Duplicating 546 rules took more than 30 seconds - I expected 3-4 seconds
• Selecting custom rules (546) filter, took about 10 seconds - I expected <1 second
• Exporting 546 rules seemed to take about 90 seconds - I expected 10 seconds or so
• The Custom Rules (546) filter button is not sticky. Filter disappeared and caused 1092 rules to be shown.

Screenshots (if relevant):

Errors in browser console (if relevant): None

Provide logs and/or server output (if relevant): None

Any additional context:

cc: @MadameSheema @dontcallmesherryli @paulewing

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[MikePaquette]

This experiment was performed on a default ESS configuration with 1GB memory for Kibana. The problem is mitigated when larger configurations of Kibana are configured for the deployment. Tested with Two zones @ 4GB each for Kibana.

[xcrzx]

Verified the rule table's behavior after these two PRs were merged: #99678, #100554

• Changing page size in rules to 600 took about 12 seconds - I expected 1-2 seconds

The maximum available number of rows has changed to 100. But switching to 100 rows still takes a sufficient amount of time, 2-3 seconds of network IO and 2-3 seconds of the main thread work.

• Duplicating 546 rules took more than 30 seconds - I expected 3-4 seconds

Rule duplication now works slightly faster (down to 10-15 seconds) but still far from the expected 3-4 seconds.

• Selecting custom rules (546) filter, took about 10 seconds - I expected <1 second

It is faster (5-6 seconds) but still not less than 1 sec.

• Exporting 546 rules seemed to take about 90 seconds - I expected 10 seconds or so

Nice improvement here. I observe < 5 seconds to export 535 rules.

• The Custom Rules (546) filter button is not sticky. Filter disappeared and caused 1092 rules to be shown.

I cannot reproduce it anymore.

[MikePaquette]

Closing this as rule management performance is good in 8.2.0