[ML] Add Anomaly Explorer charts embeddable

[qn895]

Summary

Adds the anomaly charts from the Anomaly Explorer as an embeddable to Kibana's Dashboard.

Initialize:

With query:

With filters:

With custom time range:

Context menu:

Need to do:

• Re-add tests for AnomalyExplorerChartsService (formerly explorer_charts_container_service.test)
• Remove recentlyAccessed cache dependency
• Add index patterns as embeddable's output to prevent filter bars from showing the warning
• Add unit tests
• Fix support for dark theme

Checklist

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support

[darnautov]

I've noticed that the horizontal scroll bar is always presented, we need to fix some CSS

[darnautov]

I think the Severity threshold value should persist. At the moment it always falls back to the "minor" after a page refresh.

[darnautov]

do we need this method? it doesn't do anything with the response

[qn895]

I updated this getCombinedJobs a bit here now 47509f8, mainly to avoid getting unnecessary stats. However, considering that this is a charts service, maybe this would be better in AnomalyDetectorService?

[darnautov]

I'd recommend writing some unit tests for the input resolver. It's the most important and complex part of this feature

[qn895]

Not sure if the sizing of the charts in a row needs some adjustment. If I make my browser window larger, I can end up with whitespace, rather than the preferred behavior of the charts filling the available space:

I have added a fix for that issue here 3e2f4fc (#94396). Although this fix only works when we have one chart visible or if the number of charts visible is less than the calculated charts per row. Let me know if you find any other edge cases.

[peteharverson]

I have added a fix for that issue here 3e2f4fc (#94396). Although this fix only works when we have one chart visible or if the number of charts visible is less than the calculated charts per row. Let me know if you find any other edge cases.

The sizing behavior is much better now. I still see the occasional odd behavior, such as when maximizing the window, when the x axis time range doesn't obey the time range in the dashboard.

But I'm happy for any other changes in this area to be done in a follow up.

[qn895]

The sizing behavior is much better now. I still see the occasional odd behavior, such as when maximizing the window, when the x axis time range doesn't obey the time range in the dashboard.

I have added a note for that on the meta issue here #94650

[pheyos]

Good to see functional tests coming in as part of this PR! 🎉
Left a few comments and suggestions.

[peteharverson]

Tested latest changes and LGTM.

[darnautov]

Assuming unresolved comments will be addressed in a follow up PR, LGTM! Great job 🚀

[pheyos]

Functional tests LGTM 🎉

[kibanamachine]

💛 Build succeeded, but was flaky

• continuous-integration/kibana-ci/pull-request

• Commit: 1a9a7c0

• Storybooks Preview

• Flaky suites:

  • xpack-kibana-ciGroup6

---

Test Failures

Kibana Pipeline / general / X-Pack API Integration Tests.x-pack/test/api_integration/apis/security_solution/kpi_network·ts.apis SecuritySolution Endpoints Kpi Network With filebeat Make sure that we get KpiNetwork networkEvents data

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook in "apis"
[00:05:28]           └-: SecuritySolution Endpoints
[00:05:28]             └-> "before all" hook in "SecuritySolution Endpoints"
[00:05:31]             └-: Kpi Network
[00:05:31]               └-> "before all" hook in "Kpi Network"
[00:05:31]               └-: With filebeat
[00:05:31]                 └-> "before all" hook for "Make sure that we get KpiNetwork uniqueFlows data"
[00:05:31]                 └-> "before all" hook for "Make sure that we get KpiNetwork uniqueFlows data"
[00:05:31]                   │ info [filebeat/default] Loading "mappings.json"
[00:05:31]                   │ info [filebeat/default] Loading "data.json.gz"
[00:05:31]                   │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xxl-1616770851388747980] [filebeat-7.0.0-iot-2019.06] creating index, cause [api], templates [], shards [1]/[0]
[00:05:31]                   │ info [filebeat/default] Created index "filebeat-7.0.0-iot-2019.06"
[00:05:31]                   │ debg [filebeat/default] "filebeat-7.0.0-iot-2019.06" settings {"index":{"lifecycle":{"name":"filebeat-7.0.0","rollover_alias":"filebeat-7.0.0"},"mapping":{"total_fields":{"limit":"10000"}},"number_of_replicas":"0","number_of_shards":"1","query":{"default_field":["tags","message","agent.version","agent.name","agent.type","agent.id","agent.ephemeral_id","client.address","client.mac","client.domain","client.geo.continent_name","client.geo.country_name","client.geo.region_name","client.geo.city_name","client.geo.country_iso_code","client.geo.region_iso_code","client.geo.name","cloud.provider","cloud.availability_zone","cloud.region","cloud.instance.id","cloud.instance.name","cloud.machine.type","cloud.account.id","container.runtime","container.id","container.image.name","container.image.tag","container.name","destination.address","destination.mac","destination.domain","destination.geo.continent_name","destination.geo.country_name","destination.geo.region_name","destination.geo.city_name","destination.geo.country_iso_code","destination.geo.region_iso_code","destination.geo.name","ecs.version","error.id","error.message","error.code","event.id","event.kind","event.category","event.action","event.outcome","event.type","event.module","event.dataset","event.hash","event.timezone","file.path","file.target_path","file.extension","file.type","file.device","file.inode","file.uid","file.owner","file.gid","file.group","file.mode","group.id","group.name","host.hostname","host.name","host.id","host.mac","host.type","host.architecture","host.os.platform","host.os.name","host.os.full","host.os.family","host.os.version","host.os.kernel","host.geo.continent_name","host.geo.country_name","host.geo.region_name","host.geo.city_name","host.geo.country_iso_code","host.geo.region_iso_code","host.geo.name","http.request.method","http.request.body.content","http.request.referrer","http.response.body.content","http.version","log.level","network.name","network.type","network.iana_number","network.transport","network.application","network.protocol","network.direction","network.community_id","observer.mac","observer.hostname","observer.vendor","observer.version","observer.serial_number","observer.type","observer.os.platform","observer.os.name","observer.os.full","observer.os.family","observer.os.version","observer.os.kernel","observer.geo.continent_name","observer.geo.country_name","observer.geo.region_name","observer.geo.city_name","observer.geo.country_iso_code","observer.geo.region_iso_code","observer.geo.name","organization.name","organization.id","os.platform","os.name","os.full","os.family","os.version","os.kernel","process.name","process.args","process.executable","process.title","process.working_directory","server.address","server.mac","server.domain","server.geo.continent_name","server.geo.country_name","server.geo.region_name","server.geo.city_name","server.geo.country_iso_code","server.geo.region_iso_code","server.geo.name","service.id","service.name","service.type","service.state","service.version","service.ephemeral_id","source.address","source.mac","source.domain","source.geo.continent_name","source.geo.country_name","source.geo.region_name","source.geo.city_name","source.geo.country_iso_code","source.geo.region_iso_code","source.geo.name","url.original","url.full","url.scheme","url.domain","url.path","url.query","url.fragment","url.username","url.password","user.id","user.name","user.full_name","user.email","user.hash","user.group.id","user.group.name","user_agent.original","user_agent.name","user_agent.version","user_agent.device.name","user_agent.os.platform","user_agent.os.name","user_agent.os.full","user_agent.os.family","user_agent.os.version","user_agent.os.kernel","agent.hostname","error.type","cloud.project.id","kubernetes.pod.name","kubernetes.pod.uid","kubernetes.namespace","kubernetes.node.name","kubernetes.container.name","kubernetes.container.image","log.file.path","log.source.address","stream","input.type","syslog.severity_label","syslog.facility_label","process.program","log.flags","user_agent.os.full_name","fileset.name","apache.access.ssl.protocol","apache.access.ssl.cipher","apache.error.module","user.terminal","user.audit.id","user.audit.name","user.audit.group.id","user.audit.group.name","user.effective.id","user.effective.name","user.effective.group.id","user.effective.group.name","user.filesystem.id","user.filesystem.name","user.filesystem.group.id","user.filesystem.group.name","user.owner.id","user.owner.name","user.owner.group.id","user.owner.group.name","user.saved.id","user.saved.name","user.saved.group.id","user.saved.group.name","auditd.log.old_auid","auditd.log.new_auid","auditd.log.old_ses","auditd.log.new_ses","auditd.log.items","auditd.log.item","auditd.log.tty","auditd.log.a0","elasticsearch.component","elasticsearch.cluster.uuid","elasticsearch.cluster.name","elasticsearch.node.id","elasticsearch.node.name","elasticsearch.index.name","elasticsearch.index.id","elasticsearch.shard.id","elasticsearch.audit.layer","elasticsearch.audit.origin.type","elasticsearch.audit.realm","elasticsearch.audit.user.realm","elasticsearch.audit.user.roles","elasticsearch.audit.action","elasticsearch.audit.url.params","elasticsearch.audit.indices","elasticsearch.audit.request.id","elasticsearch.audit.request.name","elasticsearch.gc.phase.name","elasticsearch.gc.tags","elasticsearch.slowlog.logger","elasticsearch.slowlog.took","elasticsearch.slowlog.types","elasticsearch.slowlog.stats","elasticsearch.slowlog.search_type","elasticsearch.slowlog.source_query","elasticsearch.slowlog.extra_source","elasticsearch.slowlog.total_hits","elasticsearch.slowlog.total_shards","elasticsearch.slowlog.routing","elasticsearch.slowlog.id","elasticsearch.slowlog.type","haproxy.frontend_name","haproxy.backend_name","haproxy.server_name","haproxy.bind_name","haproxy.error_message","haproxy.source","haproxy.termination_state","haproxy.mode","haproxy.http.response.captured_cookie","haproxy.http.response.captured_headers","haproxy.http.request.captured_cookie","haproxy.http.request.captured_headers","haproxy.http.request.raw_request_line","icinga.debug.facility","icinga.main.facility","icinga.startup.facility","iis.access.site_name","iis.access.server_name","iis.access.cookie","iis.error.reason_phrase","iis.error.queue_name","iptables.fragment_flags","iptables.input_device","iptables.output_device","iptables.tcp.flags","iptables.ubiquiti.input_zone","iptables.ubiquiti.output_zone","iptables.ubiquiti.rule_number","iptables.ubiquiti.rule_set","kafka.log.component","kafka.log.class","kafka.log.trace.class","kafka.log.trace.message","kibana.log.tags","kibana.log.state","logstash.log.module","logstash.log.thread","text","logstash.slowlog.module","logstash.slowlog.thread","text","logstash.slowlog.event","text","logstash.slowlog.plugin_name","logstash.slowlog.plugin_type","logstash.slowlog.plugin_params","text","mongodb.log.component","mongodb.log.context","mysql.slowlog.query","mysql.slowlog.schema","mysql.slowlog.current_user","mysql.slowlog.last_errno","mysql.slowlog.killed","mysql.slowlog.log_slow_rate_type","mysql.slowlog.log_slow_rate_limit","mysql.slowlog.innodb.trx_id","netflow.type","netflow.exporter.address","netflow.source_mac_address","netflow.post_destination_mac_address","netflow.destination_mac_address","netflow.post_source_mac_address","netflow.interface_name","netflow.interface_description","netflow.sampler_name","netflow.application_description","netflow.application_name","netflow.class_name","netflow.wlan_ssid","netflow.vr_fname","netflow.metro_evc_id","netflow.nat_pool_name","netflow.p2p_technology","netflow.tunnel_technology","netflow.encrypted_technology","netflow.observation_domain_name","netflow.selector_name","netflow.information_element_description","netflow.information_element_name","netflow.virtual_station_interface_name","netflow.virtual_station_name","netflow.sta_mac_address","netflow.wtp_mac_address","netflow.user_name","netflow.application_category_name","netflow.application_sub_category_name","netflow.application_group_name","netflow.dot1q_customer_source_mac_address","netflow.dot1q_customer_destination_mac_address","netflow.mib_context_name","netflow.mib_object_name","netflow.mib_object_description","netflow.mib_object_syntax","netflow.mib_module_name","netflow.mobile_imsi","netflow.mobile_msisdn","netflow.http_request_method","netflow.http_request_host","netflow.http_request_target","netflow.http_message_version","netflow.http_user_agent","netflow.http_content_type","netflow.http_reason_phrase","osquery.result.name","osquery.result.action","osquery.result.host_identifier","osquery.result.calendar_time","postgresql.log.timestamp","postgresql.log.database","postgresql.log.query","redis.log.role","redis.slowlog.cmd","redis.slowlog.key","redis.slowlog.args","santa.action","santa.decision","santa.reason","santa.mode","santa.disk.volume","santa.disk.bus","santa.disk.serial","santa.disk.bsdname","santa.disk.model","santa.disk.fs","santa.disk.mount","certificate.common_name","certificate.sha256","hash.sha256","suricata.eve.event_type","suricata.eve.app_proto_orig","suricata.eve.tcp.tcp_flags","suricata.eve.tcp.tcp_flags_tc","suricata.eve.tcp.state","suricata.eve.tcp.tcp_flags_ts","suricata.eve.fileinfo.sha1","suricata.eve.fileinfo.state","suricata.eve.fileinfo.sha256","suricata.eve.fileinfo.md5","suricata.eve.dns.type","suricata.eve.dns.rrtype","suricata.eve.dns.rrname","suricata.eve.dns.rdata","suricata.eve.dns.rcode","suricata.eve.flow_id","suricata.eve.email.status","suricata.eve.http.redirect","suricata.eve.http.protocol","suricata.eve.http.http_content_type","suricata.eve.in_iface","suricata.eve.alert.category","suricata.eve.alert.signature","suricata.eve.ssh.client.proto_version","suricata.eve.ssh.client.software_version","suricata.eve.ssh.server.proto_version","suricata.eve.ssh.server.software_version","suricata.eve.tls.issuerdn","suricata.eve.tls.sni","suricata.eve.tls.version","suricata.eve.tls.fingerprint","suricata.eve.tls.serial","suricata.eve.tls.subject","suricata.eve.app_proto_ts","suricata.eve.flow.state","suricata.eve.flow.reason","suricata.eve.app_proto_tc","suricata.eve.smtp.rcpt_to","suricata.eve.smtp.mail_from","suricata.eve.smtp.helo","suricata.eve.app_proto_expected","system.auth.ssh.method","system.auth.ssh.signature","system.auth.sudo.error","system.auth.sudo.tty","system.auth.sudo.pwd","system.auth.sudo.user","system.auth.sudo.command","system.auth.useradd.home","system.auth.useradd.shell","traefik.access.user_identifier","traefik.access.frontend_name","traefik.access.backend_url","zeek.session_id","zeek.connection.state","zeek.connection.history","zeek.connection.orig_l2_addr","zeek.resp_l2_addr","zeek.vlan","zeek.inner_vlan","zeek.dns.query","zeek.dns.qclass_name","zeek.dns.qtype_name","zeek.dns.rcode_name","zeek.dns.answers","zeek.http.status_msg","zeek.http.info_msg","zeek.http.filename","zeek.http.tags","zeek.http.proxied","zeek.http.client_header_names","zeek.http.server_header_names","zeek.http.orig_fuids","zeek.http.orig_mime_types","zeek.http.orig_filenames","zeek.http.resp_fuids","zeek.http.resp_mime_types","zeek.http.resp_filenames","zeek.files.fuid","zeek.files.session_ids","zeek.files.source","zeek.files.analyzers","zeek.files.mime_type","zeek.files.filename","zeek.files.parent_fuid","zeek.files.md5","zeek.files.sha1","zeek.files.sha256","zeek.files.extracted","zeek.ssl.version","zeek.ssl.cipher","zeek.ssl.curve","zeek.ssl.server_name","zeek.ssl.next_protocol","zeek.ssl.cert_chain","zeek.ssl.cert_chain_fuids","zeek.ssl.client_cert_chain","zeek.ssl.client_cert_chain_fuids","zeek.ssl.issuer","zeek.ssl.client_issuer","zeek.ssl.validation_status","zeek.ssl.subject","zeek.ssl.client_subject","zeek.ssl.last_alert","fields.*"]},"refresh_interval":"5s"}}
[00:05:32]                   │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-debian-tests-xxl-1616770851388747980] [.kibana_8.0.0_001/FwJL5TpPQqK3hwR7DpRnSg] update_mapping [_doc]
[00:05:33]                   │ info [filebeat/default] Indexed 6157 docs into "filebeat-7.0.0-iot-2019.06"
[00:05:33]                 └-> Make sure that we get KpiNetwork uniqueFlows data
[00:05:33]                   └-> "before each" hook: global before each for "Make sure that we get KpiNetwork uniqueFlows data"
[00:05:33]                   └- ✓ pass  (37ms) "apis SecuritySolution Endpoints Kpi Network With filebeat Make sure that we get KpiNetwork uniqueFlows data"
[00:05:33]                 └-> Make sure that we get KpiNetwork networkEvents data
[00:05:33]                   └-> "before each" hook: global before each for "Make sure that we get KpiNetwork networkEvents data"
[00:05:33]                   └- ✖ fail: apis SecuritySolution Endpoints Kpi Network With filebeat Make sure that we get KpiNetwork networkEvents data
[00:05:33]                   │       Error: expected 0 to sort of equal 6157
[00:05:33]                   │       + expected - actual
[00:05:33]                   │ 
[00:05:33]                   │       -0
[00:05:33]                   │       +6157
[00:05:33]                   │       
[00:05:33]                   │       at Assertion.assert (/dev/shm/workspace/parallel/9/kibana/node_modules/@kbn/expect/expect.js:100:11)
[00:05:33]                   │       at Assertion.eql (/dev/shm/workspace/parallel/9/kibana/node_modules/@kbn/expect/expect.js:244:8)
[00:05:33]                   │       at Context.<anonymous> (test/api_integration/apis/security_solution/kpi_network.ts:107:45)
[00:05:33]                   │       at Object.apply (/dev/shm/workspace/parallel/9/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)
[00:05:33]                   │ 
[00:05:33]                   │ 

Stack Trace

Error: expected 0 to sort of equal 6157
    at Assertion.assert (/dev/shm/workspace/parallel/9/kibana/node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/parallel/9/kibana/node_modules/@kbn/expect/expect.js:244:8)
    at Context.<anonymous> (test/api_integration/apis/security_solution/kpi_network.ts:107:45)
    at Object.apply (/dev/shm/workspace/parallel/9/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16) {
  actual: '0',
  expected: '6157',
  showDiff: true
}

---

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
ml	1744	1779	+35

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
ml	6.0MB	6.1MB	+72.1KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
ml	64.1KB	64.6KB	+501.0B

Unknown metric groups

async chunk count

id	before	after	diff
ml	21	23	+2

History

• 💚 Build #115385 succeeded 32d70bf
• 💔 Build #115229 failed 82e02c6
• 💚 Build #115086 succeeded 7b31acf
• 💚 Build #114741 succeeded 6b5f157
• 💚 Build #114441 succeeded 8bdd67b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @qn895

[kibanamachine]

💔 Backport failed

❌ 7.x: Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 94396