From 9abc91257c7b62d89b9ea892baf8bca85834688d Mon Sep 17 00:00:00 2001 From: Patrick Mueller Date: Fri, 13 Dec 2019 16:33:17 -0500 Subject: [PATCH] convert the RFC to a README --- .../plugins/event_log/README.md | 225 ++++++++---------- 1 file changed, 99 insertions(+), 126 deletions(-) rename rfcs/text/0007_event_log.md => x-pack/plugins/event_log/README.md (57%) diff --git a/rfcs/text/0007_event_log.md b/x-pack/plugins/event_log/README.md similarity index 57% rename from rfcs/text/0007_event_log.md rename to x-pack/plugins/event_log/README.md index 993e8ec44a0aba0..d2d67ffc27d947a 100644 --- a/rfcs/text/0007_event_log.md +++ b/x-pack/plugins/event_log/README.md @@ -1,12 +1,69 @@ -- Start Date: 2019-09-12 -- RFC PR: currently https://github.com/elastic/kibana/pull/45081 -- Kibana Issue: (leave this empty) +# Event Log -See also issue https://github.com/elastic/kibana/issues/45083, which is an -umbrella issue tracking ongoing initial work. +## Overview +The purpose of this plugin is to provide a way to persist a history of events +occuring in Kibana, initially just for the Make It Action project - alerts +and actions. -# Summary + +## Basic Usage - Logging Events + +Follow these steps to use `event_log` in your plugin: + +1. Declare `event_log` as a dependency in `kibana.json`: + +```json +{ + ... + "requiredPlugins": ["event_log"], + ... +} +``` + +2. Register provider / actions, and create your plugin's logger, using service +API provided in the `setup` stage: + +```typescript +... +import { IEventLogger, IEventLogService } from '../../event_log/server'; +interface PluginSetupDependencies { + event_log: IEventLogService; +} +... +public setup(core: CoreSetup, { event_log }: PluginSetupDependencies) { + ... + event_log.registerProviderActions('my-plugin', ['action-1, action-2']); + const eventLogger: IEventLogger = event_log.getLogger({ event: { provider: 'my-plugin' } }); + ... +} +... +``` + +4. To log an event, call `logEvent()` on the `eventLogger` object you created: + +```typescript +... + eventLogger.logEvent({ event: { action: 'action-1' }, tags: ['fe', 'fi', 'fo'] }); +... +``` + + +## Testing + +### Unit tests + +From `kibana-root-folder/x-pack`, run: +```bash +$ node node scripts/jest plugins/event_log +``` + +### API Integration tests + +None yet! + + +## Background For the Make It Action alerting / action plugins, we will need a way to persist data regarding alerts and actions, for UI and investigative purposes. @@ -85,19 +142,18 @@ Here's the event written to the event log index: } ``` +The shape of the document written to the index is a subset of [ECS][] with an +extended field of `kibana` with some Kibana-related properties contained within +it. -# Motivation +The ES mappings for the ECS data, and the config-schema for the ECS data, are +generated by a script, and available here: -The existing designs for Make It Action describe UIs which show some amount -of alert "history". Up till now, we had no where to write this history, -and so a new elasticsearch index is required for that. Because we already -have two known clients of this plugin, which are related but different, -the design of the event documents is very generic, which means it could easily -be used by other plugins that would like to record "events", for the same -general purposes. +- [`generated/mappings.json`](generated/mappings.json) +- [`generated/schemas.ts`](generated/schemas.ts) -The shape of the document written to the index is [ECS][] with an extended field -of `kibana` with some Kibana-related properties contained within it. +It's anticipated that these interfaces will grow over time, hopefully adding +more ECS fields but adding Kibana extensions as required. Since there are some security concerns with the data, we are currently restricting access via known saved object ids. That is, you can only query @@ -105,7 +161,6 @@ history records associated with specific saved object ids. [ECS]: https://www.elastic.co/guide/en/ecs/current/index.html -# Detailed design ## API @@ -123,6 +178,8 @@ export interface IEventLogService { export interface IEventLogger { logEvent(properties: IEvent): void; + startTiming(event: IEvent): void; + stopTiming(event: IEvent): void; } ``` @@ -159,6 +216,27 @@ method, nor what it would do if the method threw an error. All the error processing involved with getting the data into the index is handled internally, and logged to the server log as appropriate. +The `startTiming()` and `stopTiming()` methods can be used to set the timing +properties `start`, `end`, and `duration` in the event. For example: + +```typescript + const loggedEvent: IEvent = { event: { action: 'foo' } }; + + // sets event.start + eventLogger.startTiming(loggedEvent); + + longRunningFunction(); + + // sets event.end and event.duration + eventLogger.stopTiming(loggedEvent); + + eventLogger.logEvent(loggedEvent); + +``` + +It's anticipated that more "helper" methods like this will be provided in the +future. + ## Stored data @@ -168,8 +246,9 @@ with older event documents deleted, turned cold, frozen, etc. We'll supply some default values, but customers will be able to tweak these. The index template, mappings, config-schema types, etc for the index can -be found in the [x-pack/plugins/event_log/generated directory](../../x-pack/plugins/event_log/generated). These files are generated from a script which takes as -input the ECS properties to use, and the Kibana extensions. +be found in the [generated directory](generated). These files are generated +from a script which takes as input the ECS properties to use, and the Kibana +extensions. See [ilm rollover action docs][] for more info on the `is_write_index`, and `index.lifecycle.*` properties. @@ -182,6 +261,7 @@ Of particular note in the `mappings`: We may change some of that before releasing. + ## ILM setup We'll want to provide default ILM policy, this seems like a reasonable first @@ -217,110 +297,3 @@ For more relevant information on ILM, see: [getting started with ILM doc]: https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html [write index alias behavior]: https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html#indices-rollover-is-write-index - -# Drawbacks - -Leaves things up to the user to customize the ILM policy directly. - - -# Alternatives - -## Using Saved Objects instead of a new index - -See [issue 51223](https://github.com/elastic/kibana/issues/51223) for a -discussion on whether to try to use saved objects as the primary store for -event log documents, versus using a new stand-alone index. - -Summary: we'll use a new stand-alone index, but use the "link" saved object -pattern to only allow access via a saved object. Ie, you will need to have -access to a saved object, to get it's id, to then use in a search through -the event log index. - -## Concern with overlap with eventual audit log capability - -In an early review, concern was raised that there is a lot of overlap with -where we eventually be with audit logging. - -The issues with that are: - -- audit logs are currently just an `audit-log` tag on messages written to the - standard Kibana log; there is no infrastructure to allow them to be queried - -- security differences; the users able to read audit logs are probably much - more constrained than users that need to be able to read these event logs - -So, we don't really have an _alternative_ here, but ... a lot of overlap. -Long-term, it may make sense to make the eventual audit logging code -parameterizable, so that the core code could be used for other purposes, like -this event log. We're just not there yet. Or implement the audit log with -the event log :-) - -## Write to a events to a file and ingest via FileBeat - -This came up as _"why don't we use our own stuff to do this since that's what -it's designed for"_. Yup! However, the only thing that really changes over -writing our index is: - -- we'd write to some other log (could be console.log, or a file) instead of - an index, directly -- we'd have to arrange to run FileBeat alongside Kibana, ingesting these - logged events, which would send them into elasticsearch - -We still need to arrange to initialize all the elasticsearch resources (ilm, -index template, etc). - -The big kicker though is: where would we right these that FileBeat could -find them? Writing to a file might be problematic for users that run -Kibana with no writable file system, including folks using docker-based -deployments. - -Lots of unknown unknowns, it seems. - -Interestingly enough, this kind of story might work out extremely well for -Elastic Cloud, where we have a lot more control over the deployments! - -# Adoption strategy - -If we implement this proposal, how will existing Kibana developers adopt it? Is -this a breaking change? Can we write a codemod? Should we coordinate with -other projects or libraries? - -The API surface has been kept to a minimum to make it easy for other plugins -make use of it. But that's just a nice-to-have right now anyway, since -the only known clients will be actions and alerting. - - -# How we teach this - -**What names and terminology work best for these concepts and why? How is this -idea best presented? As a continuation of existing Kibana patterns?** - -"event" and "log" are probably the most controversial names in use here. -Previously, we've referred to this as an "audit log" or "history log". -Audit log is not a great name due to the existance of the audit logging -facility already available in Kibana. History or History log seems like -a closer fit. Event log works for me, as it provides just a touch more -information than history - it's a history of events - but of course "events" -as suitable vague enough that it doesn'tt add thatt much more. - -**Would the acceptance of this proposal mean the Kibana documentation must be -re-organized or altered? Does it change how Kibana is taught to new developers -at any level?** - -no - -**How should this feature be taught to existing Kibana developers?** - -Follow examples of other plugins using it. - -The API surface is very small, it's designed to be easy for other developers -to reuse. - - -# Unresolved questions - -**Optional, but suggested for first drafts. What parts of the design are still -TBD?** - -There is a task list of resolved and unresolved TODO items and future-looking -items in the initial event log PR https://github.com/elastic/kibana/issues/45083 \ No newline at end of file