Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] Installed Packages should attempt to create data_streams when installed or integration added to Policy #90672

Open
kevinlog opened this issue Feb 8, 2021 · 4 comments
Assignees
Labels
Feature:Fleet Fleet team's agent central management project Team:Defend Workflows “EDR Workflows” sub-team of Security Solution

Comments

@kevinlog
Copy link
Contributor

kevinlog commented Feb 8, 2021

Describe the feature:
Currently, when packages are installed via Fleet, any data_streams contained within are created when the Agent or sub-process first attempts to send a document to ES. We should investigate if there's a way to create the data_streams earlier.

This bug illustrates an issue when we're unable to create data_streams before the first document is sent: #90401

Describe a specific use case for the feature:
As described in #90401

The logs-endpoint.alerts-* index pattern and corresponding mapping does not get created until the Elastic Endpoint sends the first event of type alert to the system. This means that the Elastic Endpoint Security rule will fail execution until the first alert gets sent from an endpoint.

This bug could be fixed if we were able to create data_streams earlier in the lifecycle.

@kevinlog kevinlog added the Team:Defend Workflows “EDR Workflows” sub-team of Security Solution label Feb 8, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

@kevinlog kevinlog added the Team:Fleet Team label for Observability Data Collection Fleet team label Feb 8, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@kevinlog kevinlog self-assigned this Feb 8, 2021
@kevinlog
Copy link
Contributor Author

kevinlog commented Feb 8, 2021

I spoke with @scunningham about potential solutions for this one, but they seem complicated.

Options:

  • Attempt to install data_streams included with a package when they are first installed or updated
    • This may work for static data_streams, but many rely on context within a deployed Agent Policy to get the naming correct
  • Attempt to install data_streams when a corresponding integration is added to an Agent Policy.
    • This may be complicated because we would need to analyze all other integrations and settings assigned to the Policy

cc\ @ruflin @ph @peluja1012 @spong

@spong
Copy link
Member

spong commented Feb 9, 2021

For the best Detections UX the only index that needs to be created on install of the Endpoint Security integration would be the logs-endpoint.alerts-* index, if that makes this any easier (or could be done as a one-off?).

@jen-huang jen-huang added Feature:Fleet Fleet team's agent central management project and removed Team:Fleet Team label for Observability Data Collection Fleet team labels Apr 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Fleet Fleet team's agent central management project Team:Defend Workflows “EDR Workflows” sub-team of Security Solution
Projects
None yet
Development

No branches or pull requests

4 participants