Endpoint Security Rules are Failing on On-Prem usage (seen on 7.11.0 BC6 and 8.0/master)

[ghost]

Describe the bug
Endpoint Security Rules are Failing on 7.11.0 BC6 On-Prem

Build Details:

Platform: Production
Version: 7.11.0 BC6
Commit : 80030db683333591cf7de76bc6780c96e8b733a2
Build: 37890

Browser Details
All

Preconditions
1.Elastic Stack environment version 7.11.0 BC6 should be available.
2.Install the Endpoint security on above environment

Steps to Reproduce

1. Navigate to Security App from Left Navigatation bar.
2. Go to Security App > Detection tab.
3. Click on Manage Detection rules
4. Observed the Failure on Endpoint Security Rules

Error: The following index patterns did not match any indices: ["logs-endpoint.alerts-*"]

Actual Result
Endpoint Security Rules are Failing on 7.11.0 BC6 On-Prem

Expected Result
Endpoint Security Rules should not Fail on 7.11.0 BC6 On-Prem

Whats Working

• N/A

Whats Not Working

• N/A

Screenshots

logs

• GET _cat/indices
  Observed that required index "logs-endpoint.alerts-" mentioned in error message in missing in below response

yellow open .ds-logs-endpoint.events.registry-default-2021.02.05-000001           qb5DHpKZTPqpvWb3pqb-ug 1 1    90    0  162.2kb  162.2kb
yellow open .ds-logs-endpoint.events.file-default-2021.02.05-000001               r_OSUirDSE6DeP3FKSlcWg 1 1 13785    0    2.9mb    2.9mb
yellow open .ds-metrics-elastic_agent.metricbeat-default-2021.02.05-000001        -OO9StYYQPKi_w1DLTXBYQ 1 1    58    0  150.4kb  150.4kb
yellow open .ds-metrics-endpoint.policy-default-2021.02.05-000001                 LInhwQ0ST16uvlfwEUqwtA 1 1     3    0     23kb     23kb
yellow open .ds-metrics-system.uptime-default-2021.02.05-000001                   hFG-loG0QVm9gfHGdZ2ONg 1 1    31    0  165.4kb  165.4kb
yellow open .lists-default-000001                                                 zyvVivimR26FtAyLWzCq2g 1 1     0    0     208b     208b
yellow open .ds-metrics-system.socket_summary-default-2021.02.05-000001           ldeqQ9diRqitrpbR3vjL-w 1 1    31    0  208.9kb  208.9kb
yellow open .ds-logs-endpoint.events.network-default-2021.02.05-000001            A3995KsiSZKJD1G3LvJy1g 1 1   231    0  187.4kb  187.4kb
green  open .transform-internal-005                                               kmtlMfyVTOexU0xB3ONE-w 1 0     5    1   58.1kb   58.1kb
green  open .apm-agent-configuration                                              NERREv1GS7O7aOWNim47Ow 1 0     0    0     208b     208b
yellow open .ds-metrics-endpoint.metadata-default-2021.02.05-000001               w8WomAh1SHWnuv7fuQG5PA 1 1     3    0   20.2kb   20.2kb
yellow open .ds-metrics-elastic_agent.filebeat-default-2021.02.05-000001          JmnUHSbhRgi9LJFIcwGhqA 1 1    58    0  203.5kb  203.5kb
green  open .kibana_1                                                             Qeaha8efTSW4be8_AxvMdw 1 0  1603 1041    3.2mb    3.2mb
yellow open .ds-metrics-system.process-default-2021.02.05-000001                  Lg72xGpBTjakJVa5LsX_aA 1 1   204    0  500.5kb  500.5kb
yellow open .ds-metrics-system.cpu-default-2021.02.05-000001                      ZS7TiwU8QkeAzEP5NNux0g 1 1    31    0  193.7kb  193.7kb
yellow open .ds-metrics-system.memory-default-2021.02.05-000001                   EE-gwv0iTb2TGR4cvz7KOw 1 1    31    0  200.5kb  200.5kb
green  open metrics-endpoint.metadata_current_default                             sgXmscwIRGqnUpNaYba1xA 1 0     1    1   47.6kb   47.6kb
green  open .security-7                                                           dGE3STT3TaKn3e9iulgdXg 1 0    63    0  165.9kb  165.9kb
yellow open .ds-metrics-system.filesystem-default-2021.02.05-000001               1UzQAqx1S9eK4FLNDIcNsQ 1 1     6    0   99.6kb   99.6kb
yellow open .ds-metrics-system.diskio-default-2021.02.05-000001                   PZuoyiTWRuC0ndrNO4dWNA 1 1    61    0  182.9kb  182.9kb
yellow open .ds-metrics-endpoint.metrics-default-2021.02.05-000001                Xis5qOT7T8SV-qlhGyAIjw 1 1     3    0   41.4kb   41.4kb
yellow open .ds-metrics-system.process_summary-default-2021.02.05-000001          dre3-KSHSbOUL4y57qkZ5w 1 1    31    0  178.5kb  178.5kb
yellow open .ds-metrics-system.fsstat-default-2021.02.05-000001                   8rPtQ3TfR_2O3ggLDNFGXQ 1 1     6    0     99kb     99kb
yellow open .ds-logs-elastic_agent.metricbeat-default-2021.02.05-000001           SDV4sZUwRECPiKSQiYgnpQ 1 1    55    0    128kb    128kb
green  open .async-search                                                         e05AjcDfST2bOY2lkUn2Rw 1 0     1    1    7.7kb    7.7kb
yellow open .ds-logs-system.system-default-2021.02.05-000001                      ZrQIIubiQ7e1TeEb2iEk9w 1 1  9909    0    3.4mb    3.4mb
yellow open .ds-logs-system.application-default-2021.02.05-000001                 d3h0JMivSKW72Vht80S5QQ 1 1   185    0  216.2kb  216.2kb
yellow open .ds-logs-system.security-default-2021.02.05-000001                    Y-qKo8wwRkqI8xAjBHIriQ 1 1 29045    0    7.8mb    7.8mb

• Kibana error
  

• Endpoint Log
  endpoint-000000.zip

[ghost]

@manishgupta-qasource Please review

[manishgupta-qasource]

Reviewed & Assigned to @MadameSheema

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[ghost]

As 7.11.0 BC6 is now also on gcp-europe-west1 prod , we have observed the same issue on 7.11.0 BC6 Cloud as well. Agent and Endpoint Security State is Healthy/Online

Build Details:

Platform: Production
Version: 7.11.0 BC6
Commit : 80030db683333591cf7de76bc6780c96e8b733a2
Build: 37890

Snap-Shoot:

[peluja1012]

Thanks for reporting this issue. @karanbirsingh-qasource does the error go away after the windows Endpoint starts sending data to the server?

[peluja1012]

The logs-endpoint.alerts-* index pattern and corresponding mapping does not get created until the Elastic Endpoint sends the first event of type alert to the system. This means that the Elastic Endpoint Security rule will fail execution until the first alert gets sent from an endpoint.

You can verify in Kibana Dev Tools by running the following query after installing the endpoint but before it sends the first alert.

# Before first alert gets sent
GET logs-endpoint.alerts-*/_mapping

# Result
{ }

Then you can run the same query again after the Endpoint sends the first alert to verify the mapping is now present.

# After first alert gets sent
GET logs-endpoint.alerts-*/_mapping

# Result
{
  ".ds-logs-endpoint.alerts-default-2021.02.05-000001" : {
    "mappings" : {
      "dynamic" : "false",
      "_meta" : {
        "package" : {
          "name" : "endpoint"
        },
        "managed_by" : "ingest-manager",
        "managed" : true
      } ....

After the mapping and index pattern are present, the rule will execute successfully:

[kevinlog]

@peluja1012 @spong

Following from the meeting, I created this issue to attempt to create data_streams before the Endpoint sends the first document. #90672

After speaking with @scunningham the fix may not be straightforward since there is a bit of context involved in naming the data_streams correctly (i.e. what is the namespace setting for the Agent Policy that you add the integration to). However, looking to address this problem is some way is certainly worthwhile, so I think we should keep a thread going.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[ghost]

Hi @peluja1012

We have validate this issue on 7.11.1 BC1 On-Cloud and found that issue is still occurring . Endpoint security detection's rules are Failing.

However Endpoint Security Rule got to success state after first alert signals generated for the respective rule.

Build Details:

Platform: Production
Version: 7.11.1
Commit : 3f71ce7177a41e067ddb1e670ec4ace5f6d4f5fe
Build : 37897

Snap-Shoots:

C.C @kevinlog

[dhurley14]

This is a feature 😉

I have merged a branch to 7.11.2 which will replace these failing statuses with a warning. But we have already heard from users that they didn't know they didn't get the endpoint integration out of the box so this has been helpful. Definitely not a bug though.

[cyberpescadito]

Hi there, i'm having the same issue with a fresh install.

I only have linux hosts with agents installed on (no windows), and i'm trying to generate an alert to initiate the logs-endpoint.alerts index without success.
Initially my rules was showing an error state and was missing both auditbeat index and logs-endpoints, since I installed manually an auditbeat on a client, the rules ares in "success" except "endpoint security" that is in "warning" state and displaying "no index matching: ["logs-endpoint.alerts-*"] was found. "

I have been trying to trigger an alert using out of the box elastic detection rules regarding nmap process detection and telnet activity (easy to trigger).
Using GET logs-endpoint.alerts-*/_mapping in dev tools i'm still getting an empty result; Then i have no alerts in my detections tab.

Additionnally if I search in logs for nmap, I can see from metricbeat source the events I generated on namespace default, but I can't see the events on custom namespaces (Using different policies i got agent on default namespace and agents on custom namespaces; I generated the 'malicious activity' from both).

Any idea of what is going on there?

[cyberpescadito]

After days of troubleshooting, finally tried with a VM instead of a container.
It works like a charm in the VM.
wondering if the endpoint is supposed to work with container or not