[Heartbeat] Handle data streams

[andrewvc]

Handles data streams from fleet, the current heartbeat code doesn't handle data_streams at all, this fixes that. Additionally, it hoists the id from the top level of the yaml config, and merges both levels of data_streams, since one is at the input level, and the other is at the stream level.

Additionally, this cleans up the data streams code significantly, introducing a new add_data_streams_index processor that:

1. More efficiently formats index names for data streams
2. Allows individual events to override the dataset (useful for synthetics where we have a base browser dataset, but also browser_screenshot and browser_network for extended data that can take up lots of space, and often requires a different ILM policy).

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Use the following heartbeat.yml and verify the output sets the correct monitor ID and writes to the correct data streams

# Configure monitors inline
heartbeat.monitors:
- id: 4ae879a9-b5da-4132-94ba-ab6e1fcbdc6e
  name: Sample monitor 2
  revision: 1
  type: synthetics/http
  use_output: default
  meta:
    package:
    name: synthetics
    version: 0.1.28
  data_stream:
    namespace: default
  streams:
   - id: synthetics/http-http-4ae879a9-b5da-4132-94ba-ab6e1fcbdc6e
     name: Sample monitor 2
     type: http
     data_stream:
       dataset: http
       type: synthetics
     urls: 'http://elastic.co'
     service.name: APM Service Name
     schedule: '@every 5s'
     timeout: 1600
     max_redirects: 1
     proxy_url: 'http://elastic.co'
     tags:
       - tag
       - tag2 

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #24223 updated

• Start Time: 2021-03-04T16:18:01.089+0000

• Duration: 52 min 39 sec

• Commit: 15ead22

Test stats 🧪

Test	Results
Failed	0
Passed	45789
Skipped	4916
Total	50705

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	45789
Skipped	4916
Total	50705

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[blakerouse]

This looks really good. Didn't see any issues and looks well tested.

Only thing is the imports seem a little bit jumbled. It would just make the code cleaner to fix the groupings.

[urso]

I wonder if we can rather have agent (or the spec file) to produce a complete/correct configuration (for all Beats). The merging code here makes it look like the responsibility of understanding streams from Fleet configurations is shared between agent+heartbeat.

[andrewvc]

I'm not sure if I know the answer to that since I'm unfamiliar with how the parsing for the other beats works. For heartbeat we decided to keep the agent logic minimal and put more in heartbeat. That decision has worked out well since debugging issues in heartbeat is much simpler without fleet involved. I know that effectively heartbeat.monitors is equivalent to inputs in agent, so I can copy / paste fleet configs straight in for manual testing to confirm.

[urso]

Would be nice if you can add the change to Filebeat/Packetbeat/Metricbeat right away in order to reduce fragmentation.