[Ingest Manager] Support updated package output structure

[jen-huang]

‼️ Do not merge, PR needs to be merged in tandem with elastic/package-registry#514

Summary

Part of elastic/beats#19082. This PR makes the necessary changes for Ingest Manager to support package registry changes done in elastic/package-registry#514:

• Update EPM's registry types and schemas to reflect registry package output changes
  • dataset.id was renamed to dataset.name, PR updates all references
• Update data source saved object mappings to change stream[].dataset: string; to stream[].dataset: { name: string; type: string}
  • Associated types and schemas were also updated
• Inline agent templates were removed from package registry in favor of template_path, datasourceService.assignPackageStream() and related methods were updated to fetch the stream templates
• Moved inputs[].package information in agent config yaml to inputs[].meta.package
• Updated associated tests for everything above

Example updated agent config yaml

Expand me (yaml for default config with system integration)

id: c455d4d0-b71f-11ea-a989-ef5894976394
revision: 2
settings:
  monitoring:
    enabled: true
    use_output: default
    logs: true
    metrics: true
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://localhost:9200'
inputs:
  - id: c8a076d0-b71f-11ea-a989-ef5894976394
    name: system-1
    type: logs
    use_output: default
    meta:
      package:
        name: system
        version: 0.3.0
    dataset:
      namespace: default
    streams:
      - id: logs-system.auth
        dataset:
          name: system.auth
          type: logs
        paths:
          - /var/log/auth.log*
          - /var/log/secure*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.5.0
      - id: logs-system.syslog
        dataset:
          name: system.syslog
          type: logs
        paths:
          - /var/log/messages*
          - /var/log/syslog*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.5.0
  - id: c8a076d0-b71f-11ea-a989-ef5894976394
    name: system-1
    type: system/metrics
    use_output: default
    meta:
      package:
        name: system
        version: 0.3.0
    dataset:
      namespace: default
    streams:
      - id: system/metrics-system.cpu
        dataset:
          name: system.cpu
          type: metrics
        metricsets:
          - cpu
        core.metrics: null
        cpu.metrics:
          - percentages
          - normalized_percentages
        period: 10s
      - id: system/metrics-system.diskio
        dataset:
          name: system.diskio
          type: metrics
        metricsets:
          - diskio
      - id: system/metrics-system.load
        dataset:
          name: system.load
          type: metrics
        metricsets:
          - load
        core.metrics: null
        cpu.metrics: null
        period: 10s
        process.include_top_n.by_cpu: null
        process.include_top_n.by_memory: null
        processes: null
      - id: system/metrics-system.memory
        dataset:
          name: system.memory
          type: metrics
        metricsets:
          - memory
        core.metrics: null
        cpu.metrics: null
        period: 10s
        process.include_top_n.by_cpu: null
        process.include_top_n.by_memory: null
        processes: null
      - id: system/metrics-system.network
        dataset:
          name: system.network
          type: metrics
        metricsets:
          - network
        core.metrics: null
        cpu.metrics: null
        period: 10s
        process.include_top_n.by_cpu: null
        process.include_top_n.by_memory: null
        processes: null
      - id: system/metrics-system.process
        dataset:
          name: system.process
          type: metrics
        metricsets:
          - process
        period: 10s
        process.include_top_n.by_cpu: 5
        process.include_top_n.by_memory: 5
        processes:
          - .*
      - id: system/metrics-system.process_summary
        dataset:
          name: system.process_summary
          type: metrics
        metricsets:
          - process_summary
        core.metrics: null
        cpu.metrics: null
        period: 10s
        process.include_top_n.by_cpu: null
        process.include_top_n.by_memory: null
        processes: null
      - id: system/metrics-system.socket_summary
        dataset:
          name: system.socket_summary
          type: metrics
        metricsets:
          - socket_summary
        core.metrics: null
        cpu.metrics: null
        period: 10s
        process.include_top_n.by_cpu: null
        process.include_top_n.by_memory: null
        processes: null
      - id: system/metrics-system.uptime
        dataset:
          name: system.uptime
          type: metrics
        metricsets:
          - uptime
        core.metrics: null
        cpu.metrics: null
        period: 10s
        processes: null

Testing

Use:

• Local build of package registry from Update package output structure package-registry#514 with xpack.ingestManager.epm.registryUrl: 'http://localhost:8080'
• Latest agent snapshot
• Fresh ES instance with no previous data

All functionality should work normally with no UI changes, apart from slight agent config YAML view changes 😄

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[paul-tavares]

I will try to check this out locally today to see if its a matter of just updating the test case to match the new full agent config output

[jen-huang]

@paul-tavares The test failed because I moved package info under a new field, meta, so the snapshot needs to be updated. Working on it now!

[paul-tavares]

@jen-huang cool. I assume all that moving around is being coordinated with Agent and that the Endpoint specific data will still reach the endpoint, so 👍

[ruflin]

@michalpristas for your awareness. As Agent ignores this at the moment, it should not have any effect.

[ruflin]

@elasticmachine merge upstream

[ruflin]

@jen-huang I tested this PR together with the registry and the newest agent and everything seems to work as expected. I suggest we get the two PR's in in parallel and open quickly a backport for KB.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: e06bf3f

Build metrics

✅ unchanged

History

• 💚 Build #56744 succeeded cd6f2d1
• 💔 Build #56505 failed 3b4e75e
• 💚 Build #56129 succeeded f34e642

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream