[Metrics UI] Add AWS Metricsets to Inventory Models

[simianhacker]

Summary

This PR adds the AWS metricset to the Inventory Models for EC2, RDS, S3 and SQS. It also cleans up a few areas that were missed in the Inventory Model PR.

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

• This was checked for cross-browser compatibility, including a check against IE11
• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• This was checked for keyboard-only and screenreader accessibility

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: e0bb941

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: f8db271

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: 2b80ccf

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: 1dcc91c

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: da9359d

[simianhacker]

@sorantis What "actions" should we have for the AWS EC2 metrics? Here is what we have now:

I think the question is "Can we correlate on EC2 data with our other pillars of Observability? If so what are the fields used for each option?"

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: a9d4684

[sorantis]

@simianhacker good question.

• View metrics should be possible as we [collect a bunch of them].(https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-metricset-aws-ec2.html)
• View logs is a bit trickier. Currently we have support for S3 logs, and the plan is to add CloudWatch support. @kaiyan-sheng what would we need from AWS filebeat module to ensure correlation with logs? Can it be done already today? If so, what fields can we use?
• View EC2 APM traces - I don't think we have any special handling for EC2. If an APM agent is installed on an EC2 VM, I'd assume we have similar correlation as with any host.
• View EC2 in Uptime - Looks like Uptime can interact with AWS and get cloud metadata.

[kaiyan-sheng]

@sorantis What kind of correlation are you thinking? By looking at the ec2 log example, there will be the instanceId field that can be directly link back to ec2 metrics from Metricbeat.

I will start the investigation work around cloudwatch logs after I get the permissions I need from infra team.

[sorantis]

@kaiyan-sheng something that could tie AWS logs to metrics so that the View Logs would link to something meaningful. I believe the instance id could be it. @simianhacker wdyt?

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: e0aef39

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: 5414ec3

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: 6ce5c0c

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[phillipb]

What about creating some util funcs for these modelCreators. For example:

const createAWSEC2Model = (indexPattern: string, timeField: string, series: any) => {
    return {
      requires: ['aws.ec2'],
      .....
      series
    }
}

createAWSEC2Model([
    {
      id: 'write',
      split_mode: 'everything',
      metrics: [
        {
          field: 'aws.ec2.diskio.write.bytes_per_sec',
          id: 'avg-write',
          type: 'avg',
        },
      ],
    },
    {
      id: 'read',
      split_mode: 'everything',
      metrics: [
        {
          field: 'aws.ec2.diskio.read.bytes_per_sec',
          id: 'avg-read',
          type: 'avg',
        },
        {
          id: 'calculation-rate',
          type: 'calculation',
          variables: [{ id: 'rate-var', name: 'rate', field: 'avg-read' }],
          script: 'params.rate * -1',
        },
      ],
    },
  ]);

Doing that could save about 70 lines of code in this PR alone.

[simianhacker]

That would work... I was also thinking for all the basic ones where it's just an avg/max/min we could just create a function that takes the field name. I'm sure that would reduce things down some too.

[phillipb]

Looks great! Lots of code, but I think that might just be the nature of what we're trying to build. Couple of tiny questions.

[phillipb]

Does it?

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 53b507a

History

• 💚 Build #15009 succeeded 09555e5
• 💚 Build #15002 succeeded 9b23276
• 💚 Build #14951 succeeded f662730
• 💚 Build #14862 succeeded cab5a3a
• 💔 Build #14804 failed ce4eda5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream