Skip to content

Commit

Permalink
Add support for appinsights in the metricbeat azure module (elastic#1…
Browse files Browse the repository at this point in the history
…8940)

* app

* changelog

* update

* Revert "update"

This reverts commit 06db76b.

* avoid winio

* fix text

(cherry picked from commit 2efea84)
  • Loading branch information
narph committed Jul 13, 2020
1 parent a1e598c commit 9b74900
Show file tree
Hide file tree
Showing 24 changed files with 725 additions and 13 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -619,6 +619,7 @@ field. You can revert this change by configuring tags for the module and omittin
- The `elasticsearch/index` metricset now collects metrics for hidden indices as well. {issue}18639[18639] {pull}18703[18703]
- Added `performance` and `query` metricsets to `mysql` module. {pull}18955[18955]
- The `elasticsearch-xpack/index` metricset now reports hidden indices as such. {issue}18639[18639] {pull}18706[18706]
- Adds support for app insights metrics in the azure module. {issue}18570[18570] {pull}18940[18940]

*Packetbeat*

Expand Down
47 changes: 47 additions & 0 deletions metricbeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -4516,6 +4516,53 @@ type: keyword
Azure metric dimensions.


type: object

--

[float]
=== app_insights

application insights



*`azure.app_insights.application_id`*::
+
--
The application ID


type: keyword

--

*`azure.app_insights.start_date`*::
+
--
The start date


type: date

--

*`azure.app_insights.end_date`*::
+
--
The end date


type: date

--

*`azure.app_insights.metrics.*.*`*::
+
--
The metrics


type: object

--
Expand Down
16 changes: 15 additions & 1 deletion metricbeat/docs/modules/azure.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,9 @@ so the `period` for `container_service` metricset should be `300s` or multiples
This metricset will collect relevant metrics from specified database accounts, these metrics will have a timegrain every 5 minutes,
so the `period` for `database_account` metricset should be `300s` or multiples of `300s`.

[float]
=== `app_insights`
This metricset will collect application insights metrics, the `period` (interval) for the `app-insights` metricset is set by default at `300s`.

[float]
== Additional notes about metrics and costs
Expand Down Expand Up @@ -198,7 +201,6 @@ metricbeat.modules:
tenant_id: '${AZURE_TENANT_ID:""}'
subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
- module: azure
metricsets:
- database_account
Expand All @@ -209,13 +211,23 @@ metricbeat.modules:
tenant_id: '${AZURE_TENANT_ID:""}'
subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
- module: azure
metricsets:
- app_insights
enabled: true
period: 300s
application_id: ''
api_key: ''
----

[float]
=== Metricsets

The following metricsets are available:

* <<metricbeat-metricset-azure-app_insights,app_insights>>

* <<metricbeat-metricset-azure-compute_vm,compute_vm>>

* <<metricbeat-metricset-azure-compute_vm_scaleset,compute_vm_scaleset>>
Expand All @@ -232,6 +244,8 @@ The following metricsets are available:

* <<metricbeat-metricset-azure-storage,storage>>

include::azure/app_insights.asciidoc[]

include::azure/compute_vm.asciidoc[]

include::azure/compute_vm_scaleset.asciidoc[]
Expand Down
23 changes: 23 additions & 0 deletions metricbeat/docs/modules/azure/app_insights.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
////
This file is generated! See scripts/mage/docs_collector.go
////

[[metricbeat-metricset-azure-app_insights]]
=== Azure app_insights metricset

beta[]

include::../../../../x-pack/metricbeat/module/azure/app_insights/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-azure,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../../x-pack/metricbeat/module/azure/app_insights/_meta/data.json[]
----
3 changes: 2 additions & 1 deletion metricbeat/docs/modules_list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ This file is generated! See scripts/mage/docs_collector.go
|<<metricbeat-metricset-aws-usage,usage>> beta[]
|<<metricbeat-metricset-aws-vpn,vpn>> beta[]
|<<metricbeat-module-azure,Azure>> |image:./images/icon-yes.png[Prebuilt dashboards are available] |
.8+| .8+| |<<metricbeat-metricset-azure-compute_vm,compute_vm>>
.9+| .9+| |<<metricbeat-metricset-azure-app_insights,app_insights>> beta[]
|<<metricbeat-metricset-azure-compute_vm,compute_vm>>
|<<metricbeat-metricset-azure-compute_vm_scaleset,compute_vm_scaleset>>
|<<metricbeat-metricset-azure-container_instance,container_instance>>
|<<metricbeat-metricset-azure-container_registry,container_registry>>
Expand Down
1 change: 1 addition & 0 deletions x-pack/metricbeat/include/list.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 8 additions & 1 deletion x-pack/metricbeat/metricbeat.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,6 @@ metricbeat.modules:
tenant_id: '${AZURE_TENANT_ID:""}'
subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'


- module: azure
metricsets:
- database_account
Expand All @@ -324,6 +323,14 @@ metricbeat.modules:
tenant_id: '${AZURE_TENANT_ID:""}'
subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'

- module: azure
metricsets:
- app_insights
enabled: true
period: 300s
application_id: ''
api_key: ''


#--------------------------------- Beat Module ---------------------------------
- module: beat
Expand Down
9 changes: 8 additions & 1 deletion x-pack/metricbeat/module/azure/_meta/config.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,6 @@
tenant_id: '${AZURE_TENANT_ID:""}'
subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'


- module: azure
metricsets:
- database_account
Expand All @@ -84,3 +83,11 @@
tenant_id: '${AZURE_TENANT_ID:""}'
subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'

- module: azure
metricsets:
- app_insights
enabled: true
period: 300s
application_id: ''
api_key: ''

8 changes: 8 additions & 0 deletions x-pack/metricbeat/module/azure/_meta/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,3 +90,11 @@
# tenant_id: '${AZURE_TENANT_ID:""}'
# subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
# refresh_list_interval: 600s

#- module: azure
# metricsets:
# - app_insights
# enabled: true
# period: 300s
# application_id: ''
# api_key: ''
3 changes: 3 additions & 0 deletions x-pack/metricbeat/module/azure/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,9 @@ so the `period` for `container_service` metricset should be `300s` or multiples
This metricset will collect relevant metrics from specified database accounts, these metrics will have a timegrain every 5 minutes,
so the `period` for `database_account` metricset should be `300s` or multiples of `300s`.

[float]
=== `app_insights`
This metricset will collect application insights metrics, the `period` (interval) for the `app-insights` metricset is set by default at `300s`.

[float]
== Additional notes about metrics and costs
Expand Down
31 changes: 31 additions & 0 deletions x-pack/metricbeat/module/azure/app_insights/_meta/data.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"azure" : {
"app_insights" : {
"metrics" : {
"requests_failed" : {
"sum" : 182
},
"request_name" : "GET /favicon.ico"
},
"start_date" : "2020-07-12T10:52:11.831Z",
"end_date" : "2020-07-12T12:52:11.831Z",
"application_id" : "42cb59a9-d5be-400b-a5c4-69b0a0026ac6"
}
},
"cloud": {
"provider": "azure"
},
"event": {
"dataset": "azure.app_insights",
"duration": 115000,
"module": "azure"
},
"metricset": {
"name": "app_insights",
"period": 10000
},
"service": {
"type": "azure"
}
}
61 changes: 61 additions & 0 deletions x-pack/metricbeat/module/azure/app_insights/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
This is the app_insights metricset.

This metricset allows users to retrieve application insights metrics from specified applications.

[float]
==== Config options to identify resources

`application_id`:: (_[]string_) ID of the application. This is Application ID from the API Access settings blade in the Azure portal.

`api_key`:: (_[]string_) The API key which will be generated, more on the steps here https://dev.applicationinsights.io/documentation/Authorization/API-key-and-App-ID.

[float]
==== App insights metric configurations

`metrics`:: List of different metrics to collect information

`id`:: (_[]string_) IDs of the metrics that's being reported. Usually, the id is descriptive enough to help identify what's measured.
A list of metric names can be entered as well.
Default metricsets include: `requests/count` `requests/duration` `requests/failed` `users/count``users/authenticated`
`pageViews/count` `pageViews/duration` `customEvents/count`
`browserTimings/processingDuration` `browserTimings/receiveDuration` `browserTimings/networkDuration` `browserTimings/sendDuration`
`browserTimings/totalDuration` `dependencies/count` `dependencies/duration` `dependencies/failed`
`exceptions/count` `exceptions/browser` `exceptions/server` `sessions/count` `performanceCounters/requestExecutionTime` `performanceCounters/requestsPerSecond`
`performanceCounters/requestsInQueue` `performanceCounters/memoryAvailableBytes` `performanceCounters/exceptionsPerSecond` `performanceCounters/processCpuPercentage`
`performanceCounters/processIOBytesPerSecond` `performanceCounters/processPrivateBytes` `performanceCounters/processorCpuPercentage` `availabilityResults/count`
`availabilityResults/availabilityPercentage` `availabilityResults/duration`

`interval`:: (_string_) The time interval to use when retrieving metric values. This is an ISO8601 duration.
If interval is omitted, the metric value is aggregated across the entire timespan.
If interval is supplied, the result may adjust the interval to a more appropriate size based on the timespan used for the query.

`aggregation`:: (_[]string_) The aggregation to use when computing the metric values.
To retrieve more than one aggregation at a time, separate them with a comma.
If no aggregation is specified, then the default aggregation for the metric is used.

`segment`:: (_[]string_) The name of the dimension to segment the metric values by.
This dimension must be applicable to the metric you are retrieving.
In this case, the metric data will be segmented in the order the dimensions are listed in the parameter.

`top`:: (_int_) The number of segments to return. This value is only valid when segment is specified.

`order_by`:: (_string_) The aggregation function and direction to sort the segments by.
This value is only valid when segment is specified.

`filter`:: (_string_) An expression used to filter the results.
This value should be a valid OData filter expression where the keys of each clause should be applicable dimensions for the metric you are retrieving.

Users can select the options to retrieve all metrics from a specific namespace using the following:

["source","yaml"]
----
metrics:
- id: ["*"]
timespan: "Microsoft.Storage/storageAccounts"
----



A default non configurable timegrain of 5 min is set so users are advised to configure an interval of 300s or a multiply of it.


24 changes: 24 additions & 0 deletions x-pack/metricbeat/module/azure/app_insights/_meta/fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
- name: app_insights
type: group
release: beta
description: >
application insights
fields:
- name: application_id
type: keyword
description: >
The application ID
- name: start_date
type: date
description: >
The start date
- name: end_date
type: date
description: >
The end date
- name: metrics.*.*
type: object
object_type: float
object_type_mapping_type: "*"
description: >
The metrics
Loading

0 comments on commit 9b74900

Please sign in to comment.