Cannot create custom metric alert via azure cli.

[konrad-jamrozik]

Note: this pertains to
https://docs.microsoft.com/en-us/cli/azure/monitor/metrics?view=azure-cli-latest#az-monitor-metrics-list-definitions
and
https://docs.microsoft.com/en-us/cli/azure/monitor/metrics/alert?view=azure-cli-latest#az-monitor-metrics-alert-create

Describe the bug
I ran

$aiid = "/subscriptions/CENSORED/resourceGroups/CENSORED/providers/microsoft.insights/components/CENSORED"
az monitor metrics list-definitions --resource $aiid

and expected to see my custom metric definitions (in custom namespaces), but didn't found any custom metrics.

To Reproduce
Log a custom metric from the source code using ApplicationIsnights. Observe you can create an alert using it it via azure portal, or view it in metrics view in AI. However, it is not listed via list-definitions, nor you can create an alert via Azure CLI using it, as it is not found.

Expected behavior
The custom metric (and its namespace) to be listed, and accepted when creating an alert with az monitor metrics alert create.

Environment summary

az --version
azure-cli                         2.0.66

Additional context
Confirmed the custom metrics also cannot be found via

Get-AzMetricDefinition -ResourceId $aiid

[tjprescott]

The command returns whatever the service does. Will pass along to the service for comment on whether this is the expected behavior or not.

[konrad-jamrozik]

To provide additional context:

I created a metric alert via azure portal, using my custom metric. When I viewed it like this:
az monitor metrics alert show --name "my custom alert" --resource-group my-resource-group

It had entry like

 "allOf": [
      {
        "dimensions": [],
        "metricName": "MyMetricName",
        "metricNamespace": "MyMetricNamespace",
        "monitorTemplateType": 8,
        "name": "Metric1",
        "operator": "GreaterThan",
        "threshold": 1000.0,
        "timeAggregation": "Maximum"
      }

Observe the metricNamespace.

However, all the metrics returned by list-definitions had namespace: "namespace": "microsoft.insights/components",.

Once I created a metric (via az cli) using one of the available ones, it has shown:

        "metricName": "requests/failed",
        "metricNamespace": "microsoft.insights/components",

So perhaps the issue is the command doesn't recognize custom namespaces, which are there?

[konrad-jamrozik]

I have a "diff repro" showing this works with Az PS1 module, but not with az CLI.

Here it is working:

$resourceGroup = "my-ai-rg"
$MyAIId = "/subscriptions/mySubId/resourceGroups/my-ai-rg/providers/microsoft.insights/components/my-ai-instance"
$actionGroupName = "myActionGroupName"

# https://docs.microsoft.com/en-us/cli/azure/monitor/action-group?view=azure-cli-latest#az-monitor-action-group-create
az monitor action-group create --name $actionGroupName --resource-group $resourceGroup
<# { (...)
  "groupShortName": "...",
  "id": "/subscriptions/mySubId/resourceGroups/my-ai-rg/providers/microsoft.insights/actionGroups/myActionGroupName",
  "name": "myActionGroupName",
  "resourceGroup": "my-ai-rg",
  "type": "Microsoft.Insights/ActionGroups",
} #>

# https://docs.microsoft.com/en-us/powershell/module/az.monitor/get-azactiongroup?view=azps-2.2.0
$actionGroup = Get-AzActionGroup -ResourceGroupName $resourceGroup -Name $actionGroupName

# https://docs.microsoft.com/en-us/powershell/module/az.monitor/new-azactiongroup?view=azps-2.2.0
$actionGroupId = New-AzActionGroup -ActionGroupId $actionGroup.Id

# https://docs.microsoft.com/en-us/powershell/module/az.monitor/new-azmetricalertrulev2criteria?view=azps-2.2.0
$cond = New-AzMetricAlertRuleV2Criteria `
   -MetricName "MyMetric" `
   -MetricNamespace "MyNamespace/Foo" `
   -TimeAggregation "Maximum" `
   -Operator "GreaterThan" `
   -Threshold "1000"

# https://docs.microsoft.com/en-us/powershell/module/az.monitor/add-azmetricalertrulev2?view=azps-2.2.0
Add-AzMetricAlertRuleV2 `
-Name "myalert" -ResourceGroupName $resourceGroup -TargetResourceId $MyAIId `
-Description "MyDesc" -Severity 4 -Condition $cond -ActionGroup $actionGroupId `
-WindowSize 0:5 -Frequency 0:5

while this fails

az monitor metrics alert create -n "myalert" -g $resourceGroup --description "mydesc" --scopes $MyAIId `
--condition "max MyNamespace/Foo.MyMetric > 500" --disabled

with
The metric names were not found MyNamespace/Foo.MyMetric.. [Code: "BadRequest"]

[konrad-jamrozik]

Hey @tjprescott, do you have an estimate when I can expect some response about this? I understand it might take time, just want to set my expectations accordingly. I just discovered I cannot create dynamic threshold alerts via Az PS1 module either ("not yet supported"), so now I will have to fallback to ARM templates or REST API (ungh ;( ).

[konrad-jamrozik]

Looks like ARM template client, both PS1 and Azure CLI, suffers from the same issue, e.g.:

New-AzResourceGroupDeployment -ResourceGroupName "myresgroup" -TemplateFile "alert.json" -TemplateParameterFile "parameters.json"
(...)
New-AzResourceGroupDeployment : 7:45:42 PM - Resource Microsoft.Insights/metricAlerts 'myalertname' failed with message '{
  "Code": "BadRequest",
  "Message": "The metric names were not found MyCustomMetric."
}'

[tjprescott]

@snehithm for comment.

@konrad-jamrozik if you know the API structure, you could also use the generic resource create/update commands as an alternative to ARM templates or direct REST (also there's a new az rest command to help with direct REST calls)

[konrad-jamrozik]

@tjprescott I managed to create a dynamic threshold alert based on my custom metric using the raw REST API. Thus, I have a workaround, but it is not ideal, as using the REST API directly requires creating, securing and managing a service principal. The az rest seems to have a broken logic for parsing body json. I have filed an issue for that: #9742. Hope that helps!

[tjprescott]

@konrad-jamrozik resource create/update have been around longer than az rest, so you might try giving that a shot.

[konrad-jamrozik]

@tjprescott I made it to work!

az resource create --name $ruleName --resource-group $resourceGroupName_appInsights --subscription $subscriptionId --namespace "Microsoft.Insights" --resource-type "metricAlerts" --is-full-object --properties "@dynamic_threshold_metric_alert_body.json"

For reference, I also made it to work via az rest as I explained in #9742.

This means I am unblocked. Thank you so much! This issue is still technically not addressed, though.

[tjprescott]

@konrad-jamrozik happy you are unblocked! I will keep this issue open to track the feature request.

[pajzo]

I'm having the same issue right now (custom application insights metric), and I think it's sad to have a custom REST call to create this alert when all my other alerts can be created with native methods :(

[snehithm]

Hi @konrad-jamrozik , @pajzo, thanks for reporting this. We are looking into this.

[pajzo]

If this is any help here is what i found out:

az monitor metrics alert create `
    --name "DEV INPAX Hangfire Failed" `
    --scopes "/subscriptions/$subscriptionId/resourceGroups/$resourceGroup/providers/microsoft.insights/components/$appInsightsResource" `
    --description "Failed jobs" `
    --condition "max azure.applicationinsights.INPAXHangfireFailed > 0" `
    --severity 1 `
    --evaluation-frequency 1m `
    --window-size 1m `
    --resource-group "$resourceGroup" `
    --subscription "$subscriptionId" `

But in debug mode i can see that the namespace is wrong. The Azure CLI is setting applicationinsights as namespace, but the correct is azure.applicationinsights

[{
	"location": "global",
	"properties": {
		"description": "Failed jobs",
		"severity": 1,
		"enabled": true,
		"scopes": ["/subscriptions/xxxxx/resourceGroups/xxxxx/providers/microsoft.insights/components/xxxxx"],
		"evaluationFrequency": "PT1M",
		"windowSize": "PT1M",
		"criteria": {
			"odata.type": "Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria",
			"allOf": [{
				"name": "cond0",
				"metricName": "INPAXHangfireFailed",
				"metricNamespace": "applicationinsights",
				"operator": "GreaterThan",
				"timeAggregation": "Maximum",
				"threshold": 0.0,
				"dimensions": []
			}]
		}
	}
}]

[K31D]

If you pass an empty string to -DynamicThreshold it lets you create the criteria unfortunatelly when you create the alert rule it comes with static "default" threshold.

protected override void ProcessRecordInternal() |  
-- | --
  | {
  | if(String.IsNullOrWhiteSpace(this.DynamicThreshold))
  | {
  | List<MetricDimension> metricDimensions = new List<MetricDimension>();
  |  
  | if (this.DimensionSelection!= null && this.DimensionSelection.Length > 0)
  | {
  | foreach (var dimension in this.DimensionSelection)
  | {
  | if (dimension.IncludeValues != null && dimension.IncludeValues.Count() > 0)
  | {
  | metricDimensions.Add(new MetricDimension(dimension.Dimension, "Include", dimension.IncludeValues));
  | }
  | if (dimension.ExcludeValues != null && dimension.ExcludeValues.Count() > 0)
  | {
  | metricDimensions.Add(new MetricDimension(dimension.Dimension, "Exclude", dimension.ExcludeValues));
  | }
  | }
  | }
  | else
  | {
  | metricDimensions = null;
  | }
  | MetricCriteria metricCriteria = new MetricCriteria(name: "metric1", metricName: this.MetricName, operatorProperty: this.Operator, timeAggregation: this.TimeAggregation, threshold: this.Threshold, metricNamespace: this.MetricNamespace, dimensions: metricDimensions);
  | PSMetricCriteria result = new PSMetricCriteria(metricCriteria);
  | WriteObject(sendToPipeline: result);
  | }
  | else
  | {
  | WriteExceptionError(new Exception("Creating   criteria for Dynamic Threshold is not yet supported"));
  | }

[mmyyrroonn]

@snehithm Hello. Any update about this issue?

[snehithm]

Hi @myronfanqiu , this issue should be fixed now based on #10108. @tjprescott can you confirm?

[harelbr]

Creating a metric alert rule on a custom metric is supported starting version 2.0.71 (released on Aug 13, 2019)