DynamoDB with Global table v2019.11.21 and PROVISIONED and autscaling generates Table Capacity and/or GSI capacityValidationException

[onema]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.24
provider.aws v2.59.0

Affected Resource(s)

• aws_dynamodb_table
• aws_appautoscaling_policy
• aws_appautoscaling_target

Terraform Configuration Files

resource "aws_dynamodb_table" "table" {
  billing_mode     = "PROVISIONED"
  hash_key         = "PK"
  name             = "test-one"
  read_capacity    = 10
  write_capacity   = 5
  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"
  attribute {
    name = "PK"
    type = "S"
  }
  replica {
    region_name = "us-west-2"
  }
}

resource "aws_appautoscaling_target" "dynamodb-table_read_target" {
  max_capacity       = 100
  min_capacity       = 50
  resource_id        = "table/${aws_dynamodb_table.table.name}"
  scalable_dimension = "dynamodb:table:ReadCapacityUnits"
  service_namespace  = "dynamodb"
}

resource "aws_appautoscaling_policy" "dynamodb-table_read_policy" {
  name               = "dynamodb-read-capacity-utilization-${aws_appautoscaling_target.dynamodb-table_read_target.resource_id}"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.dynamodb-table_read_target.resource_id
  scalable_dimension = aws_appautoscaling_target.dynamodb-table_read_target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.dynamodb-table_read_target.service_namespace

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "DynamoDBReadCapacityUtilization"
    }
    target_value = 70
  }
}

resource "aws_appautoscaling_target" "dynamodb-table_write_target" {
  max_capacity       = 100
  min_capacity       = 50
  resource_id        = "table/${aws_dynamodb_table.table.name}"
  scalable_dimension = "dynamodb:table:WriteCapacityUnits"
  service_namespace  = "dynamodb"
}

resource "aws_appautoscaling_policy" "dynamodb-table_write_policy" {
  name               = "dynamodb-write-capacity-utilization-${aws_appautoscaling_target.dynamodb-table_write_target.resource_id}"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.dynamodb-table_write_target.resource_id
  scalable_dimension = aws_appautoscaling_target.dynamodb-table_write_target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.dynamodb-table_write_target.service_namespace

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "DynamoDBWriteCapacityUtilization"
    }
    target_value = 70
  }
}

Debug Output

gist

Panic Output

NA

Expected Behavior

Replica for Gobal tables v2019.11.21 should play nicely with autoscaling:

1. create the table
2. create the autoscaling target and policies
3. apply the replication

Actual Behavior

Errors out with a message:

error creating DynamoDB Table (test-one) replicas: error creating DynamoDB Table (test-one) replicas: ValidationException: Table write capacity should either be Pay-Per-Request or AutoScaled.

Steps to Reproduce

1. terraform apply

Important Factoids

NA

References

• DynamoDB V2019.11.21 #12342 (comment)

[mgreen-sm]

We've encountered this issue as well, using pretty much the same TF configuration as outlined in the report above.

I've worked around it by initially setting up the DynamoDB resource with billing_mode set to PAY_PER_REQUEST and then use a provisioner "local-exec" { ... } section in the resource that calls a script:

• Wait until the table exists (aws dynamodb table-exists ...)
• Change the billing mode to PROVISIONED (aws dynamodb update-table ...)
• And wait for the table to actually show this as the billing mode (aws dynamodb describe-table ...).

The aws_appautoscaling_target and aws_appautoscaling_policy resources are blocked from executing until the DynamoDB resource local provisioner is finished. So once the script is done (and that waiting for the billing mode is important), the auto scaling resources are also created without issue.

Note that the DynamoDB resource would also need to have a lifecycle { ignore_changes = [ billing_mode, read_capacity, write_capacity ] } section, to prevent it from 'correcting' the script's changes.

It isn't pretty, but allows us to continue with the desired state. Hopefully this will help someone until it is fixed.

[AashishAsh]

I'm encountering the same issue as well, Replica block isn't working as expected at the time of table creation.

Error: error creating DynamoDB Table (example_test) replicas: error creating DynamoDB Table (example_test) replica (us-west-2): ValidationException: Table write capacity should either be Pay-Per-Request or AutoScaled.
status code: 400, request id: JHO3C0MD070PG09U89HA1LUFKNVV4KQNSO5AEMVJF66Q9ASUAAJG

on modules/aws-dynamodb/module.tf line 11, in resource "aws_dynamodb_table" "dynamodb_table_master":
11: resource "aws_dynamodb_table" "dynamodb_table_master" {

If I add replica block after the table creation in resource aws_dynamodb_table then it works but adding replica block before in the module is throwing above error.

[jawspeak]

I have this error when I try to make a global table (the new v2 2019.11.21) type per: https://www.terraform.io/docs/providers/aws/r/dynamodb_table.html

resource "aws_dynamodb_table" "xyz" {
  provider = aws.default-as-east #my provider in us-east-1
  name = "xyz"
  billing_mode = "PROVISIONED"
  read_capacity = 5
  write_capacity = 5
  hash_key = "id"
  range_key = "created_at"

  local_secondary_index {
    name = "index_state"
    range_key = "state"
    projection_type = "ALL"
  }

  attribute {
      name = "id"
      type = "S"
  }

  attribute {
      name = "created_at"
      type = "S"
  }

  attribute {
    name = "state"
    type = "S"
  }

  # breaks with this, works without replica set below.
  replica {
    region_name = "us-west-2"
  }
}

The error is:
Error: error creating DynamoDB Table (...) replicas: error creating DynamoDB Table (...) replicas: ValidationException: Table write capacity should either be Pay-Per-Request or AutoScaled.

[artem-kosenko]

Is the any progress there? I have the same issue and looks like I'm not alone

resource "aws_dynamodb_table" "dynamodb" {
    hash_key         = var.primary_partition_key
    range_key        = var.primary_sort_key
    name             = var.name
    stream_enabled   = true
    stream_view_type = "NEW_AND_OLD_IMAGES"
    billing_mode     = "PROVISIONED"
    read_capacity    = var.read_capacity
    write_capacity   = var.write_capacity
    tags             = var.tags
    attribute {
        name = var.primary_partition_key
        type = var.primary_partition_key_type
    }
    attribute {
        name = var.primary_sort_key
        type = var.primary_sort_key_type
    }
    replica {
        region_name = "eu-west-1"
    }
    replica {
        region_name = "ap-southeast-1"
    }
    lifecycle {
        ignore_changes = [ billing_mode, read_capacity, write_capacity ]
    }
}

error creating DynamoDB Table (my_table_name) replica (eu-west-1): ValidationException: Table write capacity should either be Pay-Per-Request or AutoScaled.

[artem-kosenko]

I'm encountering the same issue as well, Replica block isn't working as expected at the time of table creation.

Error: error creating DynamoDB Table (example_test) replicas: error creating DynamoDB Table (example_test) replica (us-west-2): ValidationException: Table write capacity should either be Pay-Per-Request or AutoScaled.
status code: 400, request id: JHO3C0MD070PG09U89HA1LUFKNVV4KQNSO5AEMVJF66Q9ASUAAJG

on modules/aws-dynamodb/module.tf line 11, in resource "aws_dynamodb_table" "dynamodb_table_master":
11: resource "aws_dynamodb_table" "dynamodb_table_master" {

If I add replica block after the table creation in resource aws_dynamodb_table then it works but adding replica block before in the module is throwing above error.

this does not work on very firs creation and when I create a table without replicas blocks, add them afterwards and apply one mode time. In both cases it failed with the same error.

[artem-kosenko]

resolved it with billing_mode = "PAY_PER_REQUEST"

now creating/updating/destroying works perfect

resource "aws_dynamodb_table" "dynamodb" {

    hash_key         = var.primary_partition_key
    range_key        = var.primary_sort_key
    name             = var.name
    stream_enabled   = true
    stream_view_type = "NEW_AND_OLD_IMAGES"
    billing_mode     = "PAY_PER_REQUEST"
    tags             = var.tags

    attribute {
        name = var.primary_partition_key
        type = var.primary_partition_key_type
    }
    attribute {
        name = var.primary_sort_key
        type = var.primary_sort_key_type
    }
    dynamic "replica" {
        for_each = var.replica_regions
        content {
            region_name = replica.value
        }
    }
}

[brandybutler]

Same issue, any updates?

My workaround was using the 2017 "layered" tables approach. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/dynamodb_global_table

[gyoga99]

Seeing same issue.

[quinyx-tjeerd]

This can be solved by initially having the replication turned off, and applying everything.

In case of GSI's you need to create additional scaling targets and policies for each index, which is also described for CLI here:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.CLI.html#AutoScaling.CLI.RegisterScalableTarget

difference compared to the normal target and policy the resource id is different as is the scalable dimension.

    resource_id = "table/TestTable/index/test-index" 
    scalable_dimension = "dynamodb:index:WriteCapacityUnits"

after that is applied, enable the region and have terraform update in place.

[onema]

This can be solved by initially having the replication turned off, and applying everything.

@quinyx-tjeerd, this is really not a solution but rather a valid workaround around for this problem. The idea of this ticket is for Terraform to take care of such details or at least warn or error in the planning face and not as it is creating resources.

[mcintyremason]

Currently experiencing this issue. Does the merged PR above solve it? Also, I agree with @onema. Terraform should be able to handle creating the DDB replicas, or at a minimum return an error during the call to terraform plan.

[jisnor-rs]

I too am struggling with this at the moment. Our use-case was that we had a provisioned-throughput table with autoscaling read and write credits already created and I attempted to add a replica. Using PAY_PER_REQUEST isn't a valid solution for us because then the autoscaling targets complain.

[anGie44]

Hey all, after some investigation, I've found that this is a limitation of the DynamoDB API and managing a table+replica of this kind in terraform is troublesome to do in a single apply (validating @quinyx-tjeerd 's workaround #13097 (comment)). For a replica to be created from a table that uses PROVISIONED mode, you have to make sure write auto-scaling is enabled for that table i.e. via separate AppAutoscaling configurations managed by the appautoscaling AWS service, not dynamodb. Thus if the Dynamodb API were to expose a way to enable autoscaling during create time, then the provider could handle this directly. Unfortunately, atm, there is only a method for updates after a replica has been created: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateTableReplicaAutoScaling.html

The AWS console, for example does allow one to configure autoscaling at create time, but again, this is not exposed in the AWS Go SDK's dynamodb package which the dynamodb_table resource is built around, and behind the scenes (as seen in the screenshot's right sidebar), AppAutoScaling settings are handled for you:

With that said, I do not foresee a plausible code change to solve this issue, but rather want to reiterate the workaround by @quinyx-tjeerd as a viable interim solution to the upstream service's limitation; first apply the dynamodb table (and related resources like appautoscaling) without the replica configuration block. then once complete, re-apply the config with replica configured.

Reference:

* Best Practices and Requirements for Managing Global Tables

When using the latest version of global tables, you must either use on-demand capacity or enable autoscaling on the table. Using autoscaling or on-demand capacity ensures that you always have sufficient capacity to perform replicated writes to all regions of the global table.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.