DynamoTable creates but fails to reconcile (provisionedThroughput error)

[JasP19]

What happened?

When I apply the yaml for a DynamoTable resource, I can see (through the AWS management console) that the table is created successfully. However, I notice that reconciliation is being attempted extremely often (more than once per second) and failing with the following error message:

update failed: cannot update DynamoDB table: ValidationException: The provisioned throughput for the table will not change. The requested value equals the current value. Current ReadCapacityUnits provisioned for the table: 1. Requested ReadCapacityUnits: 1. Current WriteCapacityUnits provisioned for the table: 1. Requested WriteCapacityUnits: 1. Refer to the Amazon DynamoDB Developer Guide for current limits and how to request higher limits.

How can we reproduce it?

Apply the following sample manifest (through ArgoCD in my case):

apiVersion: database.aws.crossplane.io/v1alpha1
kind: DynamoTable
metadata:
  name: test-table
spec:
  deletionPolicy: Orphan
  forProvider:
    region: eu-west-1
    attributeDefinitions:
      - attributeName: testCol
        attributeType: S
    keySchema:
      - attributeName: testCol
        keyType: HASH
    provisionedThroughput: 
      readCapacityUnits: 1
      writeCapacityUnits: 1
    tag:
      - tag: environment
        value: dev
      - tag: provisionedBy
        value: Crossplane

The table is correctly created but describing the table shows the following:

Name:         test-table
Namespace:
Labels:       argocd.argoproj.io/instance=xxx
Annotations:  crossplane.io/external-name: test-table
API Version:  database.aws.crossplane.io/v1alpha1
Kind:         DynamoTable
Metadata:
  Creation Timestamp:  2020-12-07T07:36:33Z
  Finalizers:
    finalizer.managedresource.crossplane.io
  Generation:        2
  Resource Version:  729018380
  Self Link:         /apis/database.aws.crossplane.io/v1alpha1/dynamotables/test-table
  UID:               a1e4b45d-2038-434f-a3c1-ed3009382509
Spec:
  Deletion Policy:  Orphan
  For Provider:
    Attribute Definitions:
      Attribute Name:  testCol
      Attribute Type:  S
    Key Schema:
      Attribute Name:  testCol
      Key Type:        HASH
    Provisioned Throughput:
      Read Capacity Units:   1
      Write Capacity Units:  1
    Region:     eu-west-1
    Tag:
      Tag:    environment
      Value:  dev
      Tag:    provisionedBy
      Value:  Crossplane
  Provider Config Ref:
    Name:  default
Status:
  At Provider:
    Attribute Definitions:
      Attribute Name:  testCol
      Attribute Type:  S
    Key Schema:
      Attribute Name:  testCol
      Key Type:        HASH
    Provisioned Throughput:
      Read Capacity Units:   1
      Write Capacity Units:  1
    Table Arn:               arn:aws:dynamodb:eu-west-1:xxx:table/test-table
    Table Id:                xxx
    Table Name:              test-table
    Table Status:            ACTIVE
  Conditions:
    Last Transition Time:  2020-12-07T07:36:38Z
    Reason:                Available
    Status:                True
    Type:                  Ready
    Last Transition Time:  2020-12-07T07:37:19Z
    Message:               update failed: cannot update DynamoDB table: ValidationException: The provisioned throughput for the table will not change. The requested value equals the current value. Current ReadCapacityUnits provisioned for the table: 1. Requested ReadCapacityUnits: 1. Current WriteCapacityUnits provisioned for the table: 1. Requested WriteCapacityUnits: 1. Refer to the Amazon DynamoDB Developer Guide for current limits and how to request higher limits.
                           status code: 400, request id: 5V4M8K2F9EC0VGUI45RJETBALVVV4KQNSO5AEMVJF66Q9ASUAAJG
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
Events:
  Type     Reason                        Age   From                                            Message
  ----     ------                        ----  ----                                            -------
  Normal   CreatedExternalResource       52s   managed/dynamotable.database.aws.crossplane.io  Successfully requested creation of external resource
  Warning  CannotUpdateExternalResource  52s   managed/dynamotable.database.aws.crossplane.io  cannot update DynamoDB table: ResourceInUseException: Attempt to change a resource which is still in use: Table is being created: test-table
           status code: 400, request id: QC1U6MU46R1JPU7NTMS7C3VUDNVV4KQNSO5AEMVJF66Q9ASUAAJG
  Warning  CannotUpdateExternalResource  52s  managed/dynamotable.database.aws.crossplane.io  cannot update DynamoDB table: ResourceInUseException: Attempt to change a resource which is still in use: Table is being created: test-table
           status code: 400, request id: S49722LHS8T1NCF46PAIB3EMAFVV4KQNSO5AEMVJF66Q9ASUAAJG
  Warning  CannotUpdateExternalResource  52s  managed/dynamotable.database.aws.crossplane.io  cannot update DynamoDB table: ResourceInUseException: Attempt to change a resource which is still in use: Table is being created: test-table
           status code: 400, request id: JCIPM4J0UQ9SHG71F8M5EMRQCVVV4KQNSO5AEMVJF66Q9ASUAAJG
  Warning  CannotUpdateExternalResource  52s  managed/dynamotable.database.aws.crossplane.io  cannot update DynamoDB table: ResourceInUseException: Attempt to change a resource which is still in use: Table is being created: test-table
           status code: 400, request id: EJRTPC0CRDHMMR85R1ACU69VQRVV4KQNSO5AEMVJF66Q9ASUAAJG
  Warning  CannotUpdateExternalResource  52s  managed/dynamotable.database.aws.crossplane.io  cannot update DynamoDB table: ResourceInUseException: Attempt to change a resource which is still in use: Table is being created: test-table
           status code: 400, request id: BS6H8NHE9SG8J49A95J0OVTPHNVV4KQNSO5AEMVJF66Q9ASUAAJG
  Warning  CannotUpdateExternalResource  52s  managed/dynamotable.database.aws.crossplane.io  cannot update DynamoDB table: ResourceInUseException: Attempt to change a resource which is still in use: Table is being created: test-table
           status code: 400, request id: AL44T8C07VIPQFTA3POTR3P7AJVV4KQNSO5AEMVJF66Q9ASUAAJG
  Warning  CannotUpdateExternalResource  52s  managed/dynamotable.database.aws.crossplane.io  cannot update DynamoDB table: ResourceInUseException: Attempt to change a resource which is still in use: Table is being created: test-table
           status code: 400, request id: KIRQ2HRD661K2EV260IAL3LUEVVV4KQNSO5AEMVJF66Q9ASUAAJG
  Warning  CannotUpdateExternalResource  52s  managed/dynamotable.database.aws.crossplane.io  cannot update DynamoDB table: ResourceInUseException: Attempt to change a resource which is still in use: Table is being created: test-table
           status code: 400, request id: TKRUMH0UQMSKJ4RQR9QVKGLLAFVV4KQNSO5AEMVJF66Q9ASUAAJG
  Warning  CannotUpdateExternalResource  52s  managed/dynamotable.database.aws.crossplane.io  cannot update DynamoDB table: ResourceInUseException: Attempt to change a resource which is still in use: Table is being created: test-table
           status code: 400, request id: 7N15703VP8GALQR3DA5RB7OU6RVV4KQNSO5AEMVJF66Q9ASUAAJG
  Warning  CannotUpdateExternalResource  51s (x15 over 52s)  managed/dynamotable.database.aws.crossplane.io  (combined from similar events): cannot update DynamoDB table: ResourceInUseException: Attempt to change a resource which is still in use: Table is being created: test-table
           status code: 400, request id: CE0U49IIJELB3KR97D8Q8NDO4JVV4KQNSO5AEMVJF66Q9ASUAAJG

What environment did it happen in?

• Crossplane version: v0.14.0
• AWS EKS 1.16

[negz]

@muvaf Is this a bug in the generated code? I wonder if ACK has the same issue.

[muvaf]

This is related to the hand-crafted code. The generated one is merged as Table in dynamodb.aws.crossplane.io group. However, the update code is ported over and it seems AWS has certain expectations about UpdateTable call:

// UpdateTable API operation for Amazon DynamoDB.
//
// Modifies the provisioned throughput settings, global secondary indexes, or
// DynamoDB Streams settings for a given table.
//
// You can only perform one of the following operations at once:
//
//    * Modify the provisioned throughput settings of the table.
//
//    * Enable or disable DynamoDB Streams on the table.
//
//    * Remove a global secondary index from the table.
//
//    * Create a new global secondary index on the table. After the index begins
//    backfilling, you can use UpdateTable to perform other operations.

I'll open a PR tomorrow.