Incorrectly parsing arrays of stringvals

[mwright-pivotal]

Terraform version: v1.9.4
Kubernetes provider version: 2.32.0
Kubernetes version: v1.29.6+aba1e8d

Given:

    resource "kubernetes_manifest" "rook-operator-openshift" {
  for_each = { for k, v in provider::kubernetes::manifest_decode_multi(file("${path.module}/infra/rook/operator-openshift.yaml")) : k => v }
  manifest = each.value
  field_manager {
    # set the name of the field manager
    name = "myteam"

    # force field manager conflicts to be overridden
    force_conflicts = true
  }
}

and YAML source found here: https://github.com/mwright-pivotal/platform_automation_hcp_rhos/blob/main/infra/rook/operator-openshift.yaml
which ultimately came from: https://github.com/rook/rook/blob/master/deploy/examples/operator-openshift.yaml

Results in:

Debug Output:

{"@level":"error","@message":"Error: Provider produced inconsistent result after apply","@module":"terraform.ui","@timestamp":"2024-08-15T14:21:50.677977Z","diagnostic":{"severity":"error","summary":"Provider produced inconsistent result after apply","detail":"When applying changes to kubernetes_manifest.rook-operator-openshift[\"1\"], provider \"provider[\\\"registry.terraform.io/hashicorp/kubernetes\\\"]\" produced an unexpected new value: .object.volumes[1]: was cty.StringVal(\"projected\"), but now cty.StringVal(\"emptyDir\").\n\nThis is a bug in the provider, which should be reported in the provider's own issue tracker."},"type":"diagnostic"}
{"@level":"error","@message":"Error: Provider produced inconsistent result after apply","@module":"terraform.ui","@timestamp":"2024-08-15T14:21:50.678179Z","diagnostic":{"severity":"error","summary":"Provider produced inconsistent result after apply","detail":"When applying changes to kubernetes_manifest.rook-operator-openshift[\"1\"], provider \"provider[\\\"registry.terraform.io/hashicorp/kubernetes\\\"]\" produced an unexpected new value: .object.volumes[2]: was cty.StringVal(\"emptyDir\"), but now cty.StringVal(\"hostPath\").\n\nThis is a bug in the provider, which should be reported in the provider's own issue tracker."},"type":"diagnostic"}
{"@level":"error","@message":"Error: Provider produced inconsistent result after apply","@module":"terraform.ui","@timestamp":"2024-08-15T14:21:50.678195Z","diagnostic":{"severity":"error","summary":"Provider produced inconsistent result after apply","detail":"When applying changes to kubernetes_manifest.rook-operator-openshift[\"1\"], provider \"provider[\\\"registry.terraform.io/hashicorp/kubernetes\\\"]\" produced an unexpected new value: .object.volumes[3]: was cty.StringVal(\"hostPath\"), but now cty.StringVal(\"projected\").\n\nThis is a bug in the provider, which should be reported in the provider's own issue tracker."},"type":"diagnostic"}

Steps to Reproduce

Install Rook Ceph using instructions documented here: https://github.com/rook/rook/blob/master/Documentation/Getting-Started/ceph-openshift.md using the terraform kubernetes provider.

Note: the test does not likely requirement openshift kubernetes to reproduce this error if the test is isolated to simply deploying the operator yaml and removing extraneous references, like service accounts etc. Since the error is not coming from kubernetes itself and seems to be coming from the provider parsing logic, successful deployment to kubernetes is not directly a requirement for resolution.

Expected Behavior

operator-openshift.yaml is comprised of many YAML objects. kubernetes_manifest.rook-operator-openshift["0"] and kubernetes_manifest.rook-operator-openshift["1"] both have an attribute named volumes and each have a list of string values. kubernetes_manifest.rook-operator-openshift["0"] and kubernetes_manifest.rook-operator-openshift["1"] should both be parsed successfully.

Actual Behavior

kubernetes_manifest.rook-operator-openshift["0"] is successfully parsed, yet kubernetes_manifest.rook-operator-openshift["1"] is not.

Important Factoids

It appears the provider parsing is somehow conflating values for volumes from kubernetes_manifest.rook-operator-openshift["0"] and kubernetes_manifest.rook-operator-openshift["1"]

[alexsomesan]

This might actually be a side-effect of provider::kubernetes::manifest_decode_multi() function while transcoding YAML. We will have a look into it.

Regardless, we do not recommend this approach of on-the-fly transcoding from YAML documents because it discards a lot of the benefits that Terraform brings over plain YAML, such as attribute value interpolation and establishing a dependency order between resources, etc.

Instead we advise to do the transcoding in advance and save the results as *.tf files with full-fledged terraform resources. That way, the source configuration is stable and the parsing order of such types as lists and maps is reproducible.

[mwright-pivotal]

ok, thanks Alex. I will look into converting the YAML to TF. I guess what's implied here is that we will always have a manual step to perform if new YAML gets published or maybe have a separate pipeline somewhere that gets triggered from YAML commits to automate the conversion to TF, then push converted artifacts to another repo? In some cases this YAML comes from third party and may not be authored by the user, as in this case deploying Rook Ceph.

…

On Mon, Aug 26, 2024 at 6:08 AM Alex Somesan ***@***.***> wrote: This might actually be a side-effect of provider::kubernetes::manifest_decode_multi() function while transcoding YAML. We will have a look into it. Regardless, we do not recommend this approach of on-the-fly transcoding from YAML documents because it discards a lot of the benefits that Terraform brings over plain YAML, such as attribute value interpolation and establishing a dependency order between resources, etc. Instead we advise to do the transcoding in advance and save the results as *.tf files with full-fledged terraform resources. That way, the source configuration is stable and the parsing order of such types as lists and maps is reproducible. — Reply to this email directly, view it on GitHub <#2574 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHBAG3H6PE26N6VDAN7FH3ZTL5CRAVCNFSM6AAAAABMUM7HJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBZHA2DGOBUGI> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[alexsomesan]

Yes, that's correct. I'm suggesting to do the conversion in a prior step. That's just best practice, IMO.

You could either use Terraform itself with the built-in provider function or this simpler tool that makes it a bit easier: https://github.com/jrhouston/tfk8s (author is a member of our team).

The tfk8s tool also does bulk conversion of entire repos which you could then push into version control and consume as a module.

[mwright-pivotal]

Quick update... I generated https://github.com/mwright-pivotal/platform_automation_hcp_rhos/blob/main/infra/rook/operator-openshift.tf using tfk8s. It also generates the same error when resources are separated into individual kubernetes_manifests.