S3 remote for tfstate: x509: certificate signed by unknown authority

[andrejs-sisojevs]

Hello

I'm getting a very strange error, when configuring S3 remote for tfstate. My colleagues are using same commands, and don't get this error - everything works fine for them. So I guess that might be related to Terraform dependencies or OS. Any ideas where I should search for failure reason?

Terraform versions 0.7.7-0.7.13.

$ env | grep AWS
AWS_DEFAULT_PROFILE=*****
AWS_PROFILE=******
AWS_REGION=us-east-1

$ aws s3 cp s3://my-bucket/infrastructure/terraform.tfstate ./
download: s3://my-bucket/infrastructure/terraform.tfstate to ./terraform.tfstate

$ cat ./terraform.tfstate

/* proper file */

$ export TF_LOG=TRACE

$ terraform remote config \
>     -backend=s3 \
>     -backend-config=bucket=my-bucket \
>     -backend-config=key=infrastructure/terraform.tfstate \
>     -backend-config=region=us-east-1 \
>     -backend-config=skip_cert_verification=true
2016/11/29 12:17:43 [INFO] Terraform version: 0.7.11  65acdc2ea31541283e0fec2fdcc8cb49e89e4cf6
2016/11/29 12:17:43 [INFO] CLI args: []string{"/opt/terraform/terraform", "remote", "config", "-backend=s3", "-backend-config=bucket=my-bucket", "-backend-config=key=infrastructure/terraform.tfstate", "-backend-config=region=us-east-1", "-backend-config=skip_cert_verification=true"}
2016/11/29 12:17:43 [DEBUG] Detected home directory from env var: /Users/andrejs.sisojevs
2016/11/29 12:17:43 [DEBUG] Detected home directory from env var: /Users/andrejs.sisojevs
2016/11/29 12:17:43 [DEBUG] Attempting to open CLI config file: /Users/andrejs.sisojevs/.terraformrc
2016/11/29 12:17:43 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2016/11/29 12:17:43 [DEBUG] Detected home directory from env var: /Users/andrejs.sisojevs
2016/11/29 12:17:43 [TRACE] Preserving existing state lineage "c6380fd3-57a5-4139-93dd-1d6154f1cf0b"
2016/11/29 12:17:43 [TRACE] Preserving existing state lineage "c6380fd3-57a5-4139-93dd-1d6154f1cf0b"
2016/11/29 12:17:43 [TRACE] Preserving existing state lineage "c6380fd3-57a5-4139-93dd-1d6154f1cf0b"
2016/11/29 12:17:43 [TRACE] Preserving existing state lineage "c6380fd3-57a5-4139-93dd-1d6154f1cf0b"
2016/11/29 12:17:43 [TRACE] Preserving existing state lineage "c6380fd3-57a5-4139-93dd-1d6154f1cf0b"
2016/11/29 12:17:43 [WARN] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2016/11/29 12:17:43 [TRACE] Preserving existing state lineage "c6380fd3-57a5-4139-93dd-1d6154f1cf0b"
Remote configuration updated
2016/11/29 12:17:43 [TRACE] Preserving existing state lineage "c6380fd3-57a5-4139-93dd-1d6154f1cf0b"
2016/11/29 12:17:43 [TRACE] Preserving existing state lineage "c6380fd3-57a5-4139-93dd-1d6154f1cf0b"
2016/11/29 12:17:43 [WARN] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
Error while performing the initial pull. The error message is shown
2016/11/29 12:17:44 [DEBUG] plugin: waiting for all plugin processes to complete...
below. Note that remote state was properly configured, so you don't
need to reconfigure. You can now use `push` and `pull` directly.

Error reloading remote state: RequestError: send request failed
caused by: Get https://my-bucket.s3.amazonaws.com/infrastructure/terraform.tfstate: x509: certificate signed by unknown authority

Although .tfstate in S3 bucket if fine (uploaded by colleagues who do not have problems with terraform), locally it generates almost empty file

{
    "version": 3,
    "serial": 0,
    "lineage": "c6380fd3-57a5-4139-93dd-1d6154f1cf0b",
    "remote": {
        "type": "s3",
        "config": {
            "bucket": "my-bucket",
            "key": "infrastructure/terraform.tfstate",
            "region": "us-east-1",
            "skip_cert_verification": "true"
        }
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {},
            "depends_on": []
        }
    ]
}

Tried exposing bucket - did not help.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket",
                "arn:aws:s3:::my-bucket/*"
            ]
        }
    ]
}

No old plugins in OS ("terraform-*" files)

Any ideas?

[mitchellh]

This just means that Terraform can't verify the TLS cert. Terraform uses the Go standard library to check this which further uses your system CA certificates. Terraform doesn't ever touch CA certificate lookup (just uses default). If this works on colleague machines then it sounds like something may not be right with your CA certs.

[Cinderhaze]

I have a question related to this one.. I am not a go user, and my google-fu has not helped me find this answer..

Is there some environment variable that I can set to point terraform (or packer.etc) at a different cacert file?

[andrejs-sisojevs]

@Cinderhaze, not sure your question is in scope of this issue.

In my case OS keychain with globally-accepted CA certs failed. For S3 those CA are "DigiCert Baltimore CA-2 G2" (and "Baltimore CyberTrust Root" as a parent). Normally, trust to that king of CAs is distributed by OS vendors with OS security updates.
For OS X terraform uses System and SystemRoot keychains:

func certKeychains() []string {
	keychains := []string{
		"/System/Library/Keychains/SystemRootCertificates.keychain",
		"/Library/Keychains/System.keychain",
	}
	home, err := homedir.Dir()
	if err == nil {
		loginKeychain := path.Join(home, "Library", "Keychains", "login.keychain")
		keychains = append(keychains, loginKeychain)
	}
	return keychains
}

Although, I've manually downloaded "DigiCert Baltimore CA-2 G2" and "Baltimore CyberTrust Root", added them to System keychain, configured them to be always trusted, - it didn't help. (but that was just an experiment - it shouldn't be solved in such a manner)

Probably our IT support dept or security dept confused something, when hardened my work Mac.

As a temporary solution I made a Vagrant script that provisions Ubuntu with all DEV tools required - Terraform works well there...

[takieddine-sbiai]

@andrejs-sisojevs-accenture
the same issue happened to me when I started using terraform 0.8.
Before that, I used 0.7.7 without any issues.

@mitchellh
Is it possible that this is related?
golang/go#16589

For now I am downgrading to 0.7.7 until this is fixed.

[mitchellh]

@takieddine-sbiai This came up yesterday, I believe the cert security issue fixed in Go 1.7.4 is causing some issues for people. Please see the Go 1.7.4 changelog, I don't believe this is a bug on our end or Go, but a system configuration issue, though not 100% sure. We haven't touched any of the HTTP client code, etc.

[chulkilee]

Hitting this problem with terraform 0.8.1 from homebrew (https://github.com/onlydole/homebrew-core/blob/master/Formula/terraform.rb) which uses go 1.7.4..

Also potentially related: golang/go#18141 and #10718

[kevinduterne]

Using the 0.8.3 binary from https://www.terraform.io/downloads.html instead of homebrew installed version seemed to have resolved the issue.

[andrejs-sisojevs]

I had it previously installed from hashicorp binaries - didn't work. Tried with 0.8.3 binaries - same failure.

[prudhvigodithi]

Hey in my case terraform works on ec2 host but inside container using official terraform docker container and mounting the hosts directory (for certs) to the container and when i do terraform init form inside the container I get the following error, which works fine from the host any fix for this, please let me know.
thank you

Error configuring the backend "s3": RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: x509: certificate signed by unknown authority

Please update the configuration in your Terraform files to fix this error
then run this command again.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.