Unable to find zones with cross-account role using kube2iam

[ryan-dyer-sp]

Our environment is such that we have multiple AWS accounts with a kube cluster in each account. Each account/cluster has an associated hosted zone; however, these zones exist in a separate single account. ie the zone for accounts 1-3 exist in account 4.

I have been attempting to use external-dns and kube2iam to solve this problem. I am able to successfully install and configure kube2iam on my cluster such that when I kubectl exec into the external-dns pod, I can run aws route53 list-hosted-zones (after install aws-cli) and retrieve the hosted zones from the dns mgmt account without any configuration. Curling for meta-data/iam/security-credentials/ properly shows the cross account role. However when external-dns starts it gets the following:

time="2018-02-22T16:16:09Z" level=info msg="Connected to cluster at https://100.64.0.1:443"
time="2018-02-22T16:16:10Z" level=error msg="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
time="2018-02-22T16:17:10Z" level=error msg="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
time="2018-02-22T16:18:10Z" level=error msg="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
time="2018-02-22T16:19:10Z" level=error msg="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
time="2018-02-22T16:20:10Z" level=error msg="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
time="2018-02-22T16:21:10Z" level=error msg="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

Please let me know what additional info I can provide to help troubleshoot.

[mariomerco]

Same issue in my side. @ryan-dyer-sp did you solve the problem?

My scenario:
I have external-dns up and running in the cluster, then install kube2iam and both work excellent. After a certain amount of time (can be 2-3 hours, but not 100% sure, only know it is not immediately) the external-dns pod starts showing the exact same error message you are presenting.

[ryan-dyer-sp]

@mariomerco No luck on figuring anything out either. Would love for someone working on this project to chime in. If you figure anything out, please reach out as I am at a loss.

[mariomerco]

@ryan-dyer-sp I solved the problem!
Let's say Role-A is the role assigned in the kubernetes worker, and Role-B is the default role (--default-role: https://github.com/jtblin/kube2iam#options) provided when installing kube2iam. My issue was that, as external-dns doesn't have a tag of a role to use, it will try to assume the default role, but the Role-A was not allowed to assume Role-B, so had to create a trust relationship in Role-B to allow Role-A to assume it.
The service is running correctly for more than 12 hours at this point.
Hope it works!

[ryan-dyer-sp]

@mariomerco Sorry but that doesnt sound like my issue. My nodes(in account A) have an assume role permission. In account B, I have my role for route53 manipulation and have added a trust to the role in account A. In my external-dns config I have an annotation for the role in account B. Within the external-dns container I can successfully run the commands to view the route53 zones via aws-cli, but the application itself cant seem to obtain a valid credential. If this does sound the same, then I'm not sure what I'm missing from what you said.

[njuettner]

Hi just a follow-up from our side. We're currently working on a solution.