Service resolution doesn't work in minikube

[vskr]

Hello

Copying a question I have asked on stackoverflow couple of days back

I installed minikube to use kubernetes locally. I was able to create pods and services locally.

However, pods (and containers) running on them, cannot resolve services using service names.

Example: I have redis service running that acts a proxy for redis pods.

kubectl get services shows taht redis service has been created.

However, when my web application tries to connect to redis-service, I get connection timeout, because web application (pod) cannot resolve redis-service.

Is there anything special taht needs to be installed to get service resolution working locally.

this is the output of running kubectl get services

 frontend            10.0.0.250                 80/TCP      3h
 kubernetes          10.0.0.1     <none>        443/TCP     3h
 redis-service       10.0.0.156   <none>        6379/TCP    3h
 rethinkdb-service   10.0.0.89    <none>        28015/TCP   3h






kubectl describe services --namespace=kube-system

Name:                   kube-dns
Namespace:              kube-system
Labels:                 k8s-app=kube-dns,kubernetes.io/name=KubeDNS
Selector:               <none>
Type:                   ClusterIP
IP:                     10.0.0.10
Port:                   dns     53/UDP
Endpoints:              10.0.2.15:53
Port:                   dns-tcp 53/TCP
Endpoints:              10.0.2.15:53
Session Affinity:       None
No events.

Name:                   kubernetes-dashboard
Namespace:              kube-system
Labels:                 app=kubernetes-dashboard,kubernetes.io/cluster-service=true
Selector:               app=kubernetes-dashboard
Type:                   NodePort
IP:                     10.0.0.156
Port:                   <unset> 80/TCP
NodePort:               <unset> 30000/TCP
Endpoints:              172.17.0.2:9090
Session Affinity:       None
No events.

However, same yaml files when executd in Google Container Engine work fine. They are able to find services

[jimmidyson]

Can you share the logs from your application just to confirm it is DNS resolution & not service timing out or something like that? Can you also confirm that the redis pods are up & that the redis-service has valid endpoints as expected?

[dlorenc]

To check DNS, could you try something like this:

$ kubectl exec -it <pod> bash

Then run:
nslookup kubernetes.default

That should return something like 10.0.0.1 if DNS is working.

[dlorenc]

Any luck here?

[vskr]

I switched from 'dns' to 'env'. It worked then.

When I switched back from 'env' to 'dns', it continued working. Not sure what changed (other than discovery type), but it continued working with 'dns'.

Fyi (not sure if it matters), I am running all of this locally on a mac.

Thanks !!

[vskr]

More info:

My docker image is a pretty minimal ubuntu image . It doesn't have nslookup installed on it, but when I installed and ran nslookup kubernetes.default, it did show 10.0.**, which means DNS is working?

Should be a separate thread, I guess, but what are some of the best practices in debugging services. Sometimes, my services says it is in pending state on the dashboard (minikube dashboard).

And sometimes, service doesn't respond at all and times out, and I don't see any logs on my application pods (pods which are being proxied by the service). So quite sure, requests haven't reached the application pod, but are being interrupted before..

[vskr]

Closing this issue, since this particular issue is not a issue anymore.

[erulabs]

This is closed, but its the first thing that comes up when a newbie searches "kubernetes services dont work in minikube". Turns out, in minikube, you have to provide a GET_HOSTS_FROM environment variable.

In Kube on AWS, kube-on-mac, and every other kube environment I've used, setting no GET_HOSTS_FROM would ideally mean that GET_HOSTS_FROM defaults to dns. However, in minikube, setting:

      containers:
      - name: app
        image: myimage
        env:
          - name: GET_HOSTS_FROM
            value: dns

Seems to make services work! I believe this is still an open bug (because IMO you shouldnt have to provide that env line)

Hope this helps someone else banging their head against minikube like I was :)

[denisw]

Thank you so much @erulabs! That caused a lot of confusion for me.

How come GET_HOSTS_FROM is required, but not documented in the Kubernetes tutorials I looked at (the ones linked from the tour)? Is there an GitHub issue that tracks this?