unable to access tectonic console and containers are not running on one of the master nodes

[sriramnulu]

Hi All,

we tried to deploy tectonic on vmware environment by following the below link
https://coreos.com/tectonic/docs/latest/install/vmware/vmware-terraform.html

we did not observe any failures when "terraform apply ../../platforms/vmware" was run.

the deployment configured 3-etcd, 2-master, 2-worker nodes on the vCenter.

we could not access the tectonic console and we are getting below output when run kubectl

root@VM1:~/tectonic_1.8.4-tectonic.3/build/bcluster# kubectl cluster-info
Kubernetes master is running at https://bcluster.biarcatectonic.com:443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server bcluster.biarcatectonic.com:443 was refused - did you specify the right host or port?

root@VM1:~/tectonic_1.8.4-tectonic.3/build/bcluster# curl https://bcluster.biarcatectonic.com:443
curl: (7) Failed to connect to bcluster.biarcatectonic.com port 443: Connection refused
root@VM1:~/tectonic_1.8.4-tectonic.3/build/bcluster#

On the master-0 vm when we looked at possible docker containers observed that some of them are in pause state.
Do we need the last 3 conatiners mentioned below?

core@bcluster-master-0 /opt/tectonic $ docker ps
CONTAINER ID        IMAGE                                                                                              COMMAND                  CREATED             STATUS              PORTS               NAMES
bfec220f5932        quay.io/coreos/hyperkube@sha256:0d659da66e80396f7fbe1352878d1013b30f15a728924ee3eaff1cc79a38b825   "./hyperkube contr..."   39 minutes ago      Up 39 minutes                           k8s_kube-controller-manager_bootstrap-kube-controller-manager-bcluster-master-0_kube-system_ea1461f702b58f80dec7afcbe6d84c33_0
5d78b3c932e4        quay.io/coreos/hyperkube@sha256:0d659da66e80396f7fbe1352878d1013b30f15a728924ee3eaff1cc79a38b825   "./hyperkube sched..."   39 minutes ago      Up 39 minutes                           k8s_kube-scheduler_bootstrap-kube-scheduler-bcluster-master-0_kube-system_9ed9a738aa21e46d4aa2be533a40fe37_0
77e9b47740a5        quay.io/coreos/hyperkube@sha256:0d659da66e80396f7fbe1352878d1013b30f15a728924ee3eaff1cc79a38b825   "/usr/bin/flock /v..."   39 minutes ago      Up 39 minutes                           k8s_kube-apiserver_bootstrap-kube-apiserver-bcluster-master-0_kube-system_3072371fbcbbe31c6efa513b2f4dd5ca_0
74ed446d9ff9        gcr.io/google_containers/pause-amd64:3.0                                                           "/pause"                 43 minutes ago      Up 43 minutes                           k8s_POD_bootstrap-kube-controller-manager-bcluster-master-0_kube-system_ea1461f702b58f80dec7afcbe6d84c33_0
5b10e21da101        gcr.io/google_containers/pause-amd64:3.0                                                           "/pause"                 43 minutes ago      Up 43 minutes                           k8s_POD_bootstrap-kube-scheduler-bcluster-master-0_kube-system_9ed9a738aa21e46d4aa2be533a40fe37_0
9bd9a6db0ba2        gcr.io/google_containers/pause-amd64:3.0                                                           "/pause"                 43 minutes ago      Up 43 minutes                           k8s_POD_bootstrap-kube-apiserver-bcluster-master-0_kube-system_3072371fbcbbe31c6efa513b2f4dd5ca_0
core@bcluster-master-0 /opt/tectonic $

on the second master (master-1) no containers are running. and we did not find /opt/tectonic in master-1 node.

Below is our environment details in terraform.tfvars file:

base_domain : "biarcatectonic.com"
vCenter cluser : "vmcluster"
tectonic_cluster: "bcluster"
etcd nodes : "bcluster-etcd-0"
			 "bcluster-etcd-1"
			 "bcluster-etcd-2"

master nodes: "bcluster-master-0"
			  "bcluster-master-1"

worker nodes: "bcluster-worker-0"
			  "bcluster-worker-1"
tectonic_vmware_controller_domain = "bcluster.biarcatectonic.com"
tectonic_vmware_ingress_domain = "biarcatectonic.com"			  

could you please clarify what exactly we need to give for controller domain and ingress domain.

Also when we ran the below command on master-0 node and got certification validation error.

core@bcluster-master-0 /opt/tectonic $ curl https://bcluster-etcd-2.biarcatectonic.com:2379
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it.

Do we need to create certificates manually? If yes how?

Here we are attaching terraform.tfvar(in .txt format) file and our DNS snippet.

Could you please validate the terraform.tfvar file and let us know whether the given parameters are valid or not.

terraform.txt

[ihavespoons]

I am having the same issues and I believe other users over at https://groups.google.com/forum/#!searchin/coreos-user/dns$20VMware%7Csort:date/coreos-user/bsmWjYqdOCs/gX0Pw9rxBQAJ are experiencing it as well.

[stromvirvel]

@sriramnulu @ihavespoons
I'm experiencing the same issue on my vSphere platform. Did you make it work in the meantime? Any hints?