How to expose services using TCP ports in nginx_values and set Load balancer hostname in one pass?

[vinnytwice]

Description

Hi, and thanks for the project once again.

I’m trying setting up nginx controller using the nginx_values parameter as I need need to both assign a hostname (for Certificates HTTP-01 LE challenge) and expose a Service in the default namespace using two TCP ports.

As read in this issue solution suggestion #354 (comment) I’m setting the load balancer hostname in the controller.service.annotations portion in nginx_values and I made sure to wait enough for my A and AAAA DNS records to propagate.
I realised that when I set the "load-balancer.hetzner.cloud/hostname": “<my A record>.<my domain>” then provisioning the infrastructure hangs until Terraform throws the error:

│ Error: remote-exec provisioner error
│ 
│   with module.kube-hetzner.null_resource.kustomization,
│   on .terraform/modules/kube-hetzner/init.tf line 287, in resource "null_resource" "kustomization":
│  287:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_623444587.sh": Process exited with status 124

If I instead apply it in a second pass by uncommenting the line and then apply the change it succeeds with no errors.
Is it only settable in a second pass?

Now, for the TCP port exposing part, for my cluster in Azure I installed ingress-nginx controller using Helm chart directly by a “helm_release” terraform resource with the TCP ports to service exposing done as:

resource "helm_release" "nginx" {
  name      = "ingress-nginx"
  namespace = "default"

  repository = "https://kubernetes.github.io/ingress-nginx" 
  chart = "ingress-nginx"
  set {
    name  = "version"
    value = "4.4.2"

  }
### expose tcp ports to neo4j services

  set {
    name = "tcp.7687"
    value = "default/cluster:7687"
  }
  set {
    name = "tcp.7474"
    value = "default/cluster:7474"
  }
  set {
    name  = "controller.extraArgs.default-ssl-certificate"
    value = "default/tls-secret"
  }

  set {
    name  = "controller.service.externalTrafficPolicy"
    value = "Local"
  }
  set {
    name = "controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-internal"
    value = "true"
  }
  set {
    name  = "controller.service.loadBalancerIP"
    value = var.public_ip_address
  }
  set {
    name = "controller.service.annotations.service.beta.kubernetes.io/azure-dns-label-name"
    value = “azure dns label”   }
  set {
    name  = "controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"
    value = "/healthz"
  }
} 

I know I could manually use a ConfigMap, edit the nginx-ingress-nginx-controller Service and Deployment created in the nginx namespace as per docs at https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/ and done in this issue #818 (comment) but I rather edit the nginx_values as I see suggested in the same issue here #818 (comment) as I prefer to set up the infra directly in one go (or two considering the load balancer hostname parte above) with Terraform.

I’m trying setting the nginx_values as

     nginx_values = <<EOT
controller:
  watchIngressWithoutClass: "true"
  kind: "DaemonSet"
  config:
    "use-forwarded-headers": "true"
    "compute-full-forwarded-for": "true"
    "use-proxy-protocol": "true"
  service:
    annotations:
      "load-balancer.hetzner.cloud/name": "server"
      "load-balancer.hetzner.cloud/use-private-ip": "true"
      "load-balancer.hetzner.cloud/disable-private-ingress": "true"
      "load-balancer.hetzner.cloud/location": "fsn1"
      "load-balancer.hetzner.cloud/type": "lb11"
      "load-balancer.hetzner.cloud/uses-proxyprotocol": "true"
    #   "load-balancer.hetzner.cloud/hostname": "<my A record>.<my domain>"
    # tcp:
    #   7687: default/cluster:7687
    #   7474: default/cluster:7474  
    # nodePorts:
    #   tcp:
    #     7687: default/cluster:7687
    #     7474: default/cluster:7474  

  extraArgs:
    default-ssl-certificate: "default/tls-secret"  
       

  tcp:
    7687: default/cluster:7687
    7474: default/cluster:7474  
  # nodePorts:
  #   tcp:
  #     7687: default/cluster:7687
  #     7474: default/cluster:7474  

  EOT 

As you can see I tried various indentations or parameters ( looking at the chart here https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml) but I guess that my tried approach `tcp: 7687: “default/tls-secret” is only valid for installing nginx controller using Helm charts, as when I check I don’t see those ports opened :

% kubectl --kubeconfig server_kubeconfig.yaml -n nginx get all                                     
NAME                                       READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-nginx-controller-82fxx   1/1     Running   0          13h

NAME                                               TYPE           CLUSTER-IP      EXTERNAL-IP                           PORT(S)                      AGE
service/nginx-ingress-nginx-controller             LoadBalancer   10.43.126.228   128.140.25.107,2a01:4f8:c01e:19f::1   80:31693/TCP,443:30007/TCP   13h
service/nginx-ingress-nginx-controller-admission   ClusterIP      10.43.110.222   <none>                                443/TCP                      13h

NAME                                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/nginx-ingress-nginx-controller   1         1         1       1            1           kubernetes.io/os=linux   13h

How do I set those ports in nginx_values ?
Many many thanks.

Vincenzo

Kube.tf file

locals {
  hcloud_token = "<hidden>"
}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token


  source = "kube-hetzner/kube-hetzner/hcloud"
  ssh_public_key = file("~/.ssh/id_rsa.pub")
  ssh_private_key = file("~/.ssh/id_rsa")
  network_region = "eu-central" 

  control_plane_nodepools = [
    {
      name        = "Control-Plane",
      server_type = "cax21",
      location    = "fsn1",
      labels      = [
        "node.kubernetes.io/server-type=control-plane"
      ],
      taints      = [],
      count       = 1
    },
  ]

  agent_nodepools = [
    {
      name        = "Nodejs-Server",
      server_type = "cax21",
      location    = "fsn1",
      labels      = [
        "node.kubernetes.io/server-type=server"
      ],
      taints      = [],
      count       = 1
    },
    {
      name        = "MongoDb",
      server_type = "cax21",
      location    = "fsn1",
      labels      = [
        "node.kubernetes.io/server-type=mongodb"
      ],
      taints      = [],
      count       = 1
    },
    {
      name        = "Neo4j",
      server_type = "cax21",
      location    = "fsn1",
      labels      = ["node.kubernetes.io/server-type=neo4j"],
      taints      = [],
      count       = 1
    },
  ]
  control_planes_custom_config = {
   etcd-expose-metrics = true,
   kube-controller-manager-arg = "bind-address=0.0.0.0",
   kube-proxy-arg ="metrics-bind-address=0.0.0.0",
   kube-scheduler-arg = "bind-address=0.0.0.0",
  }

  # enable_wireguard = true

  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"

  ingress_controller = "nginx"
  ingress_replica_count = 1

  automatically_upgrade_k3s = false
  automatically_upgrade_os = false
  cluster_name = "server"  
  use_cluster_name_in_node_name = false
  restrict_outbound_traffic = false
  # Always be careful to not commit this file!
  # create_kubeconfig = false

  # create_kustomization = false

  # Cert manager, all cert-manager helm values can be found at https://github.com/cert-manager/cert-manager/blob/master/deploy/charts/cert-manager/values.yaml
  # The following is an example, please note that the current indentation inside the EOT is important.
#      cert_manager_values = <<EOT
# installCRDs: true
# replicaCount: 1
# webhook:
#   replicaCount: 1
# cainjector:
#   replicaCount: 1
#   EOT 



  # Nginx, all Nginx helm values can be found at https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml
  # You can also have a look at https://kubernetes.github.io/ingress-nginx/, to understand how it works, and all the options at your disposal.
  # The following is an example, please note that the current indentation inside the EOT is important.
# /name annotation must be the same as cluster_name if set
     nginx_values = <<EOT
controller:
  watchIngressWithoutClass: "true"
  kind: "DaemonSet"
  config:
    "use-forwarded-headers": "true"
    "compute-full-forwarded-for": "true"
    "use-proxy-protocol": "true"
  service:
    annotations:
      "load-balancer.hetzner.cloud/name": "server"
      "load-balancer.hetzner.cloud/use-private-ip": "true"
      "load-balancer.hetzner.cloud/disable-private-ingress": "true"
      "load-balancer.hetzner.cloud/location": "fsn1"
      "load-balancer.hetzner.cloud/type": "lb11"
      "load-balancer.hetzner.cloud/uses-proxyprotocol": "true"
    #   "load-balancer.hetzner.cloud/hostname": "<hidden>"
    # tcp:
    #   7687: default/cluster:7687
    #   7474: default/cluster:7474  
    # nodePorts:
    #   tcp:
    #     7687: default/cluster:7687
    #     7474: default/cluster:7474  

  extraArgs:
    default-ssl-certificate: "default/tls-secret"  
       

  tcp:
    7687: default/cluster:7687
    7474: default/cluster:7474  
  # nodePorts:
  #   tcp:
  #     7687: default/cluster:7687
  #     7474: default/cluster:7474  

  EOT 



}

provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

terraform {
  required_version = ">= 1.4.0"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.41.0"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

variable "hcloud_token" {
  sensitive = true
  default   = ""
}

Screenshots

No response

Platform

MacBook Air M1

[vinnytwice]

Little update on exposing a service in the default namespace through TCP ports.
as for the nginx helm chart values

tcp: {}
#  8080: "default/example-tcp-svc:9000"

indeed I was trying the correct method, just wrongly indented it in nginx_values ending under controller: parameters.
Once properly indented as

     nginx_values = <<EOT
controller:
  watchIngressWithoutClass: "true"
  kind: "DaemonSet"
  config:
    "use-forwarded-headers": "true"
    "compute-full-forwarded-for": "true"
    "use-proxy-protocol": "true"
  service:
    annotations:
      "load-balancer.hetzner.cloud/name": "server"
      "load-balancer.hetzner.cloud/use-private-ip": "true"
      "load-balancer.hetzner.cloud/disable-private-ingress": "true"
      "load-balancer.hetzner.cloud/location": "fsn1"
      "load-balancer.hetzner.cloud/type": "lb11"
      "load-balancer.hetzner.cloud/uses-proxyprotocol": "true"
    #   "load-balancer.hetzner.cloud/hostname": "server.fixit.bike"
  extraArgs:
    default-ssl-certificate: "default/tls-secret"  
       

tcp:
  7687: "default/cluster:7687" 
  7474: "default/cluster:7474" 

  EOT 

I can see the two TCP ports opened..

pod/nginx-ingress-nginx-controller-9vv4q   1/1     Running   0          14m

NAME                                               TYPE           CLUSTER-IP      EXTERNAL-IP                           PORT(S)                                                    AGE
service/nginx-ingress-nginx-controller             LoadBalancer   10.43.101.221   128.140.25.122,2a01:4f8:c01e:2b8::1   80:30907/TCP,443:31191/TCP,7474:32170/TCP,7687:31670/TCP   14m
service/nginx-ingress-nginx-controller-admission   ClusterIP      10.43.249.66    <none>                                443/TCP                                                    14m

NAME                                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/nginx-ingress-nginx-controller   1         1         1       1            1           kubernetes.io/os=linux   14m

On the console the 2 new services (ports) appear as unhealthy dough.
Getting close, the port mapping and the unhealthy status on the 2 new services is still to be sorted out

[mysticaltech]

@vinnytwice Don't worry about the health status, we do not have the health check endpoints well defined for all ports, that you need to do yourself because it changes depending on your ports.

[mysticaltech]

Also check this PR #908, the changes will be shipped in the next release coming in a day or two.