Consul template reload signal on Windows 10 fails

[woodytec]

Consul Template version

Version 0.25.1

Configuration

consul {
  
  address = "https://127.0.0.1:8501"
  retry {
    
    enabled = true
    attempts = 12
    backoff = "90s"
    max_backoff = "3m"
  }

  ssl {
    enabled = true
    verify = false

    cert = "C:\\Program Files\\\MyApp\\certificates\\internal\\cert.pem"
    key  = "C:\\Program Files\\MyApp\\certificates\\internal\\key.pem"
    ca_cert = "C:\\Program Files\\MyApp\\certificates\\internal\root.pem"

  }
}

reload_signal = "SIGHUP"
kill_signal = "SIGINT"

max_stale = "10m"

block_query_wait = "60s"
log_level = "DEBUG"

wait {
  min = "5s"
  max = "10s"
}

exec {

  command = "'C:\\Program Files\\MyApp\\supervision\\prometheus\\prometheus.exe' --config.file='C:\\Program Files\\MyApp\\config\\prometheus.yaml' --storage.tsdb.path='C:\\ProgramData\\MyApp\\prometheus' --web.listen-address=:30632 --storage.tsdb.retention.time=7d"
  splay = "5s"

  reload_signal = "SIGHUP"
  kill_signal = "SIGINT"

  kill_timeout = "5s"
}

template {

  source = "C:\\Program Files\\MyApp\\config\\prometheus-tmpl.yaml"
  destination = "C:\\Program Files\\MyAPP\\config\\prometheus.yaml"

  create_dest_dirs = true
  command_timeout = "60s"
  error_on_missing_key = false
  perms = 0600
  backup = false
  left_delimiter  = "<{"
  right_delimiter = "}>"
  function_denylist = []
  sandbox_path = ""

  wait {
    min = "2s"
    max = "10s"
  }
}

global:
  scrape_interval:     10s 
  evaluation_interval: 10s
  #query_log_file: "C:\\Consul\\prometheus"


# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: loki
    scheme: "http"
    metrics_path: /metrics
    static_configs:
      - targets: [<{ range service "loki" }>"<{ .Address }>:<{ .Port }>",<{ end }>]

  - job_name: consul
    scheme: "https"
    metrics_path: /v1/agent/metrics
    params:
      format: ['prometheus']
    tls_config:
      insecure_skip_verify: true
    static_configs:
      - targets: [<{ range service "consul_metrics" }>"<{ .Address }>:<{ .Port }>",<{ end }>]

  - job_name: traefik
    scheme: "http"
    metrics_path: /metrics
    static_configs:
      - targets: [<{ range service "traefik_metrics" }>"<{ .Address }>:<{ .Port }>",<{ end }>]

  - job_name: windows_exporter
    scheme: "http"
    metrics_path: /metrics
    static_configs:
      - targets: [<{ range service "windows_exporter" }>"<{ .Address }>:<{ .Port }>",<{ end }>]

Command

C:\Program Files\MyApp\discovery\consul-template.exe -config "C:\Program Files\MyApp\config\consul-template-prometheus.hcl" 

Expected behavior

Prometheus should be reloaded by CT when values change in Consul.

Actual behavior

Consul-template is working pretty well until a reload happens. Prometheus is not reloaded. CT seems to try to send the signal specified in its configuration file but logs show :

2021/01/08 16:37:16.786062 [INFO] (child) reloading process
2021/01/08 16:37:16.787057 [ERR] (cli) 1 error occurred:
        * not supported by windows

This happens whatever the reload signal is. I first thought it was because SIGHUP is not implemented on Windows, but trying with SIGINT showed the same output.

[eikenb]

Hey @woodytec, thanks for reporting this.

That error bubbles up from Go's syscall library. Digging around a bit my first thought is that neither SIGHUP nor SIGINT are supported. The windows docs [1] I found lists the signals windows respects and SIGHUP isn't there. SIGINT is in that list but the Go docs [2] call it out as not supported.

Maybe try one of the other signals in the list at [1].

[1] https://docs.microsoft.com/en-us/previous-versions/xdkz3x12(v=vs.140)
[2] https://golang.org/pkg/os/#Process.Signal

[woodytec]

Hey @eikenb,

Thanks for your answer. I've tried all the signals mentionned in the list at [1], they all show the exact same output :

2021/01/18 09:01:35.152333 [INFO] (child) reloading process
2021/01/18 09:01:37.665449 [ERR] (cli) 1 error occurred:
	* not supported by windows

Anyway, sending another signal than SIGHUP would not help me a lot as Prometheus' reload signal cannot be changed... Another solution would be to be able to make and API call from Consul-template instead of sending a signal, but as far as I know it's not possible, is it ?

[woodytec]

I found a quick workaround to solve Prometheus' reload which is to let Consul-template kill and respawn it (what means I did not specify any reload signal in CT configuration file).

This seems to do the job for Prometheus, but it is kind of problematic in other usecases. For example Vector (that I am using to ship logs from text files to Loki) includes an option to automatically watch for config changes and reload itself when necessary. Therefore I don't want it to be killed and respawned by CT nor to receive a reload signal (what would cause the above bugs for the moment...).

I feel like a first step toward the solution would be to implement a flag on CT to disable reloading the child process.

[eikenb]

I'm adding 'enhancement' label here to address the idea of having a way to disable reloading (for use on Windows). Though I'm not 100% on that and might just disable reloading on windows by default. Unless there is a way to get it working... which is why I'm leaving the 'bug' label, as it'd be nice to actually fix this if possible, but I have no clue as it seems like it should work based on the docs I've seen. Maybe someone with more Windows knowledge will help out.

[eikenb]

There was a bug with setting the reload signal to nothing that was fixed by #1610. I'm pretty sure that change fixes this issue by allowing the empty setting to actually disable the reload signal as advertised.