Add initial support for stream upstream metrics

[eriksw]

This includes only the subset of per-upstream/peer metrics that already
exist for non-stream upstreams which are applicable to stream/UDP
upstreams. There are additional stream/UDP-specific metrics that should
be added later, but this is a useful start.

Note: This propagates the existing incorrect help text for the "fails"
metric (it shouldn't be "Active connections") because I'm not sure what
the correct help text should be.

[pleshakov]

@eriksw thx for the PR. we'll review it and get back to you shortly

[pleshakov]

@eriksw Thanks for the PR. It is a good start for supporting stream upstreams. However, the correct implementation must use the stream specific data strucures.

Currently, we're in the process of migrating the prometheus exporter to use our public SDK client https://github.com/nginxinc/nginx-plus-go-sdk. Also, we're planning to add the stream related data structures to our public SDK client.

Would you be OK to extend the PR to support a few additional stream metrics once our public is SDK is added to this project and updated with stream upstream metrics. Your existing code should continue to work.

Regarding the docs for "fails" method, thanks for catching that. The correct value is "Unsuccessful attempts to communicate with the server", we got it from http://nginx.org/en/docs/http/ngx_http_api_module.html#def_nginx_http_upstream.

[eriksw]

Yes, I'd be happy to rebase and extend this work to cover the rest of the metrics once the external SDK client is integrated into the exporter.

[pleshakov]

@eriksw great! thank you

[avthart]

@eriksw Great! Thank you for your contribution. We also need stream metrics and I already wanted to contribute and create a pull request but that 's not needed anymore. Let me know if you need help.

@pleshakov
I am the maintainer of https://github.com/avthart/nginxplus_exporter which is created 4 months ago. We are using this exporter in production for a while, but we would like to use the official exporter so that we don't have to maintain our own exporter an get support (if needed).

[eriksw]

I've been waiting for a switch to the external SDK client to land (is that being actively worked on by anyone, @pleshakov?) before reworking this to support all the possible metrics. This PR is a lot less useful than an updated one that supports all the upstream metrics would be.

[pleshakov]

@avthart thanks for sharing your implementation. I'm glad that you find our implementation useful. Our goal is to have a long-term supported solution.

@eriksw we just added support for stream upstreams and server zones to the SDK - nginxinc/nginx-plus-go-client@c9fb4ac . It will be added to this project in a couple of days. Thanks again for bringing this PR. We'll ping you once the updated SDK is added to this project.

@eriksw @avthart except for the stream-related metrics, do you have any additional requirements/proposals for the exporter?

[avthart]

I compared both exporters, see below:

nginx/nginx-prometheus-exporter:0.1.0

nginxplus_connections_accepted
nginxplus_connections_active 
nginxplus_connections_dropped 
nginxplus_connections_idle 
nginxplus_http_requests_current 
nginxplus_http_requests_total 
nginxplus_server_zone_discarded 
nginxplus_server_zone_processing 
nginxplus_server_zone_received 
nginxplus_server_zone_requests 
nginxplus_server_zone_responses 
nginxplus_server_zone_sent 
nginxplus_ssl_handshakes
nginxplus_ssl_handshakes_failed
nginxplus_ssl_session_reuses
nginxplus_upstream_keepalives 
nginxplus_upstream_server_active 
nginxplus_upstream_server_fails 
nginxplus_upstream_server_header_time 
nginxplus_upstream_server_received 
nginxplus_upstream_server_requests 
nginxplus_upstream_server_response_time 
nginxplus_upstream_server_responses 
nginxplus_upstream_server_sent 
nginxplus_upstream_server_state 
nginxplus_upstream_server_unavail 
nginxplus_upstream_zombies 

avthart/nginxplus-exporter:v0.1.0

nginx_http_requests 
nginx_http_requests_total
nginx_http_server_zone_bytes_received 
nginx_http_server_zone_bytes_sent 
nginx_http_server_zone_discarded 
nginx_http_server_zone_processing 
nginx_http_server_zone_requests 
nginx_http_server_zone_responses 
nginx_http_server_zone_responses_total 
nginx_http_upstream_peer_active 
nginx_http_upstream_peer_bytes_received 
nginx_http_upstream_peer_bytes_sent 
nginx_http_upstream_peer_fails 
nginx_http_upstream_peer_header_time 
nginx_http_upstream_peer_healthcheck_checks 
nginx_http_upstream_peer_healthcheck_fails 
nginx_http_upstream_peer_healthcheck_last_passed 
nginx_http_upstream_peer_healthcheck_unhealthy 
nginx_http_upstream_peer_max_conns 
nginx_http_upstream_peer_requests 
nginx_http_upstream_peer_response_time 
nginx_http_upstream_peer_responses 
nginx_http_upstream_peer_responses_total 
nginx_http_upstream_peer_state 
nginx_http_upstream_peer_unavail 
nginx_http_upstream_peer_up 
nginx_http_upstream_peer_weight 
nginx_server_connections 
nginx_server_ssl_handshakes
nginx_server_ssl_handshakes_failed
nginx_server_ssl_session_reuses
nginx_stream_server_zone_bytes_received 
nginx_stream_server_zone_bytes_sent 
nginx_stream_server_zone_connections 
nginx_stream_server_zone_discarded 
nginx_stream_server_zone_processing 
nginx_stream_server_zone_sessions 
nginx_stream_server_zone_sessions_total 
nginx_stream_upstream_peer_active 
nginx_stream_upstream_peer_bytes_received 
nginx_stream_upstream_peer_bytes_sent 
nginx_stream_upstream_peer_connect_time 
nginx_stream_upstream_peer_connections 
nginx_stream_upstream_peer_fails 
nginx_stream_upstream_peer_first_byte_time 
nginx_stream_upstream_peer_healthcheck_checks 
nginx_stream_upstream_peer_healthcheck_fails 
nginx_stream_upstream_peer_healthcheck_last_passed 
nginx_stream_upstream_peer_healthcheck_unhealthy 
nginx_stream_upstream_peer_max_conns 
nginx_stream_upstream_peer_response_time 
nginx_stream_upstream_peer_state 
nginx_stream_upstream_peer_unavail 
nginx_stream_upstream_peer_up 
nginx_stream_upstream_peer_weight 

As you can see some metrics are not available yet in the official exporter.

Besides that there are some implementation differences. We have included the state (unavail,down,unhealthy,up,checking) as a label of the metric nginx_stream_upstream_peer_state so that you can differentiate between these states.

[pleshakov]

@eriksw the exporter started using the NGINX Plus client from our go sdk project -- https://github.com/nginxinc/nginx-plus-go-sdk

The client includes the GetStats method that returns stream related metrics for server zones and upstreams.

I reviewed your PR. Please see the following suggestions:

1. Rebase your code against the master.

2. Update the following existing nginxplus_stream_upstream_server metrics:

• fails, change the doc to "Number of unsuccessful attempts to communicate with the server"
• unavail, change the doc to "How many times the server became unavailable for client connections (state 'unavail') due to the number of unsuccessful attempts reaching the max_fails threshold"

3. Update the following existing stream_upstream metrics:

• zombies, change the doc to "The current number of servers removed from the group but still processing active client connections"

4. Add the following nginxplus_stream_upstream_server metrics:

• connections, Total client connections
• connect_time, Average time to connect to the upstream server
• first_byte_time, Average time to receive the first byte of data
• response_time, Average time to receive the last byte of data
• downtime, Total time the server was in the 'unavail', 'checking', and 'unhealthy' states

[pleshakov]

@eriksw we'd like to proceed with this feature. If it is OK with you, we will implement that in a separate PR and close this one?

[eriksw]

Sorry, I've been too busy to work on this. Feel free to close this and reimplement in another PR.

[Dean-Coakley]

@eriksw I've opened a PR with the extra metrics implemented: #16

Thanks for this work, it was a great base for me to work on!