Unix Domain Socket support (StatsD)

[kgoralski]

java-dogstatsd-client is going to support Unix Domain Socket (see DataDog/java-dogstatsd-client#42)

is Micrometer is going to support it too?

https://docs-staging.datadoghq.com/xvello/dsd_uds/developers/dogstatsd/unix_socket/
https://github.com/DataDog/datadog-agent/wiki/Unix-Domain-Sockets-support
https://github.com/DataDog/java-dogstatsd-client#unix-domain-socket-support

[jonnywray]

This would be really useful in a Kubernetes environment for adding various deployment details, container, and k8s tags to the metrics.

We've experimented with different approaches for the DD reporter for Dropwizard metrics and UDS was the easiest to get up and running. Support here (and so for Spring Boot 2.x) would be great.

[jkschneider]

This would be really useful in a Kubernetes environment for adding various deployment details, container, and k8s tags to the metrics.

@jonnywray Could you elaborate on why a different transport is related to additional tags?

[jonnywray]

For sure. In k8s the application knows nothing about the deployment environment, but of course it's useful for custom metrics coming from an app to have those, e.g. k8s namespace, host name, container details, etc. Using the DD API is out as that means the app needs the values.

There are two recommended ways of dealing with it outlined in the DD docs (https://docs.datadoghq.com/agent/kubernetes/dogstatsd/) and we've found the unix domain socket approach much easier to get going. And I think is the recommended approach now. Once this is working the various docker and k8s tags are added automatically to your custom metrics.

I tried the other approach, using a k8s hostPort and DogStatsD instance, but could never get it to work.

There is a work around using DD API. You can use the k8s downward api to add environment variable to your application deployment and use them as custom tags in a MeterRegistryCustomizer. But, you don't get all available tags this way (e.g. docker image details) and automated tagging would be great.

[jkschneider]

Cool, thanks! Sounds like a very useful feature.

[PandaXass]

This feature could potentially work around containernetworking/plugins#123 for Datadog agent?

[Simwar]

@PandaXass I believe so, as the connection will be through the socket, not via the port mapped via hostPort.
Looks like it is planned, good to know!

[aantono]

Something like this would need to be added: https://github.com/Frankqsy/netty-sample/blob/master/netty-client/src/main/java/org/daisy/stevin/netty/sample/client/uds/UdsClient.java

[liu0013]

looking forward to be implemented, it is very useful for k8s integrate with Datadog!

[aantono]

@jkschneider Any updates on the progress of this?

[shakuzen]

Any updates on the progress of this?

Due to the apparent interest in this new feature, I'm tentatively scheduling this for the 1.5 release. It still has the help wanted label as a pull request implementing this could help get this integrated sooner.

[pboado]

Hi, isn't the implementation really simple, just changing StatsdMeterRegistry.prepareTcpClient to have a slightly different connection for UDS?

private void prepareTcpClient(Publisher<String> publisher) {
    AtomicReference<TcpClient> tcpClientReference = new AtomicReference<>();
    TcpClient tcpClient =
        getTcpClient(statsdConfig)
            .handle((in, out) -> out.sendString(publisher).neverComplete())
            .doOnDisconnected(
                connection -> {
                  Boolean connectionDisposed =
                      connection.channel().attr(CONNECTION_DISPOSED).getAndSet(Boolean.TRUE);
                  if (connectionDisposed == null || !connectionDisposed) {
                    connectAndSubscribe(tcpClientReference.get());
                  }
                });
    tcpClientReference.set(tcpClient);
    connectAndSubscribe(tcpClient);
  }

  private TcpClient getTcpClient(StatsdConfig config) {
    if (config.host().startsWith("unix:")) {
      return TcpClient.create().remoteAddress(() -> new DomainSocketAddress(config.host()));
    } else {
      return TcpClient.create().host(config.host()).port(config.port());
    }
  }

[shakuzen]

@pboado yes, I suppose it is that simple. Though to be fair, that support was added to reactor-netty long after this issue was originally opened. We would need to upgrade reactor-netty for that to be available, which we had issues doing before and so the upgrade got backed out. We're planning to upgrade for the Micrometer 1.8 release, so I'll target this at the 1.8 release now that it is clear how we can easily support this with minimal maintenance on our side.

[pboado]

I'm pretty sure the shaded version of reactor-netty packed with micrometer already includes the functionality.

[shakuzen]

While the netty class may be available, support for UDS was not added to reactor-netty until 1.0.0. See reactor/reactor-netty#403

[pboado]

Fair enough, thanks for looking into this.

[shakuzen]

Now that we have the upgrade to Reactor 2020.0 in main, I was trying to implement this using the netty/reactor-netty features. Unfortunate news is that dogstatsd is listening on a datagram type domain socket, and netty only supports stream domain sockets currently. There is netty/netty#6737 open to add datagram support but no release targeted. This means we can't just use the features offered by netty/reactor-netty to get this, as we hoped. It might be easier to implement this ourselves with JEP 380 but that is only available from JDK 16.

[jhohertz]

Hello! I maintain a fork of a dropwizard plugin for datadog that was originally done to enable the UDS support within.

I say the following with little regard for the impacts, but If you were to split of the datadog implementation to be it's "own thing" and make use of the "official" dogstatsd library you can put the responsibility for the odd implementation there. Not sure if there are other... liberties... to the statsd implementation they have taken, that could also be leveraged by using their library.

If you make use of that, they did a cute (?) thing where you specify the port as zero, and the hostname becomes a pathname to the socket.

I just noticed you were considering differing implementations, and thought I would throw this out as a potential approach to isolate the datadog-isms and keep your statsd client implementation from having to deal with those kinds of concerns.

That said, perhaps UDS is useful elsewhere too. Their implementation makes use of the jnr-unixsocket library. Nice to know future JDKs will have some native support for UDS, as it does get a bit awkward to use from java today/

Anyways, just wanted to reach out and see if I could do anything however small to help. We make some heavy use of UDS support and there is a lot of interest in micrometer too, so there are several sets of eyeballs tracking this item over here, and possibly able to help in some way.

[jhohertz]

Looks like the needed support in reactor just dropped in reactor/reactor-netty#1741

[shakuzen]

It's been a long wait, but support for datagram Unix domain socket protocol has just now landed for the StatsdMeterRegistry. It is available for early testing in 1.8.0-SNAPSHOT versions and should be in the upcoming 1.8.0-M2 milestone release. You can test it with snapshots by configuring the protocol to the newly added UDS_DATAGRAM and setting the path to the socket as the host config. Let us know if there are any issues or feedback on it. A huge thank you to @violetagg who contributed the underlying support that made this possible to the Netty project and subsequently added support on top of that in Reactor Netty.
I did some ad hoc testing with a Datadog agent running locally on my machine and running an app in a Kubernetes cluster with the datadog agent configured.