Richer key semantics to allow backend-specific transformations

[maxvt]

Hi, I'm coming to this via DataDog/dd-agent#2959 and after seeing some of go-metric's code, in particular a related request in #9.

The problem is that different backends have different features that imply formatting rules. statsd, the simplest, only has a metric name and value. Datadog has name, value and key-value pairs ("tags"). Prometheus appears to have name, value and key-value pairs ("labels"). However, software using go-metrics would want to not have any backend-specific code (perhaps not even configuration as suggested by #9, because then different configurations for prometheus, datadog, and so on would need to be done, with potentially different APIs/settings based on features and community standards of the particular backend).

My suggestion is to pass richer metadata into the backends using the existing API, allowing backends to transform the metric according to the backend's preferences. The new rule is that a particular starting character in a name component identifies a key/tag/variable, and the next name component is interpreted as the corresponding value.

Before the change:

met.SetGauge([]string{"nomad", "myapp", "memory", "used"}, float32(1))

is the amount of memory used by job "myapp" in Nomad. Job names are set by users, so that component in the key is changeable. With the suggested change, this call would become:

met.SetGauge([]string{"nomad", ":appname", "myapp", "memory", "used"}, float32(1))

thus telling all backends that there is a structured piece of information passed within the key, with name "appname" and value "myapp".

Immediately we can construct very simple but idiomatic default transforms:

• statsd: suppresses all names from the key string and retains the values, since it doesn't support any richer formatting, resulting in nomad.myapp.memory.used|1.
• datadog: removes all structured pieces and emits them as tags: equivalent to output of SetGaugeWithTags({"nomad", "memory", "used"}, 1, {"appname:myapp"})
• prometeus: equivalent to output of NewGaugeVec({"nomad", "memory", "used"}, 1, {"appname:myapp"})

Note how this can be immediately applied to structured pieces added by go-metrics itself:
met.HostName = "test"
met.EnableHostname = true
met.SetGauge([]string{"key"}, float32(1))

would call
m.sink.SetGauge({":hostname", m.HostName, key}, val)

and Datadog/Prometeus would automatically emit the hostname as tag, which most users of these services would be very happy about, while statsd output would be unchanged, without any changes on the gometrics library users.

Observe that saying "we can just pass key/value pairs separately via a different API" is not enough to match this functionality, because the statsd backend would not have any rules on how to transform this information into a metric key -- it would require a "format string" or to adopt one standard behavior which might or might not be a breaking change to many users.

This can also contribute to the issue faced by @juliusv, @stapelberg, @jeinwag and possibly others who would like to do richer transforms and extraction -- except, instead of having to extract meaningful key/value pairs with regex (and potentially facing issues with unintentional matches, etc) the key/value pairs are already provided in the call to the backend. Further transforms can then be applied on this richer source information if desired.

I'm curious if this sounds reasonable and whether a PR implementing this is likely to be considered. I would be willing to give this a stab. Thanks for your feedback!

[evan2645]

Something like this would be super useful for us