java agent reporting high cardinality metrics for http_client_duration_bucket

[gautam-nutalapati]

Hi,
I am using otel java agent 1.10.1
And I see metrics like below generated for my http traffic:

http_client_duration_bucket{http_flavor="1.1",http_method="GET",http_status_code="200",http_url="https://idp.vault.dev-local.prisidio.net/.well-known/jwks.json",le="10"} 1
http_client_duration_bucket{http_flavor="1.1",http_method="GET",http_status_code="500",http_url="http://localhost:9900/stux/v1/users/92108123607311744",le="10"} 1

The http_url is logging full URL path, which produces high cardinality metrics.
Looking at specs, I thought http.url should be using variable for path parameter to prevent high cardinality metrics.
Is this the default expected behaviour?
Is it possible to disable http.client.duration metric? or disabling default http metrics altogether?
Any guidance is much appreciated. I couldn't find much resources/discussions regarding this.

[anuraaga]

Hi @gautam-nutalapati - we have been looking at cardinality of server metrics but may have forgotten about client. I agree that we shouldn't be populating the URL there. @mateuszrzeszutek does that sound right?

[gautam-nutalapati]

Thank you for the response! Is there a way to disable these HTTP client metrics meanwhile when using auto-instrumentation java agent?

[mateuszrzeszutek]

Is there a way to disable these HTTP client metrics meanwhile when using auto-instrumentation java agent?

No, unfortunately there is no simple way to do that. Perhaps it could be achieved with implementing an extension and registering a custom view, e.g.

meterProviderBuilder
  .registerView(
    InstrumentSelector.builder()
      .setInstrumentName("http.client.duration")
      .build(),
    View.builder()
      .filterAttributes(key -> !key.equals("http.url"))
      .build())

I agree that we shouldn't be populating the URL there. @mateuszrzeszutek does that sound right?

Yes, we should definitely fix that. Currently we're already trimming the query from the http.url attribute; should we remove the path too? Or just include the net.* attributes instead of the URL.

CC @jsuereth

[anuraaga]

I think we should omit completely rather than trimming query or path since it ends up being the same as net attributes at that point

[anuraaga]

Maybe this is a good time to try to set a "default view" in the agent. Always recording high cardinality metrics is a no go, but having no way of recording them is also not ideal. We would like to be able to opt in to it, based on knowledge that a certain codepath does not have cardinality, so we probably want to set a default view which a user can override, inconveniently now using an extension and conveniently in the future using configuration.

[irizzant]

Prometheus allows for relabelling and allows removing http-url, but I definitively agree this label creates high cardinality and also high churn rates in Prometheus because of short-living metrics.
In my opinion http-url should be removed

[mateuszrzeszutek]

Maybe this is a good time to try to set a "default view" in the agent.

Hint API? 😄

Right now our "views" are just a simple hack, so for now we should probably replace the url attribute with net.peer* attrs

[anuraaga]

Definitely not hint API since API is in user control but this is about instrumentation / agent ;) I mean initially there was some problem with the SDK but now we should be able to register views in it when initializing the SDK right? I can't remember exactly what the problem was before but think we fixed something.