Add support for KafkaProducerMetrics (similar to KafkaConsumerMetrics)

[OleksiiBondar]

It would be great to publish Kafka producer metrics which already available through JMX beans.
List of attribute names - https://docs.confluent.io/current/kafka/monitoring.html#producer-metrics

[OleksiiBondar]

If no objections, I'd like to work on this improvement.

[jkschneider]

Sounds great. Part of the reason we started with consumer metrics is the tagging is difficult to figure out sometimes. It's possible it will be easier for producer side metrics. See the lengthy discussion on #878 for how to infer partition tags. The Kafka docs at least are highly questionable on this feature, so I recommend implementing off of what you see in JMX.

[larsduelfer]

@OleksiiBondar Please check the PR #1166 to fix the consumer metrics. Have you started on this yet? I can support you on this if you want.

[OleksiiBondar]

@larsduelfer I've started implementing ProducerMetrics using JMX (similar to Consumer) but as a part of discussion #512 (comment) it was suggested to use more native way (using org.apache.kafka.common.metrics.MetricsReporter interface) which seems to be less error-prone and requires less maintanance in a future. I was able to create prototype and verify briefly consumer metrics, but haven't extend this implementation to support both producer and streams.
I've put this activity on hold for a little bit, should get back to this within a week or so.
You can look into this if interested

[mmaia]

I can try helping here with this implementation. But after reading the linked issues and discussions it's still not clear to me which is the final decision. JMX or implementing MetricsReporter directly?
Indeed it seems nice to be possible to bypass JMX but in this case the kafka client will need to be added as an optional dependency of the micrometer core and clients will have to explicitly enable the metrics configuring their kafka clients (consumers, producers or streams). If this is ok then it looks like the best approach indeed.

[shakuzen]

I can try helping here with this implementation. But after reading the linked issues and discussions it's still not clear to me which is the final decision. JMX or implementing MetricsReporter directly?

I think the MetricsReporter route would be better going forward. #1173 is open with an implementation of this, but we have a couple things to figure out before we can merge it still. Feel free to take a look and help out.

[XiyuanHu]

Hi, does it mean that to merge this PR, it requires #1173 to be merged first? Is there any branch already contains the change for this PR but just keeps unmerged? Thanks!

[shakuzen]

Hi, does it mean that to merge this PR, it requires #1173 to be merged first?

That is the current thinking. At least the approach taken by it is the direction we want to go moving forward.

Is there any branch already contains the change for this PR but just keeps unmerged? Thanks!

No, there isn't any branch with the changes yet.

[OneCricketeer]

Status?

[shakuzen]

Thank you everyone for the interest in this. This has been implemented by #1835 as an incubating feature in 1.4 (see the KafkaClientMetrics binder). Please try it out (available in snapshots now, soon to be released) and give feedback. With enough feedback, we can promote this to a non-incubating feature.