KafkaProducer Metrics missing TOPIC name tag

[vijaycse]

Issue Description
KakfaConsumerMetrics (MBean implementation) has a way to get the TOPIC name added part of the tags.

micrometer/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/kafka/KafkaConsumerMetrics.java

Line 357 in 6262af1

String topic = name.getKeyProperty("topic");

but I understand from this issue #1095 that we are not going with MBean implementation anymore.

We have a producer that produces into multiple topics but there is no way for us to differentiate the metrics by TOPICs(unlike consumer)

As you can see the topic name is emitted by the MBean from kakfa
https://docs.confluent.io/platform/current/kafka/monitoring.html#per-topic-metrics

MBean: kafka.producer:type=producer-topic-metrics,client-id=([-.w]+),topic=([-.w]+)

It seems to me that MetricReporter(https://github.com/micrometer-metrics/micrometer/pull/1173/files) implementation not getting everything like MBean.

Rationale
I thing it would be better to add TOPIC name (either by MetricReporter implementation or MBean (not ideal though)
here

micrometer/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/kafka/KafkaMetrics.java

Line 238 in 6262af1

private List<Tag> meterTags(Metric metric, boolean includeCommonTags) {

[jonatan-ivanov]

@vijaycse What metrics do you want to differentiate with the topic name? Kafka only adds the topic name to these metrics:

{name='kafka.producer.topic.record.error.rate', tags=[tag(client.id=producer-1),tag(kafka.version=2.6.0),tag(topic=test)]}
{name='kafka.producer.topic.byte.total', tags=[tag(client.id=producer-1),tag(kafka.version=2.6.0),tag(topic=test)]}
{name='kafka.producer.topic.record.retry.rate', tags=[tag(client.id=producer-1),tag(kafka.version=2.6.0),tag(topic=test)]}
{name='kafka.producer.topic.byte.rate', tags=[tag(client.id=producer-1),tag(kafka.version=2.6.0),tag(topic=test)]}
{name='kafka.producer.topic.record.error.total', tags=[tag(client.id=producer-1),tag(kafka.version=2.6.0),tag(topic=test)]}
{name='kafka.producer.topic.record.send.rate', tags=[tag(client.id=producer-1),tag(kafka.version=2.6.0),tag(topic=test)]}
{name='kafka.producer.topic.record.send.total', tags=[tag(client.id=producer-1),tag(kafka.version=2.6.0),tag(topic=test)]}
{name='kafka.producer.topic.record.retry.total', tags=[tag(client.id=producer-1),tag(kafka.version=2.6.0),tag(topic=test)]}
{name='kafka.producer.topic.compression.rate', tags=[tag(client.id=producer-1),tag(kafka.version=2.6.0),tag(topic=test)]}

[jonatan-ivanov]

@vijaycse I'm closing this since the meters above has the topic tag, if you want more meters to have it, I think that should be changed on Kafka side. Please let us know if I misunderstood something and you want the issue to be reopened.

[vijaycse]

Thanks @jonatan-ivanov . I will get back to you. I am still not seeing the topic tag. perhaps its on our side or version. I need to dive deep. thanks for taking a look

[ash432ash]

@vijaycse Do you find any solution for adding the topic tag to the producer metrics? Like for these metrics
request-latency-avg
request-latency-max
We are not getting data topic wise

[vijaycse]

It was a long time ago. Sorry, I do not remember exactly. Did you attempt to try latest kafka version etc?

…

On Wed, Aug 7, 2024 at 3:59 AM Ayush Verma ***@***.***> wrote: @vijaycse <https://github.com/vijaycse> Do you find any solution for adding the topic tag to the producer metrics? — Reply to this email directly, view it on GitHub <#2746 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDARNBQUTXSEA5A77NOKZLZQHOXBAVCNFSM6AAAAABMD6JD5CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZSHE3TGNJZGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Regards, Vijay R