Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kafkajs]: Instrument Consumer #2218

Closed
bizob2828 opened this issue May 28, 2024 · 1 comment · Fixed by #2244
Closed

[kafkajs]: Instrument Consumer #2218

bizob2828 opened this issue May 28, 2024 · 1 comment · Fixed by #2244
Assignees

Comments

@bizob2828
Copy link
Member

bizob2828 commented May 28, 2024

Description

We should add instrumentation to support eachMessage. See docs

Consumer

Each time a message is consumed by the consumer, a transaction MUST be
started/stopped. If possible, a transaction SHOULD be started and stopped around a
single message. If not possible, a transaction MUST be started and stopped around
multiple messages.

Transaction duration MUST include time of the following:

  • The customer's handling of the message(s).
    Transaction duration MAY include time of the following:
  • Open a connection as needed
  • Fetch the data
  • Close the connection as needed

Transaction Name

The transaction name MUST be in the following format:
Message/Kafka/Topic/Consume/Named/{topic_name}

Metric Names

The following metrics MUST be captured for each transaction and the metric names
MUST be in the following format:

Attributes Description
Message/Kafka/Topic/Named/{topic_name}/Received/Bytes The number of Bytes received in the message for each topic.
Message/Kafka/Topic/Named/{topic_name}/Received/Messages The number of messages received for each topic.

Agent Attributes

The following attributes SHOULD be included in each transaction:

Attributes Description
kafka.consume.byteCount The number of Bytes received in the request.
kafka.consume.client_id The client id of the consumer receiving the data.

Deserialization Metrics

Because serializing and deserializing is such a common thing to do with Kafka, many
implementations include this as an option directly on the producer/consumer or as a
specialized Serializingproducer Deserializingconsumer. It very useful to capture the
time it takes to deserialize the consumer's message as there may be significant slow
downs due to this type of operation. A message consists of both a key and a value which
are deserialized independently. If deserialization monitoring is possible, the following
metrics MUST be captured:

  • MessageBroker/Kafka/Topic/Named/{topic_name}/Serialization/Value
  • MessageBroker/Kafka/Topic/Named/{topic_name}/Serialization/Key

Additional context

  • The deserialization metrics may not be possible with kafkajs.
  • We should be able to use the message-shim recordConsume method with the appropriate spec
@workato-integration
Copy link

@bizob2828 bizob2828 changed the title [node-rdkafka]: Instrument Consumer [kafkajs]: Instrument Consumer May 29, 2024
@bizob2828 bizob2828 self-assigned this May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant