Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Switch partition metricset from client to broker #3029

Merged
merged 6 commits into from
Dec 1, 2016
Merged

Switch partition metricset from client to broker #3029

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
03cb921
Switch paritition metricset from client to broker
ruflin Nov 17, 2016
9559407
Add additional close
ruflin Nov 30, 2016
2490e31
Add retry to metadata queries
Nov 30, 2016
bd46985
kafka broker connection settings
Nov 30, 2016
cae454b
kafka partitions metricset: list topics to be queried
Nov 30, 2016
fb3ef8c
rename error to error.code
Nov 30, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion glide.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ import:
- package: github.com/miekg/dns
version: 5d001d020961ae1c184f9f8152fdc73810481677
- package: github.com/Shopify/sarama
version: fix/sasl-handshake
version: enh/offset-replica-id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking, does this branch contain the sasl fix?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, the branch is directly branched off from sasl-handshake.

repo: https://github.com/urso/sarama
- package: github.com/rcrowley/go-metrics
version: ab2277b1c5d15c3cba104e9cbddbdfc622df5ad8
Expand Down
2 changes: 2 additions & 0 deletions libbeat/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ services:
expose:
- 9092
- 2181
environment:
- ADVERTISED_HOST=kafka

# Overloading kibana with a simple image as it is not needed here
kibana:
Expand Down
21 changes: 21 additions & 0 deletions metricbeat/_meta/beat.full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,27 @@ metricbeat.modules:
#period: 10s
#hosts: ["localhost:9092"]

#client_id: metricbeat

#metadata.retries: 3
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it we should use metadata namespace here, as in the end from a user perspective it is just retrieds. Metadata is like an internal "logic" thing.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense.

#metadata.backoff: 250ms

# List of Topics to query metadata for. If empty, all topics will be queried.
#topics: []

# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"

# SASL authentication
#username: ""
#password: ""

#------------------------------- MongoDB Module ------------------------------
#- module: mongodb
Expand Down
67 changes: 65 additions & 2 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1959,15 +1959,78 @@ Oldest offset of the partition.
[float]
=== kafka.partition.partition
=== kafka.partition.offset.error
type: long
Error code from fetching offset.
[float]
== partition Fields
Partition data.
[float]
=== kafka.partition.partition.id
type: long
Partition id.
[float]
=== kafka.partition.topic
=== kafka.partition.partition.leader
type: long
Leader id (broker).
[float]
=== kafka.partition.partition.isr
type: list
List of isr ids.
[float]
=== kafka.partition.partition.replica
type: long
Replica id (broker).
[float]
=== kafka.partition.partition.insync_replica
type: boolean
Indicates if replica is included in the in-sync replicate set (ISR).
[float]
=== kafka.partition.partition.error
type: long
Error code from fetching partition.
[float]
=== kafka.partition.topic.error
type: long
topic error.
[float]
=== kafka.partition.topic.name
type: keyword
Expand Down
21 changes: 21 additions & 0 deletions metricbeat/docs/modules/kafka.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,27 @@ metricbeat.modules:
#period: 10s
#hosts: ["localhost:9092"]

#client_id: metricbeat

#metadata.retries: 3
#metadata.backoff: 250ms

# List of Topics to query metadata for. If empty, all topics will be queried.
#topics: []

# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"

# SASL authentication
#username: ""
#password: ""
----

[float]
Expand Down
21 changes: 21 additions & 0 deletions metricbeat/metricbeat.full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,27 @@ metricbeat.modules:
#period: 10s
#hosts: ["localhost:9092"]

#client_id: metricbeat

#metadata.retries: 3
#metadata.backoff: 250ms

# List of Topics to query metadata for. If empty, all topics will be queried.
#topics: []

# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"

# SASL authentication
#username: ""
#password: ""

#------------------------------- MongoDB Module ------------------------------
#- module: mongodb
Expand Down
34 changes: 30 additions & 4 deletions metricbeat/metricbeat.template-es2x.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -941,6 +941,9 @@
},
"offset": {
"properties": {
"error": {
"type": "long"
},
"newest": {
"type": "long"
},
Expand All @@ -950,12 +953,35 @@
}
},
"partition": {
"type": "long"
"properties": {
"error": {
"type": "long"
},
"id": {
"type": "long"
},
"insync_replica": {
"type": "boolean"
},
"leader": {
"type": "long"
},
"replica": {
"type": "long"
}
}
},
"topic": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
"properties": {
"error": {
"type": "long"
},
"name": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
}
}
}
}
}
Expand Down
32 changes: 29 additions & 3 deletions metricbeat/metricbeat.template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -948,6 +948,9 @@
},
"offset": {
"properties": {
"error": {
"type": "long"
},
"newest": {
"type": "long"
},
Expand All @@ -957,11 +960,34 @@
}
},
"partition": {
"type": "long"
"properties": {
"error": {
"type": "long"
},
"id": {
"type": "long"
},
"insync_replica": {
"type": "boolean"
},
"leader": {
"type": "long"
},
"replica": {
"type": "long"
}
}
},
"topic": {
"ignore_above": 1024,
"type": "keyword"
"properties": {
"error": {
"type": "long"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
}
Expand Down
21 changes: 21 additions & 0 deletions metricbeat/module/kafka/_meta/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,24 @@
#period: 10s
#hosts: ["localhost:9092"]

#client_id: metricbeat
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove all tehse config options from the short config.


#metadata.retries: 3
#metadata.backoff: 250ms

# List of Topics to query metadata for. If empty, all topics will be queried.
#topics: []

# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"

# SASL authentication
#username: ""
#password: ""
17 changes: 11 additions & 6 deletions metricbeat/module/kafka/partition/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,19 @@
"id": 0
},
"offset": {
"newest": 13,
"newest": 11,
"oldest": 0
},
"partition": 0,
"replicas": [
0
],
"topic": "testtopic"
"partition": {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs update. I will do a PR as soon as this is merged.

"error": 0,
"id": 0,
"insync_replica": true,
"leader": 0,
"replica": 0
},
"topic": {
"name": "test-metricbeat-8760238589576171408"
}
}
},
"metricset": {
Expand Down
45 changes: 43 additions & 2 deletions metricbeat/module/kafka/partition/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,53 @@
type: long
description: >
Oldest offset of the partition.
- name: error
type: long
description: >
Error code from fetching offset.
- name: partition
type: group
description: >
Partition data.
fields:
- name: id
type: long
description: >
Partition id.
- name: leader
type: long
description: >
Leader id (broker).
- name: isr
type: list
description: >
List of isr ids.
- name: replica
type: long
description: >
Replica id (broker).
- name: insync_replica
type: boolean
description: >
Indicates if replica is included in the in-sync replicate set (ISR).
- name: error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe error_code for clarity?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, if we switch this one to error_code, we would have to change the other 2 errors too. WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error_code seems a bit more natural to me, but I don't feel to strong about it.

type: long
description: >
Error code from fetching partition.
- name: topic.error
type: long
description: >
Partition id.
- name: topic
topic error.
- name: topic.name
type: keyword
description: >
Topic name
- name: broker.id
type: long
description: >
Expand All @@ -32,3 +71,5 @@
type: keyword
description: >
Broker address
Loading