Jolokia Module with dynamic JMX Metricset #3570

ruflin · 2017-02-10T09:49:27Z

This is the implementation of a module for Jolokia which contains a dynamic jmx metricset.

An example configuration looks as following:

- module: jolokia
  metricsets: ["jmx"]
  enabled: true
  period: 1s
  hosts: ["localhost:8778"]
  namespace: "metrics"
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime
    - mbean: 'java.lang:type=GarbageCollector,name=ConcurrentMarkSweep'
      attributes:
        - attr: CollectionTime
          field: gc.cms_collection_time
        - attr: CollectionCount
          field: gc.cms_collection_count
    - mbean: 'java.lang:type=Memory'
      attributes:
        - attr: HeapMemoryUsage
          field: memory.heap_usage
        - attr: NonHeapMemoryUsage
          field: memory.non_heap_usage

For each mbeat the attributes which should be fetched can be defined. The field defines under which field name the event will be put. The namespace defines the metricset namespace.

This PR replaces elastic#3051

Further changes:

Added support for method and body to http helper
Handle empty fields in generators. This happens for a module which only contains dynamic metricsets which is currently the case for jolokia.

TODO:

Add system tests
Check documentation
Add integration test
Open issue for metricset which contains basic memory info Structured metricset for Jolokia Module #3585

tsg · 2017-02-13T09:21:37Z

metricbeat/module/jolokia/_meta/docs.asciidoc

@@ -0,0 +1,4 @@
+== jolokia Module
+


I suggest adding a beta[] tag here as well.

tsg · 2017-02-13T09:26:45Z

metricbeat/module/jolokia/jmx/config.go

+	return marshalJSONRequest(requestBodyStructure), responseMapping
+}
+
+func marshalJSONRequest(this SliceSet) string {


What's the reason for the custom marshaling? I suspect performance is unlikely an issue here.

@vas78 Can you share some insights on this one?

No specific reason to use custom marshaling here, this was just a quick-and-dirty solution to test how it would work that eventually was left behind.

tsg · 2017-02-13T09:29:20Z

metricbeat/module/jolokia/jmx/config_test.go

+	assert.Equal(t, "", jolokiaConfig[1].Instance)
+	assert.Equal(t, "", jolokiaConfig[1].Application)
+	assert.Equal(t, mappingConfigSpecial, jolokiaConfig[1].Mapping)
+	assert.Equal(t, expectedBodySpecial, actualBodySpecial)*/


Why is this commented out?

This is a left over from the "old" code base where the config had a special implementation. I removed the file now.

tsg · 2017-02-13T09:34:47Z

metricbeat/module/jolokia/jmx/data.go

+			event[aliasStructure[0]] = map[string]interface{}{aliasStructure[1]: attibuteValue}
+		}
+	default:
+		_ = fmt.Errorf("Mapping failed, alias nesting depth exceeds 1: %d", len(aliasStructure)-1)


This error seems completely silenced? Should it be a TODO entry? I guess the implementation shouldn't be difficult here.

Will try to simplify and cleanup this part further.

tsg · 2017-02-13T09:38:27Z

metricbeat/module/jolokia/jmx/jmx.go

+	config := struct {
+		Namespace string        `config:"namespace" validate:"required"`
+		Mapping   []MetricSetup `config:"jmx.mappings" validate:"required"`
+		// TODO: What are these settings for exactly?


Heh, I wanted to complain that they are not documented :).

I'm still in progress of figuring this one out: #3051 (comment) I added the in progress label again to the PR to make sure it does not get merged by accident.

tsg · 2017-02-13T10:10:49Z

LGTM, left some minor comments.

tsg · 2017-02-13T22:15:44Z

metricbeat/module/jolokia/jmx/data.go

+		return fmt.Errorf("No key found for metric: '%s', skipping...", metricName)
+	}
+
+	_, err := event.Put(key, attibuteValue)


Ah, I see why you were happy about the Put method 👍

tsg · 2017-02-13T22:19:33Z

metricbeat/module/jolokia/jmx/_meta/docs.asciidoc

+=== Limitations
+No authentication against Jolokia is supported yet.
+No wildcards in Jolokia requests supported yet.
+You can get max 30 attributes from the same MBean (see Add function in config.go).


Is this still a limitation? At list there's not Add function in config.go to check :)

Good point, this limitation was removed.

tsg

LGTM, happy to merge when you give the sign.

This is the implementation of a module for Jolokia which contains a dynamic jmx metricset. An example configuration looks as following: ``` - module: jolokia metricsets: ["jmx"] enabled: true period: 1s hosts: ["localhost:8778"] namespace: "metrics" jmx.mappings: - mbean: 'java.lang:type=Runtime' attributes: - attr: Uptime field: uptime - mbean: 'java.lang:type=GarbageCollector,name=ConcurrentMarkSweep' attributes: - attr: CollectionTime field: gc.cms_collection_time - attr: CollectionCount field: gc.cms_collection_count - mbean: 'java.lang:type=Memory' attributes: - attr: HeapMemoryUsage field: memory.heap_usage - attr: NonHeapMemoryUsage field: memory.non_heap_usage ``` For each mbeat the attributes which should be fetched can be defined. The field defines under which field name the event will be put. The namespace defines the metricset namespace. This PR replaces elastic#3051 Further changes: * Added support for method and body to http helper * Handle empty fields in generators. This happens for a module which only contains dynamic metricsets which is currently the case for jolokia. TODO: * [x] Add system tests * [x] Check documentation * [x] Add integration test * [ ] Open issue for metricset which contains basic memory info

kkmathigir · 2018-04-12T21:52:02Z

Is this consumable for Production?

ruflin · 2018-04-13T06:23:34Z

@kkmathigir It will be shipped with 6.3. If you already want to try it let me know and I can provide you a snapshot.

nielsen-at-cgt · 2018-05-21T16:57:20Z

Can I use it with Elasticsearch 5.6?

ruflin · 2018-05-22T13:08:56Z

@nielsen-at-cgt Yes, that should work.

ruflin added in progress Pull request is currently in progress. Metricbeat Metricbeat labels Feb 10, 2017

ruflin mentioned this pull request Feb 10, 2017

Add a generic jmx/jolokia metricset to metricbeat #3051

Closed

ruflin force-pushed the jmx-module branch 2 times, most recently from 913b10c to 4458863 Compare February 10, 2017 13:02

ruflin force-pushed the jmx-module branch from 4458863 to 7e1babe Compare February 10, 2017 13:07

ruflin changed the title ~~Add Jmx module~~ Jolokia Module with dynamic JMX Metricset Feb 10, 2017

ruflin force-pushed the jmx-module branch from 7e1babe to 0185998 Compare February 10, 2017 13:54

ruflin force-pushed the jmx-module branch from 0185998 to 48bcef4 Compare February 10, 2017 15:18

ruflin force-pushed the jmx-module branch from 48bcef4 to fc196ba Compare February 10, 2017 15:49

ruflin force-pushed the jmx-module branch from fc196ba to b9fb34b Compare February 13, 2017 08:34

ruflin added review and removed in progress Pull request is currently in progress. labels Feb 13, 2017

tsg reviewed Feb 13, 2017

View reviewed changes

ruflin added the in progress Pull request is currently in progress. label Feb 13, 2017

ruflin force-pushed the jmx-module branch 2 times, most recently from 2692fd8 to 0c84f51 Compare February 13, 2017 20:20

tsg reviewed Feb 13, 2017

View reviewed changes

tsg approved these changes Feb 13, 2017

View reviewed changes

ruflin force-pushed the jmx-module branch from 0c84f51 to 7ac79ae Compare February 14, 2017 07:29

ruflin removed the in progress Pull request is currently in progress. label Feb 14, 2017

ruflin mentioned this pull request Feb 14, 2017

Structured metricset for Jolokia Module #3585

Closed

tsg merged commit 9d5b11b into elastic:master Feb 14, 2017

olafmol mentioned this pull request Feb 17, 2017

Add metricbeat with JMX support to Vamp Docker packages magneticio/vamp-docker-images#21

Closed

ruflin deleted the jmx-module branch April 13, 2018 06:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jolokia Module with dynamic JMX Metricset #3570

Jolokia Module with dynamic JMX Metricset #3570

ruflin commented Feb 10, 2017 •

edited

Loading

tsg Feb 13, 2017

ruflin Feb 13, 2017

tsg Feb 13, 2017

ruflin Feb 13, 2017

vas78 Feb 13, 2017

tsg Feb 13, 2017

ruflin Feb 13, 2017

tsg Feb 13, 2017

ruflin Feb 13, 2017

tsg Feb 13, 2017

ruflin Feb 13, 2017

tsg commented Feb 13, 2017

tsg Feb 13, 2017

tsg Feb 13, 2017

ruflin Feb 14, 2017

tsg left a comment

kkmathigir commented Apr 12, 2018

ruflin commented Apr 13, 2018

nielsen-at-cgt commented May 21, 2018

ruflin commented May 22, 2018

Jolokia Module with dynamic JMX Metricset #3570

Jolokia Module with dynamic JMX Metricset #3570

Conversation

ruflin commented Feb 10, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsg commented Feb 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsg left a comment

Choose a reason for hiding this comment

kkmathigir commented Apr 12, 2018

ruflin commented Apr 13, 2018

nielsen-at-cgt commented May 21, 2018

ruflin commented May 22, 2018

ruflin commented Feb 10, 2017 •

edited

Loading