Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus remote write endpoint #16609

Merged
merged 34 commits into from
Mar 12, 2020
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
4039278
WIP
May 30, 2019
a356dd7
remove remote_read functionality
ChrsMark Feb 27, 2020
8cc5380
make update
ChrsMark Feb 27, 2020
0b35dcd
Add docs and fix config
ChrsMark Feb 28, 2020
5bf8324
Add data.json
ChrsMark Feb 28, 2020
da960f3
Merge remote-tracking branch 'upstream/master' into prom_remote_endpoint
ChrsMark Mar 4, 2020
db3bad5
rebase
ChrsMark Mar 4, 2020
0adda8c
add dep using go mod
ChrsMark Mar 4, 2020
d51a12a
Add python system test
ChrsMark Mar 5, 2020
58e41ca
refactoring
ChrsMark Mar 5, 2020
d052651
refactor handler functions
ChrsMark Mar 6, 2020
fd89b1f
optimisations
ChrsMark Mar 6, 2020
3e92bfb
fix http helper tests
ChrsMark Mar 6, 2020
2e9af93
update config
ChrsMark Mar 6, 2020
87a5946
go mod tidy
ChrsMark Mar 9, 2020
a0880f3
mage vendor
ChrsMark Mar 9, 2020
59f4b41
make notice
ChrsMark Mar 9, 2020
14d2643
Merge remote-tracking branch 'upstream/master' into prom_remote_endpoint
ChrsMark Mar 9, 2020
ae87c0b
fixes
ChrsMark Mar 9, 2020
d542b14
fix notice
ChrsMark Mar 9, 2020
c4ebbfc
Add python requirements
ChrsMark Mar 9, 2020
efbf774
Remove python deps and tune docker env
ChrsMark Mar 9, 2020
e5d7c20
Add system test
ChrsMark Mar 9, 2020
5a7049b
fixes
ChrsMark Mar 9, 2020
276a287
update fields.yml
ChrsMark Mar 9, 2020
45d6c11
remove leftover command
ChrsMark Mar 10, 2020
001e4d8
Merge remote-tracking branch 'upstream/master' into prom_remote_endpoint
ChrsMark Mar 10, 2020
18c6ce2
bump up container version
ChrsMark Mar 10, 2020
eb755ac
review changes
ChrsMark Mar 10, 2020
841fc7c
fixes
ChrsMark Mar 10, 2020
453e808
refactor NewHttpServer and NewHttpServerWithHandler
ChrsMark Mar 10, 2020
02c7902
fix run.sh script
ChrsMark Mar 11, 2020
6402466
remove leftover
ChrsMark Mar 11, 2020
05bdecd
Merge remote-tracking branch 'upstream/master' into prom_remote_endpoint
ChrsMark Mar 11, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,7 @@ addons:
- libxml2-utils
- libsystemd-journal-dev
- librpm-dev
- libsnappy-dev
# From deadsnakes PPA
- python3.6
- python3.6-venv
Expand Down
2 changes: 2 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ require (
github.com/gorhill/cronexpr v0.0.0-20161205141322-d520615e531a
github.com/gorilla/mux v1.7.2 // indirect
github.com/gorilla/websocket v1.4.1 // indirect
github.com/grpc-ecosystem/grpc-gateway v1.13.0 // indirect
github.com/hashicorp/golang-lru v0.5.2-0.20190520140433-59383c442f7d // indirect
github.com/insomniacslk/dhcp v0.0.0-20180716145214-633285ba52b2
github.com/jcmturner/gofork v1.0.0 // indirect
Expand Down Expand Up @@ -118,6 +119,7 @@ require (
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4
github.com/prometheus/common v0.7.0
github.com/prometheus/procfs v0.0.9-0.20191208103036-42f6e295b56f
github.com/prometheus/prometheus v2.5.0+incompatible // indirect
github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a
github.com/reviewdog/reviewdog v0.9.17
github.com/samuel/go-parser v0.0.0-20130731160455-ca8abbf65d0e // indirect
Expand Down
11 changes: 11 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ github.com/andrewkroh/goja v0.0.0-20190128172624-dd2ac4456e20 h1:7rj9qZ63knnVo2Z
github.com/andrewkroh/goja v0.0.0-20190128172624-dd2ac4456e20/go.mod h1:cI59GRkC2FRaFYtgbYEqMlgnnfvAwXzjojyZKXwklNg=
github.com/andrewkroh/sys v0.0.0-20151128191922-287798fe3e43 h1:WFwa9pqou0Nb4DdfBOyaBTH0GqLE74Qwdf61E7ITHwQ=
github.com/andrewkroh/sys v0.0.0-20151128191922-287798fe3e43/go.mod h1:tJPYQG4mnMeUtQvQKNkbsFrnmZOg59Qnf8CcctFv5v4=
github.com/antihax/optional v0.0.0-20180407024304-ca021399b1a6/go.mod h1:V8iCPQYkqmusNa815XgQio277wI47sdRh1dUOLdyC6Q=
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
github.com/aws/aws-lambda-go v1.6.0 h1:T+u/g79zPKw1oJM7xYhvpq7i4Sjc0iVsXZUaqRVVSOg=
Expand Down Expand Up @@ -261,6 +262,7 @@ github.com/frankban/quicktest v1.4.1/go.mod h1:36zfPVQyHxymz4cH7wlDmVwDrJuljRB60
github.com/garyburd/redigo v1.0.1-0.20160525165706-b8dc90050f24 h1:nREVDi4H8mwnNqfxFU9NMzZrDCg8TXbEatMvHozxKwU=
github.com/garyburd/redigo v1.0.1-0.20160525165706-b8dc90050f24/go.mod h1:NR3MbYisc3/PwhQ00EMzDiPmrwpPxAn5GI05/YaO1SY=
github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0 h1:wDJmvq38kDhkVxi50ni9ykkdUr1PKgqKOoi01fa0Mdk=
Expand Down Expand Up @@ -371,6 +373,8 @@ github.com/gorilla/mux v1.7.2/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2z
github.com/gorilla/websocket v1.4.1 h1:q7AeDBpnBk8AogcD4DSag/Ukw/KV+YhzLj2bP5HvKCM=
github.com/gorilla/websocket v1.4.1/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/gregjones/httpcache v0.0.0-20170728041850-787624de3eb7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
github.com/grpc-ecosystem/grpc-gateway v1.13.0 h1:sBDQoHXrOlfPobnKw69FIKa1wg9qsLLvvQ/Y19WtFgI=
github.com/grpc-ecosystem/grpc-gateway v1.13.0/go.mod h1:8XEsbTttt/W+VvjtQhLACqCisSPWTxCZ7sBRjU6iH9c=
github.com/hashicorp/errwrap v0.0.0-20141028054710-7554cd9344ce/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-multierror v0.0.0-20161216184304-ed905158d874/go.mod h1:JMRHfdO9jKNzS/+BTlxCjKNQHg/jZAft8U7LloJvN7I=
github.com/hashicorp/go-uuid v1.0.1 h1:fv1ep09latC32wFoVwnqcnKJGnMSdBanPczbHAYm1BE=
Expand Down Expand Up @@ -552,13 +556,16 @@ github.com/prometheus/procfs v0.0.3/go.mod h1:4A/X28fw3Fc593LaREMrKMqOKvUAntwMDa
github.com/prometheus/procfs v0.0.5/go.mod h1:4A/X28fw3Fc593LaREMrKMqOKvUAntwMDaekg4FpcdQ=
github.com/prometheus/procfs v0.0.9-0.20191208103036-42f6e295b56f h1:i2BUTcG1g7lSgF/xVL4BJBAdrtNp4zL8woVTGHpEG4g=
github.com/prometheus/procfs v0.0.9-0.20191208103036-42f6e295b56f/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A=
github.com/prometheus/prometheus v2.5.0+incompatible h1:7QPitgO2kOFG8ecuRn9O/4L9+10He72rVRJvMXrE9Hg=
github.com/prometheus/prometheus v2.5.0+incompatible/go.mod h1:oAIUtOny2rjMX0OWN5vPR5/q/twIROJvdqnQKDdil/s=
github.com/rakyll/statik v0.1.6/go.mod h1:OEi9wJV/fMUAGx1eNjq75DKDsJVuEv1U0oYdX6GX8Zs=
github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a h1:9ZKAASQSHhDYGoxY8uLVpewe1GDZ2vu2Tr/vTdVAkFQ=
github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/reviewdog/errorformat v0.0.0-20200109134752-8983be9bc7dd h1:fvaEkjpr2NJbtnFRCft7D6y/mQ5/2OQU0pKJLW8dwFA=
github.com/reviewdog/errorformat v0.0.0-20200109134752-8983be9bc7dd/go.mod h1:giYAXnpegRDPsXUO7TRpDKXJo1lFGYxyWRfEt5iQ+OA=
github.com/reviewdog/reviewdog v0.9.17 h1:MKb3rlQZgkEXr3d85iqtYNITXn7gDJr2kT0IhgX/X9A=
github.com/reviewdog/reviewdog v0.9.17/go.mod h1:Y0yPFDTi9L5ohkoecJdgbvAhq+dUXp+zI7atqVibwKg=
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/samuel/go-parser v0.0.0-20130731160455-ca8abbf65d0e h1:hUGyBE/4CXRPThr4b6kt+f1CN90no4Fs5CNrYOKYSIg=
github.com/samuel/go-parser v0.0.0-20130731160455-ca8abbf65d0e/go.mod h1:Sb6li54lXV0yYEjI4wX8cucdQ9gqUJV3+Ngg3l9g30I=
Expand Down Expand Up @@ -692,6 +699,7 @@ golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191002035440-2ec189313ef0/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191021144547-ec77196f6094/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191112182307-2180aed22343/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
Expand Down Expand Up @@ -800,12 +808,14 @@ google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRn
google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/genproto v0.0.0-20190927181202-20e1ac93f88c/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb h1:ADPHZzpzM4tk4V4S5cnCrr5SwzvlrPRmqqCuJDB8UTs=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.24.0/go.mod h1:XDChyiUovWa60DnaeDeZmSW86xtLtjtZbwvSiRnRtcA=
google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.1 h1:zvIju4sqAGvwKspUQOhwnpcqSbzi7/H6QomNNjTL4sk=
google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
Expand Down Expand Up @@ -839,6 +849,7 @@ gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWD
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.7 h1:VUgggvou5XRW9mHwD/yXxIYSMtY0zoKQf/v226p2nyo=
Expand Down
1 change: 1 addition & 0 deletions metricbeat/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ RUN \
python3 \
python3-pip \
python3-venv \
libsnappy-dev \
&& rm -rf /var/lib/apt/lists/*

ENV PYTHON_ENV=/tmp/python-env
Expand Down
14 changes: 14 additions & 0 deletions metricbeat/docs/modules/prometheus.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ in <<configuration-metricbeat>>. Here is an example configuration:
metricbeat.modules:
- module: prometheus
period: 10s
metricsets: ["collector"]
hosts: ["localhost:9090"]
metrics_path: /metrics
#metrics_filters:
Expand All @@ -42,6 +43,15 @@ metricbeat.modules:
#bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
#ssl.certificate_authorities:
# - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

- module: prometheus
metricsets: ["remote_write"]
host: "localhost"
port: "9201"

# Secure settings for the server using TLS/SSL:
#ssl.certificate: "/etc/pki/server/cert.pem"
#ssl.key: "/etc/pki/server/cert.key"
----

This module supports TLS connections when using `ssl` config field, as described in <<configuration-ssl>>.
Expand All @@ -54,5 +64,9 @@ The following metricsets are available:

* <<metricbeat-metricset-prometheus-collector,collector>>

* <<metricbeat-metricset-prometheus-remote_write,remote_write>>

include::prometheus/collector.asciidoc[]

include::prometheus/remote_write.asciidoc[]

23 changes: 23 additions & 0 deletions metricbeat/docs/modules/prometheus/remote_write.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
////
This file is generated! See scripts/mage/docs_collector.go
////

[[metricbeat-metricset-prometheus-remote_write]]
=== Prometheus remote_write metricset

beta[]

include::../../../module/prometheus/remote_write/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-prometheus,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../module/prometheus/remote_write/_meta/data.json[]
----
3 changes: 2 additions & 1 deletion metricbeat/docs/modules_list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,8 @@ This file is generated! See scripts/mage/docs_collector.go
|<<metricbeat-metricset-postgresql-database,database>>
|<<metricbeat-metricset-postgresql-statement,statement>>
|<<metricbeat-module-prometheus,Prometheus>> |image:./images/icon-yes.png[Prebuilt dashboards are available] |
.1+| .1+| |<<metricbeat-metricset-prometheus-collector,collector>>
.2+| .2+| |<<metricbeat-metricset-prometheus-collector,collector>>
|<<metricbeat-metricset-prometheus-remote_write,remote_write>> beta[]
|<<metricbeat-module-rabbitmq,RabbitMQ>> |image:./images/icon-yes.png[Prebuilt dashboards are available] |
.4+| .4+| |<<metricbeat-metricset-rabbitmq-connection,connection>>
|<<metricbeat-metricset-rabbitmq-exchange,exchange>>
Expand Down
23 changes: 20 additions & 3 deletions metricbeat/helper/server/http/http.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,12 @@ type HttpEvent struct {
meta server.Meta
}

// MetricSetFactory accepts a BaseMetricSet and returns a MetricSet. If there
// was an error creating the MetricSet then an error will be returned. The
// returned MetricSet must also implement either EventFetcher or EventsFetcher
// (but not both).
type HandlerFactory func(h *HttpServer) func(writer http.ResponseWriter, req *http.Request)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you really need a factory? The metricset is already initializing the server, so you could pass a private metricset.handleFunc which implements func(writer http.ResponseWriter, req *http.Request)


func (h *HttpEvent) GetEvent() common.MapStr {
return h.event
}
Expand All @@ -52,7 +58,15 @@ func (h *HttpEvent) GetMeta() server.Meta {
return h.meta
}

func NewHttpServer(mb mb.BaseMetricSet) (server.Server, error) {
func (h *HttpEvent) SetEvent(event common.MapStr) {
h.event = event
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having that you are now passing a custom handler when initializing (which I think is a good idea), do you need an HttpEvent construct anymore? This was used to pass data between a generic HTTP server and the metricset. But now the metricset is in control of the requests, and can directly build and send events to the output. That should make things simpler, WDYT?


func (h *HttpEvent) SetMeta(meta server.Meta) {
h.meta = meta
}

func NewHttpServer(mb mb.BaseMetricSet, factory HandlerFactory) (server.Server, error) {
config := defaultHttpConfig()
err := mb.Module().UnpackConfig(&config)
if err != nil {
Expand All @@ -71,10 +85,9 @@ func NewHttpServer(mb mb.BaseMetricSet) (server.Server, error) {
ctx: ctx,
stop: cancel,
}

httpServer := &http.Server{
Addr: net.JoinHostPort(config.Host, strconv.Itoa(int(config.Port))),
Handler: http.HandlerFunc(h.handleFunc),
Handler: http.HandlerFunc(factory(h)),
}
if tlsConfig != nil {
httpServer.TLSConfig = tlsConfig.BuildModuleConfig(config.Host)
Expand Down Expand Up @@ -116,6 +129,10 @@ func (h *HttpServer) GetEvents() chan server.Event {
return h.eventQueue
}

func (h *HttpServer) WriteEvents(event server.Event) {
h.eventQueue <- event
}

func (h *HttpServer) handleFunc(writer http.ResponseWriter, req *http.Request) {
switch req.Method {
case "POST":
Expand Down
1 change: 1 addition & 0 deletions metricbeat/include/list_common.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 10 additions & 0 deletions metricbeat/metricbeat.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -711,6 +711,7 @@ metricbeat.modules:
#------------------------------ Prometheus Module ------------------------------
- module: prometheus
period: 10s
metricsets: ["collector"]
hosts: ["localhost:9090"]
metrics_path: /metrics
#metrics_filters:
Expand All @@ -724,6 +725,15 @@ metricbeat.modules:
#ssl.certificate_authorities:
# - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

- module: prometheus
metricsets: ["remote_write"]
host: "localhost"
port: "9201"

# Secure settings for the server using TLS/SSL:
#ssl.certificate: "/etc/pki/server/cert.pem"
#ssl.key: "/etc/pki/server/cert.key"

#------------------------------- RabbitMQ Module -------------------------------
- module: rabbitmq
metricsets: ["node", "queue", "connection"]
Expand Down
52 changes: 50 additions & 2 deletions metricbeat/module/http/server/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,14 @@ package server

import (
"fmt"
"net/http"
"github.com/elastic/beats/libbeat/common"
"github.com/elastic/beats/libbeat/logp"
"github.com/elastic/beats/metricbeat/helper/server"
"io/ioutil"

jsoriano marked this conversation as resolved.
Show resolved Hide resolved
serverhelper "github.com/elastic/beats/v7/metricbeat/helper/server"
"github.com/elastic/beats/v7/metricbeat/helper/server/http"
httpserver "github.com/elastic/beats/v7/metricbeat/helper/server/http"
"github.com/elastic/beats/v7/metricbeat/mb"
)

Expand Down Expand Up @@ -52,7 +57,7 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
return nil, err
}

svc, err := http.NewHttpServer(base)
svc, err := httpserver.NewHttpServer(base, getHandleFunc)
if err != nil {
return nil, err
}
Expand Down Expand Up @@ -94,3 +99,46 @@ func (m *MetricSet) Run(reporter mb.PushReporterV2) {
}
}
}

func getHandleFunc(h *httpserver.HttpServer) func(writer http.ResponseWriter, req *http.Request) {
return func(writer http.ResponseWriter, req *http.Request) {
switch req.Method {
case "POST":
meta := server.Meta{
"path": req.URL.String(),
}

contentType := req.Header.Get("Content-Type")
if contentType != "" {
meta["Content-Type"] = contentType
}

body, err := ioutil.ReadAll(req.Body)
if err != nil {
logp.Err("Error reading body: %v", err)
http.Error(writer, "Unexpected error reading request payload", http.StatusBadRequest)
return
}

payload := common.MapStr{
server.EventDataKey: body,
}

event := &httpserver.HttpEvent{}
event.SetEvent(payload)
event.SetMeta(meta)
h.WriteEvents(event)
writer.WriteHeader(http.StatusAccepted)

case "GET":
writer.WriteHeader(http.StatusOK)
if req.TLS != nil {
writer.Write([]byte("HTTPS Server accepts data via POST"))
} else {
writer.Write([]byte("HTTP Server accepts data via POST"))
}

}
}
}

10 changes: 10 additions & 0 deletions metricbeat/module/prometheus/_meta/config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
- module: prometheus
period: 10s
metricsets: ["collector"]
hosts: ["localhost:9090"]
metrics_path: /metrics
#metrics_filters:
Expand All @@ -12,3 +13,12 @@
#bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
#ssl.certificate_authorities:
# - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

- module: prometheus
jsoriano marked this conversation as resolved.
Show resolved Hide resolved
metricsets: ["remote_write"]
host: "localhost"
port: "9201"

# Secure settings for the server using TLS/SSL:
#ssl.certificate: "/etc/pki/server/cert.pem"
#ssl.key: "/etc/pki/server/cert.key"
2 changes: 1 addition & 1 deletion metricbeat/module/prometheus/fields.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading