Move s3_daily_storage and s3_request metricsets to use cloudwatch input #21703

kaiyan-sheng · 2020-10-09T14:41:45Z

What does this PR do?

This PR is to move s3_daily_storage and s3_request metricsets to use cloudwatch input. In order to make it not a breaking change, rename processor is used in both manifest.yml files.

Why is it important?

This way we only need to maintain code in cloudwatch input instead of maintaining code in each metricset.

Checklist

My code follows the style guidelines of this project
~~I have commented my code, particularly in hard-to-understand areas~~
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

elasticmachine · 2020-10-09T14:42:16Z

Pinging @elastic/integrations-platforms (Team:Platforms)

elasticmachine · 2020-10-09T14:49:02Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: [Pull request #21703 updated]
Start Time: 2020-10-13T20:30:51.104+0000
Duration: 65 min 4 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2629
Skipped	597
Total	3226

jsoriano

Are data.json files updated?

jsoriano · 2020-10-13T17:42:47Z

x-pack/metricbeat/module/aws/s3_daily_storage/_meta/data.json

+        "s3": {
+            "metrics": {
+                "NumberOfObjects": {
+                    "avg": 57828


Is rename not being applied here?

Yeah it's applied when I run metricbeat for testing but not applied when I tried to create the event for data.json. Is there any way I can make the rename processor work with TestData function?

Oh ok, maybe processors are not initialized on testing, if that is the case we should fix that, but not on this PR. Could you please open an issue about that?

But interestingly in the iis module there is a rename that seems to be applied 🤔

beats/x-pack/metricbeat/module/iis/website/manifest.yml

Lines 36 to 40 in 3e5a167

- rename:

ignore_missing: true

fields:

- from: "iis.website.instance"

to: "iis.website.name"

beats/x-pack/metricbeat/module/iis/website/_meta/data.json

Line 27 in 3e5a167

"name": "Default Web Site"

Interesting. @narph I saw you generated the data.json file for iis website. Does the field iis.website.name generated automatically by TestData function? For my case, TestData function ignores the rename processor.

I am setting the iis.website.name using the rename processor inside the manifest.yml file

x-pack/metricbeat/module/aws/s3_request/_meta/data.json

x-pack/metricbeat/module/aws/s3_request/s3_request_integration_test.go

jsoriano

Looks good, but we should check why processors are not being executed on TestData.

jsoriano · 2020-10-14T11:04:44Z

x-pack/metricbeat/module/aws/s3_daily_storage/_meta/data.json

+        "s3": {
+            "metrics": {
+                "NumberOfObjects": {
+                    "avg": 57828


Oh ok, maybe processors are not initialized on testing, if that is the case we should fix that, but not on this PR. Could you please open an issue about that?

But interestingly in the iis module there is a rename that seems to be applied 🤔

beats/x-pack/metricbeat/module/iis/website/manifest.yml

Lines 36 to 40 in 3e5a167

- rename:

ignore_missing: true

fields:

- from: "iis.website.instance"

to: "iis.website.name"

beats/x-pack/metricbeat/module/iis/website/_meta/data.json

Line 27 in 3e5a167

"name": "Default Web Site"

* upstream/master: (29 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...

* upstream/master: (216 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...

* upstream/master: (156 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...

…csets to use cloudwatch input (#21821) (cherry picked from commit 99b4172) * backport #21828

Move s3_daily_storage and s3_request metricsets to use cloudwatch input

1c4db89

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 9, 2020

kaiyan-sheng self-assigned this Oct 9, 2020

kaiyan-sheng added needs_backport PR is waiting to be backported to other branches. review Team:Platforms Label for the Integrations - Platforms team labels Oct 9, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 9, 2020

kaiyan-sheng mentioned this pull request Oct 9, 2020

Migrate metricsets in aws module to use cloudwatch input as lightweight module #21422

Closed

6 tasks

add changelog

7d78d64

jsoriano reviewed Oct 13, 2020

View reviewed changes

Move TestFetch back

7e13f83

jsoriano approved these changes Oct 14, 2020

View reviewed changes

kaiyan-sheng mentioned this pull request Oct 14, 2020

[Metricbeat] Initialize processors in manifest.yml with TestData function #21817

Closed

kaiyan-sheng merged commit 99b4172 into elastic:master Oct 14, 2020

kaiyan-sheng deleted the s3_metriccsets branch October 14, 2020 19:30

kaiyan-sheng mentioned this pull request Oct 14, 2020

Cherry-pick #21703 to 7.x: Move s3_daily_storage and s3_request metricsets to use cloudwatch input #21821

Merged

6 tasks

kaiyan-sheng added v7.11.0 and removed needs_backport PR is waiting to be backported to other branches. labels Oct 14, 2020

kaiyan-sheng added a commit that referenced this pull request Oct 15, 2020

Cherry-pick #21703 to 7.x: Move s3_daily_storage and s3_request metri…

f282f7f

…csets to use cloudwatch input (#21821) (cherry picked from commit 99b4172) * backport #21828

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move s3_daily_storage and s3_request metricsets to use cloudwatch input #21703

Move s3_daily_storage and s3_request metricsets to use cloudwatch input #21703

kaiyan-sheng commented Oct 9, 2020 •

edited

Loading

elasticmachine commented Oct 9, 2020

elasticmachine commented Oct 9, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

jsoriano left a comment

jsoriano Oct 13, 2020

kaiyan-sheng Oct 13, 2020

jsoriano Oct 14, 2020

kaiyan-sheng Oct 14, 2020

narph Oct 15, 2020

jsoriano left a comment

jsoriano Oct 14, 2020

	- rename:
	ignore_missing: true
	fields:
	- from: "iis.website.instance"
	to: "iis.website.name"

Move s3_daily_storage and s3_request metricsets to use cloudwatch input #21703

Move s3_daily_storage and s3_request metricsets to use cloudwatch input #21703

Conversation

kaiyan-sheng commented Oct 9, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Oct 9, 2020

elasticmachine commented Oct 9, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano Oct 13, 2020

Choose a reason for hiding this comment

kaiyan-sheng Oct 13, 2020

Choose a reason for hiding this comment

jsoriano Oct 14, 2020

Choose a reason for hiding this comment

kaiyan-sheng Oct 14, 2020

Choose a reason for hiding this comment

narph Oct 15, 2020

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano Oct 14, 2020

Choose a reason for hiding this comment

kaiyan-sheng commented Oct 9, 2020 •

edited

Loading

elasticmachine commented Oct 9, 2020 •

edited by jenkins-beats-ci bot

Loading