[Filebeat] sync checkpoint module with Fleet integration #31076

andrewkroh · 2022-03-31T01:48:39Z

What does this PR do?

This adds network.transport to events derived from the IANA number. It brings in the change related to elastic/integrations#2463.

The Filebeat module was missing definitions for these four fields.

checkpoint.comment
checkpoint.conn_direction
checkpoint.db_ver
checkpoint.update_status

On difference between Fleet and Filebeat is that the Filebeat pipeline is setting client and server based on the source and destination. That behavior was kept. Another difference is event.ingested is required in the Filebeat version of the pipeline. Otherwise the pipelines are the same.

This commit was used:

elastic/integrations@2aee5ee

Filebeat had some ~10000 test log lines, but only the first 100 are checked in test_modules.py so I removed 9900 of them.

Why is it important?

Consistency is nice.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Note to reviewers

I think it would be easier to review the diff commit by commit. And turn off whitespace for the pipeline YAML diff.

Related issues

mergify · 2022-03-31T01:48:44Z

This pull request does not have a backport label. Could you fix it @andrewkroh? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2022-03-31T01:55:34Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2022-03-31T02:03:19Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-04-04T21:05:32.193+0000
Duration: 68 min 10 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6191
Skipped	728
Total	6919

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

The Filebeat module was missing definitions for these four fields. - checkpoint.comment - checkpoint.conn_direction - checkpoint.db_ver - checkpoint.update_status The Filebeat pipeline is setting client and server based on the source and destination. That behavior was kept. Otherwise the pipelines are the same. This commit was used: elastic/integrations@2aee5ee

According to https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk144192, Check Point module field checkpoint.source_object should be mapped as a string type instead of integer. Syncs change from: elastic/beats@a5e6e5b Relates: elastic/beats#25124 Other changes: - use ECS `log.file.path` - add `event.original` mapping - sort fields.yml by field name This was observed while preparing elastic/beats#31076.

The same field is defined in the CEF module which also supports Check Point data.

andrewkroh · 2022-03-31T18:05:23Z

At this point the failures appear to be caused by to the fact that Beats main branch now requires 8.3.0 ES versions for testing purposes (due to output.elasticsearch.allow_older_versions: false). So until there is an 8.3.0-SNAPSHOT build AND beats is updated to use it, then this will not pass.

…c-check-point

v1v · 2022-04-04T17:56:32Z

/test

v1v · 2022-04-04T18:13:39Z

/test

v1v · 2022-04-04T18:22:23Z

/test

v1v · 2022-04-04T18:58:42Z

/test

v1v · 2022-04-04T19:01:09Z

/test

andrewkroh · 2022-04-04T21:05:00Z

@Mergifyio update

mergify · 2022-04-04T21:05:10Z

update

✅ Branch has been successfully updated

efd6 · 2022-04-05T00:53:15Z

x-pack/filebeat/module/checkpoint/fields.go

@@ -19,5 +19,5 @@ func init() {
 // AssetCheckpoint returns asset data.
 // This is the base64 encoded zlib format compressed contents of module/checkpoint.
 func AssetCheckpoint() string {


It would be a great thing to have this embedded using embed and not requiring base64 </random-thoughts>.

efd6 · 2022-04-05T00:55:36Z

x-pack/filebeat/module/checkpoint/firewall/config/firewall.yml

-  - add_fields:
-      target: ''
-      fields:
-        ecs.version: 1.12.0


What was this?

It added ecs.version to the outgoing event. This is now being set as part of the ingest pipeline with a set processor.

* Sync Check Point module with Fleet integration The Filebeat module was missing definitions for these four fields. - checkpoint.comment - checkpoint.conn_direction - checkpoint.db_ver - checkpoint.update_status The Filebeat pipeline is setting client and server based on the source and destination. That behavior was kept. Otherwise the pipelines are the same. This commit was used: elastic/integrations@2aee5ee * Add overwrite: true for checkpoint.update_status The same field is defined in the CEF module which also supports Check Point data.

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 31, 2022

mergify bot assigned andrewkroh Mar 31, 2022

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Mar 31, 2022

andrewkroh force-pushed the feature/filebeat/sync-check-point branch from b2d6183 to 11a6a6a Compare March 31, 2022 01:49

andrewkroh added Filebeat Filebeat Team:Security-External Integrations backport-v8.3.0 Automated backport with mergify labels Mar 31, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 31, 2022

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Mar 31, 2022

andrewkroh marked this pull request as ready for review March 31, 2022 01:55

andrewkroh requested review from a team as code owners March 31, 2022 01:55

andrewkroh mentioned this pull request Mar 31, 2022

[checkpoint] Fix checkpoint.source_object mapping elastic/integrations#2951

Merged

4 tasks

andrewkroh marked this pull request as draft March 31, 2022 04:42

andrewkroh force-pushed the feature/filebeat/sync-check-point branch from cfe7fdd to 81bbaa2 Compare March 31, 2022 04:52

andrewkroh marked this pull request as ready for review March 31, 2022 04:52

Remove duplicated checkpoint.trusted_domain

84e9893

Add overwrite: true for checkpoint.update_status

0e4c645

The same field is defined in the CEF module which also supports Check Point data.

Merge remote-tracking branch 'elastic/main' into feature/filebeat/syn…

e383f18

…c-check-point

cmacknz removed the request for review from a team April 4, 2022 18:26

Merge branch 'main' into feature/filebeat/sync-check-point

c95a117

efd6 approved these changes Apr 5, 2022

View reviewed changes

andrewkroh merged commit e999979 into elastic:main Apr 5, 2022

eyapene mentioned this pull request May 25, 2023

[Filebeat][Checkpoint] - Removed PR #31076 from 7.17.8 release notes #35582

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] sync checkpoint module with Fleet integration #31076

[Filebeat] sync checkpoint module with Fleet integration #31076

andrewkroh commented Mar 31, 2022 •

edited

Loading

mergify bot commented Mar 31, 2022

elasticmachine commented Mar 31, 2022

elasticmachine commented Mar 31, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

andrewkroh commented Mar 31, 2022 •

edited

Loading

v1v commented Apr 4, 2022

v1v commented Apr 4, 2022

v1v commented Apr 4, 2022

v1v commented Apr 4, 2022

v1v commented Apr 4, 2022

andrewkroh commented Apr 4, 2022

mergify bot commented Apr 4, 2022

efd6 Apr 5, 2022

andrewkroh Apr 5, 2022

efd6 Apr 5, 2022

andrewkroh Apr 5, 2022

[Filebeat] sync checkpoint module with Fleet integration #31076

[Filebeat] sync checkpoint module with Fleet integration #31076

Conversation

andrewkroh commented Mar 31, 2022 • edited Loading

What does this PR do?

Why is it important?

Checklist

Note to reviewers

Related issues

mergify bot commented Mar 31, 2022

elasticmachine commented Mar 31, 2022

elasticmachine commented Mar 31, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

andrewkroh commented Mar 31, 2022 • edited Loading

v1v commented Apr 4, 2022

v1v commented Apr 4, 2022

v1v commented Apr 4, 2022

v1v commented Apr 4, 2022

v1v commented Apr 4, 2022

andrewkroh commented Apr 4, 2022

mergify bot commented Apr 4, 2022

✅ Branch has been successfully updated

efd6 Apr 5, 2022

Choose a reason for hiding this comment

andrewkroh Apr 5, 2022

Choose a reason for hiding this comment

efd6 Apr 5, 2022

Choose a reason for hiding this comment

andrewkroh Apr 5, 2022

Choose a reason for hiding this comment

andrewkroh commented Mar 31, 2022 •

edited

Loading

elasticmachine commented Mar 31, 2022 •

edited by jenkins-beats-ci bot

Loading

andrewkroh commented Mar 31, 2022 •

edited

Loading