[Packetbeat] Fix data stream name for network flows under Agent #28408

andrewkroh · 2021-10-13T15:23:55Z

What does this PR do?

This fixes and issue with network flows being written to the wrong
index (it should not include any date) when using the Network Packet Capture integration in Fleet.

The error was:

{"type:"security_exception", "reason":"action [indices:admin/auto_create] is unauthorized for API key id [xxx] of user [elastic/fleet-server] on indices [logs-network_traffic.flow-default-2021.10.13] …"}

The cause is that flows were setting index rather than raw_index. With
index Beats adds the date suffix, but since this is a data stream we
want raw_index where the value passes through as-is.

Why is it important?

Fleet could not ingest network flows from Packetbeat.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Links

Also reported in https://discuss.elastic.co/t/is-network-traffic-fleet-integration-the-new-packetbeat-missing-netfow/285636.

This fixes and issue with network flows being written to the wrong index when using the Network Packet Capture integration in Fleet. The error was: {"type:"security_exception", "reason":"action [indices:admin/auto_create] is unauthorized for API key id [xxx] of user [elastic/fleet-server] on indices [logs-network_traffic.flow-default-2021.10.13] …"} The cause is that flows were setting `index` rather than `raw_index`. With `index` Beats adds the date suffix, but since this is a data stream we want `raw_index` where the value passes through as-is.

elasticmachine · 2021-10-13T15:23:57Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh · 2021-10-13T15:27:10Z

There weren't any existing integrations test that I could find for this.

elasticmachine · 2021-10-13T15:31:10Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-10-13T15:25:41.424+0000
Duration: 95 min 31 sec
Commit: bbd6bb2

Test stats 🧪

Test	Results
Failed	0
Passed	4933
Skipped	45
Total	4978

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

kesslerm · 2021-10-13T17:19:25Z

~~This appears to have been fixed in the master / 8.x branch already: afaad35. Needs a backport to the 7.x and 7.15 branches.~~

Edit: no, similar fix, different issue.

* Fix data stream name for network flows under Agent This fixes and issue with network flows being written to the wrong index when using the Network Packet Capture integration in Fleet. The error was: {"type:"security_exception", "reason":"action [indices:admin/auto_create] is unauthorized for API key id [xxx] of user [elastic/fleet-server] on indices [logs-network_traffic.flow-default-2021.10.13] …"} The cause is that flows were setting `index` rather than `raw_index`. With `index` Beats adds the date suffix, but since this is a data stream we want `raw_index` where the value passes through as-is. (cherry picked from commit 5c92897)

…) (#28469) * Fix data stream name for network flows under Agent This fixes and issue with network flows being written to the wrong index when using the Network Packet Capture integration in Fleet. The error was: {"type:"security_exception", "reason":"action [indices:admin/auto_create] is unauthorized for API key id [xxx] of user [elastic/fleet-server] on indices [logs-network_traffic.flow-default-2021.10.13] …"} The cause is that flows were setting `index` rather than `raw_index`. With `index` Beats adds the date suffix, but since this is a data stream we want `raw_index` where the value passes through as-is. (cherry picked from commit 5c92897) Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

…) (#28468) * Fix data stream name for network flows under Agent This fixes and issue with network flows being written to the wrong index when using the Network Packet Capture integration in Fleet. The error was: {"type:"security_exception", "reason":"action [indices:admin/auto_create] is unauthorized for API key id [xxx] of user [elastic/fleet-server] on indices [logs-network_traffic.flow-default-2021.10.13] …"} The cause is that flows were setting `index` rather than `raw_index`. With `index` Beats adds the date suffix, but since this is a data stream we want `raw_index` where the value passes through as-is. (cherry picked from commit 5c92897) Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

…tic#28408) * Fix data stream name for network flows under Agent This fixes and issue with network flows being written to the wrong index when using the Network Packet Capture integration in Fleet. The error was: {"type:"security_exception", "reason":"action [indices:admin/auto_create] is unauthorized for API key id [xxx] of user [elastic/fleet-server] on indices [logs-network_traffic.flow-default-2021.10.13] …"} The cause is that flows were setting `index` rather than `raw_index`. With `index` Beats adds the date suffix, but since this is a data stream we want `raw_index` where the value passes through as-is.

andrewkroh added bug Packetbeat Team:Security-External Integrations labels Oct 13, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 13, 2021

Add changelog

bbd6bb2

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 13, 2021

andrewkroh added backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Oct 13, 2021

elastic deleted a comment from mergify bot Oct 13, 2021

andrewkroh requested a review from a team October 13, 2021 19:25

efd6 approved these changes Oct 13, 2021

View reviewed changes

andrewkroh merged commit 5c92897 into elastic:master Oct 15, 2021

mergify bot mentioned this pull request Oct 15, 2021

[7.x](backport #28408) [Packetbeat] Fix data stream name for network flows under Agent #28468

Merged

mergify bot mentioned this pull request Oct 15, 2021

[7.15](backport #28408) [Packetbeat] Fix data stream name for network flows under Agent #28469

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Packetbeat] Fix data stream name for network flows under Agent #28408

[Packetbeat] Fix data stream name for network flows under Agent #28408

andrewkroh commented Oct 13, 2021 •

edited

Loading

elasticmachine commented Oct 13, 2021

andrewkroh commented Oct 13, 2021

elasticmachine commented Oct 13, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

kesslerm commented Oct 13, 2021 •

edited

Loading

[Packetbeat] Fix data stream name for network flows under Agent #28408

[Packetbeat] Fix data stream name for network flows under Agent #28408

Conversation

andrewkroh commented Oct 13, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Links

elasticmachine commented Oct 13, 2021

andrewkroh commented Oct 13, 2021

elasticmachine commented Oct 13, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

kesslerm commented Oct 13, 2021 • edited Loading

andrewkroh commented Oct 13, 2021 •

edited

Loading

elasticmachine commented Oct 13, 2021 •

edited by jenkins-beats-ci bot

Loading

kesslerm commented Oct 13, 2021 •

edited

Loading