feat: add support for SCRAM authentication for kafka metricbeat module, fixes #19648 #24810
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
botelastic
bot
added
the
needs_team
|
💚 CLA has been signed |
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪Test errors
Expand to view the tests failures
|
| Test | Results |
|---|---|
| Failed | 1 |
| Passed | 45948 |
| Skipped | 5012 |
| Total | 50961 |
Genuine test errors 
💔 There are test failures but not known flaky tests, most likely a genuine test failure.
- Name:
Build&Test / filebeat-windows-windows-2019 / test_restart_recursive_glob – filebeat.tests.system.test_input.Test
|
Pinging @elastic/integrations (Team:Integrations) |
botelastic
bot
removed
the
needs_team
|
/test |
jsoriano
requested changes
Mar 30, 2021
jsoriano
reviewed
Mar 30, 2021
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
At this point input validation should have been done already.
|
CLA is signed |
jsoriano
reviewed
Mar 31, 2021
|
/test |
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
|
/test |
jsoriano
added
the
needs_backport
|
@fholzer linting failures in metricbeat are related to this change, you will need to run |
|
Failing test is not related, merging. |
6 tasks
jsoriano
pushed a commit
to jsoriano/beats
that referenced
this pull request
Apr 1, 2021
…tic#24810) (cherry picked from commit 66829ff)
jsoriano
added
v7.13.0
and removed
needs_backport
jsoriano
added a commit
that referenced
this pull request
Apr 1, 2021
v1v
added a commit
to v1v/beats
that referenced
this pull request
Apr 7, 2021
* upstream/master: (91 commits) [Filebeat] Change okta.target to nested field (elastic#24636) Add RFC5424 format support for syslog input (elastic#23954) Fix links to Beats product pages (elastic#24821) [DOCS] Fix 'make setup' instructions for a new beat (elastic#24944) Remove duplicate decode_xml entry (elastic#24941) [libbeat] Add wineventlog schema to decode_xml processor (elastic#24726) [Elastic Agent] Add check for URL set when cert and cert key. (elastic#24904) feat: stage execution cache (elastic#24780) Fix error in Journalbeat commands (elastic#24880) Add baseline ECS 1.9.0 upgrade (elastic#24909) [Elastic Agent] Cloud container legacy apm files. (elastic#24896) [Elastic Agent]: Reduce allowed socket path length (elastic#24914) Add ability to destroy indices with wildcards in testing (elastic#24915) Add status subcommand to report status of running daemon. (elastic#24856) Fix types of fields GetHits and Ops in Metricbeat module for Couchbase (elastic#23287) Add support for Filestream input in elastic agent. (elastic#24820) Implement k8s secrets provider for Agent (elastic#24789) Sort processor list in docs (elastic#24874) Add support for SCRAM authentication in kafka metricbeat module (elastic#24810) Properly update offset in case of unparasable line (elastic#22685) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds SCRAM authentication for kafka to metricbeat's kafka module. Moves shared code from kafka output plugin to
libbeat/common/kafka/Why is it important?
Currently metricbeat can't pull metrics from Kafka clusters that require authentication, which is a problem in enterprise environments where authentication is a requirement.
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.I think most of this isn't applicable, as this PR virtually just moves files around and makes couple things public in the shared code. This virtually no new code introduced.
Reg. "I have made corresponding changes to the documentation", it seems that none of the kafka module's fields in metricbeat are separately documented. Instead the config.yml is pulled into the asciidoc file, which I updated.
How to test this PR locally
As this PR mostly just moves existing files around and introduces only minimal code changes, the test procedures remain the same as those for #12867 which initially introduced SCRAM code. Though it seems those were not documented at the time. Generally, though, the procedure documented in beats/blob/master/metricbeat/module/kafka/README.md remains the same, with the exception that the test kafka cluster needs to use one of the SCAM mechanisms, eg. in
server.propertieshavelistener.name.external.sasl.enabled.mechanisms=SCRAM-SHA-256configured. (if you external listener is calledEXTERNAL)Related issues
Use cases
This PR introduces
sasl.mechanismin metricbeat's kafka input module. It works the same way as the similarly named configuration directive in libbeat's kafka output plugin.