Deprecate log input in favour of filestream #28623

kvch · 2021-10-25T14:25:30Z

What does this PR do?

This PR marks the log input as deprecated. As we are not in a hurry to remove the input because it possibly impacts lots of users, I omitted the version to remove from the log message. Our goal is to let users who are satisfied with the input use it as long as they would like to. However, if it is no longer working or found any issues in the input, we are asking them to move to filestream input.

Why is it important?

Everyone should move to filestream input because it is a great improvement over the old log input.

Checklist

My code follows the style guidelines of this project
~~- [ ] I have commented my code, particularly in hard-to-understand areas~~
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
~~- [ ] I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

mergify · 2021-10-25T14:26:00Z

This pull request does not have a backport label. Could you fix it @kvch? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

… reference

elasticmachine · 2021-10-25T14:29:22Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

elasticmachine · 2021-10-25T14:32:55Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-11-04T16:10:09.478+0000
Duration: 106 min 36 sec
Commit: 6c9d725

Test stats 🧪

Test	Results
Failed	0
Passed	15004
Skipped	2324
Total	17328

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

kvch · 2021-11-04T10:58:06Z

@ruflin I removed the version from the log message and added more details to the PR description.

ruflin

2 minor comments, overall LGTM. Approving already.

ruflin · 2021-11-04T15:08:18Z

filebeat/input/log/input.go

@@ -82,6 +85,10 @@ func NewInput(
 	outlet channel.Connector,
 	context input.Context,
 ) (input.Input, error) {
+	deprecatedNotificationOnce.Do(func() {
+		cfgwarn.Deprecate("8.0.0", "Log input. Use Filestream input instead.")


Shouldn't this be 7.16.0?

ruflin · 2021-11-04T15:09:13Z

filebeat/input/stdin/input.go

@@ -48,6 +54,10 @@ type Input struct {
 // NewInput creates a new stdin input
 // This input contains one harvester which is reading from stdin
 func NewInput(cfg *common.Config, outlet channel.Connector, context input.Context) (input.Input, error) {
+	deprecatedNotificationOnce.Do(func() {
+		cfgwarn.Deprecate("", "Stdin input is deprecated.")


Should we add this to the changelog too?

I accidentally added this file. I wanted to create a separate PR, so I only have to add a single line to the changelog.

(cherry picked from commit 97243e6)

(cherry picked from commit 97243e6) Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>

…in-the-package-binareis * upstream/master: allows disable pod events enrichment with deployment name (elastic#28521) Remove Docker input from Filebeat (elastic#28817) [breaking] Make default_field: false the default for all fields (elastic#28596) Osquerybeat: Improve osquery client connect code (elastic#28848) Add crawler metrics into the stats metricset for Enterprise Search (elastic#28790) Remove the now deprecated appsearch module from metricbeat (elastic#28850) Remove Beat generators (elastic#28816) chore: upload files to Google Storage when they exist (elastic#28836) Revert "chore(ci): disable E2E tests in Beats (elastic#28715)" (elastic#28812) Deprecate generating custom Beats (elastic#28814) [Metricbeat] upgrade flatbuffers to 1.12.1 (elastic#28094) Osquerybeat: Fix restart flags after previously bad config (elastic#28827) Force ECS and JSON logging for libbeat/logp (elastic#28573) Filebeat: Error on startup for unconfigured module (elastic#28818) Deprecate log input in favour of filestream (elastic#28623) Fix some spelling mistakes (elastic#28080)

jasonperrone · 2022-02-23T12:59:33Z

Is there anything we have to be aware of when changing a log type filebeat input to a filestream input? Is there a migration or conversion we have to do or is it simply just a matter of changing "log" to "filestream" in filebeat.yml?

derekha-igs · 2022-03-08T09:45:22Z

What I'm aware is will the registry can handle the log to filestream migration correctly?
Say, the file read first 100 lines by log input, and I stop the filebeat and migration to filestream, will the filestream able to read from line 101? Also the files processed before by log input will reprocess or not?

Please advice, thanks.

haohao6683 · 2022-05-18T05:50:07Z

Is there anything we have to be aware of when changing a log type filebeat input to a filestream input? Is there a migration or conversion we have to do or is it simply just a matter of changing "log" to "filestream" in filebeat.yml?

Please advice, thanks.

…in-the-package-binareis * upstream/master: allows disable pod events enrichment with deployment name (elastic#28521) Remove Docker input from Filebeat (elastic#28817) [breaking] Make default_field: false the default for all fields (elastic#28596) Osquerybeat: Improve osquery client connect code (elastic#28848) Add crawler metrics into the stats metricset for Enterprise Search (elastic#28790) Remove the now deprecated appsearch module from metricbeat (elastic#28850) Remove Beat generators (elastic#28816) chore: upload files to Google Storage when they exist (elastic#28836) Revert "chore(ci): disable E2E tests in Beats (elastic#28715)" (elastic#28812) Deprecate generating custom Beats (elastic#28814) [Metricbeat] upgrade flatbuffers to 1.12.1 (elastic#28094) Osquerybeat: Fix restart flags after previously bad config (elastic#28827) Force ECS and JSON logging for libbeat/logp (elastic#28573) Filebeat: Error on startup for unconfigured module (elastic#28818) Deprecate log input in favour of filestream (elastic#28623) Fix some spelling mistakes (elastic#28080)

Deprecate log input in favour of filestream

1e842aa

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 25, 2021

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 25, 2021

kvch added 2 commits October 25, 2021 16:28

Remove log input from default configuration and mark it deprecated in…

fd8add6

… reference

add changelog entry

d78a49f

kvch added the backport-v7.16.0 Automated backport with mergify label Oct 25, 2021

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Oct 25, 2021

kvch added backport-skip Skip notification from the automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Oct 25, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 25, 2021

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Oct 25, 2021

Do not specify when log input is removed

5dd405c

ruflin approved these changes Nov 4, 2021

View reviewed changes

rm accidentally added changes && really remove version number

6c9d725

kvch added the backport-v8.0.0 Automated backport with mergify label Nov 4, 2021

kvch merged commit 97243e6 into elastic:master Nov 4, 2021

mergify bot pushed a commit that referenced this pull request Nov 4, 2021

Deprecate log input in favour of filestream (#28623)

07471ef

(cherry picked from commit 97243e6)

mergify bot mentioned this pull request Nov 4, 2021

[7.16](backport #28623) Deprecate log input in favour of filestream #28823

Merged

mergify bot pushed a commit that referenced this pull request Nov 4, 2021

Deprecate log input in favour of filestream (#28623)

6dc31a3

(cherry picked from commit 97243e6)

mergify bot mentioned this pull request Nov 4, 2021

[8.0](backport #28623) Deprecate log input in favour of filestream #28824

Merged

kvch added a commit that referenced this pull request Nov 5, 2021

Deprecate log input in favour of filestream (#28623) (#28823)

4c2558e

(cherry picked from commit 97243e6) Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>

kvch added a commit that referenced this pull request Nov 5, 2021

Deprecate log input in favour of filestream (#28623) (#28824)

afa9411

(cherry picked from commit 97243e6) Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deprecate log input in favour of filestream #28623

Deprecate log input in favour of filestream #28623

kvch commented Oct 25, 2021 •

edited

Loading

mergify bot commented Oct 25, 2021

elasticmachine commented Oct 25, 2021

elasticmachine commented Oct 25, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

kvch commented Nov 4, 2021

ruflin left a comment

ruflin Nov 4, 2021

ruflin Nov 4, 2021

kvch Nov 4, 2021

jasonperrone commented Feb 23, 2022

derekha-igs commented Mar 8, 2022

haohao6683 commented May 18, 2022

Deprecate log input in favour of filestream #28623

Deprecate log input in favour of filestream #28623

Conversation

kvch commented Oct 25, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

mergify bot commented Oct 25, 2021

elasticmachine commented Oct 25, 2021

elasticmachine commented Oct 25, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

kvch commented Nov 4, 2021

ruflin left a comment

Choose a reason for hiding this comment

ruflin Nov 4, 2021

Choose a reason for hiding this comment

ruflin Nov 4, 2021

Choose a reason for hiding this comment

kvch Nov 4, 2021

Choose a reason for hiding this comment

jasonperrone commented Feb 23, 2022

derekha-igs commented Mar 8, 2022

haohao6683 commented May 18, 2022

kvch commented Oct 25, 2021 •

edited

Loading

elasticmachine commented Oct 25, 2021 •

edited by jenkins-beats-ci bot

Loading