Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #23334 to 7.x: [Filebeat] Add postgresql csv fileset #24088

Merged
merged 1 commit into from
Feb 17, 2021

Conversation

jsoriano
Copy link
Member

@jsoriano jsoriano commented Feb 17, 2021

Cherry-pick of PR #23334 to 7.x branch. Original message:

As a configuration option, create another pipeline inside the module to
parse CSV log files.

Although it's necessary to configure the database to emit such logs,
there is some advantages: configured properly, some events like
statement timeout and lock timeout will display the query in the
same event, opposed to multiple lines, one with the error message and
other with statement.

What does this PR do?

This PR creates a new fileset/pipeline inside PostgreSQL module. This new fileset parses and create events based on CSV log files from postgresql.

Why is it important?

PostgreSQL logs weren't designed to be parsed, so, without very complicated logic, it's not possible to encapsulate all information in just one event. One example of that is statement timeout:

2021-01-04 02:07:15.470 UTC [13470] [user=postgres,db=postgres,app=[unknown]] ERROR:  canceling statement due to statement timeout
2021-01-04 02:07:15.470 UTC [13470] [user=postgres,db=postgres,app=[unknown]] STATEMENT:  SELECT pg_sleep(100)

Using the CSV log, those 2 lines become 1:

2021-01-04 02:07:15.470 UTC,"postgres","postgres",13470,"127.21.1.7:28182",5ff27834.349e,1,"SELECT",2021-01-04 02:06:44 UTC,16/4181895,0,ERROR,57014,"canceling statement due to statement timeout",,,,,,"SELECT pg_sleep(100)",,,""

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Documentation

Refactor PostgreSQL module to support logs in CSV format.

The fileset has three pipelines now, first one is executed
to parse the initial part of the lines, till it decides if the logs
are plain text or CSV, once decided it invokes one of the
other two pipelines, one specific for plain text logs and
the other for CSV.

Several test cases and documentation are added for CSV
support.

Additional fields are available now, and some others have
been renamed to represent more accurately the values
they store.

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
(cherry picked from commit bb78931)
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 17, 2021
@jsoriano jsoriano self-assigned this Feb 17, 2021
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #24088 opened

  • Start Time: 2021-02-17T15:05:00.759+0000

  • Duration: 51 min 45 sec

  • Commit: 04f7c5b

Test stats 🧪

Test Results
Failed 0
Passed 13022
Skipped 2229
Total 15251

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 13022
Skipped 2229
Total 15251

@jsoriano jsoriano merged commit f269952 into elastic:7.x Feb 17, 2021
@jsoriano jsoriano deleted the backport_23334_7.x branch February 17, 2021 16:37
@zube zube bot removed the [zube]: Done label May 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants