Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filebeat: Fix o365 module issues #18948

Merged
merged 11 commits into from
Jun 9, 2020
Merged

Filebeat: Fix o365 module issues #18948

merged 11 commits into from
Jun 9, 2020

Conversation

adriansr
Copy link
Contributor

@adriansr adriansr commented Jun 3, 2020 •
edited
Loading

Assorted fixes to the o365 module:

  • Mark module as beta in docs.
  • get rid of data-loss error on startup: Bad error handling around saved-state loading (unimplemented) caused a data-loss warning on startup instead of a less scary info message:
- [ERROR] Error loading saved state. Will fetch all retained events. Depending on max_retention, this can cause event loss or duplication.
+ [INFO] No saved state found. Will fetch events for the last 168h.
  • Avoid passing API errors to the JS pipeline

Ingestion pipeline errors from o365audit input need not to go through the JS pipeline, it'll add more errors and noise.

  • Prevent dissect error about overriding client.port

  • Fix how API settings are passed to the o365 input

Passing low-level API settings between module and input was broken.

  • Document max_period using the right units.

The sample conf will use 7d which is not valid as hours is the largest supported unit.

@adriansr adriansr added bug in progress Pull request is currently in progress. Team:SIEM labels Jun 3, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 3, 2020 •
edited
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #18948 updated]

  • Start Time: 2020-06-08T16:17:45.927+0000

  • Duration: 49 min 32 sec

Test stats 🧪

Test Results
Failed 0
Passed 2226
Skipped 382
Total 2608

@AndyHunt66 AndyHunt66 mentioned this pull request Jun 8, 2020
Merged
@adriansr adriansr changed the title Fix o365 module issues [draft] Filebeat: Fix o365 module issues Jun 8, 2020
@adriansr adriansr added review and removed in progress Pull request is currently in progress. labels Jun 8, 2020
@adriansr adriansr marked this pull request as ready for review June 8, 2020 16:14
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@adriansr adriansr merged commit 83bbd57 into elastic:master Jun 9, 2020
@adriansr adriansr deleted the fix_o365_issues branch June 9, 2020 08:22
adriansr added a commit to adriansr/beats that referenced this pull request Jun 9, 2020
@adriansr
Filebeat: Fix o365 module issues (elastic#18948)
4c4f2f0 
- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit 83bbd57)
@adriansr adriansr mentioned this pull request Jun 9, 2020
Merged
@adriansr adriansr added the v7.9.0 label Jun 9, 2020
adriansr added a commit to adriansr/beats that referenced this pull request Jun 9, 2020
@adriansr
Filebeat: Fix o365 module issues (elastic#18948)
1a9063a 
- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit 83bbd57)
@adriansr adriansr mentioned this pull request Jun 9, 2020
Merged
@adriansr adriansr added the v7.8.0 label Jun 9, 2020
adriansr added a commit to adriansr/beats that referenced this pull request Jun 9, 2020
@adriansr
Filebeat: Fix o365 module issues (elastic#18948)
f0cbd8f 
- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit 83bbd57)
@adriansr adriansr mentioned this pull request Jun 9, 2020
Merged
@adriansr adriansr added the v7.7.2 label Jun 9, 2020
adriansr pushed a commit that referenced this pull request Jun 9, 2020
@AndyHunt66
o365: Additional context around the max_retention (#19037)
05935ee 
I think it would be good to get some extra information around the max_retention setting, to make it clear that unless the tenant itself has a longer retention period, then 7 days is going to be all that you can fetch.

Follow-on from investigation behind #18948
adriansr pushed a commit to adriansr/beats that referenced this pull request Jun 9, 2020
@AndyHunt66 @adriansr
o365: Additional context around the max_retention (elastic#19037)
f22603f 
I think it would be good to get some extra information around the max_retention setting, to make it clear that unless the tenant itself has a longer retention period, then 7 days is going to be all that you can fetch.

Follow-on from investigation behind elastic#18948

(cherry picked from commit 05935ee)
@adriansr adriansr mentioned this pull request Jun 9, 2020
Merged
adriansr pushed a commit to adriansr/beats that referenced this pull request Jun 9, 2020
@AndyHunt66 @adriansr
o365: Additional context around the max_retention (elastic#19037)
a9f0a76 
I think it would be good to get some extra information around the max_retention setting, to make it clear that unless the tenant itself has a longer retention period, then 7 days is going to be all that you can fetch.

Follow-on from investigation behind elastic#18948

(cherry picked from commit 05935ee)
@adriansr adriansr mentioned this pull request Jun 9, 2020
Merged
adriansr added a commit that referenced this pull request Jun 9, 2020
@adriansr
Cherry-pick #18948 to 7.x: Filebeat: Fix o365 module issues (#19061)
b20374f 
* Filebeat: Fix o365 module issues (#18948)

- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit 83bbd57)
adriansr added a commit that referenced this pull request Jun 9, 2020
@adriansr
Cherry-pick #18948 to 7.7: Filebeat: Fix o365 module issues (#19063)
f9cbac5 
* Filebeat: Fix o365 module issues (#18948)

- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit 83bbd57)
adriansr added a commit that referenced this pull request Jun 9, 2020
@adriansr @AndyHunt66
o365: Additional context around the max_retention (#19037) (#19072)
da9cb87 
I think it would be good to get some extra information around the max_retention setting, to make it clear that unless the tenant itself has a longer retention period, then 7 days is going to be all that you can fetch.

Follow-on from investigation behind #18948

(cherry picked from commit 05935ee)

Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>
adriansr added a commit that referenced this pull request Jun 9, 2020
@adriansr
Cherry-pick #18948 to 7.8: Filebeat: Fix o365 module issues (#19062)
bf707dd 
- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit 83bbd57)
adriansr added a commit that referenced this pull request Jun 9, 2020
@adriansr @AndyHunt66
o365: Additional context around the max_retention (#19037) (#19073)
6ed3cef 
I think it would be good to get some extra information around the max_retention setting, to make it clear that unless the tenant itself has a longer retention period, then 7 days is going to be all that you can fetch.

Follow-on from investigation behind #18948

(cherry picked from commit 05935ee)

Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
@adriansr @melchiormoulin
Filebeat: Fix o365 module issues (elastic#18948)
2a9c95e 
- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
@AndyHunt66 @melchiormoulin
o365: Additional context around the max_retention (elastic#19037)
51929d8 
I think it would be good to get some extra information around the max_retention setting, to make it clear that unless the tenant itself has a longer retention period, then 7 days is going to be all that you can fetch.

Follow-on from investigation behind elastic#18948
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
@adriansr
Cherry-pick elastic#18948 to 7.7: Filebeat: Fix o365 module issues (e…
e8629e8 
…lastic#19063)

* Filebeat: Fix o365 module issues (elastic#18948)

- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit b99a73c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees
No one assigned
Labels
bug review v7.7.2 v7.8.0 v7.9.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@adriansr @elasticmachine @andrewkroh