Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Automatic Import] Do not remove message field for unstructured logs #193678

Merged
merged 2 commits into from
Sep 23, 2024
Merged

[Automatic Import] Do not remove message field for unstructured logs #193678

merged 2 commits into from
Sep 23, 2024

Conversation

bhapas
Copy link
Contributor

@bhapas bhapas commented Sep 23, 2024 •
edited by kibanamachine
Loading

Summary

The pipeline template by default removes the message field. But in case of unstructured logs the grok processor adds the unstructured message part into a message field and it should be part of the final pipeline.

Hence, the remove processor is not executed in case the log_format is unstructured.

Before this PR

{
  "docs": [
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "hostname": "mymachine",
          "process": "su",
          "ecs": {
            "version": "8.11.0"
          },
          "related": {
            "hosts": [
              "mymachine"
            ]
          },
          "priority": "34",
          "event": {
            "category": [
              "authentication",
              "process"
            ],
            "type": [
              "start"
            ]
          },
          "timestamp": "Oct 11 00:14:05"
        },
        "_ingest": {
          "timestamp": "2024-09-20T16:10:01.656597092Z"
        }
      }
    }
  ]
}

After this PR

image

@bhapas bhapas added bug Fixes for quality problems that affect the customer experience backport:prev-major Backport to (8.x, 8.15) the previous major branch and all later branches still in development Team:Security-Scalability Team label for Security Integrations Scalability Team Feature:AutomaticImport labels Sep 23, 2024
@bhapas bhapas self-assigned this Sep 23, 2024
@bhapas bhapas requested a review from a team as a code owner September 23, 2024 08:33
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-scalability (Team:Security-Scalability)

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @bhapas

@bhapas bhapas merged commit 2106df3 into elastic:main Sep 23, 2024
20 checks passed
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 23, 2024
@bhapas
[Automatic Import] Do not remove message field for unstructured logs (e…
adf5fba 
…lastic#193678)

(cherry picked from commit 2106df3)
@kibanamachine kibanamachine mentioned this pull request Sep 23, 2024
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 23, 2024
@bhapas
[Automatic Import] Do not remove message field for unstructured logs (e…
e65b574 
…lastic#193678)

(cherry picked from commit 2106df3)
@kibanamachine kibanamachine mentioned this pull request Sep 23, 2024
Merged
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
✅ 8.15
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Sep 23, 2024
@kibanamachine @bhapas
[8.x] [Automatic Import] Do not remove message field for unstructured…
b3311a9 
… logs (#193678) (#193704)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Automatic Import] Do not remove message field for unstructured logs
(#193678)](#193678)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Bharat
Pasupula","email":"123897612+bhapas@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-09-23T11:08:59Z","message":"[Automatic
Import] Do not remove message field for unstructured logs
(#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Automatic
Import] Do not remove message field for unstructured
logs","number":193678,"url":"https://github.com/elastic/kibana/pull/193678","mergeCommit":{"message":"[Automatic
Import] Do not remove message field for unstructured logs
(#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193678","number":193678,"mergeCommit":{"message":"[Automatic
Import] Do not remove message field for unstructured logs
(#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9"}}]}]
BACKPORT-->

Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
weizijun added a commit to weizijun/kibana that referenced this pull request Sep 23, 2024
@weizijun
Merge branch 'main' into add-alibaba-inference
2100828 
* main: (176 commits)
  [ML][Rules] Fixes deletion in Check interval input for anomaly detection rule (elastic#193420)
  Bump maximum supported package spec version to 3.2 (elastic#193574)
  [ES|QL] new pattern for `SORT` autocomplete (elastic#193595)
  [Inventory][ECO] Entities page search bar (elastic#193546)
  [Synthetics] Remove extra overview route (elastic#192449)
  [Obs Alerts table] Fix error on clicking alert reason message (elastic#193693)
  [Migrations] Remove tests that are not applicable in 9.x (elastic#193699)
  [EDR Workflows] Set Agent Tamper Protection to false on policy unassignment (elastic#193017)
  [Inventory][ECO] Enable elastic entity model from inventory (elastic#193557)
  [EDR Workflows] The host isolation exception tab is hidden on the basic license if no artifacts (elastic#192562)
  [Entity Analytics] Ensuring definition transforms are managed (elastic#193408)
  [Automatic Import] Do not remove message field for unstructured logs (elastic#193678)
  [Fleet] Add missing permissions for connector package (elastic#193573)
  [Fleet] using @kbn/config-schema part 2 (outputs and other apis)  (elastic#193326)
  [Migrations] Provide testing archives + tooling for migrations integration tests (elastic#193328)
  [ES|QL] Renames the textbased editor to esql editor (elastic#193521)
  [ES|QL] Update function metadata (elastic#193662)
  [Security Solution][Entity Analytics] Scoping the entity store to spaces (elastic#193303)
  [Docs] Update Sharing docs (elastic#190318)
  [ML] AIOps: Move Log Rate Analysis results callout to help popover. (elastic#192243)
  ...

# Conflicts:
#	x-pack/plugins/search_inference_endpoints/public/components/all_inference_endpoints/render_table_columns/render_endpoint/endpoint_info.test.tsx
#	x-pack/plugins/search_inference_endpoints/public/components/all_inference_endpoints/render_table_columns/render_endpoint/endpoint_info.tsx
kibanamachine added a commit that referenced this pull request Sep 23, 2024
@kibanamachine @bhapas @elasticmachine
[8.15] [Automatic Import] Do not remove message field for unstructure…
2d82be0 
…d logs (#193678) (#193702)

# Backport

This will backport the following commits from `main` to `8.15`:
- [[Automatic Import] Do not remove message field for unstructured logs
(#193678)](#193678)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Bharat
Pasupula","email":"123897612+bhapas@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-09-23T11:08:59Z","message":"[Automatic
Import] Do not remove message field for unstructured logs
(#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Automatic
Import] Do not remove message field for unstructured
logs","number":193678,"url":"https://github.com/elastic/kibana/pull/193678","mergeCommit":{"message":"[Automatic
Import] Do not remove message field for unstructured logs
(#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193678","number":193678,"mergeCommit":{"message":"[Automatic
Import] Do not remove message field for unstructured logs
(#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9"}}]}]
BACKPORT-->

Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
neptunian pushed a commit to neptunian/kibana that referenced this pull request Oct 1, 2024
@kibanamachine @bhapas @neptunian
[8.x] [Automatic Import] Do not remove message field for unstructured…
fbbf5e8 
… logs (elastic#193678) (elastic#193704)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Automatic Import] Do not remove message field for unstructured logs
(elastic#193678)](elastic#193678)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Bharat
Pasupula","email":"123897612+bhapas@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-09-23T11:08:59Z","message":"[Automatic
Import] Do not remove message field for unstructured logs
(elastic#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Automatic
Import] Do not remove message field for unstructured
logs","number":193678,"url":"https://github.com/elastic/kibana/pull/193678","mergeCommit":{"message":"[Automatic
Import] Do not remove message field for unstructured logs
(elastic#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193678","number":193678,"mergeCommit":{"message":"[Automatic
Import] Do not remove message field for unstructured logs
(elastic#193678)","sha":"2106df354adaa48dae7d8457b7d3401104724fc9"}}]}]
BACKPORT-->

Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@P1llus P1llus P1llus approved these changes

Assignees

@bhapas bhapas

Labels
backport:prev-major Backport to (8.x, 8.15) the previous major branch and all later branches still in development bug Fixes for quality problems that affect the customer experience Feature:AutomaticImport release_note:fix Team:Security-Scalability Team label for Security Integrations Scalability Team v8.15.2 v8.16.0 v9.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@bhapas @elasticmachine @kibana-ci @kibanamachine @P1llus