Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added documentation for select entries, map-to-list, and trucate processors. #6660

Merged
merged 18 commits into from
Mar 27, 2024

Conversation

kkondaka
Copy link
Contributor

Description

Added documentation for

  • select-entries processor
  • truncate processor
  • map-to-list processor
    Updated documentation for
  • file sink
  • s3 sink
  • kafka buffer

Issues Resolved

List any issues this PR will resolve, e.g. Closes [...].

Checklist

  • [X ] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…d other documents

Signed-off-by: Kondaka <krishkdk@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
_data-prepper/pipelines/configuration/buffers/kafka.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/buffers/kafka.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/buffers/kafka.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/buffers/kafka.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/buffers/kafka.md Outdated Show resolved Hide resolved
@@ -0,0 +1,133 @@
---
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going to remove this in favor of #6680.

_data-prepper/pipelines/configuration/sinks/file.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/sources/s3.md Outdated Show resolved Hide resolved
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
@Naarcha-AWS
Copy link
Collaborator

Can confirm the map-to-list will work once this is merged in.

@Naarcha-AWS Naarcha-AWS added 3 - Tech review PR: Tech review in progress and removed 4 - Doc review PR: Doc review in progress labels Mar 26, 2024

# select_entries

The `select_entries` processor selects entries from a Data Prepper event.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can be more explicit here.

The select_entries processor selects entries from a Data Prepper event. Only the selected entries will remain in the event and all other ones will be removed.

Option | Required | Type | Description
:--- | :--- | :--- | :---
`entries` | Yes | String list | A list of entries to add to an event.
`source_keys` | Yes | String list | The list of sources keys that will be modified by the processor.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not required anymore.

By default, it will truncate all fields.

source_keys | Not | String list | The list of sources keys that will be modified by the processor. The default value is an empty list which indicates truncating all values.

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
@Naarcha-AWS Naarcha-AWS added 5 - Editorial review PR: Editorial review in progress and removed 3 - Tech review PR: Tech review in progress labels Mar 26, 2024
@Naarcha-AWS
Copy link
Collaborator

@natebower: This is ready for your review.

Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kkondaka @Naarcha-AWS Please see my comments and changes and let me know if you have any questions. I'd like to see reread lines 25, 62, and 69 in truncate.md before approving. Thanks!

_data-prepper/pipelines/configuration/buffers/kafka.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/buffers/kafka.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/buffers/kafka.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/buffers/kafka.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/sources/s3.md Outdated Show resolved Hide resolved
Naarcha-AWS and others added 7 commits March 27, 2024 10:16
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Naarcha-AWS Just one comma to add. Otherwise, LGTM!

Naarcha-AWS and others added 2 commits March 27, 2024 11:19
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
@Naarcha-AWS Naarcha-AWS merged commit e697522 into opensearch-project:main Mar 27, 2024
2 of 3 checks passed
@Naarcha-AWS Naarcha-AWS added the backport 2.12 PR: Backport label for 2.12 label Mar 28, 2024
@opensearch-trigger-bot
Copy link

The backport to 2.12 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.12 2.12
# Navigate to the new working tree
pushd ../.worktrees/backport-2.12
# Create a new branch
git switch --create backport/backport-6660-to-2.12
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e697522b99a1b77b83362d328e0ae940326e445e
# Push it to GitHub
git push --set-upstream origin backport/backport-6660-to-2.12
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.12

Then, create a pull request where the base branch is 2.12 and the compare/head branch is backport/backport-6660-to-2.12.

Naarcha-AWS pushed a commit that referenced this pull request Mar 28, 2024
…essors. (#6660)

* Added documentation for select entries and trucate processors. Updated other documents

Signed-off-by: Kondaka <krishkdk@amazon.com>

* Update select-entries.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update and rename truncate-processor.md to truncate.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Delete _data-prepper/pipelines/configuration/processors/map-to-list.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/mutate-event.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/buffers/kafka.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/buffers/kafka.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/buffers/kafka.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/mutate-event.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/truncate.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

---------

Signed-off-by: Kondaka <krishkdk@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
(cherry picked from commit e697522)
Naarcha-AWS added a commit that referenced this pull request Mar 28, 2024
…essors. (#6660) (#6801)

* Added documentation for select entries and trucate processors. Updated other documents

Signed-off-by: Kondaka <krishkdk@amazon.com>

* Update select-entries.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update and rename truncate-processor.md to truncate.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Delete _data-prepper/pipelines/configuration/processors/map-to-list.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/mutate-event.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/buffers/kafka.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/buffers/kafka.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/buffers/kafka.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/mutate-event.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _data-prepper/pipelines/configuration/processors/truncate.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

---------

Signed-off-by: Kondaka <krishkdk@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
(cherry picked from commit e697522)

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Krishna Kondaka <41027584+kkondaka@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5 - Editorial review PR: Editorial review in progress backport 2.12 PR: Backport label for 2.12 data-prepper 2.7
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants