Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support reading of old image for delete events on DynamoDB source #4261

Closed
graytaylor0 opened this issue Mar 11, 2024 · 1 comment · Fixed by #4275
Closed

Support reading of old image for delete events on DynamoDB source #4261

graytaylor0 opened this issue Mar 11, 2024 · 1 comment · Fixed by #4275
Labels
enhancement New feature or request
Milestone

Comments

@graytaylor0
Copy link
Member

graytaylor0 commented Mar 11, 2024

Is your feature request related to a problem? Please describe.
As a user of the DynamoDB source using a custom document_id or document_version from my DynamoDB items, I am not able to keep deletions on my DDB table in sync with OpenSearch, because Data Prepper only reads the newImage of DDB stream records, and does not write the old image

Describe the solution you'd like
A new parameter under the stream block of DDB source

source:
  dynamodb:
    stream:
      use_old_image_for_deletes: true

When this is set to true, Data Prepper would check stream records for the REMOVE action, and then it would construct the Event from the old image, rather than the new empty image.

Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

@dlvenable
Copy link
Member

We changed this to view_on_remove: OLD_IMAGE to potentially support other configurations. See the following thread:

#4275 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

Successfully merging a pull request may close this issue.

2 participants