Dynamodb source pipeline not ingesting data

[an-khoros]

I have created a dynamo db table with dynamo db stream enabled (new and old images). Added , modified and deleted few records to generate activity. Now I have setup a pipeline (LATEST as the start position) using this table as source and then the target is open search cluster. For some reason the pipeline doesn't transfer the data even though there are new activities happening on the table. Below is the log that I see and it keeps retrying.

Any help is much appreciated.

[dlvenable]

@an-khoros ,

Would you be able to share your pipeline configuration? You can obfuscate your ARNs.

I see these are CloudWatch Logs. Are you using Amazon OpenSearch Ingestion or Data Prepper? If you are using Data Prepper, you will need to configure your source coordination. Amazon OpenSearch Ingestion does this for you, but in Data Prepper, you must make this configuration.

[an-khoros]

@dlvenable Thanks for your reply. I am using AWS Open Search. And I figured out what's the issue. My AWS role didn't have some of the dynamodb:* allow permissions. The data prepper was failing because of this and there was no exception being logged. Seems like exception handling could be better?

[graytaylor0]

Which permissions exactly did you not have? If you run data prepper locally with these permissions, do you see the error get logged? I have always seen permission errors get logged in general

[an-khoros]

I think I didn't add allow permission to myTable/*. Not sure though. Somewhere in the code it needs access to myTable/stream or something like that and then it was failing.