diff --git a/docs/core/idempotency.md b/docs/core/idempotency.md new file mode 100644 index 00000000..4770ed1d --- /dev/null +++ b/docs/core/idempotency.md @@ -0,0 +1,431 @@ +--- +title: Idempotency +description: Utility +--- + +The idempotency utility provides a simple solution to convert your Lambda functions into idempotent operations which are safe to retry. + +## Terminology + +The property of idempotency means that an operation does not cause additional side effects if it is called more than once with the same input parameters. + +**Idempotent operations will return the same result when they are called multiple times with the same parameters**. This makes idempotent operations safe to retry. [Read more](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) about idempotency. + +**Idempotency key** is a hash representation of either the entire event or a specific configured subset of the event, and invocation results are **JSON serialized** and stored in your persistence storage layer. + +## Key features + +* Prevent Lambda handler function from executing more than once on the same event payload during a time window +* Ensure Lambda handler returns the same result when called with the same payload +* Select a subset of the event as the idempotency key using [JMESPath](https://jmespath.org/) expressions +* Set a time window in which records with the same payload should be considered duplicates + +## Getting started + +### Installation +You should install with NuGet: + +``` +Install-Package AWS.Lambda.Powertools.Idempotency +``` + +Or via the .NET Core command line interface: + +``` +dotnet add package AWS.Lambda.Powertools.Idempotency +``` + +### Required resources + +Before getting started, you need to create a persistent storage layer where the idempotency utility can store its state - your Lambda functions will need read and write access to it. + +As of now, Amazon DynamoDB is the only supported persistent storage layer, so you'll need to create a table first. + +**Default table configuration** + +If you're not [changing the default configuration for the DynamoDB persistence layer](#dynamodbpersistencestore), this is the expected default configuration: + +| Configuration | Value | Notes | +|--------------------|--------------|-------------------------------------------------------------------------------------| +| Partition key | `id` | | +| TTL attribute name | `expiration` | This can only be configured after your table is created if you're using AWS Console | + +!!! Tip "Tip: You can share a single state table for all functions" + You can reuse the same DynamoDB table to store idempotency state. We add your function name in addition to the idempotency key as a hash key. + +=== "template.yml" + + ```yaml hl_lines="5-13 21-23 26" title="AWS Serverless Application Model (SAM) example" + Resources: + IdempotencyTable: + Type: AWS::DynamoDB::Table + Properties: + AttributeDefinitions: + - AttributeName: id + AttributeType: S + KeySchema: + - AttributeName: id + KeyType: HASH + TimeToLiveSpecification: + AttributeName: expiration + Enabled: true + BillingMode: PAY_PER_REQUEST + + IdempotencyFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: Function + Handler: HelloWorld::HelloWorld.Function::FunctionHandler + Policies: + - DynamoDBCrudPolicy: + TableName: !Ref IdempotencyTable + Environment: + Variables: + IDEMPOTENCY_TABLE: !Ref IdempotencyTable + ``` + +!!! warning "Warning: Large responses with DynamoDB persistence layer" + When using this utility with DynamoDB, your function's responses must be [smaller than 400KB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-items). + Larger items cannot be written to DynamoDB and will cause exceptions. + +!!! info "Info: DynamoDB" + Each function invocation will generally make 2 requests to DynamoDB. If the + result returned by your Lambda is less than 1kb, you can expect 2 WCUs per invocation. For retried invocations, you will + see 1WCU and 1RCU. Review the [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/) to + estimate the cost. + +### Idempotent attribute + +You can quickly start by configuring `Idempotency` and using it with the `Idempotent` attribute on your Lambda function. + +!!! warning "Important" + Initialization and configuration of the `Idempotency` must be performed outside the handler, preferably in the constructor. + + ```csharp + public class Function + { + public Function() + { + Idempotency.Configure(builder => builder.UseDynamoDb("idempotency_table")); + } + + [Idempotent] + public Task FunctionHandler(string input, ILambdaContext context) + { + return Task.FromResult(input.ToUpper()); + } + } + ``` + +### Choosing a payload subset for idempotency + +!!! tip "Tip: Dealing with always changing payloads" + When dealing with an elaborate payload (API Gateway request for example), where parts of the payload always change, you should configure the **`EventKeyJmesPath`**. + +Use [`IdempotencyConfig`](#customizing-the-default-behavior) to instruct the Idempotent annotation to only use a portion of your payload to verify whether a request is idempotent, and therefore it should not be retried. + +> **Payment scenario** + +In this example, we have a Lambda handler that creates a payment for a user subscribing to a product. We want to ensure that we don't accidentally charge our customer by subscribing them more than once. + +Imagine the function executes successfully, but the client never receives the response due to a connection issue. It is safe to retry in this instance, as the idempotent decorator will return a previously saved response. + +!!! warning "Warning: Idempotency for JSON payloads" + The payload extracted by the `EventKeyJmesPath` is treated as a string by default, so will be sensitive to differences in whitespace even when the JSON payload itself is identical. + + To alter this behaviour, you can use the JMESPath built-in function `powertools_json()` to treat the payload as a JSON object rather than a string. + + ```csharp + Idempotency.Configure(builder => + builder + .WithOptions(optionsBuilder => + optionsBuilder.WithEventKeyJmesPath("powertools_json(Body).address")) + .UseDynamoDb("idempotency_table")); + ``` + +### Handling exceptions + +If you are using the `Idempotent` attribute on your Lambda handler or any other method, any unhandled exceptions that are thrown during the code execution will cause **the record in the persistence layer to be deleted**. +This means that new invocations will execute your code again despite having the same payload. If you don't want the record to be deleted, you need to catch exceptions within the idempotent function and return a successful response. + +!!! warning + **We will throw an `IdempotencyPersistenceLayerException`** if any of the calls to the persistence layer fail unexpectedly. + + As this happens outside the scope of your decorated function, you are not able to catch it. + +### Persistence stores + +#### DynamoDBPersistenceStore + +This persistence store is built-in, and you can either use an existing DynamoDB table or create a new one dedicated for idempotency state (recommended). + +Use the builder to customize the table structure: +```csharp title="Customizing DynamoDBPersistenceStore to suit your table structure" +new DynamoDBPersistenceStoreBuilder() + .WithTableName("TABLE_NAME") + .WithKeyAttr("idempotency_key") + .WithExpiryAttr("expires_at") + .WithStatusAttr("current_status") + .WithDataAttr("result_data") + .WithValidationAttr("validation_key") + .Build() +``` + +When using DynamoDB as a persistence layer, you can alter the attribute names by passing these parameters when initializing the persistence layer: + +| Parameter | Required | Default | Description | +|--------------------|----------|--------------------------------------|--------------------------------------------------------------------------------------------------------| +| **TableName** | Y | | Table name to store state | +| **KeyAttr** | | `id` | Partition key of the table. Hashed representation of the payload (unless **SortKeyAttr** is specified) | +| **ExpiryAttr** | | `expiration` | Unix timestamp of when record expires | +| **StatusAttr** | | `status` | Stores status of the Lambda execution during and after invocation | +| **DataAttr** | | `data` | Stores results of successfully idempotent methods | +| **ValidationAttr** | | `validation` | Hashed representation of the parts of the event used for validation | +| **SortKeyAttr** | | | Sort key of the table (if table is configured with a sort key). | +| **StaticPkValue** | | `idempotency#{LAMBDA_FUNCTION_NAME}` | Static value to use as the partition key. Only used when **SortKeyAttr** is set. | + +## Advanced + +### Customizing the default behavior + +Idempotency behavior can be further configured with **`IdempotencyOptions`** using a builder: + +```csharp +new IdempotencyOptionsBuilder() + .WithEventKeyJmesPath("id") + .WithPayloadValidationJmesPath("paymentId") + .WithThrowOnNoIdempotencyKey(true) + .WithExpiration(TimeSpan.FromMinutes(1)) + .WithUseLocalCache(true) + .WithHashFunction("MD5") + .Build(); +``` + +These are the available options for further configuration: + +| Parameter | Default | Description | +|---------------------------------------------------|---------|----------------------------------------------------------------------------------------------------------------------------------| +| **EventKeyJMESPath** | `""` | JMESPath expression to extract the idempotency key from the event record. | +| **PayloadValidationJMESPath** | `""` | JMESPath expression to validate whether certain parameters have changed in the event | +| **ThrowOnNoIdempotencyKey** | `False` | Throw exception if no idempotency key was found in the request | +| **ExpirationInSeconds** | 3600 | The number of seconds to wait before a record is expired | +| **UseLocalCache** | `false` | Whether to locally cache idempotency results (LRU cache) | +| **HashFunction** | `MD5` | Algorithm to use for calculating hashes, as supported by `System.Security.Cryptography.HashAlgorithm` (eg. SHA1, SHA-256, ...) | + +These features are detailed below. + +### Handling concurrent executions with the same payload + +This utility will throw an **`IdempotencyAlreadyInProgressException`** if we receive **multiple invocations with the same payload while the first invocation hasn't completed yet**. + +!!! info + If you receive `IdempotencyAlreadyInProgressException`, you can safely retry the operation. + +This is a locking mechanism for correctness. Since we don't know the result from the first invocation yet, we can't safely allow another concurrent execution. + +### Using in-memory cache + +**By default, in-memory local caching is disabled**, to avoid using memory in an unpredictable way. + +!!! warning Memory configuration of your function + Be sure to configure the Lambda memory according to the number of records and the potential size of each record. + +You can enable it as seen before with: +```csharp title="Enable local cache" + new IdempotencyOptionsBuilder() + .WithUseLocalCache(true) + .Build() +``` +When enabled, we cache a maximum of 255 records in each Lambda execution environment + +!!! note "Note: This in-memory cache is local to each Lambda execution environment" + This means it will be effective in cases where your function's concurrency is low in comparison to the number of "retry" invocations with the same payload, because cache might be empty. + + +### Expiring idempotency records + +!!! note + By default, we expire idempotency records after **an hour** (3600 seconds). + +In most cases, it is not desirable to store the idempotency records forever. Rather, you want to guarantee that the same payload won't be executed within a period of time. + +You can change this window with the **`ExpirationInSeconds`** parameter: +```csharp title="Customizing expiration time" +new IdempotencyOptionsBuilder() + .WithExpiration(TimeSpan.FromMinutes(5)) + .Build() +``` + +Records older than 5 minutes will be marked as expired, and the Lambda handler will be executed normally even if it is invoked with a matching payload. + +!!! note "Note: DynamoDB time-to-live field" + This utility uses **`expiration`** as the TTL field in DynamoDB, as [demonstrated in the SAM example earlier](#required-resources). + +### Payload validation + +!!! question "Question: What if your function is invoked with the same payload except some outer parameters have changed?" + Example: A payment transaction for a given productID was requested twice for the same customer, **however the amount to be paid has changed in the second transaction**. + +By default, we will return the same result as it returned before, however in this instance it may be misleading; we provide a fail fast payload validation to address this edge case. + +With **`PayloadValidationJMESPath`**, you can provide an additional JMESPath expression to specify which part of the event body should be validated against previous idempotent invocations + +=== "Function.cs" + + ```csharp + Idempotency.Configure(builder => + builder + .WithOptions(optionsBuilder => + optionsBuilder + .WithEventKeyJmesPath("[userDetail, productId]") + .WithPayloadValidationJmesPath("amount")) + .UseDynamoDb("TABLE_NAME")); + ``` + +=== "Example Event 1" + + ```json hl_lines="8" + { + "userDetail": { + "username": "User1", + "user_email": "user@example.com" + }, + "productId": 1500, + "charge_type": "subscription", + "amount": 500 + } + ``` + +=== "Example Event 2" + + ```json hl_lines="8" + { + "userDetail": { + "username": "User1", + "user_email": "user@example.com" + }, + "productId": 1500, + "charge_type": "subscription", + "amount": 1 + } + ``` + +In this example, the **`userDetail`** and **`productId`** keys are used as the payload to generate the idempotency key, as per **`EventKeyJMESPath`** parameter. + +!!! note + If we try to send the same request but with a different amount, we will raise **`IdempotencyValidationException`**. + +Without payload validation, we would have returned the same result as we did for the initial request. Since we're also returning an amount in the response, this could be quite confusing for the client. + +By using **`withPayloadValidationJMESPath("amount")`**, we prevent this potentially confusing behavior and instead throw an Exception. + +### Making idempotency key required + +If you want to enforce that an idempotency key is required, you can set **`ThrowOnNoIdempotencyKey`** to `True`. + +This means that we will throw **`IdempotencyKeyException`** if the evaluation of **`EventKeyJMESPath`** is `null`. + +=== "Function.cs" + + ```csharp + public App() + { + Idempotency.Configure(builder => + builder + .WithOptions(optionsBuilder => + optionsBuilder + // Requires "user"."uid" and "orderId" to be present + .WithEventKeyJmesPath("[user.uid, orderId]") + .WithThrowOnNoIdempotencyKey(true)) + .UseDynamoDb("TABLE_NAME")); + } + + [Idempotent] + public Task FunctionHandler(Order input, ILambdaContext context) + { + // ... + } + ``` + +=== "Success Event" + + ```json + { + "user": { + "uid": "BB0D045C-8878-40C8-889E-38B3CB0A61B1", + "name": "Foo" + }, + "orderId": 10000 + } + ``` + +=== "Failure Event" + + Notice that `orderId` is now accidentally within `user` key + + ```json + { + "user": { + "uid": "DE0D000E-1234-10D1-991E-EAC1DD1D52C8", + "name": "Joe Bloggs", + "orderId": 10000 + }, + } + ``` + +### Customizing DynamoDB configuration + +When creating the `DynamoDBPersistenceStore`, you can set a custom [`AmazonDynamoDBClient`](https://docs.aws.amazon.com/sdkfornet1/latest/apidocs/html/T_Amazon_DynamoDB_AmazonDynamoDBClient.htm) if you need to customize the configuration: + +=== "Custom AmazonDynamoDBClient" + + ```csharp + public Function() + { + AmazonDynamoDBClient customClient = new AmazonDynamoDBClient(RegionEndpoint.APSouth1); + + Idempotency.Configure(builder => + builder.UseDynamoDb(storeBuilder => + storeBuilder. + WithTableName("TABLE_NAME") + .WithDynamoDBClient(customClient) + )); + } + ``` + +### Using a DynamoDB table with a composite primary key + +When using a composite primary key table (hash+range key), use `SortKeyAttr` parameter when initializing your persistence store. + +With this setting, we will save the idempotency key in the sort key instead of the primary key. By default, the primary key will now be set to `idempotency#{LAMBDA_FUNCTION_NAME}`. + +You can optionally set a static value for the partition key using the `StaticPkValue` parameter. + +```csharp title="Reusing a DynamoDB table that uses a composite primary key" +Idempotency.Configure(builder => + builder.UseDynamoDb(storeBuilder => + storeBuilder. + WithTableName("TABLE_NAME") + .WithSortKeyAttr("sort_key") + )); +``` + +Data would then be stored in DynamoDB like this: + +| id | sort_key | expiration | status | data | +|------------------------------|----------------------------------|------------|-------------|--------------------------------------| +| idempotency#MyLambdaFunction | 1e956ef7da78d0cb890be999aecc0c9e | 1636549553 | COMPLETED | {"id": 12391, "message": "success"} | +| idempotency#MyLambdaFunction | 2b2cdb5f86361e97b4383087c1ffdf27 | 1636549571 | COMPLETED | {"id": 527212, "message": "success"} | +| idempotency#MyLambdaFunction | f091d2527ad1c78f05d54cc3f363be80 | 1636549585 | IN_PROGRESS | | + +## Testing your code + +The idempotency utility provides several routes to test your code. + +### Disabling the idempotency utility +When testing your code, you may wish to disable the idempotency logic altogether and focus on testing your business logic. To do this, you can set the environment variable `POWERTOOLS_IDEMPOTENCY_DISABLED` to true. + +## Extra resources + +If you're interested in a deep dive on how Amazon uses idempotency when building our APIs, check out +[this article](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/). diff --git a/libraries/AWS.Lambda.Powertools.sln b/libraries/AWS.Lambda.Powertools.sln index 9734269b..5f194c34 100644 --- a/libraries/AWS.Lambda.Powertools.sln +++ b/libraries/AWS.Lambda.Powertools.sln @@ -23,6 +23,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AWS.Lambda.Powertools.Traci EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AWS.Lambda.Powertools.Common.Tests", "tests\AWS.Lambda.Powertools.Common.Tests\AWS.Lambda.Powertools.Common.Tests.csproj", "{4EC48E6A-45B5-4E25-ABBD-C23FE2BD6E1E}" EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AWS.Lambda.Powertools.Idempotency", "src\AWS.Lambda.Powertools.Idempotency\AWS.Lambda.Powertools.Idempotency.csproj", "{B7AC87DF-9705-47D9-AC00-C230E577CA5D}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AWS.Lambda.Powertools.Idempotency.Tests", "tests\AWS.Lambda.Powertools.Idempotency.Tests\AWS.Lambda.Powertools.Idempotency.Tests.csproj", "{3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}" +EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AWS.Lambda.Powertools.Parameters", "src\AWS.Lambda.Powertools.Parameters\AWS.Lambda.Powertools.Parameters.csproj", "{1ECB31E8-2EF0-41E2-8C71-CB9876D207F0}" EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AWS.Lambda.Powertools.Parameters.Tests", "tests\AWS.Lambda.Powertools.Parameters.Tests\AWS.Lambda.Powertools.Parameters.Tests.csproj", "{386A9769-59BF-4BE3-99D4-A9603E300729}" @@ -136,6 +140,30 @@ Global {4EC48E6A-45B5-4E25-ABBD-C23FE2BD6E1E}.Release|x64.Build.0 = Release|Any CPU {4EC48E6A-45B5-4E25-ABBD-C23FE2BD6E1E}.Release|x86.ActiveCfg = Release|Any CPU {4EC48E6A-45B5-4E25-ABBD-C23FE2BD6E1E}.Release|x86.Build.0 = Release|Any CPU + {B7AC87DF-9705-47D9-AC00-C230E577CA5D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {B7AC87DF-9705-47D9-AC00-C230E577CA5D}.Debug|Any CPU.Build.0 = Debug|Any CPU + {B7AC87DF-9705-47D9-AC00-C230E577CA5D}.Debug|x64.ActiveCfg = Debug|Any CPU + {B7AC87DF-9705-47D9-AC00-C230E577CA5D}.Debug|x64.Build.0 = Debug|Any CPU + {B7AC87DF-9705-47D9-AC00-C230E577CA5D}.Debug|x86.ActiveCfg = Debug|Any CPU + {B7AC87DF-9705-47D9-AC00-C230E577CA5D}.Debug|x86.Build.0 = Debug|Any CPU + {B7AC87DF-9705-47D9-AC00-C230E577CA5D}.Release|Any CPU.ActiveCfg = Release|Any CPU + {B7AC87DF-9705-47D9-AC00-C230E577CA5D}.Release|Any CPU.Build.0 = Release|Any CPU + {B7AC87DF-9705-47D9-AC00-C230E577CA5D}.Release|x64.ActiveCfg = Release|Any CPU + {B7AC87DF-9705-47D9-AC00-C230E577CA5D}.Release|x64.Build.0 = Release|Any CPU + {B7AC87DF-9705-47D9-AC00-C230E577CA5D}.Release|x86.ActiveCfg = Release|Any CPU + {B7AC87DF-9705-47D9-AC00-C230E577CA5D}.Release|x86.Build.0 = Release|Any CPU + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}.Debug|Any CPU.Build.0 = Debug|Any CPU + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}.Debug|x64.ActiveCfg = Debug|Any CPU + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}.Debug|x64.Build.0 = Debug|Any CPU + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}.Debug|x86.ActiveCfg = Debug|Any CPU + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}.Debug|x86.Build.0 = Debug|Any CPU + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}.Release|Any CPU.ActiveCfg = Release|Any CPU + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}.Release|Any CPU.Build.0 = Release|Any CPU + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}.Release|x64.ActiveCfg = Release|Any CPU + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}.Release|x64.Build.0 = Release|Any CPU + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}.Release|x86.ActiveCfg = Release|Any CPU + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3}.Release|x86.Build.0 = Release|Any CPU {1ECB31E8-2EF0-41E2-8C71-CB9876D207F0}.Debug|Any CPU.ActiveCfg = Debug|Any CPU {1ECB31E8-2EF0-41E2-8C71-CB9876D207F0}.Debug|Any CPU.Build.0 = Debug|Any CPU {1ECB31E8-2EF0-41E2-8C71-CB9876D207F0}.Debug|x64.ActiveCfg = Debug|Any CPU @@ -173,5 +201,7 @@ Global {A040AED5-BBB8-4BFA-B2A5-BBD82817B8A5} = {1CFF5568-8486-475F-81F6-06105C437528} {1ECB31E8-2EF0-41E2-8C71-CB9876D207F0} = {73C9B1E5-3893-47E8-B373-17E5F5D7E6F5} {386A9769-59BF-4BE3-99D4-A9603E300729} = {1CFF5568-8486-475F-81F6-06105C437528} + {B7AC87DF-9705-47D9-AC00-C230E577CA5D} = {73C9B1E5-3893-47E8-B373-17E5F5D7E6F5} + {3E1D77BD-70AF-4767-B00A-4A321D5AB2C3} = {1CFF5568-8486-475F-81F6-06105C437528} EndGlobalSection EndGlobal diff --git a/libraries/src/AWS.Lambda.Powertools.Common/Core/Constants.cs b/libraries/src/AWS.Lambda.Powertools.Common/Core/Constants.cs index 35d33189..a3c93441 100644 --- a/libraries/src/AWS.Lambda.Powertools.Common/Core/Constants.cs +++ b/libraries/src/AWS.Lambda.Powertools.Common/Core/Constants.cs @@ -90,4 +90,19 @@ internal static class Constants /// Constant for Powertools for AWS Lambda (.NET) feature identifier fo AWS_EXECUTION_ENV environment variable /// internal const string FeatureContextIdentifier = "PT"; + + /// + /// Constant for IDEMPOTENCY_DISABLED_ENV environment variable + /// + internal const string IdempotencyDisabledEnv = "POWERTOOLS_IDEMPOTENCY_DISABLED"; + + /// + /// Constant for AWS_REGION_ENV environment variable + /// + internal const string AwsRegionEnv = "AWS_REGION"; + + /// + /// Constant for LAMBDA_FUNCTION_NAME_ENV environment variable + /// + internal const string LambdaFunctionNameEnv = "AWS_LAMBDA_FUNCTION_NAME"; } \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Common/Core/IPowertoolsConfigurations.cs b/libraries/src/AWS.Lambda.Powertools.Common/Core/IPowertoolsConfigurations.cs index a2e10cc7..86edd2ba 100644 --- a/libraries/src/AWS.Lambda.Powertools.Common/Core/IPowertoolsConfigurations.cs +++ b/libraries/src/AWS.Lambda.Powertools.Common/Core/IPowertoolsConfigurations.cs @@ -126,4 +126,10 @@ public interface IPowertoolsConfigurations /// /// void SetExecutionEnvironment(T type); + + /// + /// Gets a value indicating whether [Idempotency is disabled]. + /// + /// true if [Idempotency is disabled]; otherwise, false. + bool IdempotencyDisabled { get; } } \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Common/Core/PowertoolsConfigurations.cs b/libraries/src/AWS.Lambda.Powertools.Common/Core/PowertoolsConfigurations.cs index 1d099e4d..bcab772f 100644 --- a/libraries/src/AWS.Lambda.Powertools.Common/Core/PowertoolsConfigurations.cs +++ b/libraries/src/AWS.Lambda.Powertools.Common/Core/PowertoolsConfigurations.cs @@ -190,4 +190,8 @@ public void SetExecutionEnvironment(T type) { _systemWrapper.SetExecutionEnvironment(type); } + + /// + public bool IdempotencyDisabled => + GetEnvironmentVariableOrDefault(Constants.IdempotencyDisabledEnv, false); } \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Common/Core/PowertoolsLambdaContext.cs b/libraries/src/AWS.Lambda.Powertools.Common/Core/PowertoolsLambdaContext.cs index b07013a8..7f9b3fee 100644 --- a/libraries/src/AWS.Lambda.Powertools.Common/Core/PowertoolsLambdaContext.cs +++ b/libraries/src/AWS.Lambda.Powertools.Common/Core/PowertoolsLambdaContext.cs @@ -1,6 +1,4 @@ using System; -using System.Linq; -using System.Reflection; namespace AWS.Lambda.Powertools.Common; diff --git a/libraries/src/AWS.Lambda.Powertools.Common/InternalsVisibleTo.cs b/libraries/src/AWS.Lambda.Powertools.Common/InternalsVisibleTo.cs index f88809b3..ee178f7f 100644 --- a/libraries/src/AWS.Lambda.Powertools.Common/InternalsVisibleTo.cs +++ b/libraries/src/AWS.Lambda.Powertools.Common/InternalsVisibleTo.cs @@ -17,7 +17,9 @@ [assembly: InternalsVisibleTo("AWS.Lambda.Powertools.Logging")] [assembly: InternalsVisibleTo("AWS.Lambda.Powertools.Metrics")] +[assembly: InternalsVisibleTo("AWS.Lambda.Powertools.Idempotency")] [assembly: InternalsVisibleTo("AWS.Lambda.Powertools.Common.Tests")] [assembly: InternalsVisibleTo("AWS.Lambda.Powertools.Tracing.Tests")] [assembly: InternalsVisibleTo("AWS.Lambda.Powertools.Metrics.Tests")] -[assembly: InternalsVisibleTo("AWS.Lambda.Powertools.Logging.Tests")] \ No newline at end of file +[assembly: InternalsVisibleTo("AWS.Lambda.Powertools.Logging.Tests")] +[assembly: InternalsVisibleTo("AWS.Lambda.Powertools.Idempotency.Tests")] \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/AWS.Lambda.Powertools.Idempotency.csproj b/libraries/src/AWS.Lambda.Powertools.Idempotency/AWS.Lambda.Powertools.Idempotency.csproj new file mode 100644 index 00000000..684a6a50 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/AWS.Lambda.Powertools.Idempotency.csproj @@ -0,0 +1,46 @@ + + + + net6.0 + default + AWS.Lambda.Powertools.Idempotency + 0.0.1 + Amazon Web Services + Amazon.com, Inc + AWS Lambda Powertools for .NET + AWS Lambda Powertools for .NET - Idempotency package. + Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved. + https://github.com/awslabs/aws-lambda-powertools-dotnet + Apache-2.0 + AWS;Amazon;Lambda;Powertools + README.md + https://sdk-for-net.amazonwebservices.com/images/AWSLogo128x128.png + AWSLogo128x128.png + true + AWS.Lambda.Powertools.Idempotency + AWS.Lambda.Powertools.Idempotency + + + + true + + + + + + + + + + + + + + + + + + + + + diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyAlreadyInProgressException.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyAlreadyInProgressException.cs new file mode 100644 index 00000000..3732c573 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyAlreadyInProgressException.cs @@ -0,0 +1,44 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; + +namespace AWS.Lambda.Powertools.Idempotency.Exceptions; + +/// +/// This exception is thrown when the same payload is sent +/// while the previous one was not yet fully stored in the persistence layer (marked as COMPLETED). +/// Implements the +/// +/// +public class IdempotencyAlreadyInProgressException: Exception +{ + /// + /// Creates a new IdempotencyAlreadyInProgressException + /// + public IdempotencyAlreadyInProgressException() + { + } + + /// + public IdempotencyAlreadyInProgressException(string message) : base(message) + { + } + + /// + public IdempotencyAlreadyInProgressException(string message, Exception innerException) : base(message, innerException) + { + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyConfigurationException.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyConfigurationException.cs new file mode 100644 index 00000000..cec6f729 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyConfigurationException.cs @@ -0,0 +1,44 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; + +namespace AWS.Lambda.Powertools.Idempotency.Exceptions; + +/// +/// Exception thrown when Idempotency is not well configured: +/// - An annotated method does not return anything +/// - An annotated method does not use Task as return value +/// - An annotated method does not have parameters +/// +public class IdempotencyConfigurationException : Exception +{ + /// + /// Creates a new IdempotencyConfigurationException + /// + public IdempotencyConfigurationException() + { + } + + /// + public IdempotencyConfigurationException(string message) : base(message) + { + } + + /// + public IdempotencyConfigurationException(string message, Exception innerException) : base(message, innerException) + { + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyInconsistentStateException.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyInconsistentStateException.cs new file mode 100644 index 00000000..8478a813 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyInconsistentStateException.cs @@ -0,0 +1,42 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; + +namespace AWS.Lambda.Powertools.Idempotency.Exceptions; + +/// +/// Exception can happen under rare but expected cases +/// when persistent state changes in the small-time between put and get requests. +/// +public class IdempotencyInconsistentStateException : Exception +{ + /// + /// Creates a new IdempotencyInconsistentStateException + /// + public IdempotencyInconsistentStateException() + { + } + + /// + public IdempotencyInconsistentStateException(string message) : base(message) + { + } + + /// + public IdempotencyInconsistentStateException(string message, Exception innerException) : base(message, innerException) + { + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyItemAlreadyExistsException.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyItemAlreadyExistsException.cs new file mode 100644 index 00000000..1603bba7 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyItemAlreadyExistsException.cs @@ -0,0 +1,41 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; + +namespace AWS.Lambda.Powertools.Idempotency.Exceptions; + +/// +/// Exception thrown when trying to store an item which already exists. +/// +public class IdempotencyItemAlreadyExistsException : Exception +{ + /// + /// Creates a new IdempotencyItemAlreadyExistsException + /// + public IdempotencyItemAlreadyExistsException() + { + } + + /// + public IdempotencyItemAlreadyExistsException(string message) : base(message) + { + } + + /// + public IdempotencyItemAlreadyExistsException(string message, Exception innerException) : base(message, innerException) + { + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyItemNotFoundException.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyItemNotFoundException.cs new file mode 100644 index 00000000..8be4180d --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyItemNotFoundException.cs @@ -0,0 +1,41 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; + +namespace AWS.Lambda.Powertools.Idempotency.Exceptions; + +/// +/// Exception thrown when the item was not found in the persistence store. +/// +public class IdempotencyItemNotFoundException : Exception +{ + /// + /// Creates a new IdempotencyItemNotFoundException + /// + public IdempotencyItemNotFoundException() + { + } + + /// + public IdempotencyItemNotFoundException(string message) : base(message) + { + } + + /// + public IdempotencyItemNotFoundException(string message, Exception innerException) : base(message, innerException) + { + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyKeyException.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyKeyException.cs new file mode 100644 index 00000000..46529872 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyKeyException.cs @@ -0,0 +1,42 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; + +namespace AWS.Lambda.Powertools.Idempotency.Exceptions; + +/// +/// Exception thrown only when using +/// and if a key could not be found in the event (for example when having a bad JMESPath configured) +/// +public class IdempotencyKeyException : Exception +{ + /// + /// Creates a new IdempotencyKeyException + /// + public IdempotencyKeyException() + { + } + + /// + public IdempotencyKeyException(string message) : base(message) + { + } + + /// + public IdempotencyKeyException(string message, Exception innerException) : base(message, innerException) + { + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyPersistenceLayerException.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyPersistenceLayerException.cs new file mode 100644 index 00000000..51377386 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyPersistenceLayerException.cs @@ -0,0 +1,41 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; + +namespace AWS.Lambda.Powertools.Idempotency.Exceptions; + +/// +/// Exception thrown when a technical error occurred with the persistence layer (eg. insertion, deletion, ... in database) +/// +public class IdempotencyPersistenceLayerException : Exception +{ + /// + /// Creates a new IdempotencyPersistenceLayerException + /// + public IdempotencyPersistenceLayerException() + { + } + + /// + public IdempotencyPersistenceLayerException(string message) : base(message) + { + } + + /// + public IdempotencyPersistenceLayerException(string message, Exception innerException) : base(message, innerException) + { + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyValidationException.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyValidationException.cs new file mode 100644 index 00000000..eb03f8b5 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Exceptions/IdempotencyValidationException.cs @@ -0,0 +1,42 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; + +namespace AWS.Lambda.Powertools.Idempotency.Exceptions; + +/// +/// Exception thrown only when using is configured +/// and the payload changed between two calls (but with the same idempotency key). +/// +public class IdempotencyValidationException : Exception +{ + /// + /// Creates a new IdempotencyValidationException + /// + public IdempotencyValidationException() + { + } + + /// + public IdempotencyValidationException(string message) : base(message) + { + } + + /// + public IdempotencyValidationException(string message, Exception innerException) : base(message, innerException) + { + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Idempotency.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Idempotency.cs new file mode 100644 index 00000000..d81fb428 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Idempotency.cs @@ -0,0 +1,165 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using AWS.Lambda.Powertools.Common; +using AWS.Lambda.Powertools.Idempotency.Persistence; + +namespace AWS.Lambda.Powertools.Idempotency; + +/// +/// Holds the configuration for idempotency: +/// The persistence layer to use for persisting the request and response of the function (mandatory). +/// The general configurations for idempotency (optional, see {@link IdempotencyConfig.Builder} methods to see defaults values. +/// Use it before the function handler get called. +/// Example: Idempotency.Configure(builder => builder.WithPersistenceStore(...)); +/// +public sealed class Idempotency +{ + /// + /// The general configurations for the idempotency + /// + public IdempotencyOptions IdempotencyOptions { get; private set; } = null!; + + /// + /// The persistence layer to use for persisting the request and response of the function + /// + public BasePersistenceStore PersistenceStore { get; private set; } = null!; + + /// + /// Idempotency Constructor + /// + /// + internal Idempotency(IPowertoolsConfigurations powertoolsConfigurations) + { + powertoolsConfigurations.SetExecutionEnvironment(this); + } + /// + /// Set Idempotency options + /// + /// + private void SetConfig(IdempotencyOptions options) + { + IdempotencyOptions = options; + } + + /// + /// Set Persistence Store + /// + /// + private void SetPersistenceStore(BasePersistenceStore persistenceStore) + { + PersistenceStore = persistenceStore; + } + + /// + /// Holds the idempotency Instance: + /// + public static Idempotency Instance { get; } = new(PowertoolsConfigurations.Instance); + + /// + /// Use this method to configure persistence layer (mandatory) and idempotency options (optional) + /// + public static void Configure(Action configurationAction) + { + var builder = new IdempotencyBuilder(); + configurationAction(builder); + if (builder.Store == null) + { + throw new NullReferenceException("Persistence Layer is null, configure one with 'WithPersistenceStore()'"); + } + + Instance.SetConfig(builder.Options ?? new IdempotencyOptionsBuilder().Build()); + Instance.SetPersistenceStore(builder.Store); + } + + /// + /// Create a builder that can be used to configure and create + /// + public class IdempotencyBuilder + { + /// + /// Exposes Idempotency options + /// + internal IdempotencyOptions Options { get; private set; } + + /// + /// Exposes Persistence Store + /// + internal BasePersistenceStore Store { get; private set; } + + /// + /// Set the persistence layer to use for storing the request and response + /// + /// + /// IdempotencyBuilder + public IdempotencyBuilder WithPersistenceStore(BasePersistenceStore persistenceStore) + { + Store = persistenceStore; + return this; + } + + /// + /// Configure Idempotency to use DynamoDBPersistenceStore + /// + /// The builder being used to configure the + /// IdempotencyBuilder + public IdempotencyBuilder UseDynamoDb(Action builderAction) + { + var builder = + new DynamoDBPersistenceStoreBuilder(); + builderAction(builder); + Store = builder.Build(); + return this; + } + + /// + /// Configure Idempotency to use DynamoDBPersistenceStore + /// + /// The DynamoDb table name + /// IdempotencyBuilder + public IdempotencyBuilder UseDynamoDb(string tableName) + { + var builder = + new DynamoDBPersistenceStoreBuilder(); + Store = builder.WithTableName(tableName).Build(); + return this; + } + + /// + /// Set the idempotency configurations + /// + /// The builder being used to configure the . + /// IdempotencyBuilder + public IdempotencyBuilder WithOptions(Action builderAction) + { + var builder = new IdempotencyOptionsBuilder(); + builderAction(builder); + Options = builder.Build(); + return this; + } + + /// + /// Set the default idempotency configurations + /// + /// + /// IdempotencyBuilder + public IdempotencyBuilder WithOptions(IdempotencyOptions options) + { + Options = options; + return this; + } + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/IdempotencyOptions.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/IdempotencyOptions.cs new file mode 100644 index 00000000..ed1b4a82 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/IdempotencyOptions.cs @@ -0,0 +1,88 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +namespace AWS.Lambda.Powertools.Idempotency; + +/// +/// Configuration of the idempotency feature. Use the Builder to create an instance. +/// +public class IdempotencyOptions +{ + /// + /// A JMESPath expression to extract the idempotency key from the event record. + /// https://jmespath.org for more details + /// Common paths: + /// powertools_json(Body) for APIGatewayProxyRequest + /// Records[*].powertools_json(Body) for SQSEvent + /// Records[0].Sns.Message | powertools_json(@) for SNSEvent + /// Detail for ScheduledEvent (EventBridge / CloudWatch events) + /// + public string EventKeyJmesPath { get; } + /// + /// JMES Path of a part of the payload to be used for validation + /// See https://jmespath.org/ + /// + public string PayloadValidationJmesPath { get; } + /// + /// Boolean to indicate if we must throw an Exception when + /// idempotency key could not be found in the payload. + /// + public bool ThrowOnNoIdempotencyKey { get; } + /// + /// Whether to locally cache idempotency results, by default false + /// + public bool UseLocalCache { get; } + /// + /// The maximum number of items to store in local cache + /// + public int LocalCacheMaxItems { get; } + /// + /// The number of seconds to wait before a record is expired + /// + public long ExpirationInSeconds { get; } + /// + /// Algorithm to use for calculating hashes, + /// as supported by (eg. SHA1, SHA-256, ...) + /// + public string HashFunction { get; } + + /// + /// Constructor of . + /// + /// + /// + /// + /// + /// + /// + /// + internal IdempotencyOptions( + string eventKeyJmesPath, + string payloadValidationJmesPath, + bool throwOnNoIdempotencyKey, + bool useLocalCache, + int localCacheMaxItems, + long expirationInSeconds, + string hashFunction) + { + EventKeyJmesPath = eventKeyJmesPath; + PayloadValidationJmesPath = payloadValidationJmesPath; + ThrowOnNoIdempotencyKey = throwOnNoIdempotencyKey; + UseLocalCache = useLocalCache; + LocalCacheMaxItems = localCacheMaxItems; + ExpirationInSeconds = expirationInSeconds; + HashFunction = hashFunction; + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/IdempotencyOptionsBuilder.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/IdempotencyOptionsBuilder.cs new file mode 100644 index 00000000..721ebf29 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/IdempotencyOptionsBuilder.cs @@ -0,0 +1,124 @@ +using System; + +namespace AWS.Lambda.Powertools.Idempotency; + +/// +/// Create a builder that can be used to configure and create +/// +public class IdempotencyOptionsBuilder +{ + /// + /// Default maximum number of items in the local cache. + /// + private readonly int _localCacheMaxItems = 256; + /// + /// Local cache enabled + /// + private bool _useLocalCache; + /// + /// Default expiration in seconds. + /// + private long _expirationInSeconds = 60 * 60; // 1 hour + /// + /// Event key JMESPath expression. + /// + private string _eventKeyJmesPath; + /// + /// Payload validation JMESPath expression. + /// + private string _payloadValidationJmesPath; + /// + /// Throw exception if no idempotency key is found. + /// + private bool _throwOnNoIdempotencyKey; + /// + /// Default Hash function + /// + private string _hashFunction = "MD5"; + + /// + /// Initialize and return an instance of IdempotencyConfig. + /// Example: + /// IdempotencyConfig.Builder().WithUseLocalCache().Build(); + /// This instance must then be passed to the Idempotency.Config: + /// Idempotency.Config().WithConfig(config).Configure(); + /// + /// an instance of IdempotencyConfig + public IdempotencyOptions Build() => + new(_eventKeyJmesPath, + _payloadValidationJmesPath, + _throwOnNoIdempotencyKey, + _useLocalCache, + _localCacheMaxItems, + _expirationInSeconds, + _hashFunction); + + /// + /// A JMESPath expression to extract the idempotency key from the event record. + /// See https://jmespath.org/ for more details. + /// + /// path of the key in the Lambda event + /// the instance of the builder (to chain operations) + public IdempotencyOptionsBuilder WithEventKeyJmesPath(string eventKeyJmesPath) + { + _eventKeyJmesPath = eventKeyJmesPath; + return this; + } + + /// + /// Whether to locally cache idempotency results, by default false + /// + /// Indicate if a local cache must be used in addition to the persistence store. + /// the instance of the builder (to chain operations) + public IdempotencyOptionsBuilder WithUseLocalCache(bool useLocalCache) + { + _useLocalCache = useLocalCache; + return this; + } + + /// + /// A JMESPath expression to extract the payload to be validated from the event record. + /// See https://jmespath.org/ for more details. + /// + /// JMES Path of a part of the payload to be used for validation + /// the instance of the builder (to chain operations) + public IdempotencyOptionsBuilder WithPayloadValidationJmesPath(string payloadValidationJmesPath) + { + _payloadValidationJmesPath = payloadValidationJmesPath; + return this; + } + + /// + /// Whether to throw an exception if no idempotency key was found in the request, by default false + /// + /// boolean to indicate if we must throw an Exception when + /// idempotency key could not be found in the payload. + /// the instance of the builder (to chain operations) + public IdempotencyOptionsBuilder WithThrowOnNoIdempotencyKey(bool throwOnNoIdempotencyKey) + { + _throwOnNoIdempotencyKey = throwOnNoIdempotencyKey; + return this; + } + + /// + /// The number of seconds to wait before a record is expired + /// + /// expiration of the record in the store + /// the instance of the builder (to chain operations) + public IdempotencyOptionsBuilder WithExpiration(TimeSpan duration) + { + _expirationInSeconds = (long) duration.TotalSeconds; + return this; + } + + /// + /// Function to use for calculating hashes, by default MD5. + /// + /// Can be any algorithm supported by HashAlgorithm.Create + /// the instance of the builder (to chain operations) + public IdempotencyOptionsBuilder WithHashFunction(string hashFunction) + { + _hashFunction = hashFunction; + return this; + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/IdempotentAttribute.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/IdempotentAttribute.cs new file mode 100644 index 00000000..c14ca1af --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/IdempotentAttribute.cs @@ -0,0 +1,127 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using System.Linq; +using System.Text.Json; +using System.Threading.Tasks; +using AspectInjector.Broker; +using AWS.Lambda.Powertools.Common; +using AWS.Lambda.Powertools.Idempotency.Exceptions; +using AWS.Lambda.Powertools.Idempotency.Internal; + +namespace AWS.Lambda.Powertools.Idempotency; + +/// +/// Idempotent is used to signal that the annotated method is idempotent: +/// Calling this method one or multiple times with the same parameter will always return the same result. +/// This annotation can be placed on any method of a Lambda function +/// +/// [Idempotent] +/// public Task<string> FunctionHandler(string input, ILambdaContext context) +/// { +/// return Task.FromResult(input.ToUpper()); +/// } +/// +/// Environment variables
+/// ---------------------
+/// +/// +/// Variable name +/// Description +/// +/// +/// AWS_LAMBDA_FUNCTION_NAME +/// string, function name +/// +/// +/// AWS_REGION +/// string, AWS region +/// +/// +/// POWERTOOLS_IDEMPOTENCY_DISABLED +/// string, Enable or disable the Idempotency +/// +/// +/// +[AttributeUsage(AttributeTargets.Method)] +[Injection(typeof(UniversalWrapperAspect), Inherited = true)] +public class IdempotentAttribute : UniversalWrapperAttribute +{ + /// + /// Wraps as a synchronous operation, simply throws IdempotencyConfigurationException + /// + /// + /// The target. + /// The arguments. + /// The instance containing the event data. + /// T. + protected internal sealed override T WrapSync(Func target, object[] args, AspectEventArgs eventArgs) + { + if (PowertoolsConfigurations.Instance.IdempotencyDisabled) + { + return base.WrapSync(target, args, eventArgs); + } + + var payload = args is not null && args.Any() ? JsonDocument.Parse(JsonSerializer.Serialize(args[0])) : null; + if (payload == null) + { + throw new IdempotencyConfigurationException("Unable to get payload from the method. Ensure there is at least one parameter or that you use @IdempotencyKey"); + } + + Task ResultDelegate() => Task.FromResult(target(args)); + + var idempotencyHandler = new IdempotencyAspectHandler(ResultDelegate, eventArgs.Method.Name, payload); + if (idempotencyHandler == null) + { + throw new Exception("Failed to create an instance of IdempotencyAspectHandler"); + } + var result = idempotencyHandler.Handle().GetAwaiter().GetResult(); + return result; + } + + /// + /// Wrap as an asynchronous operation. + /// + /// + /// The target. + /// The arguments. + /// The instance containing the event data. + /// A Task<T> representing the asynchronous operation. + protected internal sealed override async Task WrapAsync( + Func> target, object[] args, AspectEventArgs eventArgs) + { + if (PowertoolsConfigurations.Instance.IdempotencyDisabled) + { + return await base.WrapAsync(target, args, eventArgs); + } + + var payload = args is not null && args.Any() ? JsonDocument.Parse(JsonSerializer.Serialize(args[0])) : null; + if (payload == null) + { + throw new IdempotencyConfigurationException("Unable to get payload from the method. Ensure there is at least one parameter or that you use @IdempotencyKey"); + } + + Task ResultDelegate() => target(args); + + var idempotencyHandler = new IdempotencyAspectHandler(ResultDelegate, eventArgs.Method.Name, payload); + if (idempotencyHandler == null) + { + throw new Exception("Failed to create an instance of IdempotencyAspectHandler"); + } + var result = await idempotencyHandler.Handle(); + return result; + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Internal/IdempotencyAspectHandler.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Internal/IdempotencyAspectHandler.cs new file mode 100644 index 00000000..0124b3b6 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Internal/IdempotencyAspectHandler.cs @@ -0,0 +1,237 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using System.Text.Json; +using System.Threading.Tasks; +using AWS.Lambda.Powertools.Idempotency.Exceptions; +using AWS.Lambda.Powertools.Idempotency.Persistence; + +namespace AWS.Lambda.Powertools.Idempotency.Internal; + +internal class IdempotencyAspectHandler +{ + /// + /// Max retries + /// + private const int MaxRetries = 2; + /// + /// Delegate to execute the calling handler + /// + private readonly Func> _target; + /// + /// Request payload + /// + private readonly JsonDocument _data; + /// + /// Persistence store + /// + private readonly BasePersistenceStore _persistenceStore; + + /// + /// IdempotencyAspectHandler constructor + /// + /// + /// + /// + public IdempotencyAspectHandler( + Func> target, + string functionName, + JsonDocument payload) + { + _target = target; + _data = payload; + _persistenceStore = Idempotency.Instance.PersistenceStore; + _persistenceStore.Configure(Idempotency.Instance.IdempotencyOptions, functionName); + } + + /// + /// Main entry point for handling idempotent execution of a function. + /// + /// function response + public async Task Handle() + { + // IdempotencyInconsistentStateException can happen under rare but expected cases + // when persistent state changes in the small time between put & get requests. + // In most cases we can retry successfully on this exception. + for (var i = 0;; i++) + { + try + { + var processIdempotency = await ProcessIdempotency(); + return processIdempotency; + } + catch (IdempotencyInconsistentStateException) + { + if (i == MaxRetries) + { + throw; + } + } + } + } + + /// + /// Process the function with idempotency + /// + /// function response + /// + private async Task ProcessIdempotency() + { + try + { + // We call saveInProgress first as an optimization for the most common case where no idempotent record + // already exists. If it succeeds, there's no need to call getRecord. + await _persistenceStore.SaveInProgress(_data, DateTimeOffset.UtcNow); + } + catch (IdempotencyItemAlreadyExistsException) + { + var record = await GetIdempotencyRecord(); + return await HandleForStatus(record); + } + catch (IdempotencyKeyException) + { + throw; + } + catch (Exception e) + { + throw new IdempotencyPersistenceLayerException( + "Failed to save in progress record to idempotency store. If you believe this is a powertools bug, please open an issue.", + e); + } + + var result = await GetFunctionResponse(); + return result; + } + + /// + /// Retrieve the idempotency record from the persistence layer. + /// + /// the record if available + /// + /// + private Task GetIdempotencyRecord() + { + try + { + return _persistenceStore.GetRecord(_data, DateTimeOffset.UtcNow); + } + catch (IdempotencyItemNotFoundException e) + { + // This code path will only be triggered if the record is removed between saveInProgress and getRecord + Console.WriteLine("An existing idempotency record was deleted before we could fetch it"); + throw new IdempotencyInconsistentStateException("saveInProgress and getRecord return inconsistent results", + e); + } + catch (IdempotencyValidationException) + { + throw; + } + catch (IdempotencyKeyException) + { + throw; + } + catch (Exception e) + { + throw new IdempotencyPersistenceLayerException( + "Failed to get record from idempotency store. If you believe this is a powertools bug, please open an issue.", + e); + } + } + + /// + /// Take appropriate action based on data_record's status + /// + /// record DataRecord + /// Function's response previously used for this idempotency key, if it has successfully executed already. + /// + /// + /// + private Task HandleForStatus(DataRecord record) + { + switch (record.Status) + { + // This code path will only be triggered if the record becomes expired between the saveInProgress call and here + case DataRecord.DataRecordStatus.EXPIRED: + throw new IdempotencyInconsistentStateException("saveInProgress and getRecord return inconsistent results"); + case DataRecord.DataRecordStatus.INPROGRESS: + throw new IdempotencyAlreadyInProgressException("Execution already in progress with idempotency key: " + + record.IdempotencyKey); + case DataRecord.DataRecordStatus.COMPLETED: + default: + try + { + var result = JsonSerializer.Deserialize(record.ResponseData!); + if (result is null) + { + throw new IdempotencyPersistenceLayerException("Unable to cast function response as " + typeof(T).Name); + } + return Task.FromResult(result); + } + catch (Exception e) + { + throw new IdempotencyPersistenceLayerException("Unable to get function response as " + typeof(T).Name, e); + } + } + } + + /// + /// Get the function's response and save it to the persistence layer + /// + /// Result from Handler delegate + /// + private async Task GetFunctionResponse() + { + T response; + try + { + response = await _target(); + } + catch(Exception handlerException) + { + // We need these nested blocks to preserve function's exception in case the persistence store operation + // also raises an exception + try + { + await _persistenceStore.DeleteRecord(_data, handlerException); + } + catch (IdempotencyKeyException) + { + throw; + } + catch (Exception e) + { + throw new IdempotencyPersistenceLayerException( + "Failed to delete record from idempotency store. If you believe this is a powertools bug, please open an issue.", + e); + } + + throw; + } + + try + { + await _persistenceStore.SaveSuccess(_data, response!, DateTimeOffset.UtcNow); + } + catch (Exception e) + { + throw new IdempotencyPersistenceLayerException( + "Failed to update record state to success in idempotency store. If you believe this is a powertools bug, please open an issue.", + e); + } + + return response; + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Internal/LRUCache.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Internal/LRUCache.cs new file mode 100644 index 00000000..3c6f898a --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Internal/LRUCache.cs @@ -0,0 +1,185 @@ + +#nullable disable + +using System.Collections.Generic; + +namespace AWS.Lambda.Powertools.Idempotency.Internal; +//source: https://github.dev/microsoft/botbuilder-dotnet/blob/main/libraries/AdaptiveExpressions/LRUCache.cs + +/// +/// A least-recently-used cache stored like a dictionary. +/// +/// The type of the key to the cached item. +/// The type of the cached item. +// ReSharper disable once InconsistentNaming +internal sealed class LRUCache +{ + /// + /// Default maximum number of elements to cache. + /// + private const int DefaultCapacity = 255; + + /// + /// Shared synchronization object + /// + private readonly object _lockObj = new(); + + /// + /// Maximum number of elements to cache. + /// + private readonly int _capacity; + + /// + /// Dictionary to record the key and its data entry (O(1)) + /// + private readonly Dictionary _cacheMap; + + /// + /// Linked list that tracks LRU items (O(1)) + /// + private readonly LinkedList _cacheList; + + /// + /// Initializes a new instance of the class. + /// + public LRUCache() + : this(DefaultCapacity) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// Maximum number of elements to cache. + public LRUCache(int capacity) + { + _capacity = capacity > 0 ? capacity : DefaultCapacity; + _cacheMap = new Dictionary(); + _cacheList = new LinkedList(); + } + + /// + /// Gets the value associated with the specified key. + /// + /// The key of the value to get. + /// When this method returns, contains the value associated with + /// the specified key, if the key is found; otherwise, the default value for the + /// type of the parameter. + /// true if contains an element with the specified key; otherwise, false. + public bool TryGet(TKey key, out TValue value) + { + lock (_lockObj) + { + if (_cacheMap.TryGetValue(key, out var entry)) + { + Touch(entry.Node); + value = entry.Value; + return true; + } + } + + value = default; + return false; + } + + /// + /// Adds the specified key and value to the cache. + /// + /// The key of the element to add. + /// The value of the element to add. + public void Set(TKey key, TValue value) + { + lock (_lockObj) + { + if (!_cacheMap.TryGetValue(key, out var entry)) + { + LinkedListNode node; + if (_cacheMap.Count >= _capacity) + { + node = _cacheList.Last; + if (node != null) + { + _cacheMap.Remove(node.Value); + _cacheList.RemoveLast(); + node.Value = key; + } + else + { + node = new LinkedListNode(key); + } + } + else + { + node = new LinkedListNode(key); + } + + _cacheList.AddFirst(node); + _cacheMap.Add(key, new Entry(node, value)); + } + else + { + entry.Value = value; + _cacheMap[key] = entry; + Touch(entry.Node); + } + } + } + + /// + /// Deletes the specified key and value to the cache. + /// + /// The key of the element to remove. + public void Delete(TKey key) + { + lock (_lockObj) + { + _cacheList.Remove(key); + _cacheMap.Remove(key); + } + } + + /// + /// Count of items in Cache + /// + public int Count + { + get + { + lock (_lockObj) + { + return _cacheList.Count; + } + } + } + + /// + /// Move to most recent spot (head) in Linked List + /// + /// + private void Touch(LinkedListNode node) + { + lock (_lockObj) + { + if (node != _cacheList.First) + { + _cacheList.Remove(node); + _cacheList.AddFirst(node); + } + } + } + + /// + /// Linked List Element + /// + private struct Entry + { + public readonly LinkedListNode Node; + public TValue Value; + + public Entry(LinkedListNode node, TValue value) + { + Node = node; + Value = value; + } + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/InternalsVisibleTo.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/InternalsVisibleTo.cs new file mode 100644 index 00000000..3d73602c --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/InternalsVisibleTo.cs @@ -0,0 +1,18 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System.Runtime.CompilerServices; + +[assembly: InternalsVisibleTo("AWS.Lambda.Powertools.Idempotency.Tests")] \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Persistence/BasePersistenceStore.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Persistence/BasePersistenceStore.cs new file mode 100644 index 00000000..8e10e632 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Persistence/BasePersistenceStore.cs @@ -0,0 +1,373 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using System.Threading.Tasks; +using AWS.Lambda.Powertools.Common; +using AWS.Lambda.Powertools.Idempotency.Exceptions; +using AWS.Lambda.Powertools.Idempotency.Internal; +using AWS.Lambda.Powertools.Idempotency.Serialization; +using DevLab.JmesPath; + +namespace AWS.Lambda.Powertools.Idempotency.Persistence; + +/// +/// Persistence layer that will store the idempotency result. +/// Base implementation. See for an implementation (default one) +/// Extend this class to use your own implementation (DocumentDB, ElastiCache, ...) +/// +public abstract class BasePersistenceStore : IPersistenceStore +{ + /// + /// Idempotency Options + /// + private IdempotencyOptions _idempotencyOptions = null!; + + /// + /// Function name + /// + private string _functionName; + /// + /// Boolean to indicate whether or not payload validation is enabled + /// + protected bool PayloadValidationEnabled; + + /// + /// LRUCache + /// + private LRUCache _cache = null!; + + /// + /// Initialize the base persistence layer from the configuration settings + /// + /// Idempotency configuration settings + /// The name of the function being decorated + public void Configure(IdempotencyOptions idempotencyOptions, string functionName) + { + var funcEnv = Environment.GetEnvironmentVariable(Constants.LambdaFunctionNameEnv); + _functionName = funcEnv ?? "testFunction"; + if (!string.IsNullOrWhiteSpace(functionName)) + { + _functionName += "." + functionName; + } + _idempotencyOptions = idempotencyOptions; + + if (!string.IsNullOrWhiteSpace(_idempotencyOptions.PayloadValidationJmesPath)) + { + PayloadValidationEnabled = true; + } + + var useLocalCache = _idempotencyOptions.UseLocalCache; + if (useLocalCache) + { + _cache = new LRUCache(_idempotencyOptions.LocalCacheMaxItems); + } + } + + /// + /// For test purpose only (adding a cache to mock) + /// + internal void Configure(IdempotencyOptions options, string functionName, LRUCache cache) + { + Configure(options, functionName); + _cache = cache; + } + + /// + /// Save record of function's execution completing successfully + /// + /// Payload + /// the response from the function + /// The current date time + public virtual async Task SaveSuccess(JsonDocument data, object result, DateTimeOffset now) + { + var responseJson = JsonSerializer.Serialize(result); + var record = new DataRecord( + GetHashedIdempotencyKey(data), + DataRecord.DataRecordStatus.COMPLETED, + GetExpiryEpochSecond(now), + responseJson, + GetHashedPayload(data) + ); + await UpdateRecord(record); + SaveToCache(record); + } + + /// + /// Save record of function's execution being in progress + /// + /// Payload + /// The current date time + /// + public virtual async Task SaveInProgress(JsonDocument data, DateTimeOffset now) + { + var idempotencyKey = GetHashedIdempotencyKey(data); + + if (RetrieveFromCache(idempotencyKey, now) != null) + { + throw new IdempotencyItemAlreadyExistsException(); + } + + var record = new DataRecord( + idempotencyKey, + DataRecord.DataRecordStatus.INPROGRESS, + GetExpiryEpochSecond(now), + null, + GetHashedPayload(data) + ); + await PutRecord(record, now); + } + + /// + /// Delete record from the persistence store + /// + /// Payload + /// The throwable thrown by the function + public virtual async Task DeleteRecord(JsonDocument data, Exception throwable) + { + var idemPotencyKey = GetHashedIdempotencyKey(data); + + Console.WriteLine("Function raised an exception {0}. " + + "Clearing in progress record in persistence store for idempotency key: {1}", + throwable.GetType().Name, + idemPotencyKey); + + await DeleteRecord(idemPotencyKey); + DeleteFromCache(idemPotencyKey); + } + + /// + /// Retrieve idempotency key for data provided, fetch from persistence store, and convert to DataRecord. + /// + /// Payload + /// + /// DataRecord representation of existing record found in persistence store + public virtual async Task GetRecord(JsonDocument data, DateTimeOffset now) + { + var idempotencyKey = GetHashedIdempotencyKey(data); + + var cachedRecord = RetrieveFromCache(idempotencyKey, now); + if (cachedRecord != null) + { + ValidatePayload(data, cachedRecord); + return cachedRecord; + } + + var record = await GetRecord(idempotencyKey); + SaveToCache(record); + ValidatePayload(data, record); + return record; + } + + /// + /// Save data_record to local cache except when status is "INPROGRESS" + /// NOTE: We can't cache "INPROGRESS" records as we have no way to reflect updates that can happen outside of the + /// execution environment + /// + /// DataRecord to save in cache + private void SaveToCache(DataRecord dataRecord) + { + if (!_idempotencyOptions.UseLocalCache) + return; + if (dataRecord.Status == DataRecord.DataRecordStatus.INPROGRESS) + return; + + _cache.Set(dataRecord.IdempotencyKey, dataRecord); + } + + /// + /// Validate that the hashed payload matches data provided and stored data record + /// + /// Payload + /// DataRecord instance + /// + private void ValidatePayload(JsonDocument data, DataRecord dataRecord) + { + if (!PayloadValidationEnabled) return; + var dataHash = GetHashedPayload(data); + + if (dataHash != dataRecord.PayloadHash) + { + throw new IdempotencyValidationException("Payload does not match stored record for this event key"); + } + } + + /// + /// Retrieve data record from cache + /// + /// Idempotency key + /// DateTime Offset + /// DataRecord instance + private DataRecord RetrieveFromCache(string idempotencyKey, DateTimeOffset now) + { + if (!_idempotencyOptions.UseLocalCache) + return null; + + if (_cache.TryGet(idempotencyKey, out var record) && record!=null) + { + if (!record.IsExpired(now)) + { + return record; + } + DeleteFromCache(idempotencyKey); + } + return null; + } + + /// + /// Deletes item from cache + /// + /// + private void DeleteFromCache(string idempotencyKey) + { + if (!_idempotencyOptions.UseLocalCache) + return; + + _cache.Delete(idempotencyKey); + } + + /// + /// Extract payload using validation key jmespath and return a hashed representation + /// + /// Payload + /// Hashed representation of the data extracted by the jmespath expression + private string GetHashedPayload(JsonDocument data) + { + if (!PayloadValidationEnabled) + { + return ""; + } + + var jmes = new JmesPath(); + jmes.FunctionRepository.Register(); + var result = jmes.Transform(data.RootElement.ToString(), _idempotencyOptions.PayloadValidationJmesPath); + var node = JsonDocument.Parse(result); + return GenerateHash(node.RootElement); + } + + + + /// + /// Calculate unix timestamp of expiry date for idempotency record + /// + /// + /// unix timestamp of expiry date for idempotency record + private long GetExpiryEpochSecond(DateTimeOffset now) + { + return now.AddSeconds(_idempotencyOptions.ExpirationInSeconds).ToUnixTimeSeconds(); + } + + /// + /// Extract idempotency key and return a hashed representation + /// + /// incoming data + /// Hashed representation of the data extracted by the jmespath expression + /// + private string GetHashedIdempotencyKey(JsonDocument data) + { + var node = data.RootElement; + var eventKeyJmesPath = _idempotencyOptions.EventKeyJmesPath; + if (eventKeyJmesPath != null) + { + var jmes = new JmesPath(); + jmes.FunctionRepository.Register(); + var result = jmes.Transform(node.ToString(), eventKeyJmesPath); + node = JsonDocument.Parse(result).RootElement; + } + + if (IsMissingIdemPotencyKey(node)) + { + if (_idempotencyOptions.ThrowOnNoIdempotencyKey) + { + throw new IdempotencyKeyException("No data found to create a hashed idempotency key"); + } + Console.WriteLine("No data found to create a hashed idempotency key. JMESPath: {0}", _idempotencyOptions.EventKeyJmesPath ?? string.Empty); + } + + var hash = GenerateHash(node); + return _functionName + "#" + hash; + } + + /// + /// Check if the provided data is missing an idempotency key. + /// + /// + /// True if the Idempotency key is missing + private static bool IsMissingIdemPotencyKey(JsonElement data) + { + return data.ValueKind == JsonValueKind.Null || data.ValueKind == JsonValueKind.Undefined + || (data.ValueKind == JsonValueKind.String && data.ToString() == string.Empty); + } + + /// + /// Generate a hash value from the provided data + /// + /// data to hash + /// Hashed representation of the provided data + /// + internal string GenerateHash(JsonElement data) + { + using var hashAlgorithm = HashAlgorithm.Create(_idempotencyOptions.HashFunction); + if (hashAlgorithm == null) + { + throw new ArgumentException("Invalid HashAlgorithm"); + } + var stringToHash = data.ToString(); + var hash = GetHash(hashAlgorithm, stringToHash); + + return hash; + } + + /// + /// Get a hash of the provided string using the specified hash algorithm + /// + /// + /// + /// Hashed representation of the provided string + private static string GetHash(HashAlgorithm hashAlgorithm, string input) + { + // Convert the input string to a byte array and compute the hash. + var data = hashAlgorithm.ComputeHash(Encoding.UTF8.GetBytes(input)); + + // Create a new Stringbuilder to collect the bytes + // and create a string. + var sBuilder = new StringBuilder(); + + // Loop through each byte of the hashed data + // and format each one as a hexadecimal string. + for (var i = 0; i < data.Length; i++) + { + sBuilder.Append(data[i].ToString("x2")); + } + + // Return the hexadecimal string. + return sBuilder.ToString(); + } + + /// + public abstract Task GetRecord(string idempotencyKey); + + /// + public abstract Task PutRecord(DataRecord record, DateTimeOffset now); + + /// + public abstract Task UpdateRecord(DataRecord record); + + /// + public abstract Task DeleteRecord(string idempotencyKey); +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Persistence/DataRecord.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Persistence/DataRecord.cs new file mode 100644 index 00000000..c35d9626 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Persistence/DataRecord.cs @@ -0,0 +1,137 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; + +namespace AWS.Lambda.Powertools.Idempotency.Persistence; + +/// +/// Data Class for idempotency records. This is actually the item that will be stored in the persistence layer. +/// +public class DataRecord +{ + /// + /// Status + /// + private readonly string _status; + + /// + /// Creates a new DataRecord + /// + /// Hash representation of either entire event or specific configured subject of the event + /// The DataRecordStatus + /// Unix timestamp of when record expires + /// JSON serialized invocation results + /// A hash representation of the entire event + public DataRecord(string idempotencyKey, + DataRecordStatus status, + long expiryTimestamp, + string responseData, + string payloadHash) + { + IdempotencyKey = idempotencyKey; + _status = status.ToString(); + ExpiryTimestamp = expiryTimestamp; + ResponseData = responseData; + PayloadHash = payloadHash; + } + + /// + /// A hash representation of either the entire event or a specific configured subset of the event + /// + public string IdempotencyKey { get; } + /// + /// Unix timestamp of when record expires + /// + public long ExpiryTimestamp { get; } + /// + /// JSON serialized invocation results + /// + public string ResponseData { get; } + /// + /// A hash representation of the entire event + /// + public string PayloadHash { get; } + + + /// + /// Check if data record is expired (based on expiration configured in the IdempotencyConfig + /// + /// + /// Whether the record is currently expired or not + public bool IsExpired(DateTimeOffset now) + { + return ExpiryTimestamp != 0 && now.ToUnixTimeSeconds() > ExpiryTimestamp; + } + + /// + /// Represents the Status + /// + public DataRecordStatus Status => + IsExpired(DateTimeOffset.UtcNow) ? DataRecordStatus.EXPIRED : Enum.Parse(_status); + + /// + /// Determines whether the specified DataRecord is equal to the current DataRecord + /// + /// The DataRecord to compare with the current object. + /// true if the specified DataRecord is equal to the current DataRecord; otherwise, false. + private bool Equals(DataRecord other) + { + return _status == other._status + && IdempotencyKey == other.IdempotencyKey + && ExpiryTimestamp == other.ExpiryTimestamp + && ResponseData == other.ResponseData + && PayloadHash == other.PayloadHash; + } + + /// + public override bool Equals(object obj) + { + if (ReferenceEquals(null, obj)) return false; + if (ReferenceEquals(this, obj)) return true; + if (obj.GetType() != GetType()) return false; + return Equals((DataRecord) obj); + } + + /// + public override int GetHashCode() + { + return HashCode.Combine(IdempotencyKey, _status, ExpiryTimestamp, ResponseData, PayloadHash); + } + + /// + /// Status of the record: + /// -- INPROGRESS: record initialized when function starts + /// -- COMPLETED: record updated with the result of the function when it ends + /// -- EXPIRED: record expired, idempotency will not happen + /// + public enum DataRecordStatus { + /// + /// record initialized when function starts + /// + // ReSharper disable once InconsistentNaming + INPROGRESS, + /// + /// record updated with the result of the function when it ends + /// + // ReSharper disable once InconsistentNaming + COMPLETED, + /// + /// record expired, idempotency will not happen + /// + // ReSharper disable once InconsistentNaming + EXPIRED + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Persistence/DynamoDBPersistenceStore.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Persistence/DynamoDBPersistenceStore.cs new file mode 100644 index 00000000..bec8f308 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Persistence/DynamoDBPersistenceStore.cs @@ -0,0 +1,457 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using System.Collections.Generic; +using System.Threading.Tasks; +using Amazon; +using Amazon.DynamoDBv2; +using Amazon.DynamoDBv2.Model; +using AWS.Lambda.Powertools.Common; +using AWS.Lambda.Powertools.Idempotency.Exceptions; + +namespace AWS.Lambda.Powertools.Idempotency.Persistence; + +/// +/// DynamoDB version of the . Will store idempotency data in DynamoDB. +/// +// ReSharper disable once InconsistentNaming +public class DynamoDBPersistenceStore : BasePersistenceStore +{ + /// + /// DynamoDB table name + /// + private readonly string _tableName; + /// + /// Key attribute + /// + private readonly string _keyAttr; + /// + /// Static partition key value + /// + private readonly string _staticPkValue; + /// + /// Sort key attribute + /// + private readonly string _sortKeyAttr; + /// + /// Expiry attribute + /// + private readonly string _expiryAttr; + /// + /// Status attribute + /// + private readonly string _statusAttr; + /// + /// Data / Payload attribute + /// + private readonly string _dataAttr; + /// + /// Validation attribute + /// + private readonly string _validationAttr; + /// + /// DynamoDB client + /// + private readonly AmazonDynamoDBClient _dynamoDbClient; + + /// + /// Creates a new instance of . + /// + /// + /// + /// + /// + /// + /// + /// + /// + /// + internal DynamoDBPersistenceStore(string tableName, + string keyAttr, + string staticPkValue, + string sortKeyAttr, + string expiryAttr, + string statusAttr, + string dataAttr, + string validationAttr, + AmazonDynamoDBClient client) + { + _tableName = tableName; + _keyAttr = keyAttr; + _staticPkValue = staticPkValue; + _sortKeyAttr = sortKeyAttr; + _expiryAttr = expiryAttr; + _statusAttr = statusAttr; + _dataAttr = dataAttr; + _validationAttr = validationAttr; + + if (client != null) + { + _dynamoDbClient = client; + } + else + { + if (PowertoolsConfigurations.Instance.IdempotencyDisabled) + { + // we do not want to create a DynamoDbClient if idempotency is disabled + // null is ok as idempotency won't be called + _dynamoDbClient = null; + + } else { + var clientConfig = new AmazonDynamoDBConfig + { + RegionEndpoint = RegionEndpoint.GetBySystemName(Environment.GetEnvironmentVariable(Constants.AwsRegionEnv)) + }; + _dynamoDbClient = new AmazonDynamoDBClient(clientConfig); + } + } + } + + + /// + public override async Task GetRecord(string idempotencyKey) + { + var getItemRequest = new GetItemRequest + { + TableName = _tableName, + ConsistentRead = true, + Key = GetKey(idempotencyKey) + }; + var response = await _dynamoDbClient!.GetItemAsync(getItemRequest); + + if (!response.IsItemSet) + { + throw new IdempotencyItemNotFoundException(idempotencyKey); + } + + return ItemToRecord(response.Item); + } + + + /// + public override async Task PutRecord(DataRecord record, DateTimeOffset now) + { + Dictionary item = new(GetKey(record.IdempotencyKey)) + { + { + _expiryAttr, new AttributeValue + { + N = record.ExpiryTimestamp.ToString() + } + }, + { _statusAttr, new AttributeValue(record.Status.ToString()) } + }; + + if (PayloadValidationEnabled) + { + item.Add(_validationAttr, new AttributeValue(record.PayloadHash)); + } + + try + { + var expressionAttributeNames = new Dictionary + { + {"#id", _keyAttr}, + {"#expiry", _expiryAttr} + }; + + var request = new PutItemRequest + { + TableName = _tableName, + Item = item, + ConditionExpression = "attribute_not_exists(#id) OR #expiry < :now", + ExpressionAttributeNames = expressionAttributeNames, + ExpressionAttributeValues = new Dictionary + { + {":now", new AttributeValue {N = now.ToUnixTimeSeconds().ToString()}} + } + }; + await _dynamoDbClient!.PutItemAsync(request); + } + catch (ConditionalCheckFailedException e) + { + throw new IdempotencyItemAlreadyExistsException( + "Failed to put record for already existing idempotency key: " + record.IdempotencyKey, e); + } + } + + + /// + public override async Task UpdateRecord(DataRecord record) + { + var updateExpression = "SET #response_data = :response_data, #expiry = :expiry, #status = :status"; + + var expressionAttributeNames = new Dictionary + { + {"#response_data", _dataAttr}, + {"#expiry", _expiryAttr}, + {"#status", _statusAttr} + }; + + var expressionAttributeValues = new Dictionary + { + {":response_data", new AttributeValue(record.ResponseData)}, + {":expiry", new AttributeValue {N=record.ExpiryTimestamp.ToString()}}, + {":status", new AttributeValue(record.Status.ToString())} + }; + + if (PayloadValidationEnabled) + { + updateExpression += ", #validation_key = :validation_key"; + expressionAttributeNames.Add("#validation_key", _validationAttr); + expressionAttributeValues.Add(":validation_key", new AttributeValue(record.PayloadHash)); + } + + var request = new UpdateItemRequest + { + TableName = _tableName, + Key = GetKey(record.IdempotencyKey), + UpdateExpression = updateExpression, + ExpressionAttributeNames = expressionAttributeNames, + ExpressionAttributeValues = expressionAttributeValues + }; + await _dynamoDbClient!.UpdateItemAsync(request); + } + + /// + public override async Task DeleteRecord(string idempotencyKey) + { + var request = new DeleteItemRequest + { + TableName = _tableName, + Key = GetKey(idempotencyKey) + }; + await _dynamoDbClient!.DeleteItemAsync(request); + } + + /// + /// Translate raw item records from DynamoDB to DataRecord + /// + /// item Item from dynamodb response + /// DataRecord instance + private DataRecord ItemToRecord(Dictionary item) + { + // data and validation payload may be null + var hasDataAttribute = item.TryGetValue(_dataAttr, out var data); + var hasValidationAttribute = item.TryGetValue(_validationAttr, out var validation); + + return new DataRecord(item[_sortKeyAttr ?? _keyAttr].S, + Enum.Parse(item[_statusAttr].S), + long.Parse(item[_expiryAttr].N), + hasDataAttribute ? data?.S : null, + hasValidationAttribute ? validation?.S : null); + } + + /// + /// Get the key to use for requests (depending on if we have a sort key or not) + /// + /// idempotencyKey + /// + private Dictionary GetKey(string idempotencyKey) + { + Dictionary key = new(); + if (_sortKeyAttr != null) + { + key[_keyAttr] = new AttributeValue(_staticPkValue); + key[_sortKeyAttr] = new AttributeValue(idempotencyKey); + } + else + { + key[_keyAttr] = new AttributeValue(idempotencyKey); + } + + return key; + } + +} + +/// +/// Use this builder to get an instance of .
+/// With this builder you can configure the characteristics of the DynamoDB Table +/// (name, key, sort key, and other field names).
+/// You can also set a custom AmazonDynamoDBClient for further tuning. +/// +// ReSharper disable once InconsistentNaming +public class DynamoDBPersistenceStoreBuilder +{ + /// + /// Lambda Function Name + /// + private static readonly string FuncEnv = Environment.GetEnvironmentVariable(Constants.LambdaFunctionNameEnv); + /// + /// DynamoDB table name + /// + private string _tableName = null!; + /// + /// Key attribute + /// + private string _keyAttr = "id"; + /// + /// Static partition key value + /// + private string _staticPkValue = $"idempotency#{FuncEnv}"; + /// + /// Sort key attribute + /// + private string _sortKeyAttr; + /// + /// Expiry attribute + /// + private string _expiryAttr = "expiration"; + /// + /// Status attribute + /// + private string _statusAttr = "status"; + /// + /// Data / Payload attribute + /// + private string _dataAttr = "data"; + /// + /// Validation attribute + /// + private string _validationAttr = "validation"; + /// + /// DynamoDB client + /// + private AmazonDynamoDBClient _dynamoDbClient; + + /// + /// Initialize and return a new instance of {@link DynamoDBPersistenceStore}. + /// Example: + /// DynamoDBPersistenceStore.builder().withTableName("idempotency_store").build(); + /// + /// + /// + public DynamoDBPersistenceStore Build() + { + if (string.IsNullOrWhiteSpace(_tableName)) + { + throw new ArgumentNullException($"Table name is not specified"); + } + + return new DynamoDBPersistenceStore(_tableName, + _keyAttr, + _staticPkValue, + _sortKeyAttr, + _expiryAttr, + _statusAttr, + _dataAttr, + _validationAttr, + _dynamoDbClient); + } + + /// + /// Name of the table to use for storing execution records (mandatory) + /// + /// tableName Name of the DynamoDB table + /// the builder instance (to chain operations) + public DynamoDBPersistenceStoreBuilder WithTableName(string tableName) + { + _tableName = tableName; + return this; + } + + /// + /// DynamoDB attribute name for partition key (optional), by default "id" + /// + /// keyAttr name of the key attribute in the table + /// the builder instance (to chain operations) + public DynamoDBPersistenceStoreBuilder WithKeyAttr(string keyAttr) + { + _keyAttr = keyAttr; + return this; + } + + /// + /// DynamoDB attribute value for partition key (optional), by default "idempotency#[function-name]". + /// This will be used if the {@link #sortKeyAttr} is set. + /// + /// staticPkValue name of the partition key attribute in the table + /// the builder instance (to chain operations) + public DynamoDBPersistenceStoreBuilder WithStaticPkValue(string staticPkValue) + { + _staticPkValue = staticPkValue; + return this; + } + + /// + /// DynamoDB attribute name for the sort key (optional) + /// + /// sortKeyAttr name of the sort key attribute in the table + /// the builder instance (to chain operations) + public DynamoDBPersistenceStoreBuilder WithSortKeyAttr(string sortKeyAttr) + { + _sortKeyAttr = sortKeyAttr; + return this; + } + + /// + /// DynamoDB attribute name for expiry timestamp (optional), by default "expiration" + /// + /// expiryAttr name of the expiry attribute in the table + /// the builder instance (to chain operations) + public DynamoDBPersistenceStoreBuilder WithExpiryAttr(string expiryAttr) + { + _expiryAttr = expiryAttr; + return this; + } + + /// + /// DynamoDB attribute name for status (optional), by default "status" + /// + /// statusAttr name of the status attribute in the table + /// the builder instance (to chain operations) + public DynamoDBPersistenceStoreBuilder WithStatusAttr(string statusAttr) + { + _statusAttr = statusAttr; + return this; + } + + /// + /// DynamoDB attribute name for response data (optional), by default "data" + /// + /// dataAttr name of the data attribute in the table + /// the builder instance (to chain operations) + public DynamoDBPersistenceStoreBuilder WithDataAttr(string dataAttr) + { + _dataAttr = dataAttr; + return this; + } + + /// + /// DynamoDB attribute name for validation (optional), by default "validation" + /// + /// validationAttr name of the validation attribute in the table + /// the builder instance (to chain operations) + public DynamoDBPersistenceStoreBuilder WithValidationAttr(string validationAttr) + { + _validationAttr = validationAttr; + return this; + } + + /// + /// Custom DynamoDbClient used to query DynamoDB (optional). + /// The default one uses UrlConnectionHttpClient as a http client and + /// + /// dynamoDbClient the DynamoDbClient instance to use + /// the builder instance (to chain operations) + // ReSharper disable once InconsistentNaming + public DynamoDBPersistenceStoreBuilder WithDynamoDBClient(AmazonDynamoDBClient dynamoDbClient) + { + _dynamoDbClient = dynamoDbClient; + return this; + } +} \ No newline at end of file diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Persistence/IPersistenceStore.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Persistence/IPersistenceStore.cs new file mode 100644 index 00000000..7e00371c --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Persistence/IPersistenceStore.cs @@ -0,0 +1,57 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using System.Threading.Tasks; +using AWS.Lambda.Powertools.Idempotency.Exceptions; + +namespace AWS.Lambda.Powertools.Idempotency.Persistence; + +/// +/// Persistence layer that will store the idempotency result. +/// In order to provide another implementation, extends {@link BasePersistenceStore}. +/// +public interface IPersistenceStore +{ + + /// + /// Retrieve item from persistence store using idempotency key and return it as a DataRecord instance. + /// + /// idempotencyKey the key of the record + /// DataRecord representation of existing record found in persistence store + /// Exception thrown if no record exists in persistence store with the idempotency key + Task GetRecord(string idempotencyKey); + + /// + /// Add a DataRecord to persistence store if it does not already exist with that key + /// + /// record DataRecord instance + /// + /// + /// if a non-expired entry already exists. + Task PutRecord(DataRecord record, DateTimeOffset now); + + /// + /// Update item in persistence store + /// + /// DataRecord instance + Task UpdateRecord(DataRecord record); + + /// + /// Remove item from persistence store + /// + /// idempotencyKey the key of the record + Task DeleteRecord(string idempotencyKey); +} diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/README.md b/libraries/src/AWS.Lambda.Powertools.Idempotency/README.md new file mode 100644 index 00000000..2ebf57fc --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/README.md @@ -0,0 +1,39 @@ +# AWS Lambda Idempotency for .NET + +The idempotency package provides a simple solution to convert your Lambda functions into idempotent operations which +are safe to retry. + +## Terminology + +The property of idempotency means that an operation does not cause additional side effects if it is called more than +once with the same input parameters. + +**Idempotent operations will return the same result when they are called multiple +times with the same parameters**. This makes idempotent operations safe to retry. [Read more](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) about idempotency. + +**Idempotency key** is a hash representation of either the entire event or a specific configured subset of the event, and invocation results are **JSON serialized** and stored in your persistence storage layer. + + +## Key features + +* Prevent Lambda handler function from executing more than once on the same event payload during a time window +* Ensure Lambda handler returns the same result when called with the same payload +* Select a subset of the event as the idempotency key using JMESPath expressions +* Set a time window in which records with the same payload should be considered duplicates + + +## Installation +You should install with NuGet: + +``` +Install-Package Amazon.Lambda.PowerTools.Idempotency +``` + +Or via the .NET Core command line interface: + +``` +dotnet add package Amazon.Lambda.PowerTools.Idempotency +``` + +## Acknowledgment +This project has been ported from the Java Idempotency PowerTool Utility diff --git a/libraries/src/AWS.Lambda.Powertools.Idempotency/Serialization/JsonFunction.cs b/libraries/src/AWS.Lambda.Powertools.Idempotency/Serialization/JsonFunction.cs new file mode 100644 index 00000000..94ed3f25 --- /dev/null +++ b/libraries/src/AWS.Lambda.Powertools.Idempotency/Serialization/JsonFunction.cs @@ -0,0 +1,42 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System.Diagnostics; +using DevLab.JmesPath.Functions; +using Newtonsoft.Json.Linq; + +namespace AWS.Lambda.Powertools.Idempotency.Serialization; + +/// +/// Creates JMESPath function powertools_json() to treat the payload as a JSON object rather than a string. +/// +public class JsonFunction : JmesPathFunction +{ + /// + public JsonFunction() + : base("powertools_json", 1) + { + } + + /// + public override JToken Execute(params JmesPathFunctionArgument[] args) + { + Debug.Assert(args.Length == 1); + Debug.Assert(args[0].IsToken); + var argument = args[0]; + var token = argument.Token; + return JToken.Parse(token.ToString()); + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/AWS.Lambda.Powertools.Idempotency.Tests.csproj b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/AWS.Lambda.Powertools.Idempotency.Tests.csproj new file mode 100644 index 00000000..3f8084de --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/AWS.Lambda.Powertools.Idempotency.Tests.csproj @@ -0,0 +1,45 @@ + + + + net6.0 + default + AWS.Lambda.Powertools.Idempotency.Tests + AWS.Lambda.Powertools.Idempotency.Tests + + + + + + + + + + + + + runtime; build; native; contentfiles; analyzers; buildtransitive + all + + + runtime; build; native; contentfiles; analyzers; buildtransitive + all + + + + + + + + + + + + PreserveNewest + + + + Always + + + + diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Handlers/IdempotencyEnabledFunction.cs b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Handlers/IdempotencyEnabledFunction.cs new file mode 100644 index 00000000..5ba3e816 --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Handlers/IdempotencyEnabledFunction.cs @@ -0,0 +1,68 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System.Threading.Tasks; +using Amazon.Lambda.Core; +using AWS.Lambda.Powertools.Idempotency.Tests.Model; + +namespace AWS.Lambda.Powertools.Idempotency.Tests.Handlers; + + +public interface IIdempotencyEnabledFunction +{ + public bool HandlerExecuted { get; set; } + Task HandleTest(Product input, ILambdaContext context); +} + +public class IdempotencyEnabledFunction : IIdempotencyEnabledFunction +{ + [Idempotent] + public async Task Handle(Product input, ILambdaContext context) + { + HandlerExecuted = true; + var basket = new Basket(); + basket.Add(input); + var result = Task.FromResult(basket); + + return await result; + } + + public bool HandlerExecuted { get; set; } + + public Task HandleTest(Product input, ILambdaContext context) + { + return Handle(input, context); + } +} + +public class IdempotencyEnabledSyncFunction : IIdempotencyEnabledFunction +{ + [Idempotent] + public Basket Handle(Product input, ILambdaContext context) + { + HandlerExecuted = true; + var basket = new Basket(); + basket.Add(input); + + return basket; + } + + public bool HandlerExecuted { get; set; } + + public Task HandleTest(Product input, ILambdaContext context) + { + return Task.FromResult(Handle(input, context)); + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Handlers/IdempotencyFunction.cs b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Handlers/IdempotencyFunction.cs new file mode 100644 index 00000000..5c500f05 --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Handlers/IdempotencyFunction.cs @@ -0,0 +1,100 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Net.Http; +using System.Text.Json; +using System.Threading.Tasks; +using Amazon.DynamoDBv2; +using Amazon.Lambda.APIGatewayEvents; + +namespace AWS.Lambda.Powertools.Idempotency.Tests.Handlers; + +public class IdempotencyFunction +{ + public bool HandlerExecuted; + + public IdempotencyFunction(AmazonDynamoDBClient client) + { + Idempotency.Configure(builder => + builder + .WithOptions(optionsBuilder => + optionsBuilder + .WithEventKeyJmesPath("powertools_json(Body).address") + .WithExpiration(TimeSpan.FromSeconds(20))) + .UseDynamoDb(storeBuilder => + storeBuilder + .WithTableName("idempotency_table") + .WithDynamoDBClient(client) + )); + } + + [Idempotent] + public async Task Handle(APIGatewayProxyRequest apigProxyEvent) + { + HandlerExecuted = true; + + var result= await InternalFunctionHandler(apigProxyEvent); + + return result; + } + private async Task InternalFunctionHandler(APIGatewayProxyRequest apigProxyEvent) + { + Dictionary headers = new() + { + {"Content-Type", "application/json"}, + {"Access-Control-Allow-Origin", "*"}, + {"Access-Control-Allow-Methods", "GET, OPTIONS"}, + {"Access-Control-Allow-Headers", "*"} + }; + + try + { + var address = JsonDocument.Parse(apigProxyEvent.Body).RootElement.GetProperty("address").GetString(); + var pageContents = await GetPageContents(address); + var output = $"{{ \"message\": \"hello world\", \"location\": \"{pageContents}\" }}"; + + return new APIGatewayProxyResponse + { + Body = output, + StatusCode = 200, + Headers = headers + }; + + } + catch (IOException) + { + return new APIGatewayProxyResponse + { + Body = "{}", + StatusCode = 500, + Headers = headers + }; + } + } + + // we could actually also put the @Idempotent annotation here + private async Task GetPageContents(string address) + { + var client = new HttpClient(); + using var response = await client.GetAsync(address); + using var content = response.Content; + var pageContent = await content.ReadAsStringAsync(); + + return pageContent; + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Handlers/IdempotencyWithErrorFunction.cs b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Handlers/IdempotencyWithErrorFunction.cs new file mode 100644 index 00000000..8dbcd331 --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Handlers/IdempotencyWithErrorFunction.cs @@ -0,0 +1,50 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using System.Threading.Tasks; +using Amazon.Lambda.Core; +using AWS.Lambda.Powertools.Idempotency.Tests.Model; + +namespace AWS.Lambda.Powertools.Idempotency.Tests.Handlers; + +public interface IIdempotencyWithErrorFunction +{ + Task HandleTest(Product input, ILambdaContext context); +} + +public class IdempotencyWithErrorFunction : IIdempotencyWithErrorFunction +{ + [Idempotent] + public Task Handle(Product input, ILambdaContext context) + => throw new IndexOutOfRangeException("Fake exception"); + + public Task HandleTest(Product input, ILambdaContext context) + { + return Handle(input, context); + } +} + +public class IdempotencyWithErrorSyncFunction : IIdempotencyWithErrorFunction +{ + [Idempotent] + public Basket Handle(Product input, ILambdaContext context) + => throw new IndexOutOfRangeException("Fake exception"); + + public Task HandleTest(Product input, ILambdaContext context) + { + return Task.FromResult(Handle(input, context)); + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/IdempotencyTest.cs b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/IdempotencyTest.cs new file mode 100644 index 00000000..c0170682 --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/IdempotencyTest.cs @@ -0,0 +1,70 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System.IO; +using System.Text.Json; +using System.Threading.Tasks; +using Amazon.DynamoDBv2; +using Amazon.DynamoDBv2.Model; +using Amazon.Lambda.APIGatewayEvents; +using AWS.Lambda.Powertools.Idempotency.Tests.Handlers; +using AWS.Lambda.Powertools.Idempotency.Tests.Persistence; +using FluentAssertions; +using Xunit; + +namespace AWS.Lambda.Powertools.Idempotency.Tests; + +public class IdempotencyTest : IClassFixture +{ + private readonly AmazonDynamoDBClient _client; + private readonly string _tableName; + + public IdempotencyTest(DynamoDbFixture fixture) + { + _client = fixture.Client; + _tableName = fixture.TableName; + } + + [Fact] + [Trait("Category", "Integration")] + public async Task EndToEndTest() + { + var function = new IdempotencyFunction(_client); + + var options = new JsonSerializerOptions + { + PropertyNameCaseInsensitive = true + }; + + var request = JsonSerializer.Deserialize(await File.ReadAllTextAsync("./resources/apigw_event2.json"),options); + + var response = await function.Handle(request); + function.HandlerExecuted.Should().BeTrue(); + + function.HandlerExecuted = false; + + var response2 = await function.Handle(request); + function.HandlerExecuted.Should().BeFalse(); + + JsonSerializer.Serialize(response).Should().Be(JsonSerializer.Serialize(response)); + response2.Body.Should().Contain("hello world"); + + var scanResponse = await _client.ScanAsync(new ScanRequest + { + TableName = _tableName + }); + scanResponse.Count.Should().Be(1); + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Internal/IdempotentAspectTests.cs b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Internal/IdempotentAspectTests.cs new file mode 100644 index 00000000..b850d82a --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Internal/IdempotentAspectTests.cs @@ -0,0 +1,228 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using System.Text.Json; +using System.Threading.Tasks; +using Amazon.Lambda.TestUtilities; +using AWS.Lambda.Powertools.Common; +using AWS.Lambda.Powertools.Idempotency.Exceptions; +using AWS.Lambda.Powertools.Idempotency.Persistence; +using AWS.Lambda.Powertools.Idempotency.Tests.Handlers; +using AWS.Lambda.Powertools.Idempotency.Tests.Model; +using FluentAssertions; +using Moq; +using Xunit; + +[assembly: CollectionBehavior(DisableTestParallelization = true)] + +namespace AWS.Lambda.Powertools.Idempotency.Tests.Internal; + +[Collection("Sequential")] +public class IdempotentAspectTests : IDisposable +{ + [Theory] + [InlineData(typeof(IdempotencyEnabledFunction))] + [InlineData(typeof(IdempotencyEnabledSyncFunction))] + public async Task Handle_WhenFirstCall_ShouldPutInStore(Type type) + { + //Arrange + var store = new Mock(); + Idempotency.Configure(builder => + builder + .WithPersistenceStore(store.Object) + .WithOptions(optionsBuilder => optionsBuilder.WithEventKeyJmesPath("Id")) + ); + + var function = Activator.CreateInstance(type) as IIdempotencyEnabledFunction; + var product = new Product(42, "fake product", 12); + + //Act + var basket = await function!.HandleTest(product, new TestLambdaContext()); + + //Assert + basket.Products.Count.Should().Be(1); + function.HandlerExecuted.Should().BeTrue(); + + store + .Verify(x=>x.SaveInProgress(It.Is(t=> t.ToString() == JsonSerializer.SerializeToDocument(product, It.IsAny()).ToString()), It.IsAny())); + + store + .Verify(x=>x.SaveSuccess(It.IsAny(), It.Is(y => y.Equals(basket)), It.IsAny())); + } + + [Theory] + [InlineData(typeof(IdempotencyEnabledFunction))] + [InlineData(typeof(IdempotencyEnabledSyncFunction))] + public async Task Handle_WhenSecondCall_AndNotExpired_ShouldGetFromStore(Type type) + { + //Arrange + var store = new Mock(); + store.Setup(x=>x.SaveInProgress(It.IsAny(), It.IsAny())) + .Throws(); + + // GIVEN + Idempotency.Configure(builder => + builder + .WithPersistenceStore(store.Object) + .WithOptions(optionsBuilder => optionsBuilder.WithEventKeyJmesPath("Id")) + ); + + var product = new Product(42, "fake product", 12); + var basket = new Basket(product); + var record = new DataRecord( + "42", + DataRecord.DataRecordStatus.COMPLETED, + DateTimeOffset.UtcNow.AddSeconds(356).ToUnixTimeSeconds(), + JsonSerializer.SerializeToNode(basket)!.ToString(), + null); + store.Setup(x=>x.GetRecord(It.IsAny(), It.IsAny())) + .ReturnsAsync(record); + + var function = Activator.CreateInstance(type) as IIdempotencyEnabledFunction; + + // Act + var resultBasket = await function!.HandleTest(product, new TestLambdaContext()); + + // Assert + resultBasket.Should().Be(basket); + function.HandlerExecuted.Should().BeFalse(); + } + + [Theory] + [InlineData(typeof(IdempotencyEnabledFunction))] + [InlineData(typeof(IdempotencyEnabledSyncFunction))] + public async Task Handle_WhenSecondCall_AndStatusInProgress_ShouldThrowIdempotencyAlreadyInProgressException(Type type) + { + // Arrange + var store = new Mock(); + + Idempotency.Configure(builder => + builder + .WithPersistenceStore(store.Object) + .WithOptions(optionsBuilder => optionsBuilder.WithEventKeyJmesPath("Id")) + ); + store.Setup(x=>x.SaveInProgress(It.IsAny(), It.IsAny())) + .Throws(); + + var product = new Product(42, "fake product", 12); + var basket = new Basket(product); + var record = new DataRecord( + "42", + DataRecord.DataRecordStatus.INPROGRESS, + DateTimeOffset.UtcNow.AddSeconds(356).ToUnixTimeSeconds(), + JsonSerializer.SerializeToNode(basket)!.ToString(), + null); + store.Setup(x=>x.GetRecord(It.IsAny(), It.IsAny())) + .ReturnsAsync(record); + + // Act + var function = Activator.CreateInstance(type) as IIdempotencyEnabledFunction; + Func act = async () => await function!.HandleTest(product, new TestLambdaContext()); + + // Assert + await act.Should().ThrowAsync(); + } + + [Theory] + [InlineData(typeof(IdempotencyWithErrorFunction))] + [InlineData(typeof(IdempotencyWithErrorSyncFunction))] + public async Task Handle_WhenThrowException_ShouldDeleteRecord_AndThrowFunctionException(Type type) + { + // Arrange + var store = new Mock(); + + Idempotency.Configure(builder => + builder + .WithPersistenceStore(store.Object) + .WithOptions(optionsBuilder => optionsBuilder.WithEventKeyJmesPath("Id")) + ); + + var function = Activator.CreateInstance(type) as IIdempotencyWithErrorFunction; + var product = new Product(42, "fake product", 12); + + // Act + Func act = async () => await function!.HandleTest(product, new TestLambdaContext()); + + // Assert + await act.Should().ThrowAsync(); + store.Verify( + x => x.DeleteRecord(It.IsAny(), It.IsAny())); + } + + [Theory] + [InlineData(typeof(IdempotencyEnabledFunction))] + [InlineData(typeof(IdempotencyEnabledSyncFunction))] + public async Task Handle_WhenIdempotencyDisabled_ShouldJustRunTheFunction(Type type) + { + + // Arrange + var store = new Mock(); + + Environment.SetEnvironmentVariable(Constants.IdempotencyDisabledEnv, "true"); + + Idempotency.Configure(builder => + builder + .WithPersistenceStore(store.Object) + .WithOptions(optionsBuilder => optionsBuilder.WithEventKeyJmesPath("Id")) + ); + + var function = Activator.CreateInstance(type) as IIdempotencyEnabledFunction; + var product = new Product(42, "fake product", 12); + + // Act + var basket = await function!.HandleTest(product, new TestLambdaContext()); + + // Assert + store.Invocations.Count.Should().Be(0); + basket.Products.Count.Should().Be(1); + function.HandlerExecuted.Should().BeTrue(); + } + + [Fact] + public void Idempotency_Set_Execution_Environment_Context() + { + // Arrange + var assemblyName = "AWS.Lambda.Powertools.Idempotency"; + var assemblyVersion = "1.0.0"; + + var env = new Mock(); + env.Setup(x => x.GetAssemblyName(It.IsAny())).Returns(assemblyName); + env.Setup(x => x.GetAssemblyVersion(It.IsAny())).Returns(assemblyVersion); + + var conf = new PowertoolsConfigurations(new SystemWrapper(env.Object)); + + // Act + var xRayRecorder = new Idempotency(conf); + + // Assert + env.Verify(v => + v.SetEnvironmentVariable( + "AWS_EXECUTION_ENV", $"{Constants.FeatureContextIdentifier}/Idempotency/{assemblyVersion}" + ), Times.Once); + + env.Verify(v => + v.GetEnvironmentVariable( + "AWS_EXECUTION_ENV" + ), Times.Once); + + Assert.NotNull(xRayRecorder); + } + + public void Dispose() + { + Environment.SetEnvironmentVariable(Constants.IdempotencyDisabledEnv, "false"); + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Internal/LRUCacheTests.cs b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Internal/LRUCacheTests.cs new file mode 100644 index 00000000..54ec5388 --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Internal/LRUCacheTests.cs @@ -0,0 +1,166 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System.Collections.Generic; +using System.Threading.Tasks; +using AWS.Lambda.Powertools.Idempotency.Internal; +using Xunit; + +//Source: https://github.dev/microsoft/botbuilder-dotnet/blob/main/tests/AdaptiveExpressions.Tests/LRUCacheTest.cs +namespace AWS.Lambda.Powertools.Idempotency.Tests.Internal; + +// ReSharper disable once InconsistentNaming +public class LRUCacheTests +{ + [Fact] + public void TestBasic() + { + var cache = new LRUCache(2); + + Assert.False(cache.TryGet(1, out var result)); + + cache.Set(1, "num1"); + + Assert.True(cache.TryGet(1, out result)); + Assert.Equal("num1", result); + } + + [Fact] + public void TestDiacardPolicy() + { + var cache = new LRUCache(2); + cache.Set(1, "num1"); + cache.Set(2, "num2"); + cache.Set(3, "num3"); + + // should be {2,'num2'} and {3, 'num3'} + Assert.False(cache.TryGet(1, out var result)); + + Assert.True(cache.TryGet(2, out result)); + Assert.Equal("num2", result); + + Assert.True(cache.TryGet(3, out result)); + Assert.Equal("num3", result); + } + + [Fact] + /* + * The average time of this test is about 2ms. + */ + public void TestDpMemorySmall() + { + var cache = new LRUCache(2); + cache.Set(0, 1); + cache.Set(1, 1); + const int fib9999 = 1242044891; + const int fib100000 = 2132534333; + const int maxIdx = 10000; + for (var i = 2; i <= maxIdx; i++) + { + cache.TryGet(i - 2, out var prev2); + cache.TryGet(i - 1, out var prev1); + cache.Set(i, prev1 + prev2); + } + + Assert.False(cache.TryGet(9998, out var result)); + + Assert.True(cache.TryGet(maxIdx - 1, out result)); + Assert.Equal(fib9999, result); + + Assert.True(cache.TryGet(maxIdx, out result)); + Assert.Equal(fib100000, result); + } + + + /* + * The average time of this test is about 3ms. + */ + [Fact] + public void TestDpMemoryLarge() + { + var cache = new LRUCache(500); + cache.Set(0, 1); + cache.Set(1, 1); + const int fib9999 = 1242044891; + const int fib100000 = 2132534333; + const int maxIdx = 10000; + for (var i = 2; i <= 10000; i++) + { + cache.TryGet(i - 2, out var prev2); + cache.TryGet(i - 1, out var prev1); + cache.Set(i, prev1 + prev2); + } + + Assert.False(cache.TryGet(1, out var result)); + + Assert.True(cache.TryGet(maxIdx - 1, out result)); + Assert.Equal(fib9999, result); + + Assert.True(cache.TryGet(maxIdx, out result)); + Assert.Equal(fib100000, result); + } + + [Fact] + /* + * The average time of this test is about 13ms(without the loop of Assert statements). + */ + public async Task TestMultiThreadingAsync() + { + var cache = new LRUCache(10); + var tasks = new List(); + const int numOfThreads = 10; + const int numOfOps = 1000; + for (var i = 0; i < numOfThreads; i++) + { + tasks.Add(Task.Run(() => StoreElement(cache, numOfOps))); + } + + await Task.WhenAll(tasks).ConfigureAwait(false); + + for (var i = numOfOps - numOfThreads; i < numOfOps; i++) + { + Assert.True(cache.TryGet(i, out _)); + } + } + + + [Fact] + public void TestDelete() + { + var cache = new LRUCache(3); + cache.Set(1, "num1"); + cache.Set(2, "num2"); + cache.Set(3, "num3"); + + cache.Delete(1); + + // should be {2,'num2'} and {3, 'num3'} + Assert.False(cache.TryGet(1, out var result)); + + Assert.True(cache.TryGet(2, out result)); + Assert.Equal("num2", result); + + Assert.True(cache.TryGet(3, out result)); + Assert.Equal("num3", result); + } + + private static void StoreElement(LRUCache cache, int numOfOps) + { + for (var i = 0; i < numOfOps; i++) + { + cache.Set(i, i); + } + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Model/Basket.cs b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Model/Basket.cs new file mode 100644 index 00000000..4b0b8d51 --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Model/Basket.cs @@ -0,0 +1,56 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System.Collections.Generic; +using System.Linq; + +namespace AWS.Lambda.Powertools.Idempotency.Tests.Model; + +public class Basket +{ + public List Products { get; } = new(); + public Basket() + { + + } + + public Basket(params Product[] products) + { + Products.AddRange(products); + } + + public void Add(Product product) + { + Products.Add(product); + } + + private bool Equals(Basket other) + { + return Products.All(other.Products.Contains); + } + + public override bool Equals(object obj) + { + if (ReferenceEquals(null, obj)) return false; + if (ReferenceEquals(this, obj)) return true; + if (obj.GetType() != GetType()) return false; + return Equals((Basket) obj); + } + + public override int GetHashCode() + { + return Products.GetHashCode(); + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Model/Product.cs b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Model/Product.cs new file mode 100644 index 00000000..eb7d4657 --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Model/Product.cs @@ -0,0 +1,57 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; + +namespace AWS.Lambda.Powertools.Idempotency.Tests.Model; + +public class Product : IEquatable +{ + // ReSharper disable once MemberCanBePrivate.Global + public long Id { get; } + // ReSharper disable once MemberCanBePrivate.Global + public string Name { get; } + // ReSharper disable once MemberCanBePrivate.Global + public double Price { get; } + + public Product(long id, string name, double price) + { + Id = id; + Name = name; + Price = price; + } + + public bool Equals(Product other) + { + if (other == null) + { + return false; + } + return Id == other.Id && Name == other.Name && Price.Equals(other.Price); + } + + public override bool Equals(object obj) + { + if (ReferenceEquals(null, obj)) return false; + if (ReferenceEquals(this, obj)) return true; + if (obj.GetType() != GetType()) return false; + return Equals((Product) obj); + } + + public override int GetHashCode() + { + return HashCode.Combine(Id, Name, Price); + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Persistence/BasePersistenceStoreTests.cs b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Persistence/BasePersistenceStoreTests.cs new file mode 100644 index 00000000..43dbbc38 --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Persistence/BasePersistenceStoreTests.cs @@ -0,0 +1,525 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using System.IO; +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Threading.Tasks; +using Amazon.Lambda.APIGatewayEvents; +using AWS.Lambda.Powertools.Idempotency.Exceptions; +using AWS.Lambda.Powertools.Idempotency.Internal; +using AWS.Lambda.Powertools.Idempotency.Persistence; +using AWS.Lambda.Powertools.Idempotency.Tests.Model; +using FluentAssertions; +using Xunit; + +namespace AWS.Lambda.Powertools.Idempotency.Tests.Persistence; + +public class BasePersistenceStoreTests +{ + class InMemoryPersistenceStore : BasePersistenceStore + { + private string _validationHash = null; + public DataRecord DataRecord; + public int Status = -1; + public override Task GetRecord(string idempotencyKey) + { + Status = 0; + var dataRecord = new DataRecord( + idempotencyKey, + DataRecord.DataRecordStatus.INPROGRESS, + DateTimeOffset.UtcNow.AddSeconds(3600).ToUnixTimeSeconds(), + "Response", + _validationHash); + return Task.FromResult(dataRecord); + } + + public override Task PutRecord(DataRecord record, DateTimeOffset now) + { + DataRecord = record; + Status = 1; + return Task.CompletedTask; + } + + public override Task UpdateRecord(DataRecord record) + { + DataRecord = record; + Status = 2; + return Task.CompletedTask; + } + + public override Task DeleteRecord(string idempotencyKey) + { + DataRecord = null; + Status = 3; + return Task.CompletedTask; + } + } + + [Fact] + public async Task SaveInProgress_WhenDefaultConfig_ShouldSaveRecordInStore() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + + persistenceStore.Configure(new IdempotencyOptionsBuilder().Build(), null); + + var now = DateTimeOffset.UtcNow; + + // Act + await persistenceStore.SaveInProgress(JsonSerializer.SerializeToDocument(request)!, now); + + // Assert + var dr = persistenceStore.DataRecord; + dr.Status.Should().Be(DataRecord.DataRecordStatus.INPROGRESS); + dr.ExpiryTimestamp.Should().Be(now.AddSeconds(3600).ToUnixTimeSeconds()); + dr.ResponseData.Should().BeNull(); + dr.IdempotencyKey.Should().Be("testFunction#5eff007a9ed2789a9f9f6bc182fc6ae6"); + dr.PayloadHash.Should().BeEmpty(); + persistenceStore.Status.Should().Be(1); + } + + [Fact] + public async Task SaveInProgress_WhenKeyJmesPathIsSet_ShouldSaveRecordInStore_WithIdempotencyKeyEqualsKeyJmesPath() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + + persistenceStore.Configure(new IdempotencyOptionsBuilder() + .WithEventKeyJmesPath("powertools_json(Body).id") + .Build(), "myfunc"); + + var now = DateTimeOffset.UtcNow; + + // Act + await persistenceStore.SaveInProgress(JsonSerializer.SerializeToDocument(request)!, now); + + // Assert + var dr = persistenceStore.DataRecord; + dr.Status.Should().Be(DataRecord.DataRecordStatus.INPROGRESS); + dr.ExpiryTimestamp.Should().Be(now.AddSeconds(3600).ToUnixTimeSeconds()); + dr.ResponseData.Should().BeNull(); + dr.IdempotencyKey.Should().Be("testFunction.myfunc#2fef178cc82be5ce3da6c5e0466a6182"); + dr.PayloadHash.Should().BeEmpty(); + persistenceStore.Status.Should().Be(1); + } + + [Fact] + public async Task SaveInProgress_WhenKeyJmesPathIsSetToMultipleFields_ShouldSaveRecordInStore_WithIdempotencyKeyEqualsKeyJmesPath() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + + persistenceStore.Configure(new IdempotencyOptionsBuilder() + .WithEventKeyJmesPath("powertools_json(Body).[id, message]") //[43876123454654,"Lambda rocks"] + .Build(), "myfunc"); + + var now = DateTimeOffset.UtcNow; + + // Act + await persistenceStore.SaveInProgress(JsonSerializer.SerializeToDocument(request)!, now); + + // Assert + var dr = persistenceStore.DataRecord; + dr.Status.Should().Be(DataRecord.DataRecordStatus.INPROGRESS); + dr.ExpiryTimestamp.Should().Be(now.AddSeconds(3600).ToUnixTimeSeconds()); + dr.ResponseData.Should().BeNull(); + dr.IdempotencyKey.Should().Be("testFunction.myfunc#5ca4c8c44d427e9d43ca918a24d6cf42"); + dr.PayloadHash.Should().BeEmpty(); + persistenceStore.Status.Should().Be(1); + } + + + [Fact] + public async Task SaveInProgress_WhenJMESPath_NotFound_ShouldThrowException() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + + persistenceStore.Configure(new IdempotencyOptionsBuilder() + .WithEventKeyJmesPath("unavailable") + .WithThrowOnNoIdempotencyKey(true) // should throw + .Build(), ""); + var now = DateTimeOffset.UtcNow; + + // Act + var act = async () => await persistenceStore.SaveInProgress(JsonSerializer.SerializeToDocument(request)!, now); + + // Assert + await act.Should() + .ThrowAsync() + .WithMessage("No data found to create a hashed idempotency key"); + + persistenceStore.Status.Should().Be(-1); + } + + [Fact] + public async Task SaveInProgress_WhenJMESpath_NotFound_ShouldNotThrowException() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + + persistenceStore.Configure(new IdempotencyOptionsBuilder() + .WithEventKeyJmesPath("unavailable") + .Build(), ""); + + var now = DateTimeOffset.UtcNow; + + // Act + await persistenceStore.SaveInProgress(JsonSerializer.SerializeToDocument(request)!, now); + + // Assert + var dr = persistenceStore.DataRecord; + dr.Status.Should().Be(DataRecord.DataRecordStatus.INPROGRESS); + persistenceStore.Status.Should().Be(1); + } + + [Fact] + public async Task SaveInProgress_WhenLocalCacheIsSet_AndNotExpired_ShouldThrowException() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + + LRUCache cache = new (2); + persistenceStore.Configure(new IdempotencyOptionsBuilder() + .WithUseLocalCache(true) + .WithEventKeyJmesPath("powertools_json(Body).id") + .Build(), null, cache); + + var now = DateTimeOffset.UtcNow; + cache.Set("testFunction#2fef178cc82be5ce3da6c5e0466a6182", + new DataRecord( + "testFunction#2fef178cc82be5ce3da6c5e0466a6182", + DataRecord.DataRecordStatus.INPROGRESS, + now.AddSeconds(3600).ToUnixTimeSeconds(), + null, null) + ); + + // Act + var act = () => persistenceStore.SaveInProgress(JsonSerializer.SerializeToDocument(request)!, now); + + // Assert + await act.Should() + .ThrowAsync(); + + persistenceStore.Status.Should().Be(-1); + } + + [Fact] + public async Task SaveInProgress_WhenLocalCacheIsSetButExpired_ShouldRemoveFromCache() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + + LRUCache cache = new (2); + persistenceStore.Configure(new IdempotencyOptionsBuilder() + .WithEventKeyJmesPath("powertools_json(Body).id") + .WithUseLocalCache(true) + .WithExpiration(TimeSpan.FromSeconds(2)) + .Build(), null, cache); + + var now = DateTimeOffset.UtcNow; + cache.Set("testFunction#2fef178cc82be5ce3da6c5e0466a6182", + new DataRecord( + "testFunction#2fef178cc82be5ce3da6c5e0466a6182", + DataRecord.DataRecordStatus.INPROGRESS, + now.AddSeconds(-3).ToUnixTimeSeconds(), + null, null) + ); + + // Act + await persistenceStore.SaveInProgress(JsonSerializer.SerializeToDocument(request)!, now); + + // Assert + var dr = persistenceStore.DataRecord; + dr.Status.Should().Be(DataRecord.DataRecordStatus.INPROGRESS); + cache.Count.Should().Be(0); + persistenceStore.Status.Should().Be(1); + } + + ////// Save Success + + [Fact] + public async Task SaveSuccess_WhenDefaultConfig_ShouldUpdateRecord() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + LRUCache cache = new (2); + persistenceStore.Configure(new IdempotencyOptionsBuilder().Build(), null, cache); + + var product = new Product(34543, "product", 42); + + var now = DateTimeOffset.UtcNow; + + // Act + await persistenceStore.SaveSuccess(JsonSerializer.SerializeToDocument(request)!, product, now); + + // Assert + var dr = persistenceStore.DataRecord; + dr.Status.Should().Be(DataRecord.DataRecordStatus.COMPLETED); + dr.ExpiryTimestamp.Should().Be(now.AddSeconds(3600).ToUnixTimeSeconds()); + dr.ResponseData.Should().Be(JsonSerializer.Serialize(product)); + dr.IdempotencyKey.Should().Be("testFunction#5eff007a9ed2789a9f9f6bc182fc6ae6"); + dr.PayloadHash.Should().BeEmpty(); + persistenceStore.Status.Should().Be(2); + cache.Count.Should().Be(0); + } + + [Fact] + public async Task SaveSuccess_WhenCacheEnabled_ShouldSaveInCache() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + LRUCache cache = new (2); + + persistenceStore.Configure(new IdempotencyOptionsBuilder() + .WithUseLocalCache(true).Build(), null, cache); + + var product = new Product(34543, "product", 42); + var now = DateTimeOffset.UtcNow; + + // Act + await persistenceStore.SaveSuccess(JsonSerializer.SerializeToDocument(request)!, product, now); + + // Assert + persistenceStore.Status.Should().Be(2); + cache.Count.Should().Be(1); + + var foundDataRecord = cache.TryGet("testFunction#5eff007a9ed2789a9f9f6bc182fc6ae6", out var record); + foundDataRecord.Should().BeTrue(); + record.Status.Should().Be(DataRecord.DataRecordStatus.COMPLETED); + record.ExpiryTimestamp.Should().Be(now.AddSeconds(3600).ToUnixTimeSeconds()); + record.ResponseData.Should().Be(JsonSerializer.Serialize(product)); + record.IdempotencyKey.Should().Be("testFunction#5eff007a9ed2789a9f9f6bc182fc6ae6"); + record.PayloadHash.Should().BeEmpty(); + } + + /// Get Record + + [Fact] + public async Task GetRecord_WhenRecordIsInStore_ShouldReturnRecordFromPersistence() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + + LRUCache cache = new(2); + persistenceStore.Configure(new IdempotencyOptionsBuilder().Build(), "myfunc", cache); + + var now = DateTimeOffset.UtcNow; + + // Act + var record = await persistenceStore.GetRecord(JsonSerializer.SerializeToDocument(request)!, now); + + // Assert + record.IdempotencyKey.Should().Be("testFunction.myfunc#5eff007a9ed2789a9f9f6bc182fc6ae6"); + record.Status.Should().Be(DataRecord.DataRecordStatus.INPROGRESS); + record.ResponseData.Should().Be("Response"); + persistenceStore.Status.Should().Be(0); + } + + [Fact] + public async Task GetRecord_WhenCacheEnabledNotExpired_ShouldReturnRecordFromCache() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + LRUCache cache = new(2); + + persistenceStore.Configure(new IdempotencyOptionsBuilder() + .WithUseLocalCache(true).Build(), "myfunc", cache); + + var now = DateTimeOffset.UtcNow; + var dr = new DataRecord( + "testFunction.myfunc#5eff007a9ed2789a9f9f6bc182fc6ae6", + DataRecord.DataRecordStatus.COMPLETED, + now.AddSeconds(3600).ToUnixTimeSeconds(), + "result of the function", + null); + cache.Set("testFunction.myfunc#5eff007a9ed2789a9f9f6bc182fc6ae6", dr); + + // Act + var record = await persistenceStore.GetRecord(JsonSerializer.SerializeToDocument(request)!, now); + + // Assert + record.IdempotencyKey.Should().Be("testFunction.myfunc#5eff007a9ed2789a9f9f6bc182fc6ae6"); + record.Status.Should().Be(DataRecord.DataRecordStatus.COMPLETED); + record.ResponseData.Should().Be("result of the function"); + persistenceStore.Status.Should().Be(-1); + } + + [Fact] + public async Task GetRecord_WhenLocalCacheEnabledButRecordExpired_ShouldReturnRecordFromPersistence() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + LRUCache cache = new(2); + persistenceStore.Configure(new IdempotencyOptionsBuilder() + .WithUseLocalCache(true).Build(), "myfunc", cache); + + var now = DateTimeOffset.UtcNow; + var dr = new DataRecord( + "testFunction.myfunc#5eff007a9ed2789a9f9f6bc182fc6ae6", + DataRecord.DataRecordStatus.COMPLETED, + now.AddSeconds(-3).ToUnixTimeSeconds(), + "result of the function", + null); + cache.Set("testFunction.myfunc#5eff007a9ed2789a9f9f6bc182fc6ae6", dr); + + // Act + var record = await persistenceStore.GetRecord(JsonSerializer.SerializeToDocument(request)!, now); + + // Assert + record.IdempotencyKey.Should().Be("testFunction.myfunc#5eff007a9ed2789a9f9f6bc182fc6ae6"); + record.Status.Should().Be(DataRecord.DataRecordStatus.INPROGRESS); + record.ResponseData.Should().Be("Response"); + persistenceStore.Status.Should().Be(0); + cache.Count.Should().Be(0); + } + + [Fact] + public async Task GetRecord_WhenInvalidPayload_ShouldThrowValidationException() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + + persistenceStore.Configure(new IdempotencyOptionsBuilder() + .WithEventKeyJmesPath("powertools_json(Body).id") + .WithPayloadValidationJmesPath("powertools_json(Body).message") + .Build(), + "myfunc"); + + var now = DateTimeOffset.UtcNow; + + // Act + Func act = () => persistenceStore.GetRecord(JsonSerializer.SerializeToDocument(request)!, now); + + // Assert + await act.Should().ThrowAsync(); + } + + // Delete Record + [Fact] + public async Task DeleteRecord_WhenRecordExist_ShouldDeleteRecordFromPersistence() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + + persistenceStore.Configure(new IdempotencyOptionsBuilder().Build(), null); + + // Act + await persistenceStore.DeleteRecord(JsonSerializer.SerializeToDocument(request)!, new ArithmeticException()); + + // Assert + persistenceStore.Status.Should().Be(3); + } + + [Fact] + public async Task DeleteRecord_WhenLocalCacheEnabled_ShouldDeleteRecordFromCache() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + var request = LoadApiGatewayProxyRequest(); + LRUCache cache = new (2); + persistenceStore.Configure(new IdempotencyOptionsBuilder() + .WithUseLocalCache(true).Build(), null, cache); + + cache.Set("testFunction#5eff007a9ed2789a9f9f6bc182fc6ae6", + new DataRecord("testFunction#5eff007a9ed2789a9f9f6bc182fc6ae6", + DataRecord.DataRecordStatus.COMPLETED, + 123, + null, null)); + + // Act + await persistenceStore.DeleteRecord(JsonSerializer.SerializeToDocument(request)!, new ArithmeticException()); + + // Assert + persistenceStore.Status.Should().Be(3); + cache.Count.Should().Be(0); + } + + [Fact] + public void GenerateHash_WhenInputIsString_ShouldGenerateMd5ofString() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + persistenceStore.Configure(new IdempotencyOptionsBuilder().Build(), null); + var expectedHash = "70c24d88041893f7fbab4105b76fd9e1"; // MD5(Lambda rocks) + + // Act + var jsonValue = JsonValue.Create("Lambda rocks"); + var generatedHash = persistenceStore.GenerateHash(JsonDocument.Parse(jsonValue!.ToJsonString()).RootElement); + + // Assert + generatedHash.Should().Be(expectedHash); + } + + [Fact] + public void GenerateHash_WhenInputIsObject_ShouldGenerateMd5ofJsonObject() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + persistenceStore.Configure(new IdempotencyOptionsBuilder().Build(), null); + var product = new Product(42, "Product", 12); + var expectedHash = "c83e720b399b3b4898c8734af177c53a"; // MD5({"Id":42,"Name":"Product","Price":12}) + + // Act + var jsonValue = JsonValue.Create(product); + var generatedHash = persistenceStore.GenerateHash(JsonDocument.Parse(jsonValue!.ToJsonString()).RootElement); + + // Assert + generatedHash.Should().Be(expectedHash); + } + + [Fact] + public void GenerateHash_WhenInputIsDouble_ShouldGenerateMd5ofDouble() + { + // Arrange + var persistenceStore = new InMemoryPersistenceStore(); + persistenceStore.Configure(new IdempotencyOptionsBuilder().Build(), null); + var expectedHash = "bb84c94278119c8838649706df4db42b"; // MD5(256.42) + + // Act + var generatedHash = persistenceStore.GenerateHash(JsonDocument.Parse("256.42").RootElement); + + // Assert + generatedHash.Should().Be(expectedHash); + } + + private static APIGatewayProxyRequest LoadApiGatewayProxyRequest() + { + var options = new JsonSerializerOptions + { + PropertyNameCaseInsensitive = true + }; + + var eventJson = File.ReadAllText("./resources/apigw_event.json"); + var request = JsonSerializer.Deserialize(eventJson, options); + return request!; + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Persistence/DataRecordTests.cs b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Persistence/DataRecordTests.cs new file mode 100644 index 00000000..3f68de75 --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Persistence/DataRecordTests.cs @@ -0,0 +1,95 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using AWS.Lambda.Powertools.Idempotency.Persistence; +using FluentAssertions; +using Xunit; + +namespace AWS.Lambda.Powertools.Idempotency.Tests.Persistence; + +public class DataRecordTests +{ + [Fact] + public void IsExpired_WhenCurrentTimeIsGreaterThanExpiryTimestamp_ShouldReturnTrue() + { + // Arrange + var now = DateTimeOffset.UtcNow; + var dataRecord = new DataRecord( + "123", + DataRecord.DataRecordStatus.INPROGRESS, + now.AddSeconds(-1).ToUnixTimeSeconds(), + "abc","123"); + + // Act + var result =dataRecord.IsExpired(now); + + // Assert + result.Should().BeTrue(); + } + [Fact] + public void IsExpired_WhenCurrentTimeIsLessThanExpiryTimestamp_ShouldReturnFalse() + { + // Arrange + var now = DateTimeOffset.UtcNow; + var dataRecord = new DataRecord( + "123", + DataRecord.DataRecordStatus.INPROGRESS, + now.AddSeconds(10).ToUnixTimeSeconds(), + "abc","123"); + + // Act + var result =dataRecord.IsExpired(now); + + // Assert + result.Should().BeFalse(); + } + + [Fact] + public void Status_WhenCurrentTimeIsGreaterThanExpiryTimestamp_ShouldBeExpired() + { + // Arrange + var now = DateTimeOffset.UtcNow; + var dataRecord = new DataRecord( + "123", + DataRecord.DataRecordStatus.INPROGRESS, + now.AddSeconds(-10).ToUnixTimeSeconds(), + "abc","123"); + + // Act + var status =dataRecord.Status; + + // Assert + status.Should().Be(DataRecord.DataRecordStatus.EXPIRED); + } + + [Fact] + public void Status_WhenCurrentTimeDidNotExpire_ShouldBeRecordStatus() + { + // Arrange + var now = DateTimeOffset.UtcNow; + var dataRecord = new DataRecord( + "123", + DataRecord.DataRecordStatus.INPROGRESS, + now.AddSeconds(10).ToUnixTimeSeconds(), + "abc","123"); + + // Act + var status =dataRecord.Status; + + // Assert + status.Should().Be(DataRecord.DataRecordStatus.INPROGRESS); + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Persistence/DynamoDBFixture.cs b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Persistence/DynamoDBFixture.cs new file mode 100644 index 00000000..5df377ee --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Persistence/DynamoDBFixture.cs @@ -0,0 +1,92 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using System.Collections.Generic; +using Amazon.DynamoDBv2; +using Amazon.DynamoDBv2.Model; +using Amazon.Runtime; +using DotNet.Testcontainers.Builders; +using DotNet.Testcontainers.Containers; + +namespace AWS.Lambda.Powertools.Idempotency.Tests.Persistence; + +// ReSharper disable once ClassNeverInstantiated.Global +public class DynamoDbFixture : IDisposable +{ + private readonly IContainer _container; + public AmazonDynamoDBClient Client { get; set; } + public string TableName { get; set; } = "idempotency_table"; + + public DynamoDbFixture() + { + Environment.SetEnvironmentVariable("TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE","/var/run/docker.sock"); + + _container = new ContainerBuilder() + .WithName(Guid.NewGuid().ToString("D")) + .WithImage("amazon/dynamodb-local:latest") + .WithPortBinding(8000, true) + .WithDockerEndpoint(Environment.GetEnvironmentVariable("DOCKER_HOST") ?? "unix:///var/run/docker.sock") + .Build(); + + + _container.StartAsync().Wait(); + + var credentials = new BasicAWSCredentials("FAKE", "FAKE"); + var amazonDynamoDbConfig = new AmazonDynamoDBConfig + { + ServiceURL = new UriBuilder("http", _container.Hostname, _container.GetMappedPublicPort(8000)).Uri.ToString(), + AuthenticationRegion = "us-east-1" + }; + + Client = new AmazonDynamoDBClient(credentials, amazonDynamoDbConfig); + + var createTableRequest = new CreateTableRequest + { + TableName = TableName, + KeySchema = new List + { + new("id", KeyType.HASH) + }, + AttributeDefinitions = new List + { + new() + { + AttributeName = "id", + AttributeType = ScalarAttributeType.S + } + }, + BillingMode = BillingMode.PAY_PER_REQUEST + }; + try + { + Client.CreateTableAsync(createTableRequest).GetAwaiter().GetResult(); + var response = Client.DescribeTableAsync(TableName).GetAwaiter().GetResult(); + if (response == null) + { + throw new NullReferenceException("Table was not created within the expected time"); + } + } + catch (ResourceInUseException e) + { + Console.WriteLine(e.Message); + } + } + + public void Dispose() + { + _container.DisposeAsync().AsTask(); + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Persistence/DynamoDBPersistenceStoreTests.cs b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Persistence/DynamoDBPersistenceStoreTests.cs new file mode 100644 index 00000000..a6f61531 --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/Persistence/DynamoDBPersistenceStoreTests.cs @@ -0,0 +1,374 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +using System; +using System.Collections.Generic; +using System.Threading.Tasks; +using Amazon.DynamoDBv2; +using Amazon.DynamoDBv2.Model; +using AWS.Lambda.Powertools.Common; +using AWS.Lambda.Powertools.Idempotency.Exceptions; +using AWS.Lambda.Powertools.Idempotency.Persistence; +using FluentAssertions; +using Xunit; + +namespace AWS.Lambda.Powertools.Idempotency.Tests.Persistence; + +[Collection("Sequential")] +[Trait("Category", "Integration")] +public class DynamoDbPersistenceStoreTests : IClassFixture +{ + private readonly DynamoDBPersistenceStore _dynamoDbPersistenceStore; + private readonly AmazonDynamoDBClient _client; + private readonly string _tableName; + + public DynamoDbPersistenceStoreTests(DynamoDbFixture fixture) + { + _client = fixture.Client; + _tableName = fixture.TableName; + _dynamoDbPersistenceStore = new DynamoDBPersistenceStoreBuilder() + .WithTableName(_tableName) + .WithDynamoDBClient(_client) + .Build(); + _dynamoDbPersistenceStore.Configure(new IdempotencyOptionsBuilder().Build(),functionName: null); + } + + //putRecord + [Fact] + public async Task PutRecord_WhenRecordDoesNotExist_ShouldCreateRecordInDynamoDB() + { + // Arrange + var now = DateTimeOffset.UtcNow; + var expiry = now.AddSeconds(3600).ToUnixTimeSeconds(); + var key = CreateKey("key"); + + // Act + await _dynamoDbPersistenceStore + .PutRecord(new DataRecord("key", DataRecord.DataRecordStatus.COMPLETED, expiry, null, null), now); + + // Assert + var getItemResponse = + await _client.GetItemAsync(new GetItemRequest + { + TableName = _tableName, + Key = key + }); + + var item = getItemResponse.Item; + item.Should().NotBeNull(); + item["status"].S.Should().Be("COMPLETED"); + item["expiration"].N.Should().Be(expiry.ToString()); + } + + [Fact] + public async Task PutRecord_WhenRecordAlreadyExist_ShouldThrowIdempotencyItemAlreadyExistsException() + { + // Arrange + var key = CreateKey("key"); + + // Insert a fake item with same id + Dictionary item = new(key); + var now = DateTimeOffset.UtcNow; + var expiry = now.AddSeconds(30).ToUnixTimeMilliseconds(); + item.Add("expiration", new AttributeValue {N = expiry.ToString()}); + item.Add("status", new AttributeValue(DataRecord.DataRecordStatus.COMPLETED.ToString())); + item.Add("data", new AttributeValue("Fake Data")); + await _client.PutItemAsync(new PutItemRequest + { + TableName = _tableName, + Item = item + }); + var expiry2 = now.AddSeconds(3600).ToUnixTimeSeconds(); + + // Act + var act = () => _dynamoDbPersistenceStore.PutRecord( + new DataRecord("key", + DataRecord.DataRecordStatus.INPROGRESS, + expiry2, + null, + null + ), now); + + // Assert + await act.Should().ThrowAsync(); + + // item was not updated, retrieve the initial one + var itemInDb = (await _client.GetItemAsync(new GetItemRequest + { + TableName = _tableName, + Key = key + })).Item; + itemInDb.Should().NotBeNull(); + itemInDb["status"].S.Should().Be("COMPLETED"); + itemInDb["expiration"].N.Should().Be(expiry.ToString()); + itemInDb["data"].S.Should().Be("Fake Data"); + } + + //getRecord + [Fact] + public async Task GetRecord_WhenRecordExistsInDynamoDb_ShouldReturnExistingRecord() + { + // Arrange + //await InitializeAsync(); + + // Insert a fake item with same id + Dictionary item = new() + { + {"id", new AttributeValue("key")} //key + }; + var now = DateTimeOffset.UtcNow; + var expiry = now.AddSeconds(30).ToUnixTimeMilliseconds(); + item.Add("expiration", new AttributeValue + { + N = expiry.ToString() + }); + item.Add("status", new AttributeValue(DataRecord.DataRecordStatus.COMPLETED.ToString())); + item.Add("data", new AttributeValue("Fake Data")); + var _ = await _client.PutItemAsync(new PutItemRequest + { + TableName = _tableName, + Item = item + }); + + // Act + var record = await _dynamoDbPersistenceStore.GetRecord("key"); + + // Assert + record.IdempotencyKey.Should().Be("key"); + record.Status.Should().Be(DataRecord.DataRecordStatus.COMPLETED); + record.ResponseData.Should().Be("Fake Data"); + record.ExpiryTimestamp.Should().Be(expiry); + } + + [Fact] + public async Task GetRecord_WhenRecordIsAbsent_ShouldThrowException() + { + //Arrange + await _dynamoDbPersistenceStore.DeleteRecord("key"); + + // Act + Func act = () => _dynamoDbPersistenceStore.GetRecord("key"); + + // Assert + await act.Should().ThrowAsync(); + } + //updateRecord + + [Fact] + public async Task UpdateRecord_WhenRecordExistsInDynamoDb_ShouldUpdateRecord() + { + // Arrange: Insert a fake item with same id + var key = CreateKey("key"); + Dictionary item = new(key); + var now = DateTimeOffset.UtcNow; + var expiry = now.AddSeconds(360).ToUnixTimeMilliseconds(); + item.Add("expiration", new AttributeValue + { + N = expiry.ToString() + }); + item.Add("status", new AttributeValue(DataRecord.DataRecordStatus.INPROGRESS.ToString())); + await _client.PutItemAsync(new PutItemRequest + { + TableName = _tableName, + Item = item + }); + // enable payload validation + _dynamoDbPersistenceStore.Configure(new IdempotencyOptionsBuilder().WithPayloadValidationJmesPath("path").Build(), + null); + + // Act + expiry = now.AddSeconds(3600).ToUnixTimeMilliseconds(); + var record = new DataRecord("key", DataRecord.DataRecordStatus.COMPLETED, expiry, "Fake result", "hash"); + await _dynamoDbPersistenceStore.UpdateRecord(record); + + // Assert + var itemInDb = (await _client.GetItemAsync(new GetItemRequest + { + TableName = _tableName, + Key = key + })).Item; + + itemInDb["status"].S.Should().Be("COMPLETED"); + itemInDb["expiration"].N.Should().Be(expiry.ToString()); + itemInDb["data"].S.Should().Be("Fake result"); + itemInDb["validation"].S.Should().Be("hash"); + } + + //deleteRecord + [Fact] + public async Task DeleteRecord_WhenRecordExistsInDynamoDb_ShouldDeleteRecord() + { + // Arrange: Insert a fake item with same id + var key = CreateKey("key"); + Dictionary item = new(key); + var now = DateTimeOffset.UtcNow; + var expiry = now.AddSeconds(360).ToUnixTimeMilliseconds(); + item.Add("expiration", new AttributeValue {N=expiry.ToString()}); + item.Add("status", new AttributeValue(DataRecord.DataRecordStatus.INPROGRESS.ToString())); + await _client.PutItemAsync(new PutItemRequest + { + TableName = _tableName, + Item = item + }); + var scanResponse = await _client.ScanAsync(new ScanRequest + { + TableName = _tableName + }); + scanResponse.Items.Count.Should().Be(1); + + // Act + await _dynamoDbPersistenceStore.DeleteRecord("key"); + + // Assert + scanResponse = await _client.ScanAsync(new ScanRequest + { + TableName = _tableName + }); + scanResponse.Items.Count.Should().Be(0); + } + + [Fact] + public async Task EndToEndWithCustomAttrNamesAndSortKey() + { + const string tableNameCustom = "idempotency_table_custom"; + try + { + var createTableRequest = new CreateTableRequest + { + TableName = tableNameCustom, + KeySchema = new List + { + new("key", KeyType.HASH), + new("sortkey", KeyType.RANGE) + }, + AttributeDefinitions = new List + { + new("key", ScalarAttributeType.S), + new("sortkey", ScalarAttributeType.S) + }, + BillingMode = BillingMode.PAY_PER_REQUEST + }; + await _client.CreateTableAsync(createTableRequest); + var persistenceStore = new DynamoDBPersistenceStoreBuilder() + .WithTableName(tableNameCustom) + .WithDynamoDBClient(_client) + .WithDataAttr("result") + .WithExpiryAttr("expiry") + .WithKeyAttr("key") + .WithSortKeyAttr("sortkey") + .WithStaticPkValue("pk") + .WithStatusAttr("state") + .WithValidationAttr("valid") + .Build(); + persistenceStore.Configure(new IdempotencyOptionsBuilder().Build(),functionName: null); + + var now = DateTimeOffset.UtcNow; + var record = new DataRecord( + "mykey", + DataRecord.DataRecordStatus.INPROGRESS, + now.AddSeconds(400).ToUnixTimeMilliseconds(), + null, + null + ); + // PUT + await persistenceStore.PutRecord(record, now); + + Dictionary customKey = new() + { + { "key", new AttributeValue("pk") }, + { "sortkey", new AttributeValue("mykey") } + }; + + var itemInDb = (await _client.GetItemAsync(new GetItemRequest + { + TableName = tableNameCustom, + Key = customKey + })).Item; + + // GET + var recordInDb = await persistenceStore.GetRecord("mykey"); + + itemInDb.Should().NotBeNull(); + itemInDb["key"].S.Should().Be("pk"); + itemInDb["sortkey"].S.Should().Be(recordInDb.IdempotencyKey); + itemInDb["state"].S.Should().Be(recordInDb.Status.ToString()); + itemInDb["expiry"].N.Should().Be(recordInDb.ExpiryTimestamp.ToString()); + + // UPDATE + var updatedRecord = new DataRecord( + "mykey", + DataRecord.DataRecordStatus.COMPLETED, + now.AddSeconds(500).ToUnixTimeMilliseconds(), + "response", + null + ); + await persistenceStore.UpdateRecord(updatedRecord); + recordInDb = await persistenceStore.GetRecord("mykey"); + recordInDb.Should().Be(updatedRecord); + + // DELETE + await persistenceStore.DeleteRecord("mykey"); + (await _client.ScanAsync(new ScanRequest + { + TableName = tableNameCustom + })).Count.Should().Be(0); + + } + finally + { + try + { + await _client.DeleteTableAsync(new DeleteTableRequest + { + TableName = tableNameCustom + }); + } + catch (Exception) + { + // OK + } + } + } + + [Fact] + public async Task GetRecord_WhenIdempotencyDisabled_ShouldNotCreateClients() + { + try + { + // Arrange + Environment.SetEnvironmentVariable(Constants.IdempotencyDisabledEnv, "true"); + + var store = new DynamoDBPersistenceStoreBuilder().WithTableName(_tableName).Build(); + + // Act + Func act = () => store.GetRecord("fake"); + + // Assert + await act.Should().ThrowAsync(); + } + finally + { + Environment.SetEnvironmentVariable(Constants.IdempotencyDisabledEnv, "false"); + } + } + private static Dictionary CreateKey(string keyValue) + { + var key = new Dictionary + { + {"id", new AttributeValue(keyValue)} + }; + return key; + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/resources/apigw_event.json b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/resources/apigw_event.json new file mode 100644 index 00000000..cf3f372e --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/resources/apigw_event.json @@ -0,0 +1,62 @@ +{ + "body": "{\"message\": \"Lambda rocks\", \"id\": 43876123454654}", + "resource": "/{proxy+}", + "path": "/path/to/resource", + "httpMethod": "POST", + "isBase64Encoded": false, + "queryStringParameters": { + "foo": "bar" + }, + "pathParameters": { + "proxy": "/path/to/resource" + }, + "stageVariables": { + "baz": "qux" + }, + "headers": { + "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", + "Accept-Encoding": "gzip, deflate, sdch", + "Accept-Language": "en-US,en;q=0.8", + "Cache-Control": "max-age=0", + "CloudFront-Forwarded-Proto": "https", + "CloudFront-Is-Desktop-Viewer": "true", + "CloudFront-Is-Mobile-Viewer": "false", + "CloudFront-Is-SmartTV-Viewer": "false", + "CloudFront-Is-Tablet-Viewer": "false", + "CloudFront-Viewer-Country": "US", + "Host": "1234567890.execute-api.us-east-1.amazonaws.com", + "Upgrade-Insecure-Requests": "1", + "User-Agent": "Custom User Agent String", + "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", + "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", + "X-Forwarded-For": "127.0.0.1, 127.0.0.2", + "X-Forwarded-Port": "443", + "X-Forwarded-Proto": "https" + }, + "requestContext": { + "accountId": "123456789012", + "resourceId": "123456", + "stage": "prod", + "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", + "requestTime": "09/Apr/2015:12:34:56 +0000", + "requestTimeEpoch": 1428582896000, + "identity": { + "cognitoIdentityPoolId": null, + "accountId": null, + "cognitoIdentityId": null, + "caller": null, + "accessKey": null, + "sourceIp": "127.0.0.1", + "cognitoAuthenticationType": null, + "cognitoAuthenticationProvider": null, + "userArn": null, + "userAgent": "Custom User Agent String", + "user": null + }, + "path": "/prod/path/to/resource", + "resourcePath": "/{proxy+}", + "httpMethod": "POST", + "apiId": "1234567890", + "protocol": "HTTP/1.1" + } +} \ No newline at end of file diff --git a/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/resources/apigw_event2.json b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/resources/apigw_event2.json new file mode 100644 index 00000000..a313815c --- /dev/null +++ b/libraries/tests/AWS.Lambda.Powertools.Idempotency.Tests/resources/apigw_event2.json @@ -0,0 +1,62 @@ +{ + "body": "{\"address\": \"https://checkip.amazonaws.com\"}", + "resource": "/{proxy+}", + "path": "/path/to/resource", + "httpMethod": "POST", + "isBase64Encoded": false, + "queryStringParameters": { + "foo": "bar" + }, + "pathParameters": { + "proxy": "/path/to/resource" + }, + "stageVariables": { + "baz": "qux" + }, + "headers": { + "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", + "Accept-Encoding": "gzip, deflate, sdch", + "Accept-Language": "en-US,en;q=0.8", + "Cache-Control": "max-age=0", + "CloudFront-Forwarded-Proto": "https", + "CloudFront-Is-Desktop-Viewer": "true", + "CloudFront-Is-Mobile-Viewer": "false", + "CloudFront-Is-SmartTV-Viewer": "false", + "CloudFront-Is-Tablet-Viewer": "false", + "CloudFront-Viewer-Country": "US", + "Host": "1234567890.execute-api.us-east-1.amazonaws.com", + "Upgrade-Insecure-Requests": "1", + "User-Agent": "Custom User Agent String", + "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", + "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", + "X-Forwarded-For": "127.0.0.1, 127.0.0.2", + "X-Forwarded-Port": "443", + "X-Forwarded-Proto": "https" + }, + "requestContext": { + "accountId": "123456789012", + "resourceId": "123456", + "stage": "prod", + "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", + "requestTime": "09/Apr/2015:12:34:56 +0000", + "requestTimeEpoch": 1428582896000, + "identity": { + "cognitoIdentityPoolId": null, + "accountId": null, + "cognitoIdentityId": null, + "caller": null, + "accessKey": null, + "sourceIp": "127.0.0.1", + "cognitoAuthenticationType": null, + "cognitoAuthenticationProvider": null, + "userArn": null, + "userAgent": "Custom User Agent String", + "user": null + }, + "path": "/prod/path/to/resource", + "resourcePath": "/{proxy+}", + "httpMethod": "POST", + "apiId": "1234567890", + "protocol": "HTTP/1.1" + } +}