Mezmo CloudWatch Lambda Function

The Mezmo AWS CloudWatch integration relies on AWS Lambda to route your CloudWatch Logs to Mezmo Pipelines.

How to Use

Deploy the Code

1. Create a new Lambda function and select Author from scratch
2. Click on the Lambda function to edit the details:

• Code entry type: Upload a .ZIP file
• Upload our Lambda function .ZIP File.
• Handler: index.handler
• Runtime: Node.js 20.x

Configuration

General Configuration

If this function is being used to stream from gzipped files:

1. Set Timeout to, at least, 10 seconds.
2. Set Memory limit to, at least, 128 MB.

Notes:

• The recommended number of retries is 0 because retrying lambda execution can result in duplicate logs. It can be modified in Configuration > Asynchronous invocation.

Triggers

Add CloudWatch Logs as a trigger with the following configuration:

• Select the CloudWatch Log Group to be sent to Mezmo.
• Choose your own custom Filter Name.
• Optional Filter Pattern option can be used to filter the logs before shipping to Mezmo.

Notes:

• You can specify only one CloudWatch Log Group in one trigger.

Permissions

For Execution role, assign a role that has the following policies:

• CloudWatchLogsReadOnlyAccess
• AWSLambdaBasicExecutionRole

Environment Variables

• PIPELINE_KEY (required): Your pipeline http source node key.
• PIPELINE_URL (required): Pipeline Ingestion URL

Notes: The following optional environment variables can also be used to tune this Lambda function for specific use cases.

• MAX_REQUEST_TIMEOUT (optional): Time limit (in milliseconds) for requests made by this HTTP Client (Default: 30000)
• MAX_REQUEST_RETRIES (optional): The maximum number of retries for sending a line when there are network failures (Default: 5)
• REQUEST_RETRY_INTERVAL (optional): How frequently (in seconds) to retry for sending a line when there are network failures (Default: 60)

Monitoring

Enabling monitoring means forwarding the metrics and logs about the execution of the CloudWatch Lambda function to CloudWatch. You can also create and use a separate CloudWatch Lambda function to monitor the performance of this CloudWatch Lambda function.

Test

You can test the configuration and code package using the following test input containing the sample event data:

{
    "awslogs": {
        "data": "H4sIAAAAAAAAEzWQQW+DMAyF/wrKmaEkJCbhhjbWCzuBtMNUVSmkNBIQRMKqqep/X6Cb5Ivfs58++45G7ZzqdfMza5Sjt6IpTh9lXReHEsXI3ia9BJnQlHHIhMSEBnmw/WGx6xwcp8Z50M9uN2q/aDUGx2vn/5oYufXs2sXM3tjp3QxeLw7lX6hS47lTz6lTO9i1uynfXkOMe5lsp9Fxzyy/9eS3hTsyXYhOGVCaEsBSgsyEYBkGzrDMAIMQlAq+gQIQSjFhBFgqJOUMAog34WAfoFFOOM8kA0Y5SSH+f0SIb67GRaHq/baosn1UmUlHF7tErxvk5wa56b2Z+iRJ0OP4+AWj9ITzSgEAAA=="
    }
}

License

Copyright © Mezmo, released under an MIT license. See the LICENSE file and https://opensource.org/licenses/MIT

Contributing

Contributions are always welcome. See the contributing guide to learn how you can help.